Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference Wenchen Han
Gingfung Matthew Yeung
Marco Barletta
University College London
Huawei
Huawei
William Toner
Amory Hoste
Adam Barker
Huawei
Huawei
Huawei
arXiv:2607.01831v1 [cs.DC] 2 Jul 2026
ABSTRACT Long-context inference is increasingly common in large language model (LLM) serving, driven by retrieval-augmented generation and agentic systems. In disaggregated inference, these workloads require transferring large Key-Value (KV) caches across the network, where decoding cannot begin until the transfer completes. Recent KV quantization techniques reduce data volume and alleviate this bottleneck, but existing schemes fail to achieve both low network-exposed latency and high inference accuracy. We challenge the assumption that the KV cache is an indivisible unit that must be fully received before use. We leverage the observation that different bits in the KV cache contribute unequally to attention computation and inference precision: the most significant bits capture the coarse structure of attention and the least significant bits refine precision. This property enables partial use of the KV cache during decoding. We present Lynx, a system that enables progressive, split-stream KV transfer by partitioning the KV cache into a high-priority Anchor stream carrying the most significant bits and a low-priority Residual stream carrying remaining precision. Decoding begins upon receipt of the Anchor stream and proceeds speculatively while the Residual stream is transferred concurrently, followed by verification that ensures equivalence to higher-precision decoding. Across multiple models and serving workloads, Lynx achieves Time-to-First-Token (TTFT) comparable to aggressive 4-bit KV quantization, while matching the accuracy of high-precision (BF16) inference, improving TTFT over standard 8-bit KV quantization by up to 1.43× and improving accuracy over state-of-the-art by up to 5.1%.
1
INTRODUCTION
Large Language Model (LLMs) inference is increasingly dominated by long-context workloads, where prompts of tens to hundreds of thousands of tokens are common in retrivalaugmented generation [54], agentic systems [6], and code intelligence [52]. State-of-the-art models such as Gemini 3.0 [13], Qwen 3 [51], and DeepSeek 3.2 [30] already support context windows of more than 1 million tokens.
In order to sustain throughput under these workloads, modern serving systems disaggregate inference into a computebound prefill stage and a memory-bound decode stage, placing them on separate accelerator instances [29, 37, 61]. While prefill-decode disaggregation improves hardware utilization, it introduces a fundamental bottleneck: the transfer of KeyValue (KV) cache1 between instances [55]. The KV cache grows linearly with context length and model depth, reaching tens of gigabytes for a single longcontext request [40, 53]. Even on high-bandwidth interconnects, transferring this state introduces substantial latency that directly inflates Time-to-First-Token (TTFT) and delays decoding. As context lengths continue to scale, KV transfer latency, not compute, becomes the dominant limiter of end-to-end inference performance in disaggregated deployments [56, 60]. Existing systems mitigate this cost by quantizing the KV cache prior to transfer, which compresses KV cache 2 to INT4 or INT8 [16, 33, 35, 56]. This effectively reduces the KV volume, which yields accelerated KV transfer over the network. As illustrated in Figure 1, while these approaches reduce data volume, they preserve a strict serialization barrier: the decode stage cannot begin until the full (compressed) KV cache has arrived and been reconstructed. Layer-wise pipelines partially overlap communication and computation, but when network bandwidth is the bottleneck, decoding still stalls at the boundary of each layer. As a result, current KV compression schemes reduce how much data is transferred by not when decoding can begin. This paper challenges a core assumption underlying KV transfer: that the KV cache is an indivisible unit that must be fully received before it can be used. Our key observation is that different bits within KV values contribute unequally to the attention probability distributions. The Most Significant Bits (MSBs) determine the magnitude of attention scores and largely preserving the relative ranking of token associations, while the Least Significant Bits (LSBs) primarily refine precision. This asymmetry suggests that a 1 KV cache encodes the context information of historical tokens and is used
for the attention computation in LLM’s forward pass computation. 2 Usually from BF16 [21] or FP16. 1
SIGCOMM’18, August 21-23, 2018, Budapest, Hungary
W. Han, G. Yeung, M. Barletta, W. Toner, A. Hoste, A. Barker
Accuracy 𝐈𝐍𝐓𝟖 𝐊𝐕
Network Compute
𝐃𝐞𝐜𝐨𝐝𝐞
System
90%
𝐈𝐍𝐓𝟖 𝐏𝐫𝐞𝐜𝐢𝐬𝐢𝐨𝐧 Network Compute
𝐈𝐍𝐓𝟒 𝐊𝐕
70%
𝐃𝐞𝐜𝐨𝐝𝐞
𝐈𝐍𝐓𝟒 𝐏𝐫𝐞𝐜𝐢𝐬𝐢𝐨𝐧/𝐂𝐚𝐜𝐡𝐞𝐆𝐞𝐧 Network Compute
𝐌𝐒𝐁 𝟒𝐛
𝐋𝐒𝐁 𝟒𝐛 𝐃𝐞𝐜𝐨𝐝𝐞
Refinement
Quantization
Transfer Protocol
TTFT
TT32T
Quality
Standard (BF16)
None
Monolithic
4.4s
5.9s
85.25%
INT8
INT8
Monolithic
2.3s
3.8s
85.06%
INT4
INT4
Monolithic
1.4s
2.9s
76.46%
CacheGen [33]
Delta Encoding
Monolithic
N/A
N/A
80.07%
Lynx (Ours)
Split-Stream
Pipelined
1.6s
3.4s
85.20%
Table 1: Comparison of Lynx against KV transfer baselines on the MMLU-Pro [47] dataset, 16K context length and Qwen 32B [51] model. Lynx significantly reduces the Time-to-firstToken (TTFT) and Time-to-32-Token (TT32T) while maintaining BF16 generation quality. Full results are shown in §6.
90%
𝐋𝐲𝐧𝐱
Figure 1: Inference timelines showing that Lynx achieves both high accuracy and low inference latency by overlapping the network communication of KV transfer with the computation of decoding.
Exposed Communication Deserialize
partial KV cache is sufficient to begin the decoding phase, provided that the missing precision is recovered before final acceptance. Based on this insight, we develop Lynx, a system that implements progressive, bit-wise split-stream KV cache transfer and overlaps 12 network communication with speculative decoding. We generalize the standard Draft-then-Verify paradigm [8, 24]—which traditionally uses a smaller model to 8 generate tentative tokens—to the network layer. In Lynx, the approximation is not derived from a separate model, but from the partial MSBs of the KV cache itself. As shown in4 Figure the KV cache into two BF16 1, Lynx INT8 splitsLynx Method independent streams: a high-priority Anchor Stream containing the MSBs of quantized KV values, and a Residual Stream carrying the remaining precision. The Anchor Stream is transmitted first, allowing the decode instance to immediately begin attention computation and generate speculative tokens. While decoding proceeds, the Residual Stream is transfered in parallel and later used to verify and correct speculative outputs, ensuring that final generation recovers high-precision fidelity. To make this execution model practical, Lynx introduces a hierarchical quantization scheme that preserves attention probability distributions under aggressive MSB truncation, and a split-stream serving runtime that supports non-blocking KV access, and efficient verification. By orchestrating these components, Lynx transforms the network from a passive bottleneck into an active pipeline for speculative execution. We evaluate Lynx on three different inference datasets with three different models including LLaMA [14], Qwen [51], and Mistral [36], and compare Lynx against state-of-the-art baselines including Cachegen’s delta encoding [33]3 , standard INT4 and INT8 quantization schemes. Our results show
Decode Rejected Spec Tokens
Time (s)
12 8
Time (s)
4
BF16
INT8 Method
Lynx
(a) LLaMA over MMLU-Pro, 128K, 25Gbps
Figure 2: Time breakdown of the disaggregated inference pipeline (Section 6.4). Prefill is omitted. ’Deserialize’ includes decompression overhead and the loading of serialized KV data into paged device memory.
that Lynx achieves INT4-level KV transfer latency while maintaining INT8/BF16-equivalent inference accuracy, reducing both TTFT and Time-to-32-nd Token by up to 30.5% and 10.7% compared to the INT8 baseline and outperforms CacheGen’s inference accuracy by up to 5.1%. Table 1 and Figure 2 preview our results. These gains increase with longer context lengths and lower available bandwidth, precisely the regimes where disaggregated inference is most challenging. This paper makes the following core contributions: • Hierarchical Split-Stream Quantization: We propose a quantization algorithm that physically partitions the KV cache into coarse-grained Anchors and fine-grained Residuals. By employing non-linear logarithmic quantization and outlier-aware chunking, we minimize the reconstruction error of the KV cache to preserve the attention output distribution within a constrained bit budget. • Split-Stream Pipelining Architecture: We propose a network-system co-design that prioritizes the transmission of critical Anchor bits. This architecture integrates a custom SerDes protocol with a non-blocking serving runtime, enabling the Decode phase to commence on partial
3 As our experiments are conducted on Ascend NPUs that CacheGen does
not support yet, we only report the generation quality of our best-effort port of CacheGen. 2
Lynx: Progressive Speculative Quantization
SIGCOMM’18, August 21-23, 2018, Budapest, Hungary
KV states while the Residual stream is transferred concurrently in the background. • Speculative Decoding Verification: We design a verification protocol leveraging the formal guarantees of speculative decoding. This ensures that the distribution of tokens generated by our inference system is identical to those drawn from the un-quantized full-precision model, guaranteeing the same expected generation quality.
2
sequence length (𝐿). For a model with hidden dimension 𝐷 (product of attention heads and head dimension) and layer count 𝑁𝑙𝑎𝑦𝑒𝑟𝑠 , the total KV cache size is defined as: Size𝐾𝑉 = 𝐾 𝑓 𝑎𝑐𝑡𝑜𝑟 · 𝐿 · 𝐷 · 𝑁𝑙𝑎𝑦𝑒𝑟𝑠 · Precision
where 𝐾 𝑓 𝑎𝑐𝑡𝑜𝑟 is 2 for standard attention and 1 for MultiHead Latent Attention. Consider a single request with a 128k context window processed by Qwen3-235B-A22B (𝑁𝑙𝑎𝑦𝑒𝑟𝑠 = 94, 𝐻𝑘𝑣 = 4, 𝐷ℎ𝑒𝑎𝑑 = 128). This single request generates approximately 23.5 GB of KV data in bfloat164 . Transferring this payload creates a massive latency penalty in two primary deployment scenarios. Scenario 1: Disaggregated Prefill-Decode. On a standard 100Gbps TCP/IP interconnect, transferring this 23.5 GB payload requires roughly 2 seconds, effectively stalling the inference pipeline. While state-of-the-art engines [23, 37] attempt layer-wise pipelining (computing layer 𝑖 while receiving 𝑖 +1), this approach fails when the transfer over the network exceeds computation time (𝑇𝑐𝑜𝑚𝑚 > 𝑇𝑐𝑜𝑚𝑝 ), negating the benefits of disaggregation. Scenario 2: Remote Context Retrieval. Similarly, serving long-context models often requires retrieving KV cache states from remote storage [32]. Even in high-performance clusters equipped with 400Gbps, the transfer consumes over 0.5 second per request. Standard inference engines operating in blocking modes suffers a direct penalty on the TTFT proportional to the network latency. Consequently, simply increasing physical bandwidth is insufficient. Reducing the data volume itself is required, leading to two primary strategies: quantization and compression.
BACKGROUND & LIMITATIONS
Large Language Model (LLM) services are built upon the Transformer architecture [45], which relies fundamentally on the Attention mechanism operating at each layer. The standard inference process is autoregressive: the model takes an input sequence (the prompt), and generates output tokens sequentially, where each new token is conditioned on the entire previous context. During each generation step, the attention mechanism at each layer must compute the relationship between the current input token, and all preceding tokens. Formally, for the current token, the model generates three projections: the Query (Q), Key (K), and Value (V) tensors. The current Q is then used to calculate an attention output 𝑂 against the K and V tensors of all previous tokens defined as: √︁ 𝑂 = softmax(𝑄𝐾 𝑇 / 𝑑𝑘 ) · 𝑉 , where 𝑑𝑘 is the model head dimension. To avoid redundant re-computation, these historical Key and Value tensors are preserved in memory as the KV Cache. This structure typically resides in high-bandwidth memory (HBM) on the device accelerators, or is offloaded to host DDR to support long-context generation [9, 32, 41].
2.1
2.3
Quantization
Quantization is an effective method to reduce data volume. In LLMs, applying quantization to both weights and activations has proven highly effective for reducing memory consumption and accelerating inference [15, 16, 19, 28, 50, 57]. Unlike weight quantization that is often handled predeployment, KV cache quantization is primarily handled at runtime, where each request corresponds to unique data. Standard & Grouped Quantization. Standard quantization maps floating-point values to integers via a linear transform: 𝑄 (X) = ⌊X/𝑆 + 𝑍 ⌉. Critical to this process is the scaling factor 𝑆, which is derived from the tensor’s dynamic range [x𝑚𝑖𝑛 , x𝑚𝑎𝑥 ]. However, relying on a global scale 𝑆 for an entire tensor is proven to be insufficient [10, 50]. Grouped Quantization [10, 57] further partitions the tensor into smaller blocks, calculating independent scales for each block to capture finer grained range of values. The Challenge of Activation Outliers. Despite these techniques, existing quantization schemes faces a critical hurdle
Prefill, Decode, and Disaggregation
Standard LLM inference partitions execution into two phases with distinct hardware requirements: a compute-intensive Prefill phase that processes the prompt in parallel, and a memory-bandwidth-bound Decode phase that generates tokens sequentially [5, 20]. To address the resource contention between these phases, state-of-the-art systems adopt Disaggregated Inference (PD-Disaggregation), routing prefill and decode requests to specialized instances [38, 59]. While this architecture optimizes throughput, it creates a rigid dependency: the decoding instance cannot begin computation until it receives the requisite KV cache data from the prefill instance, whether transferred as a monolithic or in layer-wise chunks.
2.2
(1)
The KVCache Transfer Bottleneck
Transferring the KV cache over the network creates a scalability barrier. The memory footprint grows linearly with
4 Calculation: 2 × 94 × 131, 072 × 4 × 128 × 2 ≈ 23.5 GB.
3
SIGCOMM’18, August 21-23, 2018, Budapest, Hungary
W. Han, G. Yeung, M. Barletta, W. Toner, A. Hoste, A. Barker
in LLMs: activations are dominated by extreme outliers [10, 49]. For a structure such as KV cache, the hidden dimension 𝐷ℎ𝑒𝑎𝑑 consists of independent feature columns, referred to as channels. Empirical analysis shows that a small subset of features consistently holds values up to 100× larger than the rest. These outliers break linear mapping with the standard quantization schemes because the scale 𝑆 must be expanded to accommodate the extreme magnitude 𝑚. For a non-outlier channel 𝑖 with local maximum 𝑚𝑖 and global maximum 𝑚, the effective number of bins is reduced to:
200
0
0.0
100
100
200
0
Output Tokens
100
200 -0.04
Output Tokens
(b) Compression error of INT4
(a) Attention output
Figure 3: Visualization of the attention output (left) and the corresponding INT4 quantization error (right). The error magnitude is significant relative to the feature scale, indicating substantial precision loss.
(2)
100 200
200
50
Channels
0
100
100
-50 -100
0
100
Tokens
200
(a) 𝐾 Cache, Layer 0
0
100
Tokens
200
(b) 𝐾 Cache, Layer 32
Figure 4: Visualization of KV cache values at different layers (Qwen 32B on the MMLU-Pro dataset). The persistence of extreme values in specific outlier channels (visible as dark horizontal stripes) highlights the need for a per-channel quantization approach.
Speculative Decoding
Speculative decoding accelerates LLM inference [8, 24]. It employs a lightweight approximation of the target model (often referred to as a draft model) to rapidly generate a sequence of candidate tokens. The target model then verifies the candidates in parallel, accepting a prefix of the sequence and correcting where the approximation diverges. A defining property of speculative decoding is its losslessness: it guarantees that the final output distribution is mathematically equivalent to sampling from the target model directly. The acceptance rate of speculative decoding depends on the alignment between the draft and target distributions; if the draft model is a poor approximation, frequent rejections occur, thereby diminishing the efficacy of the approach. Consequently it is critical that the draft model effectively approximate the target model.
3
Channels
Channels
100
where 2𝐵 is the total bucket count. With a 100× outlier ratio (𝑚 ≈ 100𝑚𝑖 ), the resolution for the majority of non-outlier channels collapses. In an 8-bit system (256 bins), this effectively degrades precision to roughly 1.5 bits (log2 3), causing large quantization error [50]. While quantization effectively mitigates the transfer bottleneck by reducing the bit-width of the KV cache, it is inherently lossy. Compressing the attention keys and values introduces approximation errors that accumulate during the autoregressive generation process. Relying solely on quantization therefore forces a compromise: one must sacrifice output quality to achieve lower startup latencies. A standard technique for resolving this trade-off—by allowing approximate proposals that are later verified without altering the final output distribution—is speculative decoding [8, 24].
2.4
200
Channels
𝑚𝑖 Bins𝑒 𝑓 𝑓 ≈ 2𝐵 · 𝑚
0.04
3.1
Rethinking KV Availability
Current systems suffer from a "transfer bottleneck" because they view KV cache availability as a binary state. We argue for a shift toward progressive execution, analogous to progressive rendering in web browsing. In this model, generation commences using a low-precision subset of the KV cache. These approximated outputs are then validated upon the arrival of the remaining precision bits. Hypothetically, initiating decoding with only 50% of the data volume could halve the TTFT. However, this introduces two challenges. First, a formal verification protocol is required to guarantee the accelerated output remains identical to the standard, full-precision model. Second, to ensure these approximate outputs actually pass validation, it is critical to prioritize the transmission of the most information-dense segments of the cache. While a naive method might simply transmit Most-Significant Bits (MSBs), as we show, a more robust quantization strategy is required to minimize verification failures and fully realize these latency gains.
NOT ALL BITS ARE EQUAL
This section proposes a new perspective on KV availability: initiating decoding prior to complete KV cache transmission. We argue that while early decoding is viable, standard linear quantization is inadequate to support it. 4
Lynx: Progressive Speculative Quantization
SIGCOMM’18, August 21-23, 2018, Budapest, Hungary
1.00 𝐾𝑉 Cache
0.75
CDF
Lynx Quant
NIC
Prefill Worker
0.25
NPU
-0.17 0.17 0
1
NPU
NPU
𝑸𝒓𝒆𝒔 NPU
Prefill Instance
Normalized Vals
Lynx Deserializer
NIC
Lynx Dequant Lynx Spec Decoder NPU
NPU
NPU
NPU
Decode Instance
Figure 6: System overview of Lynx.
4
Figure 5: The KV cache data distribution after tokenwise (per-channel) normalization that effectively isolates extreme outliers. The x-axis is normalized values from −1 to 1 and the y-axis is the cumulative proportion of the KV cache data whose normalized value is smaller than the given value. We see that 50% of the KV cache elements concentrate within the middle 17% of the value range.
3.2
𝑸𝒂𝒏𝒄
Lynx Serializer
0.50
0.00-1
𝐾𝑉 Cache
LYNX: ENABLING PROGRESSIVE EXECUTION
We present Lynx, a system designed to enable progressive execution during KV transfer. Lynx, depicted in Figure 6, is built upon a non-blocking runtime consisting of two main components: i) a Stream Serializer/Deserializer (SerDes) for high-throughput data handling, and ii) a Speculative Decoder for asynchronous generation. By coupling this streaming runtime with a distribution-aware quantization algorithm, Lynx effectively hides network transfer latency behind the useful work of speculative token generation. Lynx redefines the contract between prefill and decode instances. Instead of treating the KV cache as a monolithic tensor that must be transferred atomically, Lynx manages it as a bifurcated data stream. The system operates via three pipelined mechanisms:
The Limitation of Linear Quantization
We first examine the feasibility of standard low-precision quantization. As shown in Figure 3, standard INT4 quantization of the KV cache significantly distorts the attention output distribution compared to the full-precision ground truth. This degradation is largely attributable to the outliers discussed in Section 2.3. Figure 4(a) and Figure 4(b) reveal that certain feature dimensions exhibit consistently high magnitudes. Accommodating these extremes forces the quantization grid to stretch, resulting in a loss of fidelity for non-outlier elements. However, simply removing outliers does not solve the problem. As illustrated in Figure 5, even after isolating the extremes, the remaining KV cache values do not spread uniformly. Instead, they cluster densely around zero, following a sharp Laplacian-like distribution [22]. Applying a linear mapping to this non-uniform data creates two inefficiencies:
(1) Hierarchical Quantization: The prefill instance employs a novel hierarchical quantization algorithm to decompose the KV cache into two logical components: i) a high-fidelity Anchor Stream, containing the MSBs and outlier-aware scalars, and ii) a Residual Stream, containing the LSBs correction terms. (2) Prioritized Transmission: Lynx prioritizes the network transfer of the Anchor Stream, minimizing the time-tofirst-token for the decode instance. (3) Speculative Generation: Upon receiving the Anchor Stream, the decode instance immediately commences generation, using the approximate KV cache to produce draft tokens. Once the Residual Stream arrives, the instance verifies the draft tokens, confirming or possibly correcting the output.
• Wasted Capacity: A significant portion of the quantization bins are allocated to the tail ends of the distribution where almost no data points exist. • Starved Precision: The zero center, which contains the majority of the information, is forced to share a small number of bins, causing severe information loss. These observations show that KV transfer need not be a blocking atomic operation. Decoding can begin using a partial, low-precision view of the KV cache, but doing so requires careful control over approximation error to ensure that speculative outputs remain faithful to the full-precision model. In the next Section we translate these insights into concrete system design choices for Lynx, which enables progressive KV transfer through prioritized streaming and lossless speculative verification.
The following subsections detail the design of the three core mechanisms Lynx: Hierarchical Quantization algorithm (Section 4.1), the Prioritized Transmission mechanism (Section 4.2), and the Verification mechanism (Section 4.3).
4.1
Hierarchical Quantization
As discussed in Section 3.2, standard quantization relies on a linear mapping using a global or per-channel scaling factor 𝑆. However, applying directly to the KV cache is insufficient: 5
SIGCOMM’18, August 21-23, 2018, Budapest, Hungary
W. Han, G. Yeung, M. Barletta, W. Toner, A. Hoste, A. Barker
Algorithm 1: Lynx Hierarchical Quantization (Vectorized)
2) Per-channel, chunk-wise Outlier Isolation. We further subdivide the page into local chunks to isolate outliers in a per-channel, per-chunk manner. A chunk refers to a consecutive sequence of entries in a KV cache that reside within a channel and span tokenwise across a consecutive number of tokens (e.g., 𝐶 = 32). The design choice of per-channel outlier isolation is motivated by the fact that extreme outliers usually concentrate within a few channels and the entries within each channel are of the same order of magnitude (Figure 4). We further partition entries in each channel into chunks, as a lower chunk size further reduces the quantization range, resulting in a lower compression error. Specifically, for each chunk, we calculate a local mean 𝜇𝑐ℎ𝑢𝑛𝑘 and a local scalar 𝜎𝑐ℎ𝑢𝑛𝑘 . These parameters are used to center the distribution around zero (by subtracting the mean) and absorb the magnitude of any local outliers (by dividing by the scalar). The use of a local scalar ensures that an extreme value in one chunk only inflates the scaling factor 𝜎𝑐ℎ𝑢𝑛𝑘 of the local region, rather than dictating the quantization step size for the entire token page. 3) Non-Linear Transformation. We apply a logarithmic transformation 𝑙𝑛 to the normalized data using the standard 𝛼-law formulation, where 𝛼 controls the skewness: ln(1 + 𝛼 |x𝑛𝑜𝑟𝑚 |) 𝑦= · (27 − 1), 𝛼 > 0 (3) ln(1 + 𝛼) The inverse of this transformation implies that the reconstructed value 𝑥ˆ follows an exponential curve relative to the stored absolute value of integer 𝑦:
Require: Page Block X ∈ R𝐻 ×𝑃 (𝐻 : Block Size, 𝑃: Tokens) Require: Chunk Size 𝐶 Ensure: Anchor Q𝑎𝑛𝑐 , Residual Q𝑟𝑒𝑠 1: // Stage 1: Per-channel, page-Level Normalization 2: 𝜸 𝑚𝑖𝑛 ← min(X, axis=1) 3: 𝜸 𝑠𝑐𝑎𝑙𝑒 ← max(X, axis=1) − 𝜸 𝑚𝑖𝑛 4: X ← (X − 𝜸 𝑚𝑖𝑛 ) ⊘ (𝜸 𝑠𝑐𝑎𝑙𝑒 + 𝜖) 5: // Stage 2: Per-channel, chunk-wise Isolation 6: X𝑣𝑖𝑒𝑤 ← Reshape(X, [𝐻 · (𝑃/𝐶), 𝐶]) 7: 𝝁 ← Mean(X𝑣𝑖𝑒𝑤 , axis=2); X𝑐𝑒𝑛𝑡 ← X𝑣𝑖𝑒𝑤 − 𝝁 8: 𝝈 ← max(|X𝑐𝑒𝑛𝑡 |, axis=2); X 𝑓 𝑖𝑛𝑎𝑙 ← X𝑐𝑒𝑛𝑡 ⊘ (𝝈 + 𝜖) 9: // Stage 3: Non-Linear Transform ln(1+𝛼 |X 𝑓 𝑖𝑛𝑎𝑙 | ) 10: Y ← · (27 − 1) ln(1+𝛼 ) 11: I ← Round(Y) ⊲ Raw Integers 12: S ← Sign(X𝑣𝑖𝑒𝑤 ) 13: // Stage 4: Split-Stream Construction 14: I𝑏𝑖𝑎𝑠 ← I + 7 ⊲ Round-Half-Down Bias 15: V𝑚𝑎𝑔 ← I𝑏𝑖𝑎𝑠 ≫ 4 ⊲ Extract Magnitude (Anchor) 16: V𝑟𝑒𝑐𝑜𝑛 ← V𝑚𝑎𝑔 ≪ 4 ⊲ Reconstruct Base 17: Q𝑟𝑒𝑠 ← V𝑟𝑒𝑐𝑜𝑛 − I ⊲ Calculate Correction Term 18: Q𝑎𝑛𝑐 ← V𝑚𝑎𝑔 ⊙ S + min(S, 0) ⊲ Two’s Complement Map 19: return Q𝑎𝑛𝑐 , Q𝑟𝑒𝑠 , {𝜸 𝑚𝑖𝑛 , 𝜸 𝑠𝑐𝑎𝑙𝑒 , 𝝁, 𝝈 } the range of outlier channels forces 𝑆 to be large, collapsing the small values into the zero bin (𝑍 ). To overcome these drawbacks, we propose a Hierarchical Non-Linear quantization scheme. We introduce two key modifications to the standard linear quantization:
𝑦
|𝑥ˆ | ∝ (1 + 𝛼) 127
(4)
4) Split-Stream Construction. We construct the integer bits represent a functional equivalence to floating-point: • Anchor Stream (MSBs Exponent): The top 4 bits partition the dynamic range into geometric intervals. For a quantized anchor value 𝑞𝑎𝑛𝑐 ∈ Q𝑎𝑛𝑐 , the reconstructed magnitude scales with (1 + 𝛼)𝑞𝑎𝑛𝑐 . This effective exponentiation allows the Anchor Stream alone to capture the Order of Magnitude required to calculate attention scores during speculative decoding. • Residual Stream (LSBs Mantissa): The bottom 4 bits perform linear interpolation within the geometric interval defined by the anchor. To minimize quantization error, we employ a Round-Half-Down bias (+7) during extraction, ensuring the residual encodes a centered, signed correction term for maximum precision. Crucially, we employ a sign-aware mapping (Algorithm 1, line 18) to map −0 to distinct negative integers. This preserves the sign bit even for small-magnitude values, which is vital for maintaining the correct sign of attention scores in the low-precision regime.
• Granularity Shift: Instead of a single scale 𝑆, we decompose the normalization into a two-level channel-wise hierarchy (Page 𝛾 and Chunk 𝜎) to preserve and isolate the outliers spatially. • Non-Linearity: We replace the uniform linear mapping with a non-linear logarithmic transformation. This aligns the quantization bins with the distribution of the residuals, allocating higher precision to the dense region to maximize the information carried by each bit. The complete procedure is formalized in Algorithm 1 and consists of four stages. 1) Page-Level Normalization. We process the cache in logical pages (e.g. 𝑃 = 256 tokens) per channel, complying with the spread out paged attention employed by inference engines [23]. First, we normalize the floating-point distribution into a positive bounded range. We calculate the global minimum (𝛾𝑚𝑖𝑛 ) and a scaling factor (𝛾𝑠𝑐𝑎𝑙𝑒 ) for the page, mapping the values to a normalized domain [0, 1]. 6
Lynx: Progressive Speculative Quantization
SIGCOMM’18, August 21-23, 2018, Budapest, Hungary Data Plane
Data Plane Sync Control
Host
AI Chip 𝐾𝑉
Sync Control
① Send 𝑟𝑞1..𝑛
Done ③
Deq𝐮𝐚𝐧𝐭 𝐊𝐞𝐫𝐧𝐞𝐥
DMA
DMA ④
𝑄𝑟𝑒𝑠
Sync Control
AI Chip 𝐾𝑉
Pull_𝑄𝑟𝑒𝑠 ① Done⑥
⑤
Deq𝐮𝐚𝐧𝐭 𝐊𝐞𝐫𝐧𝐞𝐥
Pull② ③
Host
Done Lynx DeSerializer
𝑟𝑞1 … 𝑟𝑞𝑛
④
Async Control
Decode Lynx Spec ① Execution Decoder
⑤
Done Lynx DeSerializer ⑤
… 𝑟𝑞𝑛
③
𝑄𝑎𝑛𝑐
Host Lynx Spec Decoder Pull_𝑄𝑎𝑛𝑐 ① Done⑥
⑤
Lynx Serializer 𝑟𝑞1
Data Plane
Async Control
AI Chip 𝐾𝑉
Prefill Worker
② 𝐐𝐮𝐚𝐧𝐭 𝐊𝐞𝐫𝐧𝐞𝐥
Async Control
④
𝑟𝑞1 … 𝑟𝑞𝑛
④ ④
Pull② ③
DMA
𝑄𝑎𝑛𝑐
⑤
𝑄𝑟𝑒𝑠
𝑄𝑎𝑛𝑐
𝑄𝑟𝑒𝑠
NIC
NIC
NIC
(a) Lynx prefill instance.
(b) Anchor Phase: Receiving 𝑄 𝑎𝑛𝑐 on Lynx decode instance.
(c) Residual Phase: Speculative decoding in parallel with receiving 𝑄𝑟𝑒𝑠 on the decode instance.
Figure 7: The Lynx’s prefill and decode workflows described in §4.2, where the red arrows refer to synchronous control operations, the dashed red arrows refer to asynchronous control operations and the black arrows refer to the computation on the AI chip and the KV cache data movement. 𝑟𝑞 1, · · · , 𝑟𝑞𝑛 are pieces of control messages, each of which encodes a request to serialize and send, or fetch and deserialize a page of KV cache data.
4.2
Prioritized Transmission Mechanism
finally scattered into the KV pages ⑤. Upon completion ⑥, the model immediately accesses the Anchor state for execution, while metadata is cached for the next phase. • Residual Phase (Figure 7(c)): While the decoder executes speculatively, it concurrently issues a Pull_𝑄𝑟𝑒𝑠 command ①. The deserializer retrieves the buffered 𝑄𝑟𝑒𝑠 stream ②-③, combines it with the stored metadata, and invoke the dequantization kernel ④. The kernel refines the KV cache to full precision (𝑄 𝑓 𝑢𝑙𝑙 ) ⑤, allowing the decoder to transition the verification stage upon the done signal ⑥.
Lynx implements a non-blocking architecture composed of two logical modules (Figure 6): a Stream SerDes that manages data handling and transmission, and a Speculative Decoder that orchestrates the generation workflow. The serializer and deserializer manage the quantization and transmission of paged KV cache objects in a pipelined fashion. Prefill Instance Workflow (Figure 7(a)). The process initiates when the Prefill Runner submits requests (𝑟𝑞 1..𝑛 ) to the serializer ①. The Serializer dispatches these to the quantization kernel ②, which processes each KV object X to generate the quantized components (𝑄 𝑎𝑛𝑐 , 𝑄𝑟𝑒𝑠 ), and metadata (𝛾, 𝜇, 𝜎). To maximize DMA throughput, the kernel enforces a contiguous memory layout: writing the compacted 𝑄 𝑎𝑛𝑐 and metadata to a transmission buffer via DMA ③. Finally, the serializer pushes the 𝑄 𝑎𝑛𝑐 ④ and 𝑄𝑟𝑒𝑠 ⑤ to the NIC via two separate queues. Crucially, Lynx implements pipeline overlapping, the network transmission of object 𝑖 runs concurrently with the quantization of object 𝑖 + 1, effectively hiding the serialization overhead, ensuring the network link remains saturated. Decode Instance Workflow. On the decode instance side, the deserializer reconstructs the cache in two phases:
In our prototype, Anchor buffers are always drained before Residual buffers, enforcing strict priority at the stream level. Unlike layer-wise pipelining, which still blocks on per-layer completeness. Speculative Decoder. The decoder serves as the central serving routine, managing the SerDes operations as asynchronous background coroutines. Token generation triggers immediately upon the completion of the Anchor stream deserialization (Step ⑥ in Figure 7(b)). During this interval, the model operates on the partial state of the KV cache, marking output tokens as speculative drafts (𝑠 1, · · · , 𝑠𝑡 ). Before generating each subsequent token, the decoder checks the background Residual transfer status. Once the Residual stream is fully received and dequantized, the decoder transitions from generation to verification, where the draft sequence 𝑠 1, . . . , 𝑠𝑡 is validated against the now-available high-precision KV cache 𝑄 𝑓 𝑢𝑙𝑙 (detailed in Section 4.3).
• Anchor Phase (Figure 7(b)): Token generation commences when the Anchor state is available. The Speculative Decoder issues a Pull_𝑄 𝑎𝑛𝑐 command ①, triggering the deserializer to fetch incoming 𝑄 𝑎𝑛𝑐 data from the NIC ②. ③ Upon receiving the 𝑄 𝑎𝑛𝑐 , the deserializer then invoke the dequantization kernel with 𝑄 𝑎𝑛𝑐 and the metadata ④, 7
SIGCOMM’18, August 21-23, 2018, Budapest, Hungary
W. Han, G. Yeung, M. Barletta, W. Toner, A. Hoste, A. Barker
4.3
6
Verification and Correction
Upon the completion of the Residual transfer, Lynx transitions to the verification phase to ensure valid generation quality. We adapt the speculative decoding framework [8, 24] to the context of progressive KV transfer and quantization. The verification involves a single parallel forward pass using 𝑄 𝑓 𝑢𝑙𝑙 to validate the speculative sequence 𝑠 1, . . . , 𝑠𝑡 . Lynx identifies the longest prefix of tokens, accepting valid tokens and correcting the first divergence without re-computation. This verification protocol guarantees that the resulting output sequence follows the distribution produced by the target model (𝑄 𝑓 𝑢𝑙𝑙 ), thereby ensuring that Lynx maintains full-precision inference accuracy. While relaxed verification schemes exist that trade accuracy for higher acceptance rates, we find them unnecessary in our context. As shown in Figure 10 (Section 6), the high fidelity of the Anchor stream already yields a sufficiently high acceptance rate, allowing us to retain the guarantee of the original algorithm without performance penalty.
5
EVALUATION
This section presents the end-to-end testbed evaluation of Lynx. The evaluation includes popular LLM inference workloads with mid-size models and compares Lynx with a range of uncompressed and compressed baselines. Across three models and three long-context workloads, Lynx is the only approach that simultaneously achieves 1) BF16-equivalent accuracy and 2) INT4-level Time to First Token. Both gains hold when scaling Lynx to up to 128K context length and 50Gbps bandwidth. In contrast, CacheGen and INT4-compression sacrifice inference accuracy, while INT8-compression leads to both higher KV transfer latency and slightly larger compression error than Lynx with 𝑄 𝑓 𝑢𝑙𝑙 KV cache (Lynx-INT8).
6.1
Evaluation setups
Testbed. The experiments are conducted on a testbed comprising two Atlas A2 servers [17]. Each server is equipped with 8 high-performance 910B4 NPUs, each of which has a 32GB on-device HBM memory. Inference deployment setups. We evaluate Lynx on a disaggregated prefill-decode setup with two vLLM instances (one per server) interconnected by our testbed network. To emulate the network bottlenecks characteristic of long context scenarios (e.g., > 1M tokens) within our testbed’s memory constraints, we implement a rate-limiter on the KV connector. We restrict inter-server bandwidth from 10Gbps to 50Gbps as this effectively recreates the high transfer-tocompute latency ratios typical of very long context workloads running on standard high-speed cluster networks5 . Models. We select three popular LLMs of different model families: namely LLaMA 3.1 8B Instruct [14], Qwen 3 32B [51] and Mistral 3 24B Instruct [36]. Both LLaMA 8B and Mistral 24B support a context window up to 128K tokens. Qwen 32B instead supports a context length of 32K tokens. To handle requests longer than 32K tokens, we adopt YaRN method [39] as a RoPE scaling technique to support contexts of 128K tokens.
IMPLEMENTATION
We have implemented our Lynx prototype in ≈2𝑘 LoC of Ascend-C kernels [2] code for Ascend NPUs, and ≈2𝑘 LoC in Python, that is integrated into vLLM-Ascend [4], with LMCache-Ascend [3] as the KV transfer connector. The Python code implements Lynx’s progressive execution to achieve computation-communication overlaps and speculative verification to ensure the same accuracy as decoding with the higher precision (i.e., INT8) KV cache. The Ascend-C kernels implement the core logic of Lynx’s KV compression and decompression. Compression Kernel. Similar to GPUs, NPU-based compression is a memory-bound task limited primarily by DRAM bandwidth. To support efficient two-phase transfer, our kernel must serialize data such that each stream (Anchor and Residual) forms a contiguous block in physical memory. We implement this by computing dynamic memory offsets for the bit-packed payload: the kernel extracts the MSBs from input scalars, i.e., packing two 4-bit segments into a single uint8 byte, and writes them sequentially to the Anchor buffer, followed immediately by the shared scalars (𝛾, 𝜎). To hide memory latency, we utilize the Ascend NPU’s AI Vector core and Unified Buffer (UB) to implement a Double-Buffering pipeline (Ping-Pong) [18, 25]. As illustrated in Algorithm 1, the kernel loads a chunk of KV data into the Ping buffer while simultaneously compressing and writing out the Pong buffer, ensuring the execution units remain saturated.
Dataset MMLU-Pro Needle QMSum
Task Few-shot CoT QA Retrieval Summarization
Nsamples
Length
Metric
512
16/32/64/128K
Accuracy
512 200
10K 10K
Rouge-L Rouge-L
Table 2: LLM inference evaluation datasets and target metrics (Section §6.1).
Datasets. Lynx is evaluated against three inference tasks with different datasets, as summarized in 2. In MMLU-Pro [47], 5 Transferring the KV cache for a 128K tokens over a 10Gbps link ≈ same
latency ratio as transferring a 1M tokens over an 80Gbps link 8
Lynx: Progressive Speculative Quantization
SIGCOMM’18, August 21-23, 2018, Budapest, Hungary
we run multiple-choice question answering with chain-ofthoughts (CoT) generation [43, 48]. We modulate the shared context length (defaulting to 16K in Section 6.2 and 6.3) by varying the number of few-shot examples in the system prompt. We report the multiple-choice answer accuracy as the target metric. In the multilingual Needle-in-the-haystack dataset [46], the task is to retrieve specific information as appeared in the context. The average context length is ≈10𝑘 tokens. In QMSum [58], the target task is to generate a short summary to questions given by users from a long report. For both Needle and QMSUM, Rouge-L [26] is applied to measure the similarity between the generated response of the model and the reference response. Throughout the paper we will use the general term “inference accuracy" to refer to the respective evaluation metrics of different datasets. Baselines. We compared Lynx against 4 KV-transfer baselines atop vLLM-Ascend as the inference engine: i) the uncompressed baseline that transfers KV data directly in BF16 [21]; ii) the INT8 and iii) INT4 uniform quantization schemes, which uniformly quantize KV cache into INT8 and INT4 formats respectively for KV transfer; and iv) CacheGen [33]’s quantization strategy, which includes several optimization techniques such as delta encoding and token-wise compression. Following CacheGen’s original setup, we configured it to use a mixed INT4 and INT8 precision for quantization. Due to the lack of an official CacheGen implementation for Ascend NPUs, we developed a best-effort port to the hardware architecture. While this implementation is not fully optimized, and may exhibit higher computational overhead, it serves as a functional baseline to evaluate CacheGen’s compression logic. Thus, we only report CacheGen’s inference accuracy, demonstrating its inaccurate compression introduces considerable accuracy degradation, which is independent of hardware performance. Lynx’s configurations. We implemented three versions of Lynx prototypes, namely Lynx, Lynx-INT4 and Lynx-INT8. Lynx-INT4 and Lynx-INT8 only quantize KV cache into a single precision using our proposed hierarchical quantization algorithm (Section 4.1), without split-streaming KV transfer for overlapping KV transfer with decoding. In Lynx, we set the chunk size 𝐶 = 32 and the page size to 256. We capped the number of speculative tokens generated by Lynx to 64, as we observe the acceptance rate diminishes for speculative tokens beyond 64 (as shown in Figure 10). After generating 64 speculative tokens, Lynx waits for KV transfer completion for the LSBs portion if not finished yet. System metrics. We measured both the inference accuracy (as defined above) and latency throughout experiments. To quantify the latency benefits of progressive execution, we introduce the Time-to-𝑘th-Token (TTKT) metric, defined as the cumulative latency from the start of decoding until the generation of the 𝑘th token. For Lynx, the metric only 9
accounts for valid tokens, i.e., accepted speculative tokens or tokens generated after Lynx’s residual stream transfer and verification of speculative tokens.
6.2
End-to-end inference accuracy
We first elaborate Lynx’s end-to-end inference accuracy compared with the baselines, and explain our end-to-end accuracy results with their respective compression error that reflects the accuracy degradation of each approach. End-to-end accuracy. Figure 8 shows the inference accuracy results of the three Lynx implementations and the baselines. The accuracy of INT8, Lynx, and Lynx-INT8 are of little statistical difference (i.e., within the ±0.3% range) from BF16’s accuracy that can be regarded as random variation. This suggests that both INT8 and Lynx-INT8 are accurate enough, and LLMs can typically tolerate the minimal compression error incurred. In contrast, we observe that all of INT4, Lynx-INT4 and CacheGen’s compression algorithms incur sufficiently high accuracy degradation across our inference workloads. For example, INT4, Lynx-INT4 and CacheGen bring accuracy drops of 8.7%, 1.7% and 5.1% respectively on MMLU + Qwen, and ROUGE-L drops of 4.5%, 0.9%, and 2.7% respectively on Needle + Mistral. Furthermore, Lynx-INT4 achieves higher accuracy than CacheGen and INT4, thanks to the optimization techniques of Lynx’s quantization algorithm (Section 4.1). Method CacheGen INT4 INT8 Lynx Lynx-INT4 vNMSE 0.11 0.53 0.0042 0.00017 0.015 Table 3: vNMSE for the LLaMA 8B + MMLU workload. Lynx and its variants, outperforming standard INT8, INT4 and CacheGen quantization schemes.
Compression error. We measure the compression error in vNMSE [44]. We calculate the error via recording the output activation after each attention layer with and without compression, using the equation E(||𝑜𝑙 − 𝑜ˆ𝑙 || 2 /||𝑜ˆ𝑙 || 2 ) over all tokens in different samples and averaged over all layers. As shown in Table 3, Lynx and its variants outperform standard INT8, INT4, and CacheGen quantization schemes by at least one order of magnitude respectively. This reflects why Lynx’s prototypes achieve less accuracy loss than other compressed baseline approaches accordingly.
6.3
End-to-end inference latency
Next, we present the end-to-end inference latency of Lynx’s prototypes, comparing the results with uncompressed BF16 and the simple quantization schemes of INT8 and INT4. TTKT. Figure 9 illustrates the time each approach requires to generate the 𝑘th output token. We note that TTKT grows linearly with respect to 𝑘 for all curves (with the same modelintrinsic tangent characterizing the model’s generation speed),
SIGCOMM’18, August 21-23, 2018, Budapest, Hungary BF16 CacheGen
0.55
0.55
ROUGE-L
ROUGE-L
0.19
0.19
0.18
0.60
0.55
0.20
0.60
ROUGE-L
Accuracy
Lynx-INT4
0.20
0.60
0.80
Accuracy
Accuracy
0.65
Lynx Lynx-INT8
0.65
0.85 0.60
INT8 INT4
ROUGE-L
0.65
W. Han, G. Yeung, M. Barletta, W. Toner, A. Hoste, A. Barker
0.75
Method
(a) MMLU LLaMA 0.55 Method
Method
(b) MMLU Qwen
0.50
0.50
Method
(c) Needle LLaMA
Method
0.18 Method
(d) Needle Mistral
(e) QMsum Qwen
Method (f) QMsum Mistral
Figure 8: End-to-end inference accuracy across different LLM inference workloads. Lynx hierarchical quantization schemes achieves similar accuracy across datasets when compared to both BF16 and Int8.
BF16 INT8
INT4 Lynx
Lynx-INT8 Lynx-INT4
4 3
3 2
TTKT (s)
4
TTKT (s)
TTKT (s)
2
2
1
1
1
2
0 0 5 10 15 20 25 30 35 40 Decode Token Index
0 0 5 10 15 20 25 30 35 40 Decode Token Index
0 0 5 10 15 20 25 30 35 40 Decode Token Index
1
(a) MMLU LLaMA
(b) Needle LLaMA
(c) QMSum Mistral
0 0 5 10 15 20 25 30 35 40 Decode Token Index
0 0 5 10 15 20 25 30 35 40 Decode Token Index
5 4 3 2 1 0 0 5 10 15 20 25 30 35 40 Decode Token Index
(d) MMLU Qwen
(e) Needle Mistral
(f) QMSum Qwen
TTKT (s)
3
6 2 TTKT (s)
TTKT (s)
4
TTKT (s)
0 0 5 10 15 20 25 30 35 40 Decode Token Index
1
2
Figure 9: End-to-end inference latency in terms of Time-to-Kth-token (TTKT) across different workloads. Lynx incurs only marginal overhead compared to its standard quantization counterparts.
except for Lynx, which is subject to the speculative KV transfer and decoding process. The key observation is that Lynx, despite transmitting both 𝑄 𝑎𝑛𝑐 and 𝑄𝑟𝑒𝑠 of 8-bit-per-coordinate in total over the network, achieves consistently lower TTKT than INT8. In particular, the TT1T (also known as TTFT) of Lynx is even on par with INT4, which only transmits only 4-bit-per-coordinate. For example, under the MMLU+QWen workload, Lynx achieves 0.87s faster TT1T and 0.39s faster TT32T than INT8, which translates to approximately 29 and 13 TPOT (time-per-outputtoken) respectively. Such a reduction is attributed to the 10
computation-communication overlap between the speculative decoding of Lynx with 4-bit anchor stream precision and the parallel network transmission of the refined residual stream. This renders Lynx to outperform INT8 and LynxINT8, since the total communication time for KV transfer is the same, but Lynx leverages the waiting time to generate speculative tokens ahead of INT8 and Lynx-INT8. Lynx’s advanced quantization and speculative transfer implementations inevitably introduce extra computational overhead compared with the simple INT8 and INT4 KV quantization schemes. However, such overhead is typically of low
Lynx: Progressive Speculative Quantization
SIGCOMM’18, August 21-23, 2018, Budapest, Hungary inference accuracy, in Figure 11 and Figure 12 respectively. Here the bandwidth is set as 25Gbps.
BF16 INT8
0.15 0.10
12 TT64T (s)
0.05 13 17 21 25
TT64T (s)
5 10 20 50 100 #Accepted Spec Tokens 12 # Proposed Spec Tokens
12
8 4
8 4
(b) Actual spec-accept probability
8
Figure 10: Evaluation of Lynx’s speculative tokens acceptance rate over the MMLU Qwen4workload. Figure (a) depicts 𝑃 (#accepted) ≥ 𝑘, given a sufficiently large number of pro0 (b) 10 depicts 25 the50heatmap posed speculative tokens. Figure Bandwidth (Gbps) showing the distribution between #proposed speculative tokens and #accepted tokens, where the sum of the diagonal indices equals to 64.8%.
0 32K 64K 128K Context Length
0
(a) Varying context length
(b) Varying bandwidth
10 25 50 Bandwidth (Gbps)
Figure 11: TT64T for the MMLU LLaMA workload across varying context length and bandwidth. Lynx incurs only marginal overhead compared to its standard quantization counterparts.
The acceptance rate of the speculative tokens. As mentioned earlier, Lynx’s improved TTKT performance compared to INT8 KV transfer schemes stems from the number of accepted speculative tokens generated in parallel with the residual stream’s transmission. Theoretically, the number of speculative tokens is upper-bounded by the constant time of the residual stream’s transmission divided by the TPOT of the model. Therefore, it is crucial for Lynx to ensure high acceptance rate of the speculative tokens. Figure 10 depicts that Lynx achieves a sufficiently high acceptance rate in our end-to-end experiments. In the MMLU Qwen workload, we find that the model generates on average 21.43 speculative tokens, of which 19.38 tokens are accepted; with 64.8% probability, the whole sequence of the speculative tokens is fully accepted. This acceptance rate is sufficient to fully hide the residual KV transfer latency in all evaluated settings, explaining why Lynx consistently outper0.65 total number of forms INT8 despite transmitting the same bits. We further evaluate the theoretical acceptance rate by measuring the length of the accepted prefix when generating 0.60tokens using the a sufficiently large number of speculative anchor stream KV cache. As shows in Figure 10(a), there is an 88% probability that at least 10 tokens are accepted, and a 70% probability that at least 20 tokens are0.55accepted.
Regarding inference accuracy, Lynx and Lynx-INT8’s accuracy remain consistently within the ±0.5% range of BF16’s accuracy. Conversely, both CacheGen and INT4 suffer from increasingly severe accuracy degradation with respect to longer contexts, e.g., from 2.5% to 5.3% for CacheGen raising the context length from 32K tokens to 128K tokens. The vanilla INT8 scheme also begins to experience an accuracy loss of up to 0.9%, due to its compression algorithm being less accurate than Lynx’s optimized algorithm (Table 3). BF16 CacheGen
0.56
Lynx Lynx-INT8
Accuracy
Accuracy
Lynx-INT4
0.35
0.44
0.50
Accuracy
INT8 INT4
0.48
0.53
0.30
0.40 Method
Method
(a) 32K tokens
(b) 64K tokens
0.25
Method
(c) 128K tokens
Figure 12: The inference accuracy varying the context length from 32K tokens to 128K tokens, running LLaMA 8B over the MMLU dataset.
Method
6.4
Lynx-INT8 Lynx-INT4
Accuracy
(a) Theoretical acceptance rate
INT4 Lynx
TT64T (s)
25 21 17 13
CCDF
1.0 0.8 0.6 0.4 0.2 0.0 1
# Accepted Tokens
level, e.g., 0.13s in MMLU Qwen, compared with Lynx’s improved inference latency overall.
Scalability analysis
Scaling context lengths to 128K tokens. We evaluate Lynx’s scalability against long context by varying the context length of our MMLU-Pro workload from 32𝐾 tokens to 128K tokens, and plot the TTKT results (where 𝐾 = 64) and
Regarding inference latency, Lynx’s TT64T outperforms vanilla INT8 consistently across different context lengths, and such gain increases from 0.22s with 32K tokens to 0.84s 11
SIGCOMM’18, August 21-23, 2018, Budapest, Hungary
W. Han, G. Yeung, M. Barletta, W. Toner, A. Hoste, A. Barker
with 128K tokens. In general, Lynx yields more inference latency reduction for longer contexts, as the residual stream’s transmission lasts longer and Lynx thus generates more speculative tokens in parallel. We note that the end-to-end results have already taken into account both the computational overhead of decompression and time for correcting the rejected speculative tokens. Scaling bandwidth to 50Gbps. We also evaluate how bandwidth affects inference latency of Lynx and baseline schemes. We choose to experiment on the MMLU LLaMA workload with 64K context length, and the results are depicted in Figure 11(b). It can be observed that Lynx consistently achieves lower TT64T than INT8 across the range of 10 to 50Gbps bandwidth, though arguably such gain shrinks from 0.86s to 0.18s for 10 and 50Gbps respectively. In general, Lynx’s improved inference latency is proportional to the saved KV transfer overhead of the residual stream compared with vanilla INT8, and Lynx yields maximum speedups under the scenario of low bandwidth and long context.
Lynx avoids both by overlapping transfer with speculative execution. Unlike prior speculative decoding work, which accelerates computation by approximating model execution, Lynx applies speculative execution at the communication boundary, overlapping KV transfer with decoding. Low-precision LLM inference. Another line of work is to use lower-precision weights and arithmetics for faster and more memory-efficient LLM inference. Weight-only quantization such as GPT-Q [12] and AWQ [27] are offline approaches that take place during the post-training phase, where only the model’s weight matrices are quantized to reduce the memory footprint. The objective here is to minimize the inference quality degradation caused by quantization, and the algorithm is allowed sufficiently long time for such offline optimization. Activation quantization during serving [7, 31] is of greater challenge, where the quantization takes place at the serving runtime. Such quantization requires not only accuracy in quantization but also efficiency in the low-precision Gemm computation, where hardwarefriendly quantization schemes [1, 11] gain an advantage.
7
RELATED WORK 8
KV Compression in Transfer. Prior works propose to accelerate KV transfer by compressing the KV cache data to a more compacted form. One major class of compression schemes is KV quantization [16, 28, 35, 50, 56], which converts high-precision representations (such as FP16 and BF16) into low-precision formats such as INT8, INT4, and microscaling floating-point formats [1]. An alternative to quantization is sparsification-based compression. ScissorHands [34] and SparQ [42] retrieve the KV cache corresponds to only the most important tokens to be transmitted. The above two categories are lossy compression schemes, in that they introduce compression error, which in turn results in inaccurate inference and degraded inference quality. In general, lossy compression cannot achieve the best of both worlds: high compression ratio and high inference quality. Conversely, Lynx achieves both by overlapping KV transfer with the computation of decoding, hiding partially the KV transmission latency of the less important bits, and by enabling decoding to proceed on a partial KV state rather than waiting for KV completeness. In parallel with lossy compression algorithms are lossless encoding algorithms. For example, Cachegen [33] proposes, atop their quantization scheme, a frequency-based arithmetic encoding scheme to re-encode the KV data into a compact bit-stream. However, such encoding is not only computationally expensive but also requires that the whole encoded data be received by the decoder before the data can be successfully, so that overlapping KV transfer with decoding becomes infeasible. Existing KV transfer techniques fundamentally trade accuracy for size reduction or compression efficiency;
CONCLUSION
This paper presents Lynx, a KV-cache compression and transfer system that rethinks KV movement as a divisible, agile, and pipelined operation, rather than a blocking bottleneck. By combining hierarchical non-linear quantization with Anchor–Residual prioritization and speculative decoding, Lynx overlaps KV transfer with token generation, minimizing exposed communication overhead while preserving full-precision output quality. More broadly, Lynx reframes KV transfer from a blocking prerequisite into a progressively usable resource, allowing decoding to proceed before KV completeness. Experiments demonstrates Lynx achieves INT4-level time-to-first-token while matching BF16/INT8 accuracy for up to 128K context length, significantly outperforming existing approaches. Lynx ’s prototype uses INT4 for both the anchor and residual streams, yielding an effective INT8 representation. This design is motivated by two considerations. First, INT8 KV caches already match BF16 accuracy with negligible degradation. Second, using power-of-two bit-widths enable efficient memory-coalesced kernel implementation on modern hardware accelerators. In principle, Lynx can support alternative bit-width configurations for the anchor and residual streams (e.g., INT6 or INT2), to trade-offs inference quality and TTKT latency. We leave a systematic exploration of these configurations to future work. Finally, while Lynx is implemented on the Ascend NPU platform, its design is hardware-agnostic and readily generalizable to other AI accelerators such as NVIDIA’s GPUs. We leave such extensions to future work. By proposing that 12
Lynx: Progressive Speculative Quantization
SIGCOMM’18, August 21-23, 2018, Budapest, Hungary
REFERENCES
decoding need not wait for KV completeness, Lynx opens a new direction for communication-aware inference system design. In essence, we remove a network-induced serialization barrier by making application execution progressive with respect to communication.
9
[1] 2023. OCP Microscaling Formats (MX) Specification. https://www.op encompute.org/documents/ocp-microscaling-formats-mx-v1-0-spe c-final-pdf. [2] 2026. Ascend Documentation. https://www.hiascend.com/en/docume nt/. [3] 2026. LMCache-Ascend Repo. https://github.com/LMCache/LMCache -Ascend/commits/main/. [4] 2026. vLLM Ascend Repo. https://github.com/vllm-project/vllm-asc end/tree/main/vllm_ascend. [5] Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gulavani, and Ramachandran Ramjee. 2023. SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills. CoRR (2023). [6] Anthropic. 2025. Claude Code. https://github.com/anthropics/claude -code Accessed: 2026-02-03. [7] Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. 2023. Quantizable transformers: Removing outliers by helping attention heads do nothing. Advances in Neural Information Processing Systems 36 (2023), 75067–75096. [8] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318 (2023). [9] Weijian Chen, Shuibing He, Haoyang Qu, Ruidong Zhang, Siling Yang, Ping Chen, Yi Zheng, Baoxing Huai, and Gang Chen. 2025. {IMPRESS}: An {Importance-Informed} {Multi-Tier} Prefix {KV} Storage System for Large Language Model Inference. In 23rd USENIX Conference on File and Storage Technologies (FAST 25). 187–201. [10] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. LLM.int8 (): 8-bit matrix multiplication for transformers at scale. Advances in neural information processing systems 35 (2022), 30318– 30332. [11] Roman Dubtsov, Evarist Fomenko, and Babak Hejazi. 2023. New cuBLAS 12.0 Features and Matrix Multiplication Performance on NVIDIA Hopper GPUs. https://developer.nvidia.com/blog/new-cubla s-12-0-features-and-matrix-multiplication-performance-on-nvidi a-hopper-gpus/. NVIDIA Technical Blog. [12] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323 (2022). [13] Gemini Team, Google. 2025. Gemini 3 Technical Report. Technical Report. Google DeepMind. https://storage.googleapis.com/deepm ind-media/Model-Cards/Gemini-3-Pro-Model-Card.pdf Accessed: 2026-02-03. [14] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024). [15] Song Han, Huizi Mao, and William J. Dally. 2016. Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.). http://arxiv.org/abs/1510.00149 [16] Coleman Hooper, Sehoon Kim, H. Mohammadzadeh, Michael W. Mahoney, Kurt Keutzer, and Amir Gholami. 2024. KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization. In Advances in Neural Information Processing Systems (NeurIPS). [17] Huawei. n.d.. Atlas 800I A2 Inference Server. https://e.huawei.com/cn/ products/computing/ascend/atlas-800i-a2 Accessed: 2026-02-06.
ACKNOWLEDGMENT
We acknowledge the use of Generative AI to refine the structure and clarity of the text. All technical analysis, experimental design, and final writing decisions remain the sole responsibility of the authors. This work does not raise any ethical issues.
13
SIGCOMM’18, August 21-23, 2018, Budapest, Hungary
W. Han, G. Yeung, M. Barletta, W. Toner, A. Hoste, A. Barker
[18] Huawei Technologies Co., Ltd. 2024. Ascend C Operator Development Guide. Huawei Ascend. https://www.hiascend.com/document/detail /en/canncommercial/800/opdevg/Ascendcopdevg/atlas_ascendc_10 _0008.html Accessed: 2025-02-05. [19] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. 2018. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2704–2713. doi:10.1109/CV PR.2018.00286 [20] Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Chin-Yew Lin, et al. 2024. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. Advances in Neural Information Processing Systems 37 (2024), 52481–52515. [21] Dhiraj D. Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang, Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kundu, Misha Smelyanskiy, Bharat Kaul, and Pradeep Dubey. 2019. A Study of BFLOAT16 for Deep Learning Training. CoRR abs/1905.12322 (2019). arXiv:1905.12322 http: //arxiv.org/abs/1905.12322 [22] Andrey Kuzmin, Mart van Baalen, Yuwei Ren, Markus Nagel, Jorn Peters, and Tijmen Blankevoort. 2022. FP8 Quantization: The Power of the Exponent. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 14651–14662. https://proceedi ngs.neurips.cc/paper_files/paper/2022/file/5e07476b6bd2497e1fbd11 b8f0b2de3c-Paper-Conference.pdf [23] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles. 611–626. [24] Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning. PMLR, 19274–19286. [25] Xiaoyao Liang. 2020. Chapter 3 - Hardware architecture. In Ascend AI Processor Architecture and Programming, Xiaoyao Liang (Ed.). Elsevier, 75–100. doi:10.1016/B978-0-12-823488-4.00003-5 [26] Chin-Yew Lin and Franz Josef Och. 2004. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd annual meeting of the association for computational linguistics (ACL-04). 605–612. [27] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Guangxuan Xiao, and Song Han. 2025. AWQ: Activation-aware Weight Quantization for OnDevice LLM Compression and Acceleration. GetMobile: Mobile Comp. and Comm. 28, 4 (Jan. 2025), 12–17. doi:10.1145/3714983.3714987 [28] Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. 2025. Qserve: W4a8kv4 quantization and system co-design for efficient llm serving. Proceedings of Machine Learning and Systems 7 (2025). [29] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024). [30] Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al. 2025. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556 (2025).
[31] Lian Liu, Long Cheng, Haimeng Ren, Zhaohui Xu, Yudong Pan, Mengdi Wang, Xiaowei Li, Yinhe Han, and Ying Wang. 2025. COMET: Towards Practical W4A4KV4 LLMs Serving. Association for Computing Machinery, New York, NY, USA, 131–146. https://doi.org/10.1145/3676641.37 16252 [32] Yuhan Liu, Yihua Cheng, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaoting Feng, Yuyang Huang, Samuel Shen, Rui Zhang, Kuntai Du, et al. 2025. Lmcache: An efficient KV cache layer for enterprise-scale LLM inference. arXiv preprint arXiv:2510.09665 (2025). [33] Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, et al. 2024. Cachegen: Kv cache compression and streaming for fast large language model serving. In Proceedings of the ACM SIGCOMM 2024 Conference. 38–56. [34] Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. 2023. Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. Advances in Neural Information Processing Systems 36 (2023), 52342–52364. [35] Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024. KIVI: a tuningfree asymmetric 2bit quantization for KV cache. In Proceedings of the 41st International Conference on Machine Learning. 32332–32344. [36] Mistral AI. 2025. Mistral Small 3. https://mistral.ai/news/mistral-sma ll-3/. [37] NVIDIA. 2025. NVIDIA Dynamo: A Low-Latency Distributed Inference Framework for Scaling Reasoning AI Models. https://developer.nvidia .com/blog/introducing-nvidia-dynamo-a-low-latency-distributed-i nference-framework-for-scaling-reasoning-ai-models/. Accessed: 2025-06-15. [38] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient generative llm inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 118–132. [39] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2024. YaRN: Efficient Context Window Extension of Large Language Models. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=wHBfxhZu1u [40] Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading more storage for less computation—a {KVCache-centric} architecture for serving {LLM} chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST 25). 155–170. [41] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. 3505–3506. [42] Luka Ribar, Ivan Chelombiev, Luke Hudlass-Galley, Charlie Blake, Carlo Luschi, and Douglas Orr. 2024. SparQ Attention: BandwidthEfficient LLM Inference. In International Conference on Machine Learning. PMLR, 42558–42583. [43] Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning To Retrieve Prompts for In-Context Learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 2655–2666. doi:10.18653/v1/2022.naacl -main.191 [44] Shay Vargaftik, Ran Ben-Basat, Amit Portnoy, Gal Mendelson, Yaniv Ben-Itzhak, and Michael Mitzenmacher. 2021. Drive: One-bit distributed mean estimation. Advances in Neural Information Processing 14
Lynx: Progressive Speculative Quantization
SIGCOMM’18, August 21-23, 2018, Budapest, Hungary
Systems 34 (2021), 362–377. [45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017). [46] Weiyun Wang, Shuibo Zhang, Yiming Ren, Yuchen Duan, Tiantong Li, Shuo Liu, Mengkang Hu, Zhe Chen, Kaipeng Zhang, Lewei Lu, Xizhou Zhu, Ping Luo, Yu Qiao, Jifeng Dai, Wenqi Shao, and Wenhai Wang. 2024. Needle In A Multimodal Haystack. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37. Curran Associates, Inc., 20540–20565. doi:10.52202/079017-0649 [47] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. 2024. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574 (2024). [48] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chainof-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS ’22). Curran Associates Inc., Red Hook, NY, USA, Article 1800, 14 pages. [49] Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. 2022. Outlier suppression: Pushing the limit of low-bit transformer language models. Advances in Neural Information Processing Systems 35 (2022), 17402– 17414. [50] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. Smoothquant: Accurate and efficient post-training quantization for large language models. In International conference on machine learning. PMLR, 38087–38099. [51] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. 2025. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388 (2025). [52] John Yang, Carlos E Jimenez, Alex L Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R Narasimhan, et al. 2024. SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? CoRR (2024). [53] Xinjun Yang, Qingda Hu, Junru Li, Feifei Li, Yicong Zhu, Yuqi Zhou, Qiuru Lin, Jian Dai, Yang Kong, Jiayu Zhang, et al. 2025. Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Management. arXiv preprint arXiv:2511.20172 (2025). [54] Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. 2025. CacheBlend: Fast large language model serving for RAG with cached knowledge fusion. In Proceedings of the Twentieth European Conference on Computer Systems. 94–109. [55] Shengnan Yue, Mowei Wang, Yu Yan, Weiqiang Cheng, Zihan Jiang, and Zhenhui Zhang. 2025. RTT-or Bandwidth-Bound? Demystifying the KV Cache Transfer in Large Language Model Serving. In Proceedings of the 2nd Workshop on Networks for AI Computing. 5–7.
[56] Zeyu Zhang, Haiying Shen, Shay Vargaftik, Ran Ben Basat, Michael Mitzenmacher, and Minlan Yu. 2025. Hack: Homomorphic acceleration via compression of the key-value cache for disaggregated llm inference. In Proceedings of the ACM SIGCOMM 2025 Conference. 1245–1247. [57] Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. 2024. Atom: Low-bit quantization for efficient and accurate llm serving. Proceedings of Machine Learning and Systems 6 (2024), 196–209. [58] Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021. QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (Eds.). Association for Computational Linguistics, Online, 5905–5921. doi:10.18653/v1/2021.naacl-main.472 [59] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. {DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). 193–210. [60] Ruidong Zhu, Ziheng Jiang, Chao Jin, Peng Wu, Cesar A Stuardo, Dongyang Wang, Xinlei Zhang, Huaping Zhou, Haoran Wei, Yang Cheng, et al. 2025. Megascale-infer: Efficient mixture-of-experts model serving with disaggregated expert parallelism. In Proceedings of the ACM SIGCOMM 2025 Conference. 592–608. [61] Pengfei Zuo, Huimin Lin, Junbo Deng, Nan Zou, Xingkun Yang, Yingyu Diao, Weifeng Gao, Ke Xu, Zhangyu Chen, Shirui Lu, et al. 2025. Serving Large Language Models on Huawei CloudMatrix384. arXiv preprint arXiv:2506.12708 (2025).
15