arXiv:2607.26475v1 [cs.DC] 29 Jul 2026
DualDecoder: Accelerate Long Context LLM Inference by Predictive Prefetch Zuning Liang
Zhiyi Yao
Qi Chen
Fudan University Shanghai Innovation Institute China [email protected]
Fudan University China [email protected]
Fudan University China [email protected]
Yuedong Xu
Hao Dai
Zhiqiang Ding
Fudan University China [email protected]
Ant Group China [email protected]
Ant Group China [email protected]
Tongkai Yang
Jinlong Hou
Yuan Cheng
Ant Group China [email protected]
Fudan University Shanghai Innovation Institute China [email protected]
Fudan University Shanghai Innovation Institute China [email protected]
Abstract
1
Long-context inference is becoming a fundamental capability for modern LLM serving, especially driven by emerging agentic applications. Yet it faces a severe memory wall that the KV cache scales proportionally with increasing context length and request concurrency. Existing sparse KV cache methods offload most KV entries to host memory and retrieve only the critical KV entries needed by each decoding step. However, they commonly introduce substantial auxiliary states in GPU memory for KV retrieval management. Our measurements show that these often-overlooked auxiliary states introduce significant memory overhead and become a new bottleneck under high-concurrency workloads. In this paper, we present DualDecoder, a lightweight serving system for long-context LLM inference that enables efficient sparse KV cache retrieval from host memory. Our key insight is that the critical KV entries required for decoding the next token can be accurately predicted from the preceding speculated token. This predictability enables KV retrieval to be proactively prefetched and overlapped with decoding computation, effectively eliminating the GPU memory overhead of auxiliary states. To achieve this prefetching efficiently, DualDecoder leverages a novel dual-token decoding pipeline that accurately identifies critical KV entries with negligible computational overhead, and designs a layer-aware transfer schedule to overlap KV prefetching with model computation and a layer-scoped memory manager to reduce the GPU runtime buffer. Experimental results show that DualDecoder improves decoding throughput by up to 2.62× over state-of-the-art systems while preserving decoding latency and model quality.
In large language model (LLM) and agentic AI services, long contexts significantly improve inference accuracy and response coherence, enabling the execution of complex realworld tasks such as full-document understanding [2, 4, 11] and large-scale codebase analysis without context fragmentation [18, 41]. Modern LLMs increasingly support context lengths ranging from hundreds of thousands to even millions of tokens [3, 26]. A major challenge in long-context inference is the limited capacity of GPU memory. To avoid repeatedly performing expensive extra matrix multiplications, LLMs cache intermediate key and value (KV) tensors generated during autoregressive decoding steps [20, 27]. However, the KV cache grows proportionally with context length and request batch size, and can quickly occupy a substantial portion of the limited GPU memory [14, 15, 21]. Consequently, reducing the memory footprint of the KV cache has become a critical problem in enabling efficient long-context inference. Dynamic sparse KV cache techniques [29, 40] provide an effective direction for mitigating this memory pressure. Since each decoding step usually attends to only a small subset of historical tokens [17], instead of keeping the entire KV cache in GPU memory, these systems offload most KV entries to host memory and retrieve only the selected sparse KV entries needed by the current attention computation. This design substantially reduces the memory footprint of GPU-resident KV entries and becomes a promising approach for high-throughput, long-context LLM serving (§ 2). However, the selected KV entries must be gathered from host memory and transferred to GPU memory before attention consumes them, while CPU-to-GPU bandwidth is much lower than GPU memory bandwidth. To reduce this retrieval 1
Introduction
overhead, existing systems keep additional auxiliary states in GPU memory. For example, such states may reconstruct selected key entries [28, 40] or guide future KV prefetching [13]. These designs are effective because they reduce the amount of exposed CPU-to-GPU transfer on the decoding critical path. Yet they also introduce a new memory cost that is easy to overlook. We refer to the GPU memory occupied by such auxiliary state as KV-cache-auxiliary residency. Our measurements show that this residency can dominate the memory usage and even consume 64% of GPU memory in heavy workloads. Under high-concurrency workloads, KVcache-auxiliary residency becomes a new bottleneck that limits the maximum request batch size and prevents dynamic sparse KV cache systems from reaching the serving throughput that the hardware could otherwise support (§ 3). In this paper, our key observation is that sparse KV retrieval can be predicted across adjacent decoding steps. In dynamic sparse KV cache mechanisms, the retrieval index of a decoding step is computed from the token being processed and a static GPU-resident KV landmarks. Therefore, once a useful prediction of the next token is available, we can estimate the retrieval index of the next step before that step reaches attention. More importantly, our measurements show that even when token prediction is imperfect, it can still provide sufficient information to predict most of the KV entries needed by the following decoding step. This predictability creates a new opportunity for long-context serving. Instead of waiting for the true retrieval index and then fetching selected KV entries on the critical path, the system can issue KV retrieval earlier and overlap host-to-device transfer with decoding computation. In this way, sparse KV retrieval can be conducted fast without relying on large GPU-resident auxiliary states, leaving more GPU memory available for larger request batches (§ 4). Based on this observation, we present DualDecoder, a lightweight long-context LLM serving system that makes sparse KV retrieval predictive rather than residency heavy. The core of DualDecoder is to predict the retrieval index of future decoding work and prefetch the corresponding sparse KV entries from host memory before they are consumed by attention. This approach allows DualDecoder to overlap KV retrieval with model computation, thereby preserving the latency benefit of dynamic sparse KV cache without maintaining large GPU-resident auxiliary state. Realizing this approach in a practical decoding pipeline requires addressing three challenges. First, the system must obtain retrieval guidance without turning each decoding step into a much heavier model execution. To address this issue, DualDecoder introduces a dual-token decoding pipeline that generates the normal output token and the retrieval-guiding token within the same decoding flow (§ 5.2). Second, predicted KV entries must be fetched early enough to hide CPU-toGPU transfer latency, but not too early that they occupy much GPU memory. DualDecoder addresses this tension
with a layer-aware transfer schedule that efficiently overlaps KV movement with model layer execution and corrects prediction misses on the critical path (§ 5.3). Third, predictive prefetching must not recreate the memory pressure that it is designed to remove. DualDecoder therefore uses a layerscoped KV memory manager that significantly reduces the runtime sparse KV buffer size and coordinates asynchronous KV transfer with attention computation (§ 5.4). Contributions. Our main contributions are summarized as follows: • We identify KV retrieval predictability as a new opportunity for long-context LLM serving. In dynamic sparse KV cache systems, the retrieval index of a future decoding step can be predicted before that step reaches attention. To the best of our knowledge, this is the first work to characterize and exploit KV retrieval predictability (§ 4). • We design DualDecoder, a lightweight predictive retrieval system that replaces residency-heavy sparse KV retrieval with prefetch-based KV movement. DualDecoder effectively predicts future sparse KV demand and efficiently overlaps KV transfer with decoding computation (§ 5). • We implement DualDecoder atop of a state-of-the-art dynamic sparse KV cache system [28] and evaluate its performance on representative long-context serving workloads. The results show that DualDecoder improves decoding throughput by up to 2.62× over state-of-the-art inference systems while preserving decoding latency and generation quality (§ 6).
2
Background
2.1
LLM Inference with KV Cache
LLM inference and batching techniques. Modern LLM serving systems commonly run inference on AI-specific accelerators, like GPUs, to meet the throughput and latency requirements of online applications [14, 38, 42]. A typical request is processed in two phases: prefilling, which consumes the input prompt, and autoregressive decoding, which generates output tokens one step at a time. Since decoding is sequential within each request, the end-to-end serving throughput largely depends on how many requests can be decoded concurrently [1]. Given the growing computing capability of GPUs, applications widely adopt batched decoding techniques to improve the overall token throughput and avoid wasting GPU resources. For example, a Llama-38B service running on a single GPU may group 4 or 8 active requests into one decoding batch and process them with the same model kernels. As long as the service-level objectives (SLOs) are not violated, increasing the decoding batch size usually improves token throughput and amortizes GPU execution cost across more requests [10, 38]. KV cache and its memory footprint. Autoregressive decoding relies on the key-value (KV) cache to avoid repeatedly recomputing the attention states of previous tokens [27, 40]. 2
2.2
Host
GPU
At each decoding step, the model computes the key and value tensors of the newly generated token and appends them to the cache as KV entries. In later steps, the attention computation reuses these cached KV entries together with the current query to generate the next token. This reuse is essential because recomputing all historical key and value tensors would introduce substantial redundant computation, especially for long-context requests. However, the KV cache also becomes one of the dominant consumers of GPU highbandwidth memory (HBM) [20]. Its footprint grows with the request batch size, context length, number of layers, and model hidden dimensions. As these factors increase, especially in long-context serving, the KV cache can occupy a large fraction of the limited GPU memory capacity. Serving systems must then reduce the decoding batch size to avoid out-of-memory errors, which directly lowers token throughput and leaves expensive GPU compute resources underutilized [9, 15, 28].
K Landmark Selection Token
...
Select
Top-k
Req 3 Req 3 Req 2 Req 2 Req 1 Req 1 Values Keys
... ... ...
Residency(Low Rank K / 2-bit KV···) Reconstruct
...
Attention
NewToken
Full KV Cache
...
... ... ...
...
... ...
Sparse KV
Gather
...
...
65535 65535 65535 65535 65535 65536 65535 65536 65536 65536 65536 65536
Figure 1. LLM decoding step with dynamic sparse KV cache. many noncontiguous KV entries from host memory before or during the host-to-device transfer, which further increases retrieval latency. To reduce this overhead, recent systems introduce GPU-resident auxiliary state for KV selection, reconstruction, or communication scheduling. ShadowKV [28] keeps a low-rank representation of the key cache on the GPU and reconstructs selected key entries on the fly, while SpeCache [13] maintains a compact quantized KV representation to guide future KV prefetching. These auxiliary states help reduce exposed retrieval latency without directly evicting useful KV information. However, because they must be accessed during decoding, they also consume valuable GPU memory. In this paper, we refer to the GPU memory occupied by such auxiliary state as KV-cache-auxiliary residency.
Dynamic Sparse KV Cache
Dynamic sparse KV cache mechanisms reduce GPU memory usage by moving most KV entries to host memory and bringing back only the entries that are needed for each decoding step [19, 24, 29]. The key observation behind these mechanisms is that next-token generation does not attend to all historical tokens uniformly. Instead, for many decoding steps, the attention computation depends primarily on a small subset of important historical tokens, while many cached KV entries have little impact on the generated token [17, 33, 37]. Based on this observation, dynamic sparse KV cache systems keep the full KV cache, or most of it, in host memory and retrieve selected KV entries into GPU HBM before attention computation, as shown in Figure 1. The selection is typically guided by lightweight GPU-resident metadata. For example, ShadowKV [28] segments adjacent KV entries into chunks and maintains compressed landmarks for these chunks, so that the GPU can select important chunks according to the current query and then retrieve the corresponding sparse KV entries from host memory. Since the landmark representation is much smaller than the full KV cache, this approach saves GPU memory and enables a larger request batch size while still preserving access to the original KV entries when they are needed. However, applying dynamic sparse KV cache mechanisms in practical LLM serving systems introduces nontrivial data movement overhead [7, 15]. First, host-to-device bandwidth is much lower than GPU HBM bandwidth. For example, PCIe 5.0 provides only 64 GBps unidirectional bandwidth, which is far below the 4.8 TBps bandwidth level of modern GPU HBM. As a result, even fetching a sparse subset of critical KV entries can block decoding if the transfer is exposed on the critical path. Second, the selected KV entries are often scattered across the long context rather than stored as one contiguous memory region. The system therefore has to gather
3
Motivation
In this section, we first introduce the severe throughput issues when we try to apply the dynamic sparse KV cache mechanisms (§ 3.1). And we then present that the throughput bottleneck stems from the memory occupancy of KV-cacheauxiliary residency (§ 3.2). 3.1
Poor Serving Throughtput
We deploy model inference systems with novel dynamic sparse KV cache mechanisms, including ShadowKV and SpeCache, on a testbed that consists of 8 GPUs. The memory capacity of each GPU is 80 GB and the interconnections between GPU and CPU offer a bandwidth of up to 64 GBps. With this testbed, we serve popular LLMs, including the Llama series [22] and Qwen series [23] models, with model sizes spanning from 8B to 32B and use mainstream longcontext benchmarks such as RULER [11] to validate serving quality. Limited batch size. We first measure the maximum request batch size that each system can support without triggering out-of-memory (OOM) errors. To understand how far existing systems are from the memory capacity limit, we also compute an ideal batch size by excluding the KV cache from GPU memory while keeping model weights and necessary runtime activations. This ideal setting is not a deployable 3
(GB)
Low-rank K (36%)
K Landmark (28%) (7.5%) (20%)
Figure 2. Maximum request batch size of decoding using full KV cache, ShadowKV and ideal scaling.
(s)