TideKV: Delayed KV Eviction via Layer-Wise Chunked Prefill for Memory-Efficient Long-Context LLM Inference

TideKV: Delayed KV Eviction via Layer-Wise Chunked Prefill for Memory-Efficient Long-Context LLM Inference Yu Wanga , Zhu Wanga,∗ , Ning Hub , Yifan Guoa , Xiu Mab , Bin Guoa and Zhiwen Yua,c a Northwestern Polytechnical University, No. 127 West Youyi Road, Xian, 710129, Shaanxi, China b Xi’an Aeronautics Computing Technique Research Institute, AVIC, No. 15 Jinye 2nd Road, Xi’an, 710065, Shaanxi, China

arXiv:2610.24499v1 [cs.LG] 17 Oct 2026

c College of Computer Science and Technology, Harbin Engineering University, No. 145 Nantong Street, Harbin, 150001, Heilongjiang, China

ARTICLE INFO

ABSTRACT

Keywords: Large language model inference key-value cache compression long-context inference chunked prefill memory management inference scheduling

Long-context large language model (LLM) inference is important for modern intelligent applications but remains constrained by peak memory during prompt prefill. Existing KV cache compression reduces decoding memory but typically requires full-prompt prefill to preserve global context, retaining a high construction peak. Online chunk-level eviction lowers memory earlier but selects before each layer has observed the complete prompt, potentially degrading cache construction and decoding. We present TideKV, an on-GPU cache-construction architecture using layer-major delayed eviction. TideKV processes all prompt chunks within each Transformer layer, retains full-prompt KV only for the active layer, and evicts at layer end. This schedule preserves full-prompt-aware selection while reducing activation and temporary memory. Experiments across multiple computing platforms show that TideKV preserves the inference quality and final cache size of the evaluated compression methods. At 64K on A100, it reduces the construction-time transient footprint by 83.89%. On RTX 4090, it increases time to first token by only 1.06% at 64K and enables 128K prefill. TideKV also completes 128K on Jetson, whereas matched post-prefill compression runs out of memory on both resource-constrained platforms. These results show that delayed layer-end eviction substantially expands feasible context lengths without sacrificing cache utility.

1. Introduction Recent advances in large language models (LLMs) have extended supported context windows to 128K tokens and beyond [1–3], enabled in part by positional-encoding extensions such as rotary embeddings [4–6]. This expansion is also driven by applications that require sustained reasoning over long horizons, including retrieval-augmented generation [7], multi-turn autonomous agents [8], long-form code synthesis [9], and extended document comprehension. Such applications routinely require tens of thousands of input tokens in a single inference session and have motivated dedicated long-context benchmarks [10, 11]. At the same time, due to requirements related to data privacy, communication latency, and regulatory compliance, the deployment of LLMs is undergoing a structural shift from centralized cloud infrastructure to edge computing platforms [12–14], such as mobile robots, embodied intelligent systems [15, 16], and in-vehicle autonomous driving modules [17]. However, resource-constrained edge accelerators operate under fixed and often severely limited device memory, creating a fundamental tension with the memory demands of long-context inference. During inference, memory is consumed by multiple concurrent components: the key– value (KV) cache accumulating per-layer attention states, the full-sequence hidden-state tensor propagated through Transformer blocks [18], intermediate MLP activation tensors, ∗ Corresponding author

wangyu23@mail.nwpu.edu.cn (Y. Wang); wangzhu@nwpu.edu.cn (Z. Wang); hnhawk@163.com (N. Hu); guoyf@mail.nwpu.edu.cn (Y. Guo); 2750162952@qq.com (X. Ma); guob@nwpu.edu.cn (B. Guo); zhiwenyu@nwpu.edu.cn (Z. Yu) ORCID (s): 0009-0006-4754-164X (Y. Wang)

Wang et al.: Preprint submitted to Elsevier

and per-layer attention score matrices. As input sequence length increases, each of these components grows correspondingly, with the KV cache scaling linearly with both sequence length and layer count [19]. The aggregate memory pressure renders long-context inference on resourceconstrained edge devices increasingly infeasible without systematic memory management. The memory pressure is most acute during the prefill phase, when the entire input prompt must be processed to construct the initial KV cache before decoding begins [20]. For a representative Transformer model such as Llama-3.1-8B [1], a single 128Ktoken prompt generates approximately 16 GiB of uncompressed KV cache. During prefill, hidden states and temporary activations coexist in device memory with the KV cache [21]. The cumulative footprint routinely exceeds the physical memory capacity of edge accelerators, resulting in out-of-memory (OOM) errors that render the service unavailable. Therefore, to enable long-context reasoning on resourceconstrained devices, it is necessary not only to control the final KV cache retained during decoding but also to limit the peak memory consumption during the construction of the KV cache. Existing KV cache compression approaches fall into two paradigms, each exposing a distinct limitation under tight memory budgets, as illustrated in Figure 1. Post-prefill selectors such as SnapKV [22] score and compress each layer’s cache after processing the complete prompt at that layer. While this preserves full visibility of the prompt during the selection process, it still results in high constructionmemory peaks because each layer still processes the whole prompt at once before compressing its cache. In contrast, online methods such as KeyDiff [23] evict KV during prompt Page 1 of 15

Delayed KV Eviction via Layer-Wise Chunked Prefill

(a) Full prefill

(b) Chunk-major

(c) Online eviction

L1

L1

L1

L1

L2

L2

L2

L2

L3

L3

L3

L3

c1

c2

c1

c3

peak

c2

c3

c1 c2

c1 c2

c3

peak

peak

(d) Layer-major (ours)

c3

peak

highest peak

high peak

low peak

all layers full KV + activation

KV still fills up (no evict)

later chunks see compressed history

visibility ✓

visibility ✓

visibility ✗ (myopic)

visibility ✓ (within layer)

active compute block

peak bar width = relative peak memory

full KV resident

compressed KV

not yet computed

low peak

Figure 1: Four schedules for constructing a compressed KV cache during prefill. (a) Full prefill without intermediate cache release retains uncompressed KV across all layers until the prompt has been processed, preserving full-prompt visibility at a high construction peak. (b) Chunked prefill without eviction processes the prompt chunk by chunk, yet every chunk’s KV stays resident, so memory still grows with prompt length. (c) Chunked prefill with online eviction evicts after each chunk to cap memory, but it decides what to keep before the full prompt is seen, which can degrade quality through myopic eviction. (d) TideKV (ours) streams the prompt chunk by chunk in a layer-major order and defers eviction to each layer’s boundary, so every layer scores against the full prompt while full-prompt KV is confined to one active layer and temporary activations to one chunk, alongside the hidden-state bridge and retained caches.

processing to control memory usage. Because these methods evict tokens before observing the complete prompt, they may prematurely remove tokens whose importance depends on later context. This myopic eviction can reduce compression quality. This creates a central visibility-residency trade-off: current methods either spend substantial memory to preserve full-prompt information, or bound memory by evicting before the prompt is fully observed, which can compromise cache quality. Full-prompt visibility can be preserved while lowering construction-time peak memory by reordering prefill execution. When a Transformer layer is driven to completion over the whole prompt before the next layer begins, and eviction is deferred to that layer’s boundary, every layer scores its tokens against the uncompressed causal history of the entire prompt, so selection is not preceded by eviction of the layer’s causal prompt history. At the same time, full-prompt KV is resident for only one layer at a time. Streaming the prompt in chunks limits temporary attention/MLP activations to one chunk, which bounds the activation workspace that dominates the prefill peak. Relative to an uncompressed cache spanning all 𝑁 layers, this schedule reduces uncompressed KV residency from 𝑂(𝑁𝑆) to 𝑂(𝑆) while preserving fullprompt visibility. We present TideKV, a prefill execution architecture that realizes this principle through layer-major delayed eviction. For each layer, TideKV streams the prompt chunk by chunk against that layer’s accumulated, uncompressed KV history; only after the layer has consumed the whole prompt does it compress the layer’s KV to the target budget and release the full KV before advancing. The compression step uses a two-pass blockwise streaming top-𝑘 procedure: a first pass Wang et al.: Preprint submitted to Elsevier

accumulates log-sum-exp normalization statistics over key blocks, and a second pass derives the retained-token selection, so that scoring never materializes the full observationby-prefix attention matrix. This scoring step is a replaceable component: because the schedule is decoupled from the token-importance rule, existing KV-cache compression selectors can be used in its place. TideKV further supports layer- and head-adaptive budget allocation within the same delayed-eviction invariant, enabling fine-grained memory– quality control across heterogeneous attention configurations. All state remains on the GPU, and the output is a compressed initial decode cache. The main contributions of this work are as follows: • We propose layer-major delayed eviction, which preserves full-prompt visibility for KV selection while confining uncompressed KV to one active layer and temporary attention/MLP activations to one chunk. Eviction occurs only after the layer finishes prompt encoding. • We separate construction scheduling from prefixbased selectors compatible with layer-end selection and delayed eviction, and evaluate four selector families within this framework. To keep the compression step itself within budget, we design a memorycontrolled two-pass blockwise streaming top-𝑘 procedure whose temporary footprint is independent of prompt length. • We evaluate TideKV on Llama and Mistral models over the LongBench and RULER benchmarks together with the memory and feasibility of long-context Page 2 of 15

Delayed KV Eviction via Layer-Wise Chunked Prefill

prefill. At matched budgets it preserves downstream quality and final-cache size. At 64K on A100, TideKV reduces the construction-time transient footprint by 83.89%; on the RTX 4090, the time-to-first-token increase is 1.06% at 64K. It completes 128K-token prefill on both resource-constrained platforms where matched post-prefill compression is out of memory.

2. Related work

Online construction processes prompts in chunks and evicts before full-prompt observation (Fig. 1c). InfiniPot distills history as the cache fills, processing later input against compressed history under a fixed budget [38]; KeyDiff uses key similarity for FlashAttention-compatible eviction under a fixed cache budget [23]; EpiCache clusters conversational history into episodes with layer-budgeted eviction and highlights the peak cost of full-context prefill [39]. Related designs use look-once multimodal eviction [40] and trained retaining heads [41]. Early eviction lowers construction peaks, but later tokens see compressed history and may lose entries relevant to subsequent context. TideKV delays eviction until each layer finishes, preserving uncompressed causal history during prompt encoding and full-prompt visibility for selection.

In this section, we review the relevant research in the following three aspects:

2.3. Prefill peak-memory reduction and KV offloading

The remainder of this paper is organized as follows. Section 2 reviews related work. Section 3 defines the problem setting. Sections 4 and 5 present the TideKV framework and implementation. Section 6 reports the evaluation, and Section 7 concludes the paper.

2.1. KV-cache selection and compression KV-cache compression methods differ in which entries they retain and when they make eviction decisions. Heavyhitter eviction keeps tokens with high accumulated attention together with recent tokens [24, 25]; StreamingLLM retains a few initial attention-sink tokens plus a sliding window so that a bounded cache supports streaming beyond the nominal context [26]; SnapKV predicts from a promptfinal observation window which positions generation will attend to and compresses per head [22]; and FastGen profiles the per-head attention structure to choose a per-head compression policy [27]. A parallel line refines the granularity of the budget: PyramidKV allocates a pyramidal per-layer budget [28]; Ada-KV distributes budget across heads [29]; CAKE casts eviction as adaptive layer-preference budgeting with time-varying token importance [30]; ChunkKV retains semantic chunks rather than isolated tokens [31]. In contrast, query-aware methods load only the KV pages required by the query during the decoding phase [32]. Beyond hard eviction, several methods modify or complement it: KeepKV merges KV entries to limit output perturbation [33]; KVzip targets query-agnostic caches reusable across queries [34]; a recent direction reconstructs KV on demand from a stored residual stream [35]; and, orthogonally to eviction, low-bit quantization compresses each retained entry rather than removing it [36]. Surveys catalog these directions [37]. Post-prefill selectors preserve full-prompt visibility with a compact retained cache; layer- and head-adaptive allocation also informs TideKV’s adaptive variants. Their bottleneck is cache construction. Even if layer-wise eviction confines uncompressed KV to one layer, whole-prompt execution still produces full-sequence hidden states and MLP activations, driving the prefill peak (Section 2.3). Thus, constructing a compact cache can exceed device memory (Fig. 1a). TideKV retains the selector and changes compression scheduling to reduce this construction peak.

2.2. Chunked construction with online eviction

Wang et al.: Preprint submitted to Elsevier

A third group of methods targets peak prefill memory by reducing intermediate MLP activations, which can dominate the memory footprint [42] (Fig. 1b). Chunked prefill splits the prompt into segments processed sequentially. For prompt length 𝑆 and chunk size 𝐶, chunking reduces the corresponding temporary activation term to approximately 𝐶∕𝑆 of its full-sequence size, while introducing kernellaunch and latency overheads [43–45]; MST partitions the MLP and LM-head inputs into mini-sequences during training [46], and MOM brings mini-sequence MLP processing to inference while offloading the full KV cache to CPU [42]; PyramidInfer instead computes fewer KV entries in deeper layers during prefill [47]. Offloading-centric systems keep the full KV but relocate it: HeadInfer fetches KV head-group by head-group between CPU and GPU so that an entire layer’s KV never co-resides on the GPU, at a substantial decode-speed cost [48]; FlexGen aggregates GPU, CPU, and disk memory to run inference under tight GPU budgets [49]. Collectively, these methods reduce the peak load during the prefill phase but either retain the uncompressed decoding cache or incur the cost of data transfer between the GPU and CPU. TideKV reduces peak memory usage through chunkinner execution while keeping execution on the GPU and generating a compressed decoding cache. The approaches differ in prompt visibility and memory use during construction. Post-prefill selectors preserve fullprompt visibility but mainly reduce retained KV; online eviction also lowers the construction peak while exposing later chunks to compressed history. Activation partitioning and offloading reduce prefill residency but may retain the uncompressed decode cache or move KV to the CPU. TideKV combines layer-major execution with delayed layerend eviction: full-prompt KV is confined to one active layer, temporary activations to one chunk, and selection observes the complete prompt. Construction stays on the GPU.

3. Problem formulation This section formalizes the compressed-cache construction problem that TideKV addresses. Our concern is not Page 3 of 15

Delayed KV Eviction via Layer-Wise Chunked Prefill Table 1 Notation. Symbol

Meaning

𝑁 𝐵 𝑆 𝐶 𝐻𝑞 , 𝐻𝑘𝑣 𝐷 𝑏 𝑊 𝐾𝑙 𝑑model ̂ Π

number of Transformer layers batch size prompt (sequence) length chunk size used during prefill number of query heads, KV heads head dimension element size in bytes observation-window size used for scoring retained cache budget for layer 𝑙 model hidden dimension (= 𝐻𝑞 ⋅ 𝐷) compressed initial decode cache (the output)

𝑑ff

construction procedure with a specified schedule, selector, and budget MLP intermediate width (= 𝑂(𝑑model ))

the design of a particular token-importance rule, but the execution setting in which a compressed initial decode cache is constructed from a long prompt under a device memory budget. The formulation deliberately separates two questions that prior work often couples: which KV entries to retain (the selection problem), and in what order prefill is executed and when eviction takes effect (the scheduling problem). TideKV specifies the construction schedule and leaves the selector configurable.

active attention/MLP workspace, and Temp contains transient scoring, selection, and cache-packing tensors. The observed process peak additionally contains terms not determined solely by Π: 𝑀proc ≈ 𝑀model + 𝑀live (Π) + 𝑀backend + 𝑀allocator + 𝑀external .

(2)

where 𝑀backend includes kernel workspaces, masks, and attention buffers; 𝑀allocator includes reserved-butinactive blocks and fragmentation; and 𝑀external covers CUDA-context, page-table, runtime-metadata, and direct allocations not attributed to active PyTorch tensors. Together, Eqs. (1) and (2) separate the scheduledependent working set from runtime memory contributed by the model, kernels, and memory manager. Equation (2) is a schematic accounting relation; component peaks need not coincide. This decomposition underlies the live-allocation and process-level measurements in Section 6. 3. Temporary compression memory Temp(Π): score matrices, selection buffers, and attention intermediates. A naive layer-end compression that materializes a 𝑊 × 𝑆 observation-by-prefix score matrix can make Temp dominate 𝑀live .

We consider a pretrained decoder-only Transformer (Table 1) processing one prompt 𝑥1∶𝑆 in chunks of size 𝐶. Prefill constructs each layer’s KV; decoding uses the retained cache. We fix the model weights and treat selection as replaceable to isolate cache-construction cost.

Bounding KVres alone does not bound the construction peak because Hidden and Act also grow with 𝑆. Allocator retention and backend workspaces further determine how a smaller live working set appears in reserved and process-level memory. TideKV therefore bounds 𝑀live and Temp while retaining the same 𝑆f inal as its underlying selector. Section 6 reports allocated, reserved, and process-level memory under multiple allocators, attention backends, and persistent workloads.

3.2. Memory quantities in cache construction

3.3. Construction objective

3.1. Notation and setting

A prompt KV cache occupies 2𝑁𝐵𝐻kv 𝑆𝐷𝑏 bytes. This expression counts stored KV alone; it does not describe either the final compressed cache or the peak memory required to construct it. We therefore distinguish three memory quantities. 1. Final compressed cache size: 𝑆final (Π), the number of retained KV positions after compression; under a per-layer budget 𝐾𝑙 it equals the total nominal budget ∑ 𝐾tot ≜ 𝑙 𝐾𝑙 , while its byte footprint follows after adjustment for the number of KV heads and the cache layout. 2. Construction-time live-tensor peak: the part of the prefill footprint directly controlled by the cache construction procedure, [ 𝑀live (Π) = max KVres (𝑙) + Hidden(𝑙) 𝑙 (1) ] + Act(𝑙) + Temp(𝑙) . Here, KVres is resident prompt KV, Hidden ∝ 𝐵𝑆𝑑model is the hidden-state bridge, Act is the Wang et al.: Preprint submitted to Elsevier

A construction procedure Π takes a tokenized prompt 𝑥1∶𝑆 , the fixed model, a chunk size 𝐶, and a selector with its cache-budget specification. It returns a compressed initial decode cache ̂ and the prefill output needed to start generation. For a fixed selector 𝐹 and layer/head budget allocation, our objective is to lower construction memory while preserving the prompt information available to 𝐹 . The reference procedure applies 𝐹 after uncompressed causal prefill. Let 𝑄f ull , 𝑄𝐹 , and 𝑄Tide denote downstream scores for full-cache decoding, the selector-matched reference, and TideKV, respectively. On the same evaluation, 𝑄f ull − 𝑄Tide = (𝑄f ull − 𝑄𝐹 ) + (𝑄𝐹 − 𝑄Tide ).

(3)

The first term reflects selector compression; the second measures the additional change associated with construction scheduling. Algorithm 1 implements layer-major execution and delayed eviction with explicit residency properties. Section 4 analyzes its prompt dependencies and working set; Section 6 evaluates fidelity and selector-matched task quality. Page 4 of 15

Delayed KV Eviction via Layer-Wise Chunked Prefill

Active layer l − chunk-inner loop

single-active-layer stack

hidden states bridge layers (l → l+1) − not KV

evict only after all chunks

1 2

completed: compressed input hidden chunks

3 4

Attention + MLP

per chunk j = 1…m

active layer l —full KV

h1…h

active advances l → l+1

…

( l −1) m

1

softmax

(l ) m

Layer-end Compress (Alg. 2)

d ff

⋮

⋮

j

chunk j attends to 1…j

future: none

only one full layer at any instant

h1…h

j

1 2 ⋮

5 6

2 ⋯

output hidden chunks

MLP (FFN)

Attention

one-chunk activation

O(C  d ff )

compressed cache Ĉ[l]

full KV

store for decode

append KV[j] 1

2

•••

j−1

j

decode cache

release

full prompt KV (chunks 1…j)

C = chunk size

m = number of chunks

dff = FFN width

…

Figure 2: Layer-major delayed eviction and the corresponding tensor lifetimes. Left: full prompt KV is resident only for the active layer 𝑙; completed layers hold compressed caches and future layers hold none. Right: chunk 𝑗 attends to the accumulated uncompressed KV of chunks 1, … , 𝑗, while output hidden chunks form the bridge to layer 𝑙+1. After all 𝑚 chunks complete, ̂ and releases the full KV. Hidden-state write/read and TideKV gathers the retained KV and absolute-position metadata into [𝑙] retained-KV gather are the principal added data movements; the layer-end commit is the synchronization boundary before layer 𝑙+1.

4. The TideKV framework Compressed-cache construction requires an execution schedule that bounds the live working set 𝑀live and compression workspace Temp without sacrificing the full-prompt visibility needed for accurate selection. TideKV enforces a layer-major residency discipline: it streams the complete prompt through one layer before advancing and delays eviction until the layer boundary. This schedule is governed by the single-active-layer residency invariant: At every instant of prefill, full-prompt KV is resident for at most one layer (the active layer), temporary attention/MLP activations are limited to one chunk alongside a prompt-wide hidden-state bridge, and every completed layer holds only its compressed cache. This invariant confines uncompressed KV to one active layer and temporary activations to one chunk, while the hiddenstate bridge and completed layers’ compressed caches remain on the GPU. Delaying eviction until the layer boundary preserves full-prompt visibility for selection. The framework rests on three mechanisms, detailed below and illustrated in Figs. 2 and 3: layer-major chunked prefill that gives each layer full causal visibility (4.1), delayed layer-end eviction that confines full KV to the active layer (4.1), and a memorycontrolled compression step that keeps the eviction temporaries bounded (4.2).

4.1. Layer-major delayed eviction As shown in Fig. 2, TideKV recasts prefill as a layersequential execution schedule: a standard forward pass propagates each chunk through all 𝑁 layers, whereas TideKV Wang et al.: Preprint submitted to Elsevier

advances a single layer over all chunks, moving to the next layer only after the current one has consumed the entire prompt. During prefill it maintains three classes of state: completed layers hold compressed caches, the active layer holds full prompt KV for the chunks processed so far (the only full prompt KV resident anywhere), and future layers hold none. Hidden states bridge the layers: after the active layer processes chunk 𝑗, its output hidden chunk ℎ𝑙𝑗 replaces the consumed input chunk in the bridge; the resulting 𝐻𝑙 becomes the chunked input to layer 𝑙+1. Because only the hidden outputs, not the KV, are needed to continue the forward computation, the active layer’s full KV can be compressed and released at the layer boundary. For layer 𝑙 with input chunks 𝐻𝑙−1 = {ℎ𝑙−1 , … , ℎ𝑙−1 𝑚 }, 1 𝑚 = ⌈𝑆∕𝐶⌉, TideKV processes chunks in causal order; when chunk 𝑗 is processed, its queries attend to the KV produced by chunks 1, … , 𝑗 of the same layer, so later chunks see the same uncompressed layer-local history a fullprompt execution would provide. Only after all 𝑚 chunks are processed does TideKV compress the layer’s KV to its budget 𝐾𝑙 , store the compressed cache, and release the full KV. Eviction therefore never alters the hidden states the layer has already produced, decoupling what is retained for decoding from what is visible during prompt encoding. The pseudocode separates the schedule from both the selector inside L AYERE NDCOMPRESS and the budget schedule {𝐾𝑙 }. A SnapKV-style observation-window selector, a recent-token selector, or a head-adaptive selector can be used within the same scheduling framework; the budget can be uniform (𝐾𝑙 = 𝐾tot ∕𝑁) or allocated adaptively across layers (more capacity to harder-to-compress layers) or heads. These choices preserve the uncompressed-KV residency invariant; Page 5 of 15

Delayed KV Eviction via Layer-Wise Chunked Prefill

Algorithm 1 Layer-major chunked prefill with delayed eviction

Algorithm 2 L AYERE NDCOMPRESS (two-pass blockwise scoring)

Require: hidden chunks 𝐻0 = {ℎ01 , … , ℎ0𝑚 }; layers 𝐿1 , … , 𝐿𝑁 ; budgets 𝐾1 , … , 𝐾𝑁 ; window 𝑊 ̂ final hidden Ensure: compressed initial decode cache ; chunks 𝐻 1: ̂ ← ∅; 𝐻 ← 𝐻0 2: for 𝑙 = 1 to 𝑁 do 3: KVf ull ← ∅; 𝑄obs ← ∅ 4: for 𝑗 = 1 to 𝑚 do ⊳ causal chunk order 5: 𝐻[𝑗], KVfull ← 𝐿𝑙 .FORWARD(𝐻[𝑗], KVf ull ) ⊳ replace the consumed hidden chunk 6: retain the latest 𝑊 queries in 𝑄obs 7: end for ̂ ← LAYERE NDCOMPRESS(KVfull , 𝑄obs , 𝐾𝑙 ) 8: [𝑙] 9: release KVfull ⊳ delayed eviction 10: end for ̂ 𝐻 11: return ,

Require: full layer KV (𝐾, 𝑉 ); observation queries 𝑄 (size 𝑊 ); budget 𝐾𝑙 ; block size 𝑏𝑠 ; mandatory indices mand ̂ 𝑉̂ ) Ensure: retained index set 𝑙 and gathered (𝐾, 1: initialize 𝑚𝑞 ← −∞, 𝑍𝑞 ← 0 for all 𝑞 2: for each key block 𝑡 do ⊳ Pass 1: streaming log-sum-exp, (4)–(6) 3: update running max 𝑚𝑞 and normalizer 𝑍𝑞 for all 𝑞 4: end for 5: 𝑘 ← 𝐾𝑙 − |mand |; heap ← ∅ 6: for each key block 𝑡 do ⊳ Pass 2: streaming top-𝑘, (7)–(8) 7: compute 𝑎𝑞,𝑖 with (𝑀𝑞 , 𝑍𝑞 ); pool to 𝑝𝑖 for 𝑖 ∈ 𝑡 8: retain top-𝑘 candidates with 𝑖 ∉ mand in heap 9: end for 10: 𝑙 ← indices(heap) ∪ mand 11: return 𝑙 , GATHER (𝐾, 𝑉 , 𝑙 )

they change only which entries are retained and how the retained cache is laid out, with a head-adaptive budget yielding a variable-length cache stored in a flattened layout with perhead offsets. The selector and budget schedule are configurable within this layer-end interface. Direct integration requires preserving full layer-local history until selection; compression algorithms with other execution or cache-representation requirements need additional adaptation. Reordering execution confines full-prompt KV to one active layer and temporary attention/MLP activations to one chunk, without changing the active layer’s 𝑂(𝑆) KV term or its causal attention computation. Section 4.3 quantifies the resulting working set, and Section 6 evaluates the selector and allocation variants.

retaining logits from previous blocks: ) ( (𝑡−1) , max 𝓁𝑞,𝑖 , 𝑚(𝑡) 𝑞 = max 𝑚𝑞 𝑖∈𝑡

(𝑡−1)

𝑍𝑞(𝑡) = 𝑍𝑞(𝑡−1) 𝑒 𝑚𝑞

𝓁𝑞,𝑖 =

𝑄𝑞 ⋅ 𝐾𝑖 . √ 𝐷

(4)

For each observation query, logits for keys beyond its causal position are set to −∞. Both passes use the same absolute positions and causal mask. Pass 1 (normalization) uses an online running-maximum recurrence to accumulate normalization statistics without

Wang et al.: Preprint submitted to Elsevier

+

∑

(𝑡)

(6)

𝑒 𝓁𝑞,𝑖 −𝑚𝑞 .

𝑖∈𝑡 ) (𝑇 ) After 𝑇 blocks, 𝑀𝑞 = 𝑚(𝑇 𝑞 and 𝑍𝑞 = 𝑍𝑞 ; Pass 1 stores 𝑂(𝑊 ) scalars per batch and attention head. Pass 2 (scoring + streaming top-𝑘) revisits the blocks, forms normalized weights, pools them into per-key importance, and keeps the top 𝑘 = 𝐾𝑙 − |mand | nonmandatory entries:

𝑎𝑞,𝑖 =

𝑒 𝓁𝑞,𝑖 −𝑀𝑞 , 𝑍𝑞

4.2. Memory-controlled layer-end compression As shown in Fig. 3, the layer-end step must itself respect the invariant: a naive selector that scores 𝑊 observation queries against all 𝑆 prefix keys materializes a tensor of shape [𝐵, 𝐻𝑞 , 𝑊 , 𝑆], whose 𝑂(𝐵𝐻𝑞 𝑊 𝑆) footprint can dominate 𝑀live . TideKV avoids it with a two-pass blockwise procedure that streams over key blocks and never stores the full score tensor. Partition the prefix keys into blocks 1 , … , 𝑇 of size up to 𝑏𝑠 . For observation query 𝑞 and prefix key 𝑖, the attention logit is

(𝑡)

−𝑚𝑞

(5)

𝑝𝑖 =

𝑊 ∑

(7)

𝑤𝑞 𝑎𝑞,𝑖 ,

𝑞=1

( ) 𝑙 = Top𝑘 {𝑝𝑖 ∶ 𝑖 ∉ mand } ∪ mand , |𝑙 | = 𝐾𝑙 .

(8)

where 𝑤𝑞 are pooling weights and mand is the set of alwaysretained token indices, such as recent tokens and attention sinks, counted within the budget 𝐾𝑙 rather than added on top, so the physical cache size equals the stated budget. The retention count assumes a feasible budget |mand | ≤ 𝐾𝑙 ≤ ̂ gathers the rows indexed by 𝑆. The compressed cache [𝑙] 𝑙 (or per-head 𝑙,ℎ ). The procedure leaves causal attention unchanged and bounds the temporary ( ) compression memory to Temp = 𝑂 𝐵𝐻𝑞 (𝑊 𝑏𝑠 + 𝐾𝑙 ) , independent of 𝑆, compared with 𝑂(𝐵𝐻𝑞 𝑊 𝑆) for the dense tensor. The second blockwise pass recomputes the logits in Eq. (4) instead of storing them, and this work is included in the prefill overhead reported in Section 6. The running maximum 𝑚𝑞 keeps the streaming log-sum-exp numerically stable in BF16 by rescaling every block to a common reference. Because each pass holds only one key block of size 𝑏𝑠 , the compression workspace is Page 6 of 15

Delayed KV Eviction via Layer-Wise Chunked Prefill S prefix keys

W observation queries

avoided: W·S memory one block in flight replaces the full W ×S tensor

W S score tensor

B1

B2

Bt

•••

prefix split into blocks of

B1

•••

B2

Pass 1 (mq , Z q ) normalization scan

Bt

keys running max log-sum-exp normalizer

BT

•••

(mq , Z q )

(mq , Z q )

BT

•••

(mq , Z q )

numerically-stable streaming normalizer

Pass 2 scoring +selection scan

B1

B2

•••

Bt

normalized weights

pool

1 2 3 •••

W

bs

W

1

1

2

2

3

3

⋮

⋮

W

W

only W-length state normalizers feed Pass 2

streaming recent/sink top-k heap retained indices size Kl ∪ 1

S

2 ••• K

1

2

•••

K

gather from K,V

compre ssed

 ,V K

per-key scores independent of S

naive W·S

streaming

↓

memory usage independent of S grows

Figure 3: Memory-controlled two-pass blockwise scoring. Instead of materializing the dense 𝑊 × 𝑆 observation-by-prefix score tensor, the prefix keys are split into blocks. Pass 1 scans the blocks to build the numerically stable log-sum-exp normalizers (𝑚𝑞 , 𝑍𝑞 ), keeping only 𝑊 -length state. Pass 2 rescans to form normalized weights, pools them into per-key scores, and keeps the ̂ 𝑉̂ ). The top 𝐾𝑙 − |mand | nonmandatory entries together with the mandatory recent/sink set before gathering the retained (𝐾, temporary footprint Temp = 𝑂(𝑊 𝑏𝑠 + 𝐾𝑙 ) is independent of 𝑆.

governed by 𝑏𝑠 rather than prompt length. The parameter 𝑏𝑠 therefore trades arithmetic efficiency against temporary memory. Under grouped-query attention, per-key scores are aggregated at KV-head granularity before selection, so retained indices are shared within each KV group and the ̂ 𝑉̂ ) remain in the compact KV-head layout. gathered (𝐾,

4.3. Memory and complexity analysis TideKV changes the residency schedule rather than the causal attention computation: the prompt still passes through all 𝑁 layers, with an additional 𝑂(𝑊 𝑆) layer-end scoring pass. At the largest working set of layer 𝑙, TideKV ≈ 2𝐵𝐻kv 𝑆𝐷𝑏 + 𝐵𝑆𝑑model 𝑏 + 𝑐𝐵𝐶𝑑f f 𝑏 𝑀live

⏟⏞⏞⏞⏞⏟⏞⏞⏞⏞⏟

⏟⏞⏞⏞⏟⏞⏞⏞⏟

⏟⏞⏟⏞⏟

active KV

hidden

chunk act.

+

∑

( ) 2𝐵𝐻kv 𝐾𝑙′ 𝐷𝑏 + 𝑂 𝐵𝐻𝑞 (𝑊 𝑏𝑠 + 𝐾𝑙 ) 𝑏 . ⏟⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏟⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏟ 𝑙′ <𝑙 ⏟⏞⏞⏞⏞⏞⏞⏞⏞⏞⏟⏞⏞⏞⏞⏞⏞⏞⏞⏞⏟ compression temp compressed caches

(9) where 𝑑ff = 𝑂(𝑑model ) is the MLP intermediate width, 𝑐 counts co-resident MLP activation buffers, and the expression assumes a FlashAttention backend. Only the compressed-cache sum accumulates across layers. The active-layer KV and hidden bridge each remain linear in 𝑆, with footprints of 2𝐵𝐻kv 𝑆𝐷𝑏 and 𝐵𝑆𝑑model 𝑏 bytes. Reusing hidden chunk slots avoids a separate bridge per layer. These remaining linear terms limit scaling; layerspecific workspace and allocation behavior also affect Wang et al.: Preprint submitted to Elsevier

the measured peak. A layer-end post-prefill compressor has the same active-layer KV and hidden terms. Wholeprompt execution gives an activation term of 𝑐𝐵𝑆𝑑ff 𝑏 and a dense compression workspace of 𝐵𝐻𝑞 𝑊 𝑆𝑏. TideKV reduces these terms by 𝑆∕𝐶 and from 𝑂(𝑊 𝑆) to 𝑂(𝑊 𝑏𝑠 ), respectively. The activation term provides most of the reduction because it scales with the MLP width and the full context in whole-prompt execution. Chunk size 𝐶 and score block size 𝑏𝑠 control these memory–latency trade-offs; Section 6.6 evaluates the chunk-size choice.

4.4. Prompt fidelity and scope Layer-major delayed eviction preserves each layer’s prompt path up to compression. Consider deterministic causal layers with token-local normalization and MLPs, identical weights, tokens, positions and masks, and exact arithmetic. Assume that the input hidden sequence to layer 𝑙 matches uncompressed causal prefill. Each chunk then has the same queries and complete causal KV history, with the within-chunk mask excluding future tokens. Its output agrees with the reference, so induction over chunks gives the same complete hidden sequence 𝐻𝑙 and precompression KV. Layer-end compression affects only the retained decode cache, leaving 𝐻𝑙 unchanged; induction over layers completes the argument. Finite-precision kernels and operation order can introduce numerical differences. We evaluate hidden-state relative-𝐿2 error and cosine similarity, together with nexttoken top-1 agreement and Jensen–Shannon divergence, across full, standard-chunked, no-eviction layer-major, and Page 7 of 15

Delayed KV Eviction via Layer-Wise Chunked Prefill

for each layer l= 1…N hidden chunks → next layer same active layer l A.layer-major scheduler ⋯

B. Layer-end compressor

decode C. Cross-layer cache manager 1

layer l

completed compressed

2

per chunk (1…m)

⋮

3

active: full KV

4 5

stream chunks within a layer one active layer at a time

two-pass blockwise scoring Alg. 2

6

D. Decode adapter after all layers

⋯

decoding

future: none

release full KV at layer boundary

compressed initial cache → decoding

Memory accountant attributes peak memory to KV / hidden / act / temp; audits cache bytes

Figure 4: Logical architecture of the TideKV reference implementation. The layer-major scheduler coordinates the layer-end compressor and cross-layer cache manager to transform each layer’s full KV into a compact retained cache at the layer boundary. After all layers complete, the decode adapter exposes the assembled initial cache to generation. The memory accountant records component residency and allocator/process-level peaks.

compressed TideKV paths (Section 6.5). Selector-matched downstream scores assess the utility of the retained cache separately from this prompt-encoding property.

5. System implementation We implement TideKV as an on-GPU prefill subsystem in Hugging Face Transformers. It reuses the pretrained weights and configured Transformer operators, but replaces the standard whole-model prompt traversal with a layer-bychunk loop. Figure 4 summarizes the four logical components and the cross-cutting memory accountant. Layer-major scheduler and hidden-state bridge. At layer 𝑙, TideKV stores output chunks 𝐻𝑙 = {ℎ𝑙1 , … , ℎ𝑙𝑚 } as the hidden-state bridge to layer 𝑙 + 1 and reuses the same chunk slots as execution advances. The bridge contains 𝑂(𝐵𝑆𝑑model ) elements, whereas temporary attention and MLP activations remain bounded by one chunk. The active layer separately accumulates its uncompressed KV. Causal masks and absolute token positions are maintained for every chunk. Layer-end compression and cache management. After the last chunk, the compressor scores the complete layerlocal history, gathers the retained K/V entries and their original positions, and commits the compressed cache. The crosslayer cache manager then releases the full active-layer KV. It supports rectangular per-layer caches and a variable-length layout for head-adaptive budgets. The layer-end commit is the synchronization point before the scheduler advances to layer 𝑙 + 1.

Wang et al.: Preprint submitted to Elsevier

Decode adapter and accounting. The decode adapter exposes the compressed initial cache in the structure expected by generation and resumes from true absolute positions rather than the shortened cache length. Throughout cache construction, the memory accountant tracks active allocation, allocator-reserved memory, final physical-cache bytes, and process-level device memory. These signals connect the component lifetimes in Fig. 4 to the end-to-end memory behavior evaluated in Section 6. The layer-major order changes orchestration rather than the numerical definition of attention, normalization, or MLP operators, and therefore remains compatible with intra-layer fused kernels such as FlashAttention-2. Its overhead includes chunk dispatch, blockwise scoring, hidden-state bridge traffic, KV gathering and packing, and layer-boundary synchronization; these costs are included in the measured TTFT and quantified in Section 6.6. The components interact at each layer boundary: after the complete prompt is processed, the scheduler triggers compression, the cache manager commits retained KV and releases the full active-layer KV, and the next layer receives the hidden-state bridge. The memory accountant tracks these lifetimes. This coordination preserves full-prompt selection while limiting uncompressed KV residency to one active layer.

6. Evaluation We evaluate cache utility, memory savings, runtime and device robustness, and latency through selector-matched measurements, allocator/backend and persistent-request tests, cross-device experiments, and fidelity and timing analyses. Page 8 of 15

Delayed KV Eviction via Layer-Wise Chunked Prefill Table 2 Principal experimental configuration. Platform-specific software is held fixed within every matched comparison. Item

Configuration

Models

Llama-3.1-8B-Instruct; Mistral-7B-Instruct-v0.2; BF16 A100 PCIe 40 GB; RTX 4090 24 GB; Jetson AGX Orin 64 GB (MODE_50W) CUDA 11.8; PyTorch 2.3.1; Transformers 4.44.2; FlashAttention 2.7.4.post1 Batch 1; chunk 2,048; physical KV budget 4,096 unless stated otherwise

Accelerators A100 stack Inference

6.1. Evaluation setup Models and platforms. We evaluate Llama-3.1-8BInstruct and Mistral-7B-Instruct-v0.2 in BF16 on an NVIDIA A100 PCIe 40-GB GPU, a 24-GB GeForce RTX 4090, and a 64-GB Jetson AGX Orin in MODE-50W. Unless stated otherwise, batch size is one, the chunk size is 2,048, the physical KV budget is 4,096 tokens, and the main attention path uses FlashAttention-2. Table 2 summarizes the principal configuration; every matched pair uses the same software stack. Benchmarks and metrics. LongBench [10] contains 16 English tasks and 3,750 examples. RULER [11] contains 13 controlled retrieval and aggregation tasks; we use 100 examples per task at 4K–128K and report both mean score and completion status. Memory measurements distinguish peak allocated, peak reserved, and process-level NVML memory on discrete GPUs. On Jetson, we report CUDAmanaged allocation because the CPU and GPU share physical memory. Time to first token (TTFT) includes prefill, cache selection, and the first generated token. Baselines and matched controls. The baselines span token selection (SnapKV, PyramidKV, AdaKV, and ChunkKV), online eviction (KeyDiff [23]), KV quantization (KIVI [36]), and CPU–GPU head offloading (HeadInfer [48]). Each schedule-level comparison holds model weights, prompt tokens, precision, selector parameters, physical KV budget, decode settings, attention backend, allocator, synchronization, and measurement procedure fixed. The detailed construction analysis uses a 4,096-token budget, which produces a 512-MiB compact GQA cache for both methods.

6.2. Selector-matched quality and memory We first examine whether TideKV lowers the memory required to construct a cache without changing its utility or final size. Each selector is paired with its TideKV counterpart at physical KV budgets of 256, 1,024, and 4,096 tokens and evaluated on LongBench and RULER. Across both benchmarks, the paired variants remain close. Table 3 shows that TideKV preserves the selectorspecific LongBench quality profile. Table 4 reports the corresponding A100 RULER results from 4K to 128K. Across the 12 selector–budget pairs, the five-length mean score changes by −0.46 to +1.14 points under TideKV. Differences at individual lengths range from −2.41 to +2.61 Wang et al.: Preprint submitted to Elsevier

points. At 64K and 128K, all four selector families remain close to their matched baselines, with differences of −0.41 to +2.61 points across 24 comparisons. At budget 1,024, the mean absolute difference over these two lengths is 0.19 points. Thus, the agreement between each selector and its TideKV counterpart persists across both cache budgets and context lengths. The matched memory accounting in Fig. 5 identifies where the saving occurs. After prefill, both methods converge to the same 15.466-GiB steady allocation, which comprises the loaded model, the identical 512-MiB compact cache, and the remaining persistent runtime tensors; it is not the model-weight footprint alone. During cache construction, TideKV-Snap lowers peak allocation from 22.748 to 16.639 GiB and the NVML process peak from 24.096 to 17.283 GiB. Subtracting the common steady state isolates a transient footprint of 7.281 GiB for SnapKV and 1.173 GiB for TideKV-Snap, an 83.89% reduction. TideKV therefore reduces the temporary memory required to build the same final cache rather than shrinking the cache used for decoding.

6.3. Runtime-memory robustness We next examine whether the active-memory saving depends on the allocator or attention implementation, and whether repeated mixed-length requests cause allocator growth. The allocator sweep uses the A100, Llama-3.18B-Instruct in BF16, batch size one, a 4,096-token physical budget, 𝐶 = 2048, and FlashAttention-2. “Native” denotes PyTorch’s default caching allocator; “Async” denotes the CUDA stream-ordered cudaMallocAsync backend selected through PyTorch’s allocator configuration. The method pair and all other settings are unchanged between these runs. Allocator sensitivity. At 128K, Fig. 5(b,c) shows the same 41.26% allocated-peak reduction under Native and Async, while reserved and NVML reductions range from 12.47% to 42.09%. Thus, changing the allocator leaves the active-memory reduction unchanged but alters how much of it appears at the process level. Attention-backend sensitivity. The backend stress tests in Table 5(a) support the same distinction. Under FlashAttention-2, allocated-peak savings increase from 3.74% at 8K to 26.85% at 64K, while reserved and NVML peaks also decrease. With SDPA, allocated-peak savings increase from 3.55% to 21.94% over the same range. At 64K, however, SDPA raises the reserved and NVML peaks by 23.39% and 22.94%, respectively, showing that backend workspaces can mask lower active residency at process level. Eager attention further exposes the feasibility boundary: SnapKV is OOM at both 8K and 16K, whereas TideKVSnap completes both contexts. Persistent-workload stability. To test whether allocator history causes memory to accumulate across sequential requests, we executed a 150-request mixed-length trace for each allocator–method configuration in a persistent process. Stability requires the post-warm-up reserved-memory trajectory to reach a plateau rather than grow across repeated

Page 9 of 15

Delayed KV Eviction via Layer-Wise Chunked Prefill Table 3 Detailed results of Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.2 on LongBench. Bold scores indicate the higher average within each selector pair. Single-Doc. QA

Multi-Doc. QA

Summarization

Few-shot Learning

Synthetic

Code c Lc

RB

m

t

10.38

99.5

64.58

58.97

49.37

snap tidekv-snap pyramidkv tidekv-pyramidkv adakv tidekv-adakv chunkkv tidekv-chunkkv

26.9 26.83 29.17 29.31 26.05 25.22 25.61 25.25

31.93 32.99 36.12 35.86 32.78 33.98 28.66 30.22

52.54 53.25 52.84 53.86 52.78 54.74 52.28 53.6

54.55 54.99 54.72 53.9 55.77 54.98 53.78 54.22

43.2 43.6 43.8 43.32 43.46 42.87 43.63 41.74

28.41 28.72 29.87 30.19 27.55 29.37 27.46 29.47

55.6 55.5 54.5 54 58.5 59.5 47.5 47

85.27 86.72 76.42 76.77 86.87 87.86 89.54 88.16

40.94 41.3 41.64 40.91 40.91 41.28 41.27 41.45

6.75 5.28 6.78 5.28 6.78 5.28 5.69 5.36

99 99.5 99.5 99 99.5 99.5 99 98

61.69 62.32 60.54 60.66 62.56 62.36 61.14 62.6

54.93 55.67 51.97 52.66 57.09 56.47 54.65 54.82

44.37 44.68 44.15 44.02 44.96 45.11 43.43 43.59

snap tidekv-snap pyramidkv tidekv-pyramidkv adakv tidekv-adakv chunkkv tidekv-chunkkv

29.4 29.51 29.32 29.25 30.13 30.31 28.65 28.63

42.61 43.27 43.27 43.48 43.33 43.1 43.66 43.77

53.43 54.65 55.26 55.87 55.25 55.31 54.54 55.81

55.79 55.49 55.56 55.22 56.17 55.63 56.35 55.45

44.78 44.24 44.08 44.05 44.75 44.36 45.61 44.45

29.16 31.4 31.39 31.65 29.54 30.62 31.54 31.02

67.5 69 70.5 70.5 68.5 70 67 68.5

89.62 90.61 90.17 90.98 90.37 89.94 90.41 91

42.91 42.23 42.13 43.67 42.03 42.35 42.29 42.85

6.83 5.25 6.75 5.25 6.75 5.25 6.78 5.28

99.5 99.5 99 99.5 99.5 99.5 98.5 99.5

64.25 64.19 63.25 64.02 64.04 64.07 64.55 65.08

58.01 58.18 57 55.97 58.43 58.3 58.24 58.55

47.62 47.8 47.88 47.96 47.93 47.92 47.78 47.94

snap tidekv-snap pyramidkv tidekv-pyramidkv adakv tidekv-adakv chunkkv tidekv-chunkkv

29.72 30.2 30.4 29.59 30.16 30.32 29.13 29.83

44.45 44.64 44.25 44.64 45.04 44.68 44.43 44.81

55.3 56.4 55.43 56.5 55.5 56.03 55.56 56.37

56.51 55.34 56.47 55.39 56.23 55.53 56.07 55.1

45.14 44.63 45.14 45.14 45.04 44.98 44.65 44.59

29.89 31.65 29.88 31.03 29.5 31.2 29.77 31.31

71.5 72.5 72.5 72.5 71.5 72.5 72.5 72.5

91.48 90.98 91.65 90.98 91.48 90.98 91.65 90.68

43.62 43.41 43.77 43.62 43.59 43.55 43.04 43.4

6.75 5.25 6.75 5.25 6.75 5.25 6.75 5.25

99.5 99.5 99.5 99.5 99.5 99.5 99.5 99.5

64.61 64.81 64.49 64.59 64.68 64.73 64.52 64.86

58.95 58.66 58.9 58.45 59.11 59.19 58.35 59.25

48.87 48.9 48.89 48.85 48.92 48.95 48.72 48.92

snap tidekv-snap pyramidkv tidekv-pyramidkv chunkkv tidekv-chunkkv

17.74 18.32 18.01 17.97 16.98 18.61

21.85 20.33 19.64 21.26 19.97 20.05

43.26 42.42 41.65 41.89 44.88 45.47

26.6 26.89 27.52 30.28 27.05 27.51

16.48 16.27 15.38 15.97 16.6 15.95

12.43 11.93 12.12 12.44 14 13.29

57.5 57 68 66.5 65 64.5

85.79 85.51 84.81 84.22 86.35 85.19

40.05 39.49 40.31 39.72 38.92 39.47

2.96 2.95 2.36 1.96 3.22 4.06

76.82 76.33 77.44 77.73 53.69 53.06

54.28 54.09 52.68 52.98 53.58 54.24

51.67 50.95 49.23 47.29 48.29 49.53

35.94 35.6 36 36.07 34.67 34.64

snap tidekv-snap pyramidkv tidekv-pyramidkv chunkkv tidekv-chunkkv

20.3 20.33 20.39 19.78 19.02 20.13

27.75 26.84 27.13 27.55 26.76 27.86

45.78 45.95 45.21 45.23 45.94 46.56

32.72 33.54 32.34 32.32 32.39 33.98

19.04 18.97 19.01 19.94 18.75 19.19

16.74 16.59 15.42 14.73 14.81 16.67

68 67 69.5 68 66 68

86.34 86.28 85.96 85.64 86.34 86.34

40.93 41.37 41.28 41.64 40.82 40.37

3.29 3.08 4.36 3.6 3.53 2.72

88.56 87.29 86.08 87.33 82.6 85.92

57.59 56.91 55.67 56.07 57.14 57.08

53.41 53.67 52.57 52.25 53.61 53.2

39.8 39.61 39.37 39.31 38.79 39.62

snap tidekv-snap pyramidkv tidekv-pyramidkv chunkkv tidekv-chunkkv

21.6 21.68 21.34 21.18 21.08 21.63

29.27 30.02 29.28 29.38 29.33 29.31

47.03 46.91 46.28 46.18 46.99 46.98

35.64 35.6 34.54 34.18 36.05 35.82

21.78 21.84 21.66 21.02 21.16 21.27

70 70.5 71 71 71 71

86.24 86.23 86.17 86.28 86.24 86.25

41.74 41.82 41.89 41.68 41.85 42.23

4.04 3.41 3.11 3.66 2.79 3.05

89.17 87.96 88.21 87.12 87.94 88.06

57.23 57.2 57.41 57.25 57.37 57.31

54.4 54.27 54.45 53.64 54.8 54.11

41.18 41.17 40.97 40.81 41.04 41.15

22.22 22.55 22.6 22.28 22.67 22.4 21.39 21.82

PC

EC

s ew

23.81 23.75 22.87 23.27 23.43 24.04 22.9 23.37

N

m

Su

ue

A

Q

iM

t or ep vR

2W

vQ

22.19 21.91 23.13 23.08 22.62 21.97 20.34 20.37

P

n ou

e

Su

44.22

PR

A Q

90.23

M SA

iv Tr

73.0

ia

TR

30.89 35.09 23.55 27.23 Llama-3.1-8B-Instruct,𝐵𝑢𝑑𝑔𝑒𝑡 = 256

ti ul M

ik

47.26

M Q

A Q ot

54.68

Go

p ot

55.78

iq us M

en

H

er

44.92

FM

29.69

p as Q

Full Cache

rt N A

Ave. Score

Llama-3.1-8B-Instruct,𝐵𝑢𝑑𝑔𝑒𝑡 = 1024 27.81 27.11 28.53 28.01 27.58 27.43 26.84 27.04

24.37 24.61 23.92 24.14 24.57 24.94 23.91 24.46

25.99 25.57 25.9 25.88 25.89 25.66 25.66 25.67

Llama-3.1-8B-Instruct,𝐵𝑢𝑑𝑔𝑒𝑡 = 4096 32.27 32.29 31.55 32.57 32.38 32.6 32.3 32.47

25.28 24.96 24.53 24.83 25.41 25.22 24.36 25.7

26.95 27.12 27.03 27.05 26.79 26.99 26.93 27.05

Mistral-7B-Instruct-v0.2, 𝐵𝑢𝑑𝑔𝑒𝑡 = 256 22.16 22.44 22.75 22.76 22.2 20.47

22.57 22.1 21.71 22.03 22.08 21.29

22.92 22.55 22.38 22.08 21.96 21.64

Mistral-7B-Instruct-v0.2,𝐵𝑢𝑑𝑔𝑒𝑡 = 1024 26.87 26.7 26.4 26.5 25.95 26.78

23.82 23.5 23 22.83 21.5 23.39

25.74 25.71 25.64 25.49 25.44 25.79

Mistral-7B-Instruct-v0.2,𝐵𝑢𝑑𝑔𝑒𝑡 = 4096 19 19.42 18.66 19.1 18.86 19.39

31.2 31.31 30.86 30.5 30.96 31.5

23.67 23.68 23.89 23.94 23.27 23.76

26.87 26.84 26.78 26.92 26.87 26.84

Table 4 RULER performance (%) of Llama-3.1-8B-Instruct on A100 under different physical KV budgets and context lengths. Avg equally weights the five displayed lengths. SnapKV and PyramidKV denote selector-matched controls. Higher is better. Budget = 256

Method 4K SnapKV TideKV-Snap

16K

32K

64K 128K

Budget = 1024 Avg

4K

16K

32K

64K 128K

Budget = 4096 Avg

4K

16K

32K

64K 128K

Avg

68.47 65.68 62.97 60.20 43.31 60.13 88.56 78.78 74.46 68.96 58.92 73.94 95.18 89.50 81.12 76.72 65.90 81.68 69.70 67.88 64.62 59.95 42.94 61.02 88.25 78.57 74.28 69.14 59.07 73.86 95.00 90.47 82.07 78.07 68.51 82.82

PyramidKV 61.64 61.35 58.61 53.93 39.24 54.95 80.58 73.66 70.97 65.72 56.09 69.40 95.18 87.71 77.34 71.73 63.29 79.05 TideKV-PyramidKV 61.70 61.11 59.48 54.29 40.68 55.45 80.21 73.50 70.09 65.60 56.17 69.11 95.00 85.30 77.66 71.66 63.35 78.59 AdaKV TideKV-AdaKV

70.05 68.21 63.98 59.38 41.43 60.61 88.50 79.90 75.42 68.85 58.46 74.23 95.09 90.05 81.93 77.25 65.70 82.00 70.20 68.10 64.14 59.22 41.12 60.56 89.80 79.67 75.02 69.10 58.24 74.37 95.00 90.74 83.31 77.87 68.19 83.02

ChunkKV TideKV-ChunkKV

73.23 73.57 72.32 66.68 34.97 64.15 91.53 84.78 81.88 80.14 65.62 80.79 95.18 89.22 85.85 82.35 73.11 85.14 73.59 73.23 72.41 66.27 34.63 64.03 91.83 84.25 82.57 80.07 66.07 80.96 95.00 89.56 84.66 83.35 75.14 85.54

Wang et al.: Preprint submitted to Elsevier

Page 10 of 15

Delayed KV Eviction via Layer-Wise Chunked Prefill Post-prefill steady Transient (SnapKV)

25

Memory (GiB)

20

Transient (TideKV-Snap) NVML process peak

SnapKV

24.10

7.28

30.0

Allocated

Transient −83.89%

17.6

TideKV-Snap 30.0

−41.3%

17.6

−41.3%

17.28

1.17

15

Reserved

36.4 −12.6% 31.8

NVML

36.9 −12.5% 32.3

33.8 −42.1%

19.6

10 15.47

15.47

5 0

34.3 −41.5%

20.1

Final physical KV: 512 MiB for both

SnapKV

0

TideKV-Snap

(a) Construction-memory components

10

20

30

40

0

10

Peak memory (GiB) (b) Native

20

30

40

Peak memory (GiB) (c) Async

Figure 5: Matched construction-memory accounting on A100. (a) At 64K, both methods reach the same 15.466-GiB post-prefill allocation and retain the same 512-MiB physical cache; TideKV-Snap reduces the transient construction footprint from 7.281 to 1.173 GiB. Diamonds mark the NVML process peaks. (b,c) At 128K, the allocated-peak reduction is identical under Native and Async, while allocator retention changes the reserved and process-level realization. Percentages denote reductions from SnapKV to TideKV-Snap.

Table 5 Backend and persistent-workload robustness on A100. Positive values in panel (a) denote reductions from SnapKV to TideKVSnap; negative values denote increases. Panel (b) reports 150 requests per configuration (600 total). (a) Attention-backend stress tests Backend Context FA2 FA2 FA2 FA2 SDPA SDPA SDPA SDPA

8K 16K 32K 64K 8K 16K 32K 64K

Alloc. Reserved

NVML

3.74% 5.41% 5.26% 8.08% 11.08% 10.79% 15.53% 21.38% 20.89% 26.85% 24.59% 24.13% 3.55% 4.81% 4.68% 7.04% 9.97% 9.70% 12.96% 6.70% 6.54% 21.94% −23.39% −22.94%

(b) Persistent mixed-length traces

Allocator Method Async Async Native Native

cycles. As shown in Table 5 (b), all 600 requests complete, the fitted reserved-memory slopes are approximately zero after warm-up, and the first-to-last cycle change is 0 MiB. These time-domain measures show that the reservedmemory peaks do not accumulate with request history. With Native, TideKV-Snap further reduces maximum inactive split memory from 7.234 to 1.326 GiB (81.67%) and reduces allocation retries from one to zero. Thus, the lower construction footprint remains stable under the tested persistent single-request workload while also reducing native-allocator fragmentation.

6.4. Device feasibility and long-context quality We then measure whether the lower construction peak extends feasible prompt length on memory-constrained hardware while retaining long-context accuracy. The firsttoken capacity sweep measures prefill feasibility on the RTX 4090 and Jetson; the independent 32-token protocol is analyzed with the latency results in Section 6.6. Cross-device capacity.

Wang et al.: Preprint submitted to Elsevier

Completed Max reserved Slope Cycle Δ Inactive / retries

SnapKV TideKV-Snap SnapKV TideKV-Snap

150 150 150 150

33.781 GiB 19.625 GiB 36.828 GiB 32.154 GiB

≈0 0 ≈0 0

0 MiB 0 MiB 0 MiB 0 MiB

–/– –/– 7.234 GiB / 1 1.326 GiB / 0

Table 6 First-token capacity on resource-constrained platforms. Entries are peak memory in GiB: NVML process memory on the RTX 4090 and CUDA-managed allocation on Jetson. OOM denotes prefill failure. Every completed method retains a 512-MiB final physical KV cache. Platform

Context

SnapKV peak

TideKV-Snap peak

RTX 4090 RTX 4090 RTX 4090

32K 64K 128K

20.027 23.447 OOM

16.771 17.240 18.529

Jetson AGX Orin Jetson AGX Orin Jetson AGX Orin

32K 64K 128K

19.099 OOM OOM

16.131 16.631 17.632

On the RTX 4090, TideKV-Snap reduces the 64K process peak from 23.447 to 17.240 GiB (26.47%) and completes 128K at 18.529 GiB, whereas SnapKV is OOM (Table 6). At 32K, the matched capacity measurement gives a 15.53% allocated-peak reduction and matches the independent 32-token protocol. For batch-one BF16 Llama-3.18B, the hidden bridge occupies 0.25, 0.50 and 1.00 GiB Page 11 of 15

Delayed KV Eviction via Layer-Wise Chunked Prefill Table 7 LongBench averages and RULER scores of Llama-3.1-8BInstruct on RTX 4090. Paired RULER columns indicate context lengths. The 64K matched SnapKV score uses completed examples only. Method TideKV-Snap Matched SnapKV KeyDiff KIVI HeadInfer

LongBench 48.90 48.87 47.29 48.69 49.08

4K/8K

16K/32K

64K/128K

95.14/93.44 90.63/82.17 76.65/65.93 95.03/93.60 89.55/80.98 74.82/OOM 95.06/84.05 73.54/64.38 56.32/33.07 94.04/93.31 91.01/84.55 OOM/OOM 95.16/94.36 90.89/85.93 84.89/OOM

at 32K, 64K and 128K; active-layer KV adds 0.50 GiB at 128K. Doubling context from 64K to 128K increases the measured process peak by 1.289 GiB. The bridge and active KV remain linear in prompt length, but their storage is manageable through 128K in the tested setting. Section 6.6 quantifies the associated latency. On Jetson, TideKV-Snap reduces CUDA-managed allocation from 19.099 to 16.131 GiB at 32K (15.54%). It remains feasible at 64K and 128K, with peaks of 16.631 and 17.632 GiB, respectively, whereas SnapKV is OOM at both lengths (Table 6). TideKV-Snap therefore extends the feasible context boundary from 32K to 128K on the tested unified-memory platform. Very-long-context quality. Table 7 reports quality and completion outcomes on RTX 4090. TideKV-Snap completes all evaluated RULER lengths, scoring 76.65 at 64K and 65.93 at 128K. Matched SnapKV is close through 32K, partially completes the 64K evaluation, and is OOM at 128K. At the same 4,096-token budget, TideKVPyramidKV scores 71.50/60.16 and TideKV-AdaKV scores 76.87/66.03 at 64K/128K. Available RULER logs show similar peak allocated memory: 16.495–16.510 GiB for all three variants at 64K and 17.494–17.509 GiB for Snap/Pyramid at 128K. All five logged configurations complete without OOM. These generation-run allocated peaks are distinct from the NVML measurements in Table 6. Cross-paradigm comparison. The same table compares representative resource-management paradigms: SnapKV is the matched post-prefill selector, KeyDiff performs online KV eviction, KIVI quantizes the retained cache, and HeadInfer offloads attention heads between CPU and GPU. LongBench averages range from 47.29 to 49.08, with TideKV-Snap scoring 48.90. At 128K, KeyDiff completes but scores 33.07, while KIVI and HeadInfer are OOM; HeadInfer’s higher 64K score of 84.89 therefore does not extend to 128K. KeyDiff maintains a fixed retained-KV budget, whereas TideKV preserves the full layer-local history during prompt encoding. The observed scores illustrate the memory–quality trade-off between these construction strategies. TideKV’s lower construction peak expands the feasible context length while retaining the quality profile of its underlying selector.

6.5. Prompt fidelity of delayed eviction We isolate whether layer-major execution and delayed layer-end eviction alter the prompt representation used to Wang et al.: Preprint submitted to Elsevier

Table 8 Prompt-encoding fidelity and representation drift relative to Full-OneShot (Llama-3.1-8B, 𝑆=16K, 𝐶=512, budget 1024, 20 prompts). Columns: hidden-state relative-𝐿2 and cosine similarity; logit top-1 agreement and Jensen–Shannon (JS) divergence between the prefill next-token distributions. Top: layer-major scheduling and delayed eviction stay at the bf16 chunking floor. Bottom: with the same SnapKV selector, immediate eviction drifts an order of magnitude more. Standard post-prefill SnapKV coincides with the Full-OneShot row, as its prefill forward is uncompressed. hidden

logit

Mode (selector, eviction)

rel-𝐿2

cos

top-1

JS

Full-OneShot (ref.) Full-Chunked (no evict) Tide-NoEvict TideKV (SnapKV, del.)

0 0.012 0.012 0.012

1.000 0.9999 0.9999 0.9999

1.00 0.95 0.95 0.95

0 3 × 10−5 3 × 10−5 3 × 10−5

Block-SnapKV (SnapKV, imm.) Block-Recent (recent, imm.)

0.122 1.169

0.990 0.439

0.55 0.00

0.002 0.219

construct the cache. The comparison includes an unchunked reference (Full-OneShot), BF16 chunking without eviction (Full-Chunked), the layer-major scheduler without eviction (Tide-NoEvict), TideKV with delayed compression, and two immediate-eviction controls using SnapKV or recent-token retention. At 16K with 𝐶 = 512, Full-Chunked, Tide-NoEvict, and TideKV all yield hidden relative-𝐿2 0.012, cosine similarity 0.9999, top-1 agreement 0.95, and JS divergence 3 × 10−5 relative to Full-OneShot (Table 8). These results place the layer-major schedule and delayed eviction at the numerical floor introduced by ordinary BF16 chunking. Immediate eviction produces substantially larger drift. Block-SnapKV raises relative-𝐿2 to 0.122 and lowers top1 agreement to 0.55; recent-token eviction lowers cosine similarity to 0.439 and top-1 agreement to zero (Table 8). This lower prompt drift is consistent with the selectormatched quality results.

6.6. Latency and memory–time trade-off Finally, we quantify the latency cost of lowering the construction peak, verify decoding with the resulting same-size cache, and identify a practical chunk-size operating point. Each latency run generates 32 tokens. We report TTFT, peak memory, and subsequent-token throughput. Prefill cost and memory benefit. Table 9 gives representative matched memory–latency points: at A100 128K, the process peak falls by 43.86% for a 4.17% TTFT increase; at RTX 4090 64K, it falls by 26.47% for a 1.06% increase. Decode behavior and timing composition. The detailed measurements in Table 10 show that the additional work is concentrated before generation. Across all six reported platform–length pairs, subsequent-token rates remain comparable or higher for TideKV-Snap; representative SnapKV/TideKV-Snap rates are 30.50/37.06 token/s on A100 at 64K, 27.13/31.59 token/s on RTX 4090 at 64K, and 2.64/3.45 token/s on Jetson at 32K. These paired Page 12 of 15

Delayed KV Eviction via Layer-Wise Chunked Prefill Block-SnapKV

Block-Recent

Mean hidden relative L2 (log)

Mean hidden relative L2 (log)

TideKV & no-evict refs (floor)

1

0.1 bf16 chunking floor

0.01 0

5

10

15

20

25

30

TideKV & no-evict refs (floor)

Block-SnapKV

Block-Recent

1

0.1

0.01 256

1024

4096

Chunk index

KV budget

(a) Drift vs. chunk index.

(b) Drift vs. KV budget.

Figure 6: Representation drift relative to Full-OneShot. Immediate Block-SnapKV drift grows with chunk index and decreases with a larger budget, whereas TideKV remains at the no-eviction chunking floor.

Table 9 Matched 32-token latency and memory results. Values are medians; memory is the NVML process peak on A100/RTX 4090 and CUDA-managed allocation on Jetson. Each completed method starts decoding from a 512-MiB compressed KV cache. Platform

Context SnapKV TTFT / peak TideKV-Snap TTFT / peak TTFT change Peak change

A100 A100 RTX 4090 RTX 4090 RTX 4090 Jetson Jetson

64K 128K 32K 64K 128K 32K 64K

12.390 s / 24.096 GiB 42.037 s / 33.080 GiB 5.500 s / 20.027 GiB 15.698 s / 23.447 GiB OOM 116.659 s / 19.099 GiB OOM

14.464 s / 17.283 GiB 43.791 s / 18.572 GiB 6.191 s / 16.771 GiB 15.865 s / 17.240 GiB 47.110 s / 18.529 GiB 126.492 s / 16.131 GiB 320.467 s / 16.631 GiB

Table 10 Decode rate and layer-end compression share. Decode entries are SnapKV/TideKV-Snap in token/s. Compression share is the median per-run ratio of compression time to TTFT from separate profiling runs (five on A100/RTX 4090; three on Jetson). Platform, context A100, 16K A100, 64K RTX 4090, 32K RTX 4090, 64K Jetson, 16K Jetson, 32K

Decode rate Compression share 29.77 / 36.58 30.50 / 37.06 26.86 / 32.45 27.13 / 31.59 2.67 / 3.46 2.64 / 3.45

21.92% 13.38% 17.96% 13.73% 7.16% 5.89%

measurements confirm that decoding from the matched compact cache remains intact. In separate profiling runs, layer-end scoring and compression account for 5.89%– 21.92% of TideKV TTFT. These shares are medians of per-run ratios. They describe prefill composition, whereas Table 9 reports the TTFT increase over SnapKV. Chunk-size operating point. Chunk size controls the memory–latency operating point. The cross-platform sweep in Table 11 identifies 𝐶 = 2048 as a robust default: A100 gains modest latency at 𝐶 = 4096, whereas RTX 4090 and Jetson consume more memory without improving TTFT. A separate chunk-size sweep used Llama-3.1-8BInstruct with a 4,096-token budget. As 𝐶 varied from 512 to 4,096, TideKV-Snap’s 16-task LongBench mean ranged from 48.90 to 49.09. Overall, TideKV provides substantial constructionmemory headroom with acceptable prefill overhead across Wang et al.: Preprint submitted to Elsevier

+16.74% +4.17% +12.55% +1.06% – +8.43% –

−28.28% −43.86% −16.26% −26.47% feasible −15.54% feasible

Table 11 TideKV-Snap chunk-size Pareto sweep. Each entry is TTFT in seconds / peak memory in GiB (NVML on A100/RTX 4090; CUDA-managed on Jetson). RTX 4090 and Jetson timings were measured independently of Table 9. Platform and context A100, 64K RTX 4090, 64K Jetson, 32K

𝐶 = 1024

𝐶 = 2048

𝐶 = 4096

16.783 / 17.125 17.181 / 17.080 128.27 / 16.037

14.464 / 17.283 15.804 / 17.240 125.66 / 16.131

13.899 / 17.713 15.981 / 17.670 128.62 / 16.320

the tested platforms. Near device limits, this headroom enables much longer prompts while preserving the final cache size and stable decode-stage behavior.

7. Conclusion TideKV constructs a compressed initial KV cache by streaming prompt chunks within each layer and delaying eviction until the layer has processed the complete prompt. This layer-major schedule preserves full-prompt-aware cache selection while limiting full-prompt KV residency to one active layer and transient activations to one chunk. Matched experiments show that TideKV preserves the underlying selector’s quality profile and final cache size while reducing the memory required to construct that cache. TideKV-Snap cuts the 64K transient construction footprint by 83.89% on A100 and completes 128K inference on both an RTX 4090 and a Jetson AGX Orin; its RTX 4090 RULER score is 65.93. The allocator, backend, and persistent workload results further show that the active-memory reduction holds across the tested runtime configurations. These results Page 13 of 15

Delayed KV Eviction via Layer-Wise Chunked Prefill

make layer-major delayed eviction a practical way to lower the prefill construction peak and extend feasible context length without reducing the final decode-cache budget. The remaining linear bridge and active-layer KV costs warrant further evaluation at larger batch sizes and beyond 128K. Future work will integrate the layer-aware scheduler and layer-end cache transformation with continuous batching, prefix reuse, graph execution, and distributed serving, while exploring adaptive chunk sizing and backend-aware memory management.

CRediT authorship contribution statement Yu Wang: Writing - review & editing, Writing - original draft, Validation, Software, Methodology, Investigation, Formal analysis.. Zhu Wang: Writing-review & editing, Writing – original draft, Methodology, Formal analysis, Data curation.. Ning Hu: Supervision, Project administration, Funding acquisition, Conceptualization.. Yifan Guo: Methodology, Investigation.. Xiu Ma: Methodology, Investigation.. Bin Guo: Supervision, Project administration, Funding acquisition, Conceptualization.. Zhiwen Yu: Supervision, Project administration, Funding acquisition, Conceptualization..

Acknowledgments This work is partially supported by the National Natural Science Foundation of China (No. 62472358, 62322601), and the Fundamental Research Funds for the Central Universities(No. G2026KY06261).

Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data availability Data will be made available on request.

References [1] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. AlDahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al., The llama 3 herd of models, in: Neural Information Processing Systems, Curran Associates, 2024. [2] G. Team, P. Georgiev, V. I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wang, et al., Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, arXiv preprint arXiv:2403.05530 (2024). [3] B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu, et al., Qwen2. 5-coder technical report, arXiv e-prints (2024) arXiv–2409. [4] J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, Y. Liu, Roformer: Enhanced transformer with rotary position embedding, Neurocomputing 568 (2024) 127063. [5] B. Peng, J. Quesnelle, H. Fan, E. Shippole, Yarn: Efficient context window extension of large language models, in: International Conference on Learning Representations, volume 2024, 2024, pp. 31932– 31951.

Wang et al.: Preprint submitted to Elsevier

[6] Y. Ding, L. L. Zhang, C. Zhang, Y. Xu, N. Shang, J. Xu, F. Yang, M. Yang, Longrope: extending llm context window beyond 2 million tokens, in: Proceedings of the 41st International Conference on Machine Learning, 2024, pp. 11091–11104. [7] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, et al., Retrievalaugmented generation for knowledge-intensive nlp tasks, Advances in neural information processing systems 33 (2020) 9459–9474. [8] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, Y. Cao, React: Synergizing reasoning and acting in language models, in: International Conference on Learning Representations (ICLR), 2023. [9] B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, R. Sauvestre, T. Remez, et al., Code llama: Open foundation models for code, arXiv preprint arXiv:2308.12950 (2023). [10] Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, et al., Longbench: A bilingual, multitask benchmark for long context understanding, in: Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), 2024, pp. 3119–3137. [11] C.-P. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y. Zhang, B. Ginsburg, Ruler: What’s the real context size of your long-context language models?, arXiv preprint arXiv:2404.06654 (2024). [12] J. Xu, Z. Li, W. Chen, Q. Wang, X. Gao, Q. Cai, Z. Ling, On-device language models: A comprehensive review, arXiv e-prints (2024) arXiv–2409. [13] K. Alizadeh, S. I. Mirzadeh, D. Belenko, S. Khatamifard, M. Cho, C. C. Del Mundo, M. Rastegari, M. Farajtabar, Llm in a flash: Efficient large language model inference with limited memory, in: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 12562–12584. [14] Y. Song, Z. Mi, H. Xie, H. Chen, Powerinfer: Fast large language model serving with a consumer-grade gpu, in: Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles, 2024, pp. 590–606. [15] D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al., Palm-e: an embodied multimodal language model, in: Proceedings of the 40th International Conference on Machine Learning, 2023, pp. 8469–8488. [16] B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, et al., Rt-2: Vision-language-action models transfer web knowledge to robotic control, in: Conference on Robot Learning, PMLR, 2023, pp. 2165–2183. [17] Y. Zhu, S. Wang, W. Zhong, N. Shen, Y. Li, S. Wang, Z. Li, C. Wu, Z. He, L. Li, A survey on large language model-empowered autonomous driving, arXiv e-prints (2024) arXiv–2409. [18] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, Advances in neural information processing systems 30 (2017). [19] J. Ainslie, J. Lee-Thorp, M. De Jong, Y. Zemlyanskiy, F. Lebrón, S. Sanghai, Gqa: Training generalized multi-query transformer models from multi-head checkpoints, in: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 4895–4901. [20] A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. Gulavani, A. Tumanov, R. Ramjee, Taming {Throughput-Latency} tradeoff in {LLM} inference with {Sarathi-Serve}, in: 18th USENIX symposium on operating systems design and implementation (OSDI 24), 2024, pp. 117–134. [21] T. Dao, Flashattention-2: Faster attention with better parallelism and work partitioning, in: International Conference on Learning Representations, volume 2024, 2024, pp. 35549–35562. [22] Y. Li, Y. Huang, B. Yang, B. Venkitesh, A. Locatelli, H. Ye, T. Cai, P. Lewis, D. Chen, Snapkv: Llm knows what you are looking for before generation, Advances in Neural Information Processing Systems 37 (2024) 22947–22970. [23] J. Park, D. Jones, M. Morse, R. Goel, M. Lee, C. Lott, Keydiff: Key similarity-based kv cache eviction for long-context llm inference in

Page 14 of 15

Delayed KV Eviction via Layer-Wise Chunked Prefill resource-constrained environments, Advances in Neural Information Processing Systems 38 (2026) 5983–6019. [24] Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Ré, C. Barrett, et al., H2o: Heavy-hitter oracle for efficient generative inference of large language models, Advances in Neural Information Processing Systems 36 (2023) 34661–34710. [25] Z. Liu, A. Desai, F. Liao, W. Wang, V. Xie, Z. Xu, A. Kyrillidis, A. Shrivastava, Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time, Advances in Neural Information Processing Systems 36 (2023) 52342–52364. [26] G. Xiao, Y. Tian, B. Chen, S. Han, M. Lewis, Efficient streaming language models with attention sinks, in: International Conference on Learning Representations, volume 2024, 2024, pp. 21875–21895. [27] S. Ge, Y. Zhang, L. Liu, M. Zhang, J. Han, J. Gao, Model tells you what to discard: Adaptive kv cache compression for llms, in: International Conference on Learning Representations, volume 2024, 2024, pp. 22975–22988. [28] Z. Cai, Y. Zhang, B. Gao, Y. Liu, Y. Li, T. Liu, K. Lu, W. Xiong, Y. Dong, J. Hu, et al., Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling, arXiv preprint arXiv:2406.02069 (2024). [29] Y. Feng, J. Lv, Y. Cao, X. Xie, S. K. Zhou, Ada-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient llm inference, Advances in Neural Information Processing Systems 38 (2026) 113152–113188. [30] Z. Qin, Y. Cao, M. Lin, W. Hu, S. Fan, K. Cheng, W. Lin, J. Li, CAKE: Cascading and adaptive KV cache eviction with layer preferences, arXiv preprint arXiv:2503.12491 (2025). [31] X. Liu, Z. Tang, P. Dong, Z. Li, B. Li, X. Hu, X. Chu, Chunkkv: Semantic-preserving kv cache compression for efficient long-context llm inference, Advances in Neural Information Processing Systems 38 (2026) 28728–28778. [32] J. Tang, Y. Zhao, K. Zhu, G. Xiao, B. Kasikci, S. Han, Quest: queryaware sparsity for efficient long-context llm inference, in: Proceedings of the 41st International Conference on Machine Learning, 2024, pp. 47901–47911. [33] Y. Tian, Z. Wang, Y. Peng, A. Yuan, Z. Wang, B. Yi, X. Liu, Y. Cui, T. Yang, Keepkv: Achieving periodic lossless kv cache compression for efficient llm inference, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, 2026, pp. 33259–33267. [34] J.-H. Kim, J. Kim, S. Kwon, J. W. Lee, S. Yun, H. O. Song, Kvzip: Query-agnostic kv cache compression with context reconstruction, Advances in Neural Information Processing Systems 38 (2026) 167563–167591. [35] K. U. Qasim, J. Zhang, M. K. Shaheen, R. Alharith, H. Zhang, The residual stream is all you need: On the redundancy of the kv cache in transformer inference, arXiv preprint arXiv:2603.19664 (2026). [36] Z. Liu, J. Yuan, H. Jin, S. Zhong, Z. Xu, V. Braverman, B. Chen, X. Hu, Kivi: A tuning-free asymmetric 2bit quantization for kv cache, Proceedings of Machine Learning Research 235 (2024) 32332–32344. [37] L. Shi, H. Zhang, Y. Yao, Z. Li, H. Zhao, Keep the cost down: A review on methods to optimize llm’s kv-cache consumption, arXiv preprint arXiv:2407.18003 (2024). [38] M. Kim, K. Shim, J. Choi, S. Chang, Infinipot: Infinite context processing on memory-constrained llms, in: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 16046–16060. [39] M. Kim, Epicache: Episodic kv cache management for long conversational question answering, in: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024. [40] Z. Wan, Z. Wu, C. Liu, J. Huang, Z. Zhu, P. Jin, L. Wang, L. Yuan, Look-m: Look-once optimization in kv cache for efficient multimodal long-context inference, in: Findings of the Association for Computational Linguistics: EMNLP 2024, 2024, pp. 4065–4078. [41] Y. Huang, B. Yuan, X. Han, C. Xiao, Z. Liu, Locret: Enhancing eviction in long-context llm inference with trained retaining heads on consumer-grade devices, arXiv preprint arXiv:2410.01805 (2024).

Wang et al.: Preprint submitted to Elsevier

[42] J. Zhang, T. Zhu, C. Luo, A. Anandkumar, Mom: Memory-efficient offloaded mini-sequence inference for long context language models, arXiv preprint arXiv:2504.12526 (2025). [43] A. Agrawal, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, R. Ramjee, Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills, arXiv preprint arXiv:2308.16369 (2023). [44] G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, B.-G. Chun, Orca: A distributed serving system for {Transformer-Based} generative models, in: 16th USENIX symposium on operating systems design and implementation (OSDI 22), 2022, pp. 521–538. [45] Y. Zhong, S. Liu, J. Chen, J. Hu, Y. Zhu, X. Liu, X. Jin, H. Zhang, {DistServe}: Disaggregating prefill and decoding for goodputoptimized large language model serving, in: 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), 2024, pp. 193–210. [46] C. Luo, J. Zhao, Z. Chen, B. Chen, A. Anandkumar, Mini-sequence transformers: Optimizing intermediate memory for long sequences training, Advances in Neural Information Processing Systems 37 (2024) 97299–97327. [47] D. Yang, X. Han, Y. Gao, Y. Hu, S. Zhang, H. Zhao, Pyramidinfer: Pyramid kv cache compression for high-throughput llm inference, in: Findings of the Association for Computational Linguistics: ACL 2024, 2024, pp. 3258–3270. [48] C. Luo, Z. Cai, H. Sun, J. Xiao, B. Yuan, W. Xiao, J. Hu, J. Zhao, B. Chen, A. Anandkumar, Headinfer: Memory-efficient llm inference by head-wise offloading, arXiv e-prints (2025) arXiv–2502. [49] Y. Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. Ré, I. Stoica, C. Zhang, Flexgen: High-throughput generative inference of large language models with a single gpu, in: International Conference on Machine Learning, PMLR, 2023, pp. 31094–31116.

Page 15 of 15

Related documents

• TideKV: Delayed KV Eviction via Layer-Wise Chunked Prefill for Memory-Efficient Long-Context LLM Inference#1389445
• Behavior-Preserving KV Cache Compression#1215581
• A Simple Plug-in for Improving Eviction-Based KV Cache Compression#1344588
• Interface-Aware KV Cache Quantization for Dense On-Chip NVM in Long-Context LLM Decoding#972808
• SpecQuant: Speculative Decoding with Multi-Parent Quantization for Adaptive LLM Inference#1036094