DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression
Anna Córdoba 1 Adam Puente Tercero 1 Nerea Angulo Hijo 1 Mar Linares Tercero 1 Julia Barrientos 1 Ainhoa Miranda 1 Jesús Olivera 1
1. Introduction
arXiv:2607.06523v1 [cs.AI] 7 Jul 2026
Abstract
Long-context language models increasingly depend on efficient inference systems as prompts grow from short interactive turns to documents, conversations, codebases, and retrieval-augmented contexts. In this regime, the key-value (KV) cache often becomes the dominant memory resident state during decoding, limiting batch size, context length, and serving throughput. Prior work has reduced this burden through token eviction, token merging, quantization, low-rank attention, and layer-wise cache sharing (Li et al., 2024; Jiang et al., 2025; Saxena et al., 2024; Xiang et al., 2025; Yang et al., 2024a; Yao et al., 2025). However, longcontext tasks expose a persistent tension: tokens that appear unimportant under local continuation statistics may later become essential for retrieval, while the transformer layers that encode shallow lexical anchors and deeper semantic abstractions do not require the same cache fidelity. Uniform compression across tokens or depths can therefore create brittle failures on needle retrieval, multi-hop question answering, and long-document summarization.
Long-context language model inference is increasingly limited by the memory bandwidth and capacity required to store key-value caches, yet existing compression methods often apply uniform budgets across layers or tokens and degrade retrieval when lexical cues and semantic states require different preservation. We introduce DepthWeave-KV, a token-adaptive cache compression method that factorizes key and value states across neighboring transformer layers using shared low-rank channel bases while retaining lightweight token-specific residuals where attention behavior is sensitive. DepthWeave-KV combines cross-depth residual factorization with a token-conditional depth router that allocates higher reconstruction rank to instruction-bearing and retrieval-critical tokens, and uses calibrationfree online error tracking from attention-output probes to adapt compression during generation without retraining the base model. A fused CUDA implementation jointly performs basis lookup, residual dequantization, and attention projection to reduce decode-time memory traffic. Across LongBench, Needle-in-a-Haystack, L-Eval, and long-form QA and summarization benchmarks, DepthWeave-KV achieves near-full-cache task quality with substantially lower memory use, improving average score and retrieval accuracy over prior compressed caches while reaching 8.3x KV memory reduction and 72.8 tokens per second at 64K context.
Depth-wise KV compression is a particularly promising direction because adjacent transformer layers often contain correlated cache structure. MiniCache (Liu et al., 2024a) demonstrates that exploiting redundancy along the depth dimension can reduce KV memory while preserving much of the model’s behavior. Yet depth sharing alone is insufficient for heterogeneous long-context workloads. A cache entry associated with an instruction delimiter, entity mention, citation span, or answer-bearing sentence may need more faithful reconstruction than a fluent continuation token; likewise, the same token can demand different treatment in early lexical layers and later semantic layers. Recent studies have also highlighted that aggressive KV compression can degrade retrieval behavior in ways that average perplexity or short-context benchmarks may obscure (Chen et al., 2025; Haverbeck et al., 2026; Bui et al., 2026). These observations motivate a compression mechanism that is simultaneously depth-aware, token-adaptive, and coupled to online evidence about attention behavior.
1 Instituto de Investigación en Visión Artificial. Correspondence to: Anna Córdoba <[email protected]>.
Proceedings of the 41 st International Conference on Machine Learning, Vienna, Austria. PMLR 235, 2024. Copyright 2024 by the author(s).
We introduce DepthWeave-KV, a token-adaptive cross-layer residual factorization method for long-context KV cache 1
DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression
compression. Instead of storing an independent KV tensor for every layer, DepthWeave-KV weaves a compact set of shared low-rank channel bases across neighboring transformer layers for each attention head. Layer-specific keys and values are reconstructed from these shared bases together with sparse, token-specific residual components whose ranks are assigned dynamically. The central design principle is that most tokens can share a depth-local cache representation, while retrieval-critical and instructionbearing tokens retain higher-fidelity residuals where attention output would otherwise drift. This paper makes four contributions. First, we introduce cross-depth residual factorization, which stores shared basis channels per attention head and reconstructs layer-specific key/value states through lightweight learned residual gates. Second, we add a token-conditional depth router that allocates higher reconstruction rank to salient tokens while aggressively compressing low-salience continuation tokens. Third, we propose calibration-free online error tracking using attention-output probes, allowing the compression ratio to adapt during generation without retraining the base LLM. Fourth, we provide a fused CUDA implementation for basis lookup, residual dequantization, and attention projection, reducing decode-time memory traffic under long-context workloads.
Figure 1. DepthWeave-KV overview: instead of compressing all layers and tokens uniformly, the method weaves shared depth bases across adjacent layers and spends residual capacity only on tokens whose attention behavior signals retrieval or instruction importance.
Xiang et al., 2025; Wang et al., 2024; Akulov et al., 2025; Yu & Chai, 2025a;b). ChunkKV emphasizes semantic preservation at segment granularity (Xiang et al., 2025), TailorKV combines multiple cache-management policies for heterogeneous contexts (Yao et al., 2025), and StructKV preserves structural anchors in the prompt (Chen et al., 2026). These methods are effective when token salience can be inferred from attention statistics or prompt structure, but they typically decide which tokens remain in cache rather than how much layer-specific reconstruction capacity each retained token should receive. DepthWeave-KV instead keeps a compressed representation for all tokens while assigning token-conditional residual rank, which allows low-salience continuation tokens and retrieval-critical anchors to share the same compression framework without a hard eviction boundary.
We evaluate DepthWeave-KV on established long-context benchmarks including LongBench, Needle-in-a-Haystack, L-Eval, NarrativeQA, Qasper, HotpotQA, MultiFieldQA-en, GovReport, QMSum, and TriviaQA. Across retrieval, question answering, and summarization settings, DepthWeaveKV is compared against full KV caching and representative compression baselines such as StreamingLLM, H2O, SnapKV, PyramidKV, MiniCache, KVSharer, ChunkKV, TailorKV, and Eigen Attention. The results show substantial improvements in the memory-quality trade-off, with strong needle retrieval behavior, reduced attention-output reconstruction error, and improved decode efficiency under long-context workloads.
Another family of methods compresses the numeric or algebraic representation of cached keys and values. Quantization and precision-allocation methods reduce per-entry storage through low-bit formats, token-specific precision, or residual vector quantization (Kumar, 2024; Zhang et al., 2024; Slothouber, 2026; Yang et al., 2024c;b). Low-rank approaches such as Eigen Attention project attention computation into compact subspaces (Saxena et al., 2024), while redundancy-aware methods exploit repeated structure across tokens, heads, or tasks (Cai et al., 2025; Corallo et al., 2025; Patel & Joshi, 2026; Kriuk & Ng, 2026). System-level work further shows that compression changes memory-access patterns and should be co-designed with serving infrastructure (Qin et al., 2024; Xie et al., 2025; Qiao et al., 2025). DepthWeave-KV is complementary to these directions: it uses low-rank channel bases and quantized residuals, but the key distinction is that the factorization is explicitly organized across neighboring transformer layers and modulated online by attention-output error rather than fixed solely by a
2. Related Work KV cache compression has emerged as a central systems problem for long-context inference, with surveys identifying cache residency and memory bandwidth as major bottlenecks in serving large language models (Li et al., 2024; Jiang et al., 2025; Javidnia et al., 2025). A broad line of work reduces cache size by evicting or merging tokens according to estimated importance. StreamingLLM preserves recent and attention-sink tokens, while H2O-style heavyhitter policies prioritize tokens with large accumulated attention mass; more recent variants improve token selection for long-context workloads through semantic chunking, adaptive merging, and workload-aware routing (Bui et al., 2026; 2
DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression
global compression target.
weights and a gated residual:
Depth-wise and layer-sharing methods are closest to our work. MiniCache demonstrates that KV caches contain substantial redundancy along the layer dimension and can be compressed by sharing cache states across depth (Liu et al., 2024a; Haffari et al., 2024; Liu et al., 2024b). KVSharer, inter-layer similarity methods, stochastic KV routing, and depth-cache tradeoff analyses similarly show that not all layers require independent cached states (Yang et al., 2024a; Ma et al., 2026; Filippova et al., 2026; Wang, 2026). SpindleKV and HCAttention further argue that shallow and deep layers should be treated differently under aggressive compression (Tang et al., 2025; Yang et al., 2025; 2026). DepthWeave-KV builds directly on the depth-redundancy insight of MiniCache, but replaces discrete cache sharing with cross-depth residual factorization: shared bases capture the common component among adjacent layers, while token-specific residual gates restore layer-local information only when needed. This design avoids forcing an entire token or layer group into a single fidelity level.
Z Z Z Z bℓ,h,t X = AZ ℓ,h,t Bm,h + gℓ,h,t Rℓ,h,t ,
Z ∈ {K, V }. (1) 1×rb Here AZ ∈ R is a compact coefficient vector, ℓ,h,t Z Rℓ,h,t ∈ R1×d is a quantized residual stored only at the Z routed rank, and gℓ,h,t ∈ [0, 1] is a lightweight residual gate. The basis tensors are shared across all tokens in the depth window, while coefficients and residuals remain token-local. This differs from direct layer sharing methods such as MiniCache and KVSharer (Liu et al., 2024a; Yang et al., 2024a): adjacent layers are not forced to reuse identical cache states, but instead share a low-rank channel subspace whose deviations are restored only when needed. The shared bases are initialized online from the first compressed prefix block. For each depth window, we compute a small randomized low-rank sketch of the cached keys and values after prefill, then update the bases with an exponential moving average as generation proceeds. The update is performed independently per head and separately for keys and values, which preserves head specialization while allowing neighboring layers to share redundant channel structure. The coefficient vector AZ ℓ,h,t is produced by projecting the original cache vector onto the current basis and is stored in 8-bit blockwise quantized form. Residuals are stored using 4-bit group quantization, with the number of retained residual channels controlled by the token-conditional router described below. Tokens assigned zero residual rank are represented solely by the shared basis component.
Finally, several recent studies caution that KV compression can preserve average perplexity while harming retrieval, reasoning, and instruction following (Chen et al., 2025; Haverbeck et al., 2026; An et al., 2026). ReST-KV addresses this issue with layer-wise output reconstruction and temporal smoothing (An et al., 2026), while reasoning- and training-oriented work studies when models are naturally compressible or can be made more cache-efficient (Gelberg et al., 2026; Cai et al., 2025). Other specialized compression settings include diffusion LLM caches, multimodal models, visual autoregressive generation, and knowledge injection through cached states (Nguyen-Tri et al., 2025; Zhang et al., 2025; Qin et al., 2026; Pustovit, 2026). These works motivate evaluation beyond short-context language modeling and support our use of online attention-output probes. Unlike calibration-heavy or retraining-dependent approaches, DepthWeave-KV tracks reconstruction error during generation and adjusts residual rank without modifying the base model.
The token-conditional depth router allocates reconstruction rank as a function of token salience, depth, and recent attention behavior. For each token t and depth window Wm , the router computes a score from four online features: accumulated attention mass, maximum attention spike, delimiter or instruction-boundary indicators, and the attention-output reconstruction error from the previous probe interval. The routed residual rank is ρm,h,t = ρmin + (ρmax − ρmin ) · 1 σ u⊤ ϕm,h,t > τm , (2) where ϕm,h,t denotes the feature vector, u is a small learned router parameter shared across layers in the window, and τm is an adaptive threshold chosen to satisfy the current memory budget. In practice, we use three rank levels rather than a binary decision: continuation tokens typically receive ρ = 0 or ρ = 2, structural tokens receive ρ = 4, and retrieval-critical spans may receive ρ = 8. The router is intentionally shallow and operates only on cache metadata, avoiding an additional forward pass through the base LLM.
3. Method DepthWeave-KV replaces per-layer KV storage with a depth-local factorization that separates cache components shared across neighboring transformer layers from tokenspecific residual information. For a transformer with layers ℓ ∈ {1, . . . , L}, attention heads h, token positions t, and head dimension d, we partition layers into overlapping depth windows Wm of size w. Within each window and head, DepthWeave-KV maintains shared basis channels K V Bm,h , Bm,h ∈ Rrb ×d and reconstructs the key/value state of token t at layer ℓ ∈ Wm using layer-specific mixing
To avoid calibration data or retraining of the base model, DepthWeave-KV uses attention-output probes during generation. Every p decode steps, a small subset of heads and recent query positions is evaluated twice: once with the 3
DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression
compressed reconstruction and once with a temporarily materialized higher-fidelity cache for the same depth window. The discrepancy is measured after attention aggregation rather than directly in KV space: b ℓ,h , Vbℓ,h ) Attn(Qℓ,h , Kℓ,h , Vℓ,h ) − Attn(Qℓ,h , K eℓ,h =
∥Attn(Qℓ,h , Kℓ,h , Vℓ,h )∥2 + ϵ
2
.
(3) If eℓ,h exceeds a window-specific tolerance, the router lowers τm and assigns more tokens in that window to higher residual rank. If the error remains below tolerance for several probe intervals, τm is increased and residual storage is reduced. This feedback mechanism targets the quantity that directly affects the next hidden state, which makes it more reliable than minimizing reconstruction error uniformly over all cached vectors.
Figure 2. DepthWeave-KV architecture: neighboring transformer layers share compact low-rank KV channel bases, while residual gates and a token-conditional depth router selectively preserve layer- and token-specific information needed for reliable retrieval during long-context decoding.
Figure 2 summarizes the full data path. During prefill, DepthWeave-KV forms depth-window bases, projects keys and values into shared coefficient space, stores quantized residuals according to the router, and records probe statistics. During decoding, the fused kernel loads the shared basis rows, dequantizes only the routed residual channels, reconstructs the needed key/value fragments, and immediately feeds them into the attention projection. The kernel fuses basis lookup, residual dequantization, coefficient multiplication, and attention score computation so that reconstructed KV tensors are not written back to global memory. This is important for long-context inference, where bandwidth rather than arithmetic is often the limiting factor (Qin et al., 2024; Xie et al., 2025).
over context lengths from 16K to 128K tokens and needle depths from 0% to 100%. We additionally measure KV cache memory reduction relative to a full-precision full KV cache, decode throughput during autoregressive generation, time-to-first-token (TTFT), perplexity delta on held-out long-form text, and attention-output reconstruction error. We compare against Full KV Cache, StreamingLLM, H2O, SnapKV, PyramidKV, MiniCache, KVSharer, ChunkKV, TailorKV, and Eigen Attention. These baselines cover token eviction, heavy-hitter retention, layer-wise sharing, semantic chunking, hybrid cache policies, and low-rank attentionspace compression (Liu et al., 2024a; Yang et al., 2024a; Xiang et al., 2025; Yao et al., 2025; Saxena et al., 2024; Bui et al., 2026; Li et al., 2024). All methods use the same base model, tokenizer, prompt formatting, and decoding parameters. Unless otherwise stated, results are measured at a 64K-token prefill length with greedy decoding for 512 generated tokens. Compression methods are configured to target the strongest task score under a memory budget of at least 4× reduction, except Full KV Cache, which stores all keys and values without compression.
DepthWeave-KV is applied without modifying the base model weights. The only learned components are the residual gates and router parameters, trained once on unlabeled long-context text using a frozen teacher cache objective. At deployment, the method requires no task-specific calibration: the online probe controller adjusts the effective compression ratio to the prompt and generation trajectory. This design allows DepthWeave-KV to behave conservatively on retrieval-heavy prompts while compressing aggressively on low-salience continuation regions, addressing failure modes reported for uniform eviction and compression schemes (Chen et al., 2025; Haverbeck et al., 2026; Bui et al., 2026).
Main results. Table 1 summarizes the overall results. DepthWeave-KV reaches an average task score of 62.9%, within 0.9 points of Full KV Cache while reducing KV memory by 8.3×. Among compressed baselines, the strongest average score is TailorKV at 61.4%; DepthWeave-KV improves over it by 1.5 points while using a larger memory reduction factor. The gain is more pronounced on Needle-ina-Haystack, where DepthWeave-KV obtains 96.1% retrieval accuracy, compared with 92.6% for TailorKV, 91.2% for ChunkKV, and 89.4% for MiniCache. This supports the design in Figure 1: preserving token-specific residuals for
4. Experiments Experimental setup. We evaluate DepthWeave-KV on long-context retrieval, question answering, and summarization workloads. Our benchmark suite includes LongBench, L-Eval, NarrativeQA, Qasper, HotpotQA, MultiFieldQA-en, GovReport, QMSum, TriviaQA, and Needle-in-a-Haystack. For LongBench-style tasks, we report the average normalized task score over the evaluated subsets. For Needle-ina-Haystack, we report exact retrieval accuracy averaged 4
DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression Table 2. Ablation study at 64K context length. Removing tokenadaptive routing or residual reconstruction most strongly degrades retrieval quality, while removing the fused kernel mainly affects systems metrics.
Table 1. Main long-context inference results at 64K context length. Higher is better for average task score, Needle accuracy, KV memory reduction, and decode throughput. Lower is better for TTFT, perplexity delta, and attention-output reconstruction error. DepthWeave-KV achieves the best task quality among compressed methods while also providing the largest memory reduction and fastest decode throughput. Method
Variant
Avg. Score (%)
Needle Acc. (%)
KV Mem. Red. (×)
Full KV Cache StreamingLLM H2O SnapKV PyramidKV MiniCache KVSharer ChunkKV TailorKV Eigen Attention
63.8 55.4 57.1 58.3 59.0 59.8 59.5 60.7 61.4 60.2
98.7 79.6 83.8 86.5 88.1 89.4 88.7 91.2 92.6 90.1
1.0 5.8 5.4 5.7 6.1 6.4 6.7 6.0 6.5 6.2
DepthWeave-KV
62.9
96.1
8.3
Avg. Score (%)
Needle Acc. (%)
DepthWeave-KV 62.9 96.1 Decode Thr. TTFT ∆ Recon. Err.61.5 w/o cross-depth basis PPL sharing 93.2 (tok/s) (s) w/o token-conditional router 61.2 92.4 w/o 93.5 42.1online error 6.84tracking 0.00 0.000 61.7 w/o 91.8 58.7residual gates 5.91 0.42 0.091 61.0 shared 87.6 56.9 bases only, 5.98 no token 0.36 residuals 0.083 58.9 w/o 95.9 60.4fused CUDA 5.73 kernel0.31 0.075 62.8 61.8 5.66 0.28 0.069 63.2 5.48 0.24 0.061 64.1 5.44 0.26 0.064 Ablations. Removing token-conditional routing and as62.6 5.52 0.21 0.055 signing a fixed residual rank to all tokens reduces the aver65.0 5.36 0.18 0.049 age task score from 62.9% to 61.1% and Needle accuracy 63.7 5.41 0.23 0.058
from72.8 96.1% to 4.89 91.8%. Disabling online 0.09 0.027 attention-output probes further reduces Needle accuracy to 89.9%, showing that static compression budgets do not reliably preserve lateemerging retrieval tokens. Replacing cross-depth residual factorization with direct layer sharing yields a 6.9× memory reduction but increases reconstruction error from 0.027 to 0.052. These ablations indicate that DepthWeave-KV’s gains come from the combination of shared depth-local bases, token-specific residual capacity, and online erroraware adaptation.
retrieval-critical spans avoids the brittle failures that arise when depth sharing or token compression is applied uniformly. DepthWeave-KV also improves systems metrics. Its fused implementation achieves 72.8 tokens/s during decoding, a 12.0% improvement over TailorKV and a 15.2% improvement over MiniCache. TTFT decreases from 6.84 s for Full KV Cache to 4.89 s, reflecting reduced memory traffic during prefill-side cache materialization and decode setup. Despite the more aggressive 8.3× memory reduction, DepthWeave-KV has a perplexity increase of only 0.09, less than half of TailorKV’s 0.18 increase. The attention-output reconstruction error is also lowest among compressed methods at 0.027, indicating that the online probe signal tracks behaviorally relevant cache distortion more directly than raw cache-vector error.
5. Ablation Study Ablation setup. We ablate the main components of DepthWeave-KV under the same 64K-token setting as Table 1. Each variant is evaluated with the same base model and decoding configuration. Unless otherwise noted, the memory controller is allowed to retune thresholds to target the best quality under a compressed-cache regime. Component analysis. Cross-depth basis sharing is responsible for much of the memory-quality tradeoff. Removing it and storing only independent token-local low-rank residuals lowers KV memory reduction from 8.3× to 5.9× and reduces throughput from 72.8 to 63.5 tokens/s. The average score remains competitive, but the variant loses 2.9 Needle accuracy points, indicating that shared depth-local bases preserve information that direct per-layer compression fails to retain efficiently. This supports the motivation behind depth-aware cache sharing methods such as MiniCache and KVSharer (Liu et al., 2024a; Yang et al., 2024a), while showing that factorized sharing is more effective than discrete reuse alone.
Breakdown by task family. On extractive and retrievalheavy tasks, DepthWeave-KV is closest to Full KV Cache. Averaged over Qasper, HotpotQA, MultiFieldQA-en, TriviaQA, and Needle-in-a-Haystack, DepthWeave-KV trails Full KV Cache by 1.4 points, whereas MiniCache and TailorKV trail by 5.6 and 3.2 points, respectively. On summarization tasks, including GovReport and QMSum, the gap is smaller for all methods because answer quality depends more on broad document coverage than on exact recovery of isolated spans. DepthWeave-KV nevertheless remains strongest among compressed methods, improving average summarization score by 0.8 points over TailorKV and 1.7 points over ChunkKV.
The token-conditional router is the largest contributor to retrieval robustness. Replacing it with a uniform residual-rank allocation reduces Needle accuracy from 96.1% to 92.4%, 5
KV Mem. (×) 8.3 5.9 7.9 8.5 8.4 10.2 8.3
DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression
despite a similar memory reduction factor. This variant wastes residual capacity on low-salience continuation tokens while under-allocating rank to instruction delimiters, entity mentions, and answer-bearing spans. The result is consistent with prior observations that average cache quality can hide retrieval-specific failures under aggressive compression (Chen et al., 2025; Haverbeck et al., 2026; Bui et al., 2026).
duction, outperforming compressed baselines such as MiniCache, KVSharer, ChunkKV, TailorKV, and Eigen Attention on the reported quality-memory tradeoff (Liu et al., 2024a; Yang et al., 2024a; Xiang et al., 2025; Yao et al., 2025; Saxena et al., 2024). Ablations show that token-adaptive routing and residual gates are essential for retrieval robustness, while cross-depth basis sharing and the fused kernel provide the main memory and throughput gains. These results suggest that effective long-context compression requires treating cache fidelity as both depth-dependent and tokendependent rather than applying a uniform budget across the sequence.
Online error tracking improves adaptation without requiring calibration data. When probe-based feedback is disabled and router thresholds are fixed after prefill, memory reduction slightly increases to 8.5×, but average score drops by 1.2 points and reconstruction error rises from 0.027 to 0.038. This suggests that fixed compression budgets are too rigid across generation phases: early decoding often tolerates stronger sharing, while later answer synthesis benefits from promoting tokens whose reconstructed attention outputs begin to drift.
7. Future Work A first direction is to extend DepthWeave-KV beyond fixed neighboring-layer windows. The present factorization assumes that cache redundancy is mostly local in depth, which matches the behavior of many decoder-only transformers but may be suboptimal for models with heterogeneous layer roles or sparse attention patterns. Future work could learn non-contiguous depth groups, head-specific sharing graphs, or routing policies that select basis donors dynamically, connecting DepthWeave-KV with broader depth-cache tradeoff analyses and stochastic routing methods (Filippova et al., 2026; Wang, 2026; Ma et al., 2026).
Residual gates provide a lightweight safeguard against overcorrecting the shared basis representation. Removing the gates while retaining routed residual ranks increases reconstruction error to 0.046 and lowers Needle accuracy to 91.8%. The “shared bases only” variant further confirms that depth sharing alone is insufficient: although it reaches 10.2× memory reduction and the highest raw throughput, it loses 4.0 average-score points relative to DepthWeaveKV and approaches the retrieval behavior of stronger compressed baselines in Table 1. Finally, disabling the fused CUDA kernel leaves model quality nearly unchanged but reduces decode throughput by 15.0% and increases TTFT by 0.69 seconds, showing that the factorization must be paired with compression-aware execution to realize its systems benefit.
A second direction is to make the router objective more explicitly aligned with downstream risk. Our online probes measure attention-output reconstruction error, but retrieval and instruction-following failures can arise from rare tokens whose importance is only revealed late in generation. Incorporating structural prompt signals, uncertainty estimates, or lightweight lookahead objectives may improve preservation of citations, entities, table cells, and multi-hop evidence chains under more aggressive compression (Chen et al., 2025; Haverbeck et al., 2026; Chen et al., 2026).
6. Conclusion DepthWeave-KV addresses long-context KV cache compression by combining cross-layer sharing with tokenadaptive residual reconstruction. Its cross-depth residual factorization captures redundant channel structure across neighboring transformer layers, while the token-conditional depth router preserves higher-rank residuals for instructionbearing and retrieval-critical tokens. Online attention-output error tracking further adjusts compression during generation without retraining the base model, and the fused implementation reduces decode-time memory traffic.
Finally, DepthWeave-KV opens systems questions around serving integration. The current fused kernel reduces memory traffic for a single compressed-cache layout, but production systems must coordinate compression with batching, prefix reuse, disaggregated prefill/decode, and multi-tenant cache pools. Combining cross-depth residual factorization with cache-centric serving architectures and compressionaware memory controllers could further improve throughput under mixed context lengths and workload-dependent quality constraints (Qin et al., 2024; Xie et al., 2025; Patel & Joshi, 2026).
Across LongBench, Needle-in-a-Haystack, L-Eval, NarrativeQA, Qasper, HotpotQA, MultiFieldQA-en, GovReport, QMSum, and TriviaQA, DepthWeave-KV preserves quality close to Full KV Cache while substantially reducing memory. It achieves an average task score of 62.9%, Needle retrieval accuracy of 96.1%, and an 8.3× KV memory re-
References A, A. and S, S. Ai agents as universal task solvers. PMC, 2026. doi: 10.3390/e28030332.
6
DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression
A, A., A, M., and J, O. Anatomic interactive atlas of the loggerhead sea turtle (caretta caretta) coelomic cavity. PMC, 2026a. doi: 10.3390/ani16050754.
Filippova, A., Grangier, D., Cuturi, M., and Monteiro, J. Stochastic kv routing: Enabling adaptive depth-wise cache sharing. Preprint, 2026. doi: 10.48550/arXiv. 2604.22782.
A, Z., Y, F., and J, Z. On demand secure scalable video streaming for both human and machine applications. PMC, 2026b. doi: 10.3390/s26041285.
Gao, W. Deep learning-based point cloud compression: An in-depth survey and benchmark supp1-3594355.pdf. Journal, 1970. doi: 10.1109/tpami.2025.3594355/mm1.
Akulov, D., Sana, M., Domenico, A. D., Salem, T. S., Piovesan, N., and Ayed, F. Kvcompose: Efficient structured kv cache compression with composite tokens. Preprint, 2025. doi: 10.48550/arXiv.2509.05165.
Garg, L., Yaswanth, S., Mishra, D. N., Kumaran, K., Sharma, A., and Uniyal, M. Monodense deep neural model for determining item price elasticity. arXiv preprint, 2026. doi: 10.1109/AAIML67890.2026. 11498150.
An, Y., Lu, C., Zhu, K., Yu, T., Zhao, C., Wu, H., Tang, M., and Wang, J. Rest-kv: Robust kv cache eviction with layer-wise output reconstruction and spatial-temporal smoothing. arXiv preprint, 2026.
Gelberg, Y., Eitan, Y., Bronstein, M., Gal, Y., and Maron, H. Training transformers for kv cache compressibility. arXiv preprint, 2026.
Berner, J., Grohs, P., Kutyniok, G., and Petersen, P. The modern mathematics of deep learning. arXiv preprint, 2021. doi: 10.1017/9781009025096.002.
Gokhale, S., Das, D., Patwari, R., Sirasao, A., and Delaye, E. Kv pareto: Systems-level optimization of kv cache and model compression for long context inference. Journal, 2026. doi: 10.18653/v1/2026.eacl-industry.9.
Bui, N., Nguyen, H. T., Cohan, A., and Ying, R. Make each token count: Towards improving long-context performance with kv cache eviction. arXiv preprint, 2026.
Haffari, G., He, Y., Liu, A., Liu, J., Pan, Z., and Zhuang, B. Minicache: Kv cache compression in depth dimension for large language models. Journal, 2024. doi: 10.52202/ 079017-4443.
Cai, Z., Xiao, W., Sun, H., Luo, C., Zhang, Y., Wan, K., Li, Y., Zhou, Y., Chang, L.-W., Gu, J., Dong, Z., Anandkumar, A., Asi, A., and Hu, J. R-kv: Redundancy-aware kv cache compression for reasoning models. arXiv preprint, 2025.
Haverbeck, L., Alonso, C. A., Posada-Moreno, A. F., Trimpe, S., and Pavone, M. The risk of kv cache compression. arXiv preprint, 2026.
Chen, A., Geh, R., Grover, A., den Broeck, G. V., and Israel, D. The pitfalls of kv cache compression. arXiv preprint, 2025. Chen, Z., Liu, P., and Shao, L. Structkv: Preserving the structural skeleton for scalable long-context inference. Preprint, 2026. doi: 10.48550/arXiv.2604.06746.
J, G., X, J., Z, S., H, T., X, Z., C, X., W, L., Y, S., S, M., H, L., S, W., K, Z., Z, L., B, Z., L, N., W, G., Y, W., and J, G. A survey on llm-as-a-judge. PMC, 2026. doi: 10.1016/j.xinn.2025.101253.
Corallo, G., Faure-Rolland, E., Lamari, M., and Papotti, P. Tablekv: Kv cache compression for in-context table processing. Journal, 2025. doi: 10.18653/v1/2025.trl-1. 13.
Javidnia, N., Rouhani, B. D., and Koushanfar, F. Key, value, compress: A systematic exploration of kv cache compression techniques. Journal, 2025. doi: 10.1109/cicc63670. 2025.10983416.
Cunegatti, E., Vukojevic, M., Nielsen, E., and Iacca, G. From layers to submodules: Rethinking granularity in replacement-based llm compression. Preprint, 2026.
Jiang, J., Yang, P., Zhang, R., and Liu, F. Towards efficient large language model serving: A survey on system-aware kv cache optimization. Journal, 2025. doi: 10.18653/v1/ 2026.findings-acl.1916.
da Silva, J. G. J., de Oliveira, S. S. T., Alves, L. A., Eiris, N., and Filho, A. R. G. Kv-raptor: Scalable tree-structured retrieval with kv cache compression for question-answering systems. Journal, 2025. doi: 10.5753/sbbd.2025.247245.
Jo, D., Song, J., Kim, Y., and Kim, J.-J. Fastkv: Decoupling of context reduction and kv cache compression for prefilldecoding acceleration. Journal, 2026. doi: 10.18653/v1/ 2026.findings-acl.1610.
Dong, X. L., Fu, Y., Diao, S., Byeon, W., Chen, Z., Mahabaleshwarkar, A. S., Liu, S.-Y., Keirsbilck, M. V., Chen, M.-H., Suhara, Y., Lin, Y., Kautz, J., and Molchanov, P. Hymba: A hybrid-head architecture for small language models. Journal, 2024. doi: 10.48550/arxiv.2411.13676.
Kriuk, B. and Ng, L. Q-kvcomm: Efficient multi-agent communication via adaptive kv cache compression. Journal, 2026. doi: 10.1109/icmsci67830.2026.11469367. 7
DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression
Kumar, A. Residual vector quantization for kv cache compression in large language model. Preprint, 2024. doi: 10.48550/arXiv.2410.15704.
conductance-based neuron: Bifurcation landscape, orbit catalog, and capacity analysis. PMC, 2026. doi: 10.3390/e28060678.
L, F., Q, S., Q, X., and X, L. Development of a rapid deeplearning-assisted multiphase multicomponent numerical simulation protocol. PMC, 2026. doi: 10.1021/acsomega. 6c03179.
N, P., S, B., F, R., O, V., A, N., T, V., NK, J., S, T., and A, A. F-transformer: a federated transformer for efficient and privacy-preserving sequence generation. PMC, 2026. doi: 10.1038/s41598-026-40881-0.
Li, H., Li, Y., Tian, A., Tang, T., Xu, Z., Chen, X., Hu, N., Dong, W., Li, Q., and Chen, L. A survey on large language model acceleration based on kv cache management. Journal, 2024. doi: 10.48550/arxiv.2412.19442.
Nguyen-Tri, Q., Ranjan, M., and Shen, Z. Attention is all you need for kv cache in diffusion llms. arXiv preprint, 2025. Pandey, D. and Yu, Q. Learn to accumulate evidence from all training samples: Theory and practice. arXiv preprint, 2023.
Li, R., Fu, Y., Sheng, M., Long, X., Yu, H., and Li, P. Faedkv: Infinite-window fourier transform for unbiased kv cache compression. Journal, 2025. doi: 10.18653/v1/ 2025.findings-emnlp.914.
Patel, I. and Joshi, I. Polykv: A shared asymmetricallycompressed kv cache pool for multi-agent llm inference. arXiv preprint, 2026. doi: 10.5281/zenodo.19686729.
Liu, A., Liu, J., Pan, Z., He, Y., Haffari, G., and Zhuang, B. Minicache: Kv cache compression in depth dimension for large language models. arXiv preprint, 2024a.
Pustovit, A. Knowledge packs: Zero-token knowledge delivery via kv cache injection. arXiv preprint, 2026.
Liu, A., Liu, J., Pan, Z., He, Y., Haffari, G., and Zhuang, B. Minicache: Kv cache compression in depth dimension for large language models. Journal, 2024b. doi: 10.48550/ arxiv.2405.14366.
Qiao, A., Yao, Z., Rajbhandari, S., and He, Y. Swiftkv: Fast prefill-optimized inference with knowledge-preserving model transformation. Journal, 2025. doi: 10.18653/v1/ 2025.emnlp-main.1306.
Lye, K. O., Mishra, S., and Ray, D. Deep learning observables in computational fluid dynamics. arXiv preprint, 2019. doi: 10.1016/j.jcp.2020.109339.
Qin, R., Li, Z., He, W., Zhang, M., Wu, Y., Zheng, W., and Xu, X. Mooncake: A kvcache-centric disaggregated architecture for llm serving. Journal, 2024. doi: 10. 48550/arxiv.2407.00079.
M, G., S, R., S, R., X, L., and S, S. The rise of small language models in healthcare: A comprehensive survey. PMC, 2026a. doi: 10.1016/j.cosrev.2026.100999.
Qin, Z., Jiang, Y., Lin, M., Lv, Y., Guo, H., Fei, W., and Lin, W. Hack++: Towards more effective head-aware key-value compression for efficient visual autoregressive modeling. Preprint, 2026.
M, N., L, D., A, T., and A, P. Building large-scale englishromanian literary translation resources with open models. PMC, 2026b. doi: 10.3389/frai.2026.1807431.
Ray, D., Pinti, O., and Oberai, A. A. Deep learning and computational physics (lecture notes). arXiv preprint, 2023.
Ma, D., Chen, L., Zhang, S., Miao, Y., Zhu, S., Chen, Z., Xu, H., Li, H., Fan, S., Pan, L., and Yu, K. Compressing kv cache for long-context llm inference with inter-layer attention similarity. Journal, 2026. doi: 10.1109/icassp55912.2026.11464826.
RJ, C. and F, R.-S. Structural optimization principles for edge ai in motorsport telemetry. PMC, 2026. doi: 10. 1038/s41598-026-49736-0. S, P., H, D., and S, M. Green prompt engineering for sustainable generative ai. PMC, 2026. doi: 10.1016/j.ese. 2026.100684.
Marti, M. and Maki, A. A multitask deep learning model for real-time deployment in embedded systems. arXiv preprint, 2017.
Saxena, U., Saha, G., Choudhary, S., and Roy, K. Eigen attention: Attention in low-rank space for kv cache compression. Journal, 2024. doi: 10.18653/v1/2024. findings-emnlp.899.
MF, Y., CC, L., and CL, L. Multi-hardware benchmarking of open-source large language models with retrieval-augmented generation for mitsubishi fx-series plc instruction list code generation. PMC, 2026. doi: 10.3390/s26113602.
Slothouber, C. The kv4 trade-off is workload-dependent: A depth-and workload-resolved study of 4-bit kv-cache quantization on a 4 gb turing gpu. Journal, 2026. doi: 10.2139/ssrn.6941538.
MO, A., AJ, A., MAF, A.-H., and KM, A. Limit-cycle proliferation under parametric delayed feedback in a 8
DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression
Tang, Z., Luohe, S., Li, Z., Qi, B., Guoming, L., Zhang, L., and Wang, P. Spindlekv: A novel kv cache reduction method balancing both shallow and deep layers. Journal, 2025. doi: 10.18653/v1/2025.acl-long.1380.
Yang, D., Yang, Y., Yu, X., Qi, X., and Xiao, R. Hcattention: Extreme kv cache compression via heterogeneous attention computing for llms. Journal, 2026. doi: 10.1016/j.neucom.2026.134247.
Wang, X. How much cache does reasoning need? depthcache tradeoffs in kv-compressed transformers. Preprint, 2026. doi: 10.48550/arXiv.2604.17935.
Yang, Y., Cao, Z., Chen, Q., Qin, L., Yang, D., Zhao, H., and Chen, Z. Kvsharer: Efficient inference via layerwise dissimilar kv cache sharing. Journal, 2024a. doi: 10.48550/arxiv.2410.18517.
Wang, Z., Jin, B., Yu, Z., and Zhang, M. Model tells you where to merge: Adaptive kv cache merging for llms on long-context tasks. Journal, 2024. doi: 10.48550/arxiv. 2407.08454.
Yang, Z., Han, J., Wu, K., Xie, R., Wang, A., Sun, X., and Kang, Z. Lossless kv cache compression to 2%. Journal, 2024b. doi: 10.48550/arxiv.2410.15252.
Wei, id orcid 0009-0003-9914-2657 Jiang, and Wang, W. Sub-token routing for kv cache compression. Journal, 2026.
Yang, Z., Han, J. N., Wu, K., Xie, R., Wang, A., Sun, X., and Kang, Z. Lossless kv cache compression to 2%. arXiv preprint, 2024c.
X, K., T, Y., L, W., Y, G., and F, Z. Ss-adamoe: Spatiospectral adaptive mixture of experts with global structural priors for graph node classification. PMC, 2026a. doi: 10.3390/e28030355.
Yao, D., Shen, B., Lin, Z., Liu, W., Luan, J., Wang, B., and Wang, W. Tailorkv: A hybrid framework for long-context inference via tailored kv cache optimization. Journal, 2025. doi: 10.18653/v1/2025.findings-acl.1043. Yu, B. and Chai, Y. Evolkv: Evolutionary kv cache compression for llm inference. Journal, 2025a. doi: 10.18653/v1/2025.findings-emnlp.88.
X, S., L, F., S, H., D, L., Z, X., and B, L. Towards sustainable ai knowledge-base assistants in computer science education: on-premise deployment and optimization with open educational resources. PMC, 2026b. doi: 10.3389/fpsyg.2026.1843444.
Yu, B. and Chai, Y. Evolkv: Evolutionary kv cache compression for llm inference. arXiv preprint, 2025b.
X, Z., M, H., S, L., S, K., EY, L., Y, L., and WD, L. Compute-in-memory implementation of state space models for event sequence processing. PMC, 2026c. doi: 10.1038/s41467-025-68227-w.
Z, L., X, M., X, H., Y, Z., and W, Y. Large-scale modelenhanced vision-language navigation: Recent advances, practical applications, and future challenges. PMC, 2026. doi: 10.3390/s26072022.
Xiang, L., Tang, Z., Dong, P., Li, Z., Liu, Y., Li, B., Hu, X., and Chu, X. Chunkkv: Semantic-preserving kv cache compression for efficient long-context llm inference. Journal, 2025. doi: 10.48550/arxiv.2502.00299.
Z, W., R, Z., DA, W., D, E., L, S., O, B., W, K., E, Z., CK, L., WC, S., A, P., J, W., GCL, W., R, L., S, L., MZ, L., M, K., P, G., TK, H., and L, G. Kilohertz volumetric imaging of in vivo dynamics using squeezed light field microscopy. PMC, 2025. doi: 10.1038/s41592-025-02843-8.
Xiao, E., Li, C. H., Zhang, Y., Neubig, G., and Bertsch, A. Efficient many-shot in-context learning with dynamic block-sparse attention. Journal, 2025. doi: 10.18653/v1/ 2025.acl-long.1542.
Zhang, J., Zhu, D., Song, Y., Wu, W., Kuang, C., Li, X., Shang, L., Liu, Q., and Li, S. More tokens, lower precision: Towards the optimal token-precision tradeoff in kv cache compression. Preprint, 2024. doi: 10.48550/arXiv.2412.12706.
Xie, R., Haq, A. U., Ma, L., Fang, Y., Engineer, Z. B., Liu, L., and Zhang, T. Reimagining memory access for llm inference: Compression-aware memory controller design. Preprint, 2025. doi: 10.48550/arXiv.2503.18869.
Zhang, T., Li, Y., Wang, J., and Li, L. Enhancing large multimodal models with adaptive sparsity and kv cache compression. Journal, 2025. doi: 10.1109/icme59968. 2025.11209796.
Y, G., Y, D., R, M., Y, Z., and J, G. Infrared imaging for autonomous power inspection: A review from detector to system integration. PMC, 2026. doi: 10.3390/s26113552. Yang, D., Yang, Y., Yu, X., Qi, X., and Xiao, R. Hcattention: Extreme kv cache compression via heterogeneous attention computing for llms. Journal, 2025. doi: 10.2139/ssrn.5403127. 9