Adaptive DHiSparse: Speculative Decoding over a Load-Adaptive KV Hierarchy

Adaptive DHiSparse: Speculative Decoding over a Load-Adaptive KV Hierarchy Naji Alazhar

Wassel Alazhar

Umans AI research@umans.ai

arXiv:2610.13708v1 [cs.AI] 9 Oct 2026

Abstract

Hierarchical KV caching addresses this mismatch by retaining history in host memory and recovering only the entries selected by the model. HiSparse combines a bounded GPU cache, fused miss resolution, and prefetch for shared selections (Xie et al., 2026). A recently used row can remain in the cache and serve subsequent queries, exploiting locality while bounding each request’s device footprint. Its fixed budget imposes a deployment-level trade-off: a small working set permits more sessions but can incur avoidable host reads while HBM is available; a larger reservation reduces retrieval but consumes admission capacity under pressure. A deployment with changing load encounters both sides of this trade-off. Section 4.3.4 measures it. HiSparse also has no path for draft-model speculative decoding: it maintains no draft KV, resolves one query per request, and provides no multi-query verification staging. This matters because draft models can amortize target-model computation across several output tokens. DSpark selects request-dependent verification prefixes using confidence and engine cost, while DFlash2 verifies a configured block generated through its conditional selector (Cheng et al., 2026; Inco AI, 2026). Their benefits depend on both the return from verification and the memory available to sustain concurrent requests. A hierarchy that supports only singlequery decoding leaves this acceleration path unavailable. Verification expands the immediate KV working set because several queries can select the same historical row while retaining separate causal boundaries and producing tentative state that may later be rejected. Resolving each query independently can repeat host reads; retaining every selection independently can require substantially more staging than the hot cache. Draft KV introduces another persistent allocation, and keeping it in a full-history layout can consume the capacity released by target offloading. The KV layer must therefore manage shared historical reads, temporary verification state, and the prefix retained after acceptance as parts of the same execution path. We introduce Adaptive DHiSparse, which brings DSpark and DFlash2 to hierarchical sparse KV serving and adapts history residency to the memory available at each load. We call the same system without a

Long-context serving runs out of KV memory before the hardware reaches its useful throughput. HiSparse stores history in host memory and fetches only the selected entries, but it uses one cache budget at every load and does not handle draft-model speculative decoding such as DSpark and DFlash2. We present Adaptive DHiSparse, which supports speculative decoding over hierarchical sparse KV and adapts residency to load. It protects the required working set, retains additional history while GPU memory is spare, and recovers the rest through exact fused retrieval. DSpark and DFlash2 keep their proposal and acceptance rules: queries share missing-row retrieval, draft KV uses a window-bounded ring, and verification staging is reused across layer groups. At 100K context and concurrency 128, GLM-5.3 with DFlash2 delivers 793 output tokens/s per decode GPU, 1.45× HiSparse and 1.27× non-speculative adaptive serving. Held to the same latency bounds as HiSparse, it sustains up to 4× the concurrency and 2.3 to 3.7× the output, depending on the per-token bound. On DeepSeek-V4.1Flash at 500K context and concurrency 512, Adaptive DHiSparse with DSpark delivers 1.85× the output of resident serving with DSpark. In a load step, optional residency reduces host fetches from 14.2% to 1.4% of selected rows at low load and releases memory under pressure.

1. Introduction Long-context agents and assistants retain large histories while producing relatively short continuations; a coding request may carry material for several later decisions, leaving the cost of its KV state in place between decoding steps. Sparse attention reduces the entries read by each query, but conventional serving still reserves the complete history in GPU high-bandwidth memory (HBM). As context length and the number of sessions increase, this storage can prevent further admission before the hardware reaches its useful throughput (DeepSeek-AI, 2025, 2026; Xie et al., 2026). Reducing the computation of a query therefore makes the placement of its history increasingly important.

1

Resident

HiSparse

Adaptive HiSparse

Resident + DFlash2

Adaptive DHiSparse

4,000

3,000

2,000

1,000

c Mean time per output token (ms)

b Mean time to first token (s)

Generation throughput (tok/s)

a 300 100 30 10 3 1 0.3

0 8

32

64

128

Offered concurrency (requests)

0

1,000

2,000

3,000

4,000

Generation throughput (tok/s)

60 50 40 30 20 10 0 0

1,000

2,000

3,000

4,000

Generation throughput (tok/s)

Figure 1 | GLM-5.3 at 32K input and 8K output cap, using the input and output lengths of the HiSparse evaluation. (a) Generation throughput against offered concurrency, (b) mean TTFT on a logarithmic axis and (c) mean TPOT against generation throughput.

drafter Adaptive HiSparse. We protect a fixed working set for exact recovery, retain additional immutable history while HBM is spare, and reclaim that history when other requests need its pages. The fixed hot-cache structure remains small as optional residency grows, allowing the resolver to exploit additional hits without scanning the full resident history. Under pressure, backup and reclamation make those pages available for admission while selected entries remain recoverable from host memory. This couples the allocation decision to the execution that uses it. We follow the reuse structure of each model: compatible layers in DeepSeek-V4.1-Flash consume a shared global KV source, whereas GLM-5.3 shares sparse indices while retaining distinct layer payloads (DeepSeek-AI, 2026; Z.ai, 2026). Shared positions can consequently announce several future copies without making their bytes interchangeable. We exploit both forms of reuse and bound the additional speculative storage by its lifetime: the drafter keeps its existing visible window in a ring, while target layers reuse staging after their consumers complete. Together, these choices let verification operate over the hierarchy while preserving proposal generation, sparse selections, and acceptance. We evaluate GLM-5.3 and DeepSeek-V4.1-Flash using matched cache settings, reporting output together with first-token and per-token latency. The experiments cover fixed offered loads, a same-build load step, separate resolver measurements, and a sweep of the hot-cache budget with kernel profiles. They distinguish the capacity of an allocation from the activity that serving actually sustains, and show where more active requests improve aggregate output while slowing individual requests. Comparisons emphasize relative performance under the shared setup; absolute rates are in Table 2 and the figures. Our contributions are:

adaptive serving, with mean time to first token (TTFT) 2.1× lower than HiSparse; held to the same latency bounds as HiSparse, it sustains up to 4× the concurrency and 2.3 to 3.7× the output, depending on the bound on time per output token (TPOT). • Residency that follows load. Optional history reduces host-fetched selections from 14.2% to 1.4% at low load in the on/off experiment, yields its memory under pressure, and repopulates as requests drain. • Bounded speculative state. A windowed draft ring and reusable verification staging add 0.060 GiB of required GPU state per GLM session while supporting the full configured verification block. Figure 1 gives an early view at 32K input and an 8K output cap, matching the input/output lengths of the HiSparse evaluation. There, Adaptive DHiSparse delivers 1.7 to 1.9× the output of HiSparse from C8 to C64 and 1.45× at C128, where its mean TTFT is 2.3× lower. Resident serving with DFlash2 is 5 to 9% ahead up to C32, while every history still fits, and stops scaling at C64. Our main GLM experiments use 100K context, and the DeepSeek experiments use 500K context.

2. Background 2.1. Sparse Attention and KV Residency At query position 𝑡 , a sparse indexer assigns a score 𝐼𝑡,𝑠 to each historical KV entry 𝑠 and selects 𝑘 entries, denoted by S𝑡 . The model’s sparse-attention operator Attn combines the query hidden state h𝑡 with the selected stored rows c𝑠 to produce u𝑡 : S𝑡 = { 𝑠 | 𝐼𝑡,𝑠 ∈ Top-k( 𝐼𝑡,: )}, u𝑡 = Attn(h𝑡 , {c𝑠 : 𝑠 ∈ S𝑡 }) .

(1)

The stored rows may be compressed latent representations (DeepSeek-AI, 2025). We keep S𝑡 as the ordered list the model’s top-𝑘 returns, with its tie handling, causal masking and any additional local-attention branch, and write set operations on it when only mem-

• Speculative serving on hierarchical KV. At 100K context and offered concurrency 128 (C128), GLM5.3 with DFlash2 delivers 793 tokens/s per decode GPU, 1.45× HiSparse and 1.27× non-speculative 2

bership matters. The indexer runs before the selected payload is read, providing an explicit demand signal to the memory system. Keeping selection state on the GPU lets that computation continue over the logical history while the larger attention payload is distributed across memory tiers. The resolver then supplies the announced rows before attention consumes them. An entry omitted from S𝑡 can appear in a later selection, so sparse attention reduces what a query reads while the complete history remains logically available. HiSparse exploits this distinction by changing where entries are stored rather than which entries attention selects (Xie et al., 2026). Let R 𝑡 contain the entries ready on the GPU in the tail, hot cache, or optional history. The selected entries that require host recovery form D𝑡 = S𝑡 \ R 𝑡 .

A fixed cache keeps its configured size at every load: it fetches rows from host while memory is spare, and a larger reservation would cost admission capacity when load rises. This motivates separating the small replacement working set from additional, reclaimable history. Physical-source sharing. A KV source produces bytes consumed by one or more layers: different layers may select the same positions from distinct sources, whereas DeepSeek-V4.1-Flash shares global KV and indexer keys across compatible layers. A Full layer produces a source and a selection; a Reindex layer selects again using shared KV; a Reuse layer consumes the latest applicable selection (DeepSeek-AI, 2026). A residency manager can consequently organize payload by request and physical source, while selection reuse determines when a location vector can also be shared.

(2)

Placement changes D𝑡 while preserving S𝑡 . Retaining additional history may reduce recovery, but consumes memory that could admit another request, and populating that history also incurs work.

2.3. Parallel Drafting and Verification Speculative decoding proposes several tokens for verification in one target pass: DFlash conditions a parallel draft block on target features (Chen et al., 2026); DSpark adds a lightweight sequential head and confidencescheduled verification prefixes (Cheng et al., 2026); DFlash2 uses a conditional selector over its candidate lattice (Inco AI, 2026). The target accepts a prefix and emits a correction or bonus according to the existing verification rule (Leviathan et al., 2023). These methods determine proposal and acceptance behavior; the KV layer supplies the historical and tentative state required to execute their queries.

2.2. KV Residency Strategies Serving approaches differ in how much history they reserve on the GPU and how selected entries reach attention, with the resulting storage and retrieval costs determining the useful active batch. Full-history residency. Full-resident serving keeps every admitted request’s KV cache in HBM, giving each selected position an existing device location. For a source containing 𝐿 historical rows, payload storage grows in proportion to 𝐿. Across concurrent requests, these histories can exhaust the available cache budget and cap the active decode batch (Xie et al., 2026). Paging reduces allocation waste, and source sharing removes duplicated state, but each distinct resident history still occupies device memory (Kwon et al., 2023; DeepSeek-AI, 2026).

3. Architecture We combine load-adaptive residency, residency-aware resolution, and speculative KV-state management around the selected set in equation (1): the model chooses positions, and our runtime supplies their device locations. Figure 2 follows one decode step from selection to attention. We first describe the division between required storage and optional history, then how resolution and speculative execution use it.

Fixed-cache offloading. HiSparse retains complete history in pinned host memory and gives each request a bounded GPU cache of 𝐵 entries for each KV source (Xie et al., 2026). After the indexer selects positions, a fused kernel identifies hits, protects useful cached rows, chooses least-recently-used (LRU) victims, fetches missing entries, and returns their device locations. Compact indexer state remains on the GPU, allowing the model to select across the full history while its larger KV payload uses the hierarchy. The cache preserves temporal locality: a selected entry can remain in the hot set rather than being copied on every step. Increasing 𝐵 retains more history, whereas decreasing it leaves room for additional requests. For models that share selections across layers, HiSparse also prefetches layer-local payload after an earlier index producer announces the positions (Xie et al., 2026; Bai et al., 2026). The transfer can begin during intervening computation and complete before the consuming attention layer.

3.1. Load-Adaptive Residency We reserve a hot cache, a mutable tail and metadata for each admitted request and KV source. With a drafter we also reserve verification staging (Section 3.4). The hot cache holds up to 𝐵 recently useful rows; the tail holds newly computed state until it can become immutable history. Additional history occupies pages that can be returned when required allocations grow, letting a request retain many more than 𝐵 rows while keeping the replacement structure small. Optional history occupies reclaimable capacity within the configured KV pools: unclaimed dedicated payload pages and eligible unused index backing. Let 𝑀 be the capacity of those pools. At time 𝜏, let 𝑀req ( 𝜏) be the memory held by required alloca-

3

top-k locations

Indexer

12

98

5

31

77

63

99

40

Resolver

Attention

2

1

1

GPU HBM

moves with load 12

index state

31

40

5

optional history

63

77

hot cache

97

98

99

tail

3

Host DRAM host backing

selected row

in tail

in optional history

in hot cache

copied from host

not selected

reserve page

not backed up

Figure 2 | One decode step and where its KV lives. The indexer reads index state and emits the top-𝑘 positions, colored here by where each row is found. The resolver first takes rows that are in the tail or in optional history (1), then scans the hot cache (2). A row found in none of them is copied from host DRAM into the hot cache (3) before attention reads the selected rows. Optional history holds imported history in reclaimable capacity of the KV pools, drawn schematically as one divider that moves with load. The hot cache and the tail are kept per request and KV source and are never lent. Host backup can be deferred while HBM is spare: a row without host backing stays protected on the GPU, so every missing row has host backing. A lookup skips steps 2 and 3 only when every selected row resolves in step 1. Row positions are illustrative.

tions and 𝑀opt ( 𝜏) the memory held by optional history. Keeping an admission reserve 𝑀res , we require 𝑀req ( 𝜏) + 𝑀opt ( 𝜏) + 𝑀res ≤ 𝑀.

is much larger. Increasing optional residency retains more possible future selections without enlarging the LRU scan, so allocation can respond to available capacity while execution retains the same bounded fallback during admission and partial refill. We reclaim pages only after their rows have host backing and earlier readers complete, then return them to required allocations. Population reverses this order: reserve eligible pages, copy or retain their bytes, and publish ready locations after completion. This ordering lets a request continue through partial coverage while protecting device-only rows until their host backup completes. Every row of a request’s history is therefore, at all times, ready on the GPU or has a host copy. On admission we reserve the request’s hot cache and keep its imported history on device while HBM is spare, deferring its host backup. We call this residentfirst admission. Under pressure, we select whole requests until their optional pages cover the deficit, back up their history, and retire those pages once backup and earlier readers complete; the new request takes the freed capacity and the other requests keep their history on device. A reclaimed request then resolves its history through its hot cache, fetching misses from host. Completion releases a request’s capacity, which later population refills with history or leaves free.

(3)

Required allocations and the reserve take priority; hotcache and tail storage are never lent. Optional residency covers complete pages of a request’s imported history. Rows generated during decoding and the partial boundary page stay under the tail and hot-cache rules, and a row may be in the optional map and in the hot cache at once. Let 𝑎𝑟 be the number of optional pages held by request 𝑟 . When an admission leaves a deficit of Δ pages, the runtime reclaims whole requests: it walks the admitted requests in its reclaim order 𝑟1 , 𝑟2 , . . . and takes the pages of the first 𝑞, with ′

n

′

𝑞 = min 𝑞 :

𝑞 ∑︁

o

𝑎𝑟𝑖 ≥ Δ .

(4)

𝑖=1

The other requests keep their pages. If the optional pages do not cover Δ, the admission waits. When pages are free again, a bounded round-robin scheduler repopulates history, a limited number of pages per attempt. We track ready locations separately from allocated pages so attention uses rows only after their copies complete. A pending page remains unavailable through the optional map, while the hot-cache recovery path continues to serve selections. The fixed 𝐵 therefore governs replacement work even when optional history

3.2. Residency-Aware Resolution The fused resolver takes the model’s selected positions and returns their physical KV locations to attention. It 4

first locates rows in the tail and ready optional-history map (Figure 2, callout 1), then scans the bounded hot cache for unresolved positions, protecting selected rows and identifying unselected LRU victims (callout 2). It recovers the remaining misses from pinned host memory and emits locations in selection order (callout 3). Hit detection, replacement, retrieval, and location generation share request-local intermediate state. The copy path launches over a grid that scales with the batch, from 4 to 32 thread blocks, so that row movement uses the available parallelism. Rows resolved in the first step (callout 1) never reach replacement or host recovery. In a census of selected rows on GLM-5.3 at C8, without speculation, 69.7% resolved directly, through the optional-history map or the tail, 30.3% through the hot cache and 0.05% from host. The hot scan remains bounded by 𝐵 rather than the entire resident history. A lookup skips the hot-cache scan only when every selected row resolves through a ready mapping or the tail. Rows generated during decoding follow hot-cache rules, so keeping the whole imported history on device does not by itself make a lookup wholly direct (Section 4.3.3). We organize the cache by physical KV source so that a DeepSeek-V4.1-Flash Reuse consumer can inherit the preceding selected-location vector together with its materialized payload. A Reindex consumer chooses a fresh top-𝑘 from the shared source, reusing rows already present and recovering any new misses. The cache update follows earlier attention readers, allowing several layers to share storage safely as their selections change. A shared source therefore carries one cache state across its consumers: rows fetched for an earlier selection remain available to later layers, and a fresh selection adds only the rows absent from that state. Reusing a selection also avoids another resolution call for the same location vector. GLM-5.3 instead has 21 index producers announcing positions for 78 distinct payload layers, allowing one selection to start several follower-layer transfers while each layer retains its own bytes. We use HiSparse’s exact layer-wise prefetch (Xie et al., 2026) with sourcespecific locations, reusing selection and transfer planning through shared indices and reusing payload itself where the model shares KV. This distinction determines which consumers share a resolved vector and which need separate transient storage, whose size we bound by the simultaneously live layer group in Section 3.4.

Let 𝑛𝑟 be the number of token positions covered by committed target KV before the round. Query 𝑗, with 0 ≤ 𝑗 < 𝐾𝑟 , sits at position 𝑛𝑟 + 𝑗; 𝑗 = 0 is the anchor. Its visible token positions form V𝑟, 𝑗 = { 𝑣 : 𝑣 < 𝑛𝑟 } ∪ { 𝑛𝑟 + 𝑢 : 0 ≤ 𝑢 ≤ 𝑗 },

(5)

the committed history, served through the hierarchy, and the current block up to the query itself, read from the tail. We write S𝑟, 𝑗 for the query’s selected rows; at token granularity, S𝑟, 𝑗 ⊆ V𝑟, 𝑗 . Compressed source rows must also be complete, as described below. We keep these query boundaries when resolving historical demand, even though all queries execute in one target pass. For a fixed request and payload source, let R 𝑟 be the rows ready on the GPU, as in equation (2), and H𝑟 the rows with a complete host copy. The retrieval union U𝑟 and the missing-row set D𝑟 are U𝑟 =

𝐾Ø 𝑟 −1

S𝑟, 𝑗 ,

|U𝑟 | ≤ 𝐾𝑟 𝑘,

𝑗=0

(6)

D𝑟 = U𝑟 \ R 𝑟 ⊆ H𝑟 . The inclusion is the invariant of Section 3.1: a row that is not ready on the GPU always has a host copy. We deduplicate these misses within the request and payload source, copy each distinct row once, and return its location to every selecting query. In Figure 3, callout 2, row a is selected three times and row b twice, yet each requires one host copy while attention retains each ordered S𝑟, 𝑗 and its causal mask. Newly computed rows and incomplete boundary state remain in the tail, where their visibility follows the individual query. A union can exceed the 𝐵-row hot cache even when each single-query selection fits. We reserve staging for these overflow rows and protect all selected locations until their attention consumers finish. The bound 𝐾max 𝑘 per live request/source supplies space for disjoint selections; overlapping selections reduce the copies required within that allocation. Thus the configured hot-cache budget continues to preserve temporal locality while verification can materialize a larger immediate working set. Section 3.4 describes how layer groups reuse this staging. Write 𝑥𝑟,0 for the anchor and 𝑥𝑟,1 , . . . , 𝑥𝑟,𝐾𝑟 −1 for the proposed tokens. The existing acceptance procedure (Leviathan et al., 2023) accepts 𝐴𝑟 of the 𝐾𝑟 −1 proposals, 0 ≤ 𝐴𝑟 ≤ 𝐾𝑟 − 1, and emits a correction or bonus 𝑏𝑟 that starts the next round. The retained state and new committed frontier 𝑛𝑟′ satisfy

3.3. Speculative Verification over the Hierarchy DSpark chooses a per-request prefix with its confidence scheduler, whereas DFlash2 uses a fixed block (Cheng et al., 2026; Inco AI, 2026). For request 𝑟 , we denote the number of target queries in a round by 𝐾𝑟 , including the anchor, and the maximum supported width by 𝐾max . Our KV path leaves proposals and acceptance unchanged. Figure 3 shows how the common retrieval and state-management path handles the resulting block.

new output = [ 𝑥𝑟,1 , . . . , 𝑥𝑟,𝐴𝑟 , 𝑏𝑟 ] , committed prefix = [ 𝑥𝑟,0 , . . . , 𝑥𝑟,𝐴𝑟 ] ,

(7)

′

𝑛𝑟 = 𝑛𝑟 + 1 + 𝐴 𝑟 .

The next forward computes the target KV of 𝑏𝑟 . In Figure 3, callout 4, the target has computed D, E, F, and G; accepting E and F retains D–F, discards G, and emits G∗ as the next anchor. 5

next round

Drafter

D

D

E

F

Target model

G

a top-k

a

D

E

F

G

G*

b a b

Resolver GPU HBM hot cache a

draft KV ring

1

tail

staging arena

D

b

E

F

G

3 2

4

Host DRAM host backing

D

target token

E

draft token

a

row on device

row copied from host

b

row written this round

backup after verification

Figure 3 | One speculative round over the KV hierarchy. The drafter, conditioned on target features, extends the anchor D into a block, and the target model verifies its 𝐾 queries in one pass. Here E and F are accepted, G is rejected, and the correction G∗ opens the next round. (1) Draft KV is held in a ring bounded by the drafter’s window. (2) For each request and KV source, the resolver handles the top-𝑘 selections of the 𝐾 queries together, so each distinct missing row is copied from host once (a is selected by three queries, b by two), and each query still attends over its own selection. (3) Rows that do not fit the hot cache go to a staging arena, reused across layer groups after its readers finish and promoted rows are copied back. (4) The target writes the rows of all 𝐾 queries to the tail. Acceptance commits the computed rows of the anchor and the accepted drafts; the rejected row is dropped and G∗ has no row until the next forward. Host backup of the committed rows follows verification. Sizes are illustrative ( 𝐾 = 4, 𝑘 = 3).

For a source that packs 𝑚 consecutive token positions into each stored row (𝑚 = 1 without compression), the source-row indices 𝑠 completed by the round satisfy   ⌊ 𝑛𝑟 /𝑚⌋ ≤ 𝑠 < 𝑛𝑟′ /𝑚 . (8)

taneously live, preserving the information consumed by the models while limiting its physical footprint. A drafter that reads at most 𝑊 prior positions needs storage for that window and its current block of at most 𝐾max positions. With 𝑝 rows per page, alignment adds at most 𝑝 − 1 positions. We allocate a ring of 𝑅𝑑 rows per request, where   𝑊 + 𝑝 − 1 + 𝐾max . (9) 𝑅𝑑 = 𝑝

These rows join the immutable history, and their host backup is scheduled after verification; any remaining incomplete group stays in the tail until it is complete. Committed rows follow the accepted prefix, rejected and unverified rows are never published, and the drafter’s state advances with the same prefix. This interface separates how many queries the drafter asks the target to verify from how much history those queries fetch. DSpark can change 𝐾𝑟 across requests and rounds, while DFlash2 uses its configured width; both retain their original verification decisions. The storage system handles the resulting union and accepted state, including repetitions of a selected position across queries.

𝑝

This covers the visible interval when previous readers complete before overwrite. For request slot 𝑖, the physical ring address 𝜙 ( 𝑖, 𝑡 ) of draft position 𝑡 is 𝜙 ( 𝑖, 𝑡 ) = 𝑖𝑅 𝑑 + ( 𝑡 mod 𝑅𝑑 ) .

(10)

Two live positions of one request differ by less than 𝑅𝑑 , so they never share a row, and slots occupy disjoint ranges. Figure 3, callout 1, places this bounded draft state in HBM. Logical positions advance through the ring while the drafter continues to read the same suffix, making its KV allocation independent of the length of the target’s full context. In GLM-5.3/DFlash2, 𝑊 = 2,048, 𝑝 = 64, and 𝐾max = 8 give 𝑅𝑑 = 2,176, a page-rounded size that also covers the six- and four-query blocks. Six draft transformer layers have twelve K/V allocations total-

3.4. Bounded Speculative Storage Speculation adds persistent draft KV and temporary target-verification state, allocations that can become the next concurrency constraint once the target history is offloaded. We size the draft by its visible window and the target staging by the layer payloads that are simul-

6

Table 1 | Memory per session for GLM-5.3, in GiB. Required GPU memory at 32K and 100K context, and the full host backup at 100K. Model weights and shared workspaces are excluded. † excludes optional residency, which only uses HBM that is spare. GPU, required Method Resident HiSparse Adaptive HiSparse Resident + DFlash2 Adaptive DHiSparse

3.5. Exactness Our hierarchy changes where KV rows are stored, while each query receives the model’s ordered selection, payload and scales, and causal mask. Sharing a retrieval reuses the copy rather than merging attention inputs. For the payload source at hand, 𝜋𝑟 ( 𝑠) is the GPU location the resolver returns for row 𝑠 of request 𝑟 ; a row held in two places has one of them chosen. If 𝑠𝑟, 𝑗,𝑖 is the 𝑖-th entry in query 𝑗’s ordered selection, the resolver returns a location ℓ𝑟, 𝑗,𝑖 satisfying

Host backup

32K

100K

100K

1.456 0.259 0.259† 1.481 0.319†

4.444 0.432 0.432† 4.469 0.492†

– 4.186 4.186 – 4.186

ℓ𝑟, 𝑗,𝑖 = 𝜋𝑟 ( 𝑠𝑟, 𝑗,𝑖 ) ,

(12)

Here KV[ ℓ] is the GPU payload at location ℓ, and c𝑟,𝑠 is the original KV row 𝑠 of request 𝑟 . Each query receives its own ordered locations; queries selecting the same row share its location. We preserve target features, proposal probabilities, acceptance, and accepted-prefix updates, and make committed rows recoverable before reusing their device storage. Given equal model state and numerical execution, these properties preserve attention inputs and the verifier’s output law (Leviathan et al., 2023). The ring retains the drafter’s visible window, and staging is reused only after its readers and required copy-back complete. These are the design conditions for exact placement.

ing 12,288 bytes per row. For 35 request slots they occupy about 0.87 GiB per GPU. The target can continue selecting historical rows far outside the draft window through the KV hierarchy, while the drafter retains the suffix and target features it already uses. Target verification instead reuses temporary storage, with overflow rows in Figure 3, callout 3, occupying staging separate from the hot cache. Layers that share selected positions may still read different payloads, so each simultaneously active source receives a distinct arena. After all consumers of a layer group complete, the next group can reuse those arenas. Promoted rows are copied back to their stable hot-cache addresses before an arena is released. With 𝜒 payload sources live at once, 𝑁 request slots per GPU, and 𝑑KV bytes per stored KV row, the staging reservation is 𝑀stage = 𝜒𝑁 𝐾max 𝑘 𝑑KV .

KV[ ℓ𝑟, 𝑗,𝑖 ] = c𝑟, 𝑠𝑟, 𝑗,𝑖 .

4. Experiments We evaluate GLM-5.3 and DeepSeek-V4.1-Flash serving, then examine how optional residency responds to load, how verification width affects output, what a miss costs, and how large the hot cache should be.

(11)

The reservation 𝑀stage covers the maximum verification width, with allocator rounding, maps, and persistent cache state accounted separately, while each round uses locations selected by its live queries. GLM-5.3 processes its index-sharing layer groups sequentially, with at most four payload layers live together. Reusing four arenas instead of dedicating one to each of 78 layers divides this transient term by 19.5, from 24.0 to 1.23 GiB per GPU at 35 slots. The windowed ring and reusable staging together add 0.060 GiB of required GPU state per session in the evaluated layout. Copy-back remains part of execution cost. Table 1 gives the per-session memory: at 100K Adaptive HiSparse requires 0.432 GiB of GPU memory against 4.444 GiB for resident serving, 10.3× less; with DFlash2 the two figures are 0.492 and 4.469 GiB.

4.1. Experimental Setup Hardware and workloads. Experiments use 8×B300 nodes (269 GiB of HBM per GPU, 4.3 TB of host memory over PCIe). GLM-5.3 uses four GPUs for prefill and four for decode at 100K context, while DeepSeek-V4.1Flash uses two per role at 500K. We evaluate end-to-end serving with the closed-loop concurrency benchmark used by HiSparse (Xie et al., 2026), sweeping offered concurrency 𝐶 at a fixed input length and output cap. GLM-5.3 runs use 100K input and a 2,048-token output cap; Figure 1 uses HiSparse’s 32K input and 8K output. DeepSeek-V4.1-Flash runs use 500K input and a 2,048-token output cap. The 100K and 500K workloads reuse prepared prefixes, one for GLM-5.3 and 16 for DeepSeek-V4.1-Flash, with independent decode KV per request. We report output throughput per decode GPU, time to first token (TTFT) and time per output token (TPOT).

Capacity accounting. Speculation also adds fixed memory on each decode GPU, outside the per-session figures of Table 1: the DFlash2 drafter takes 4.9 GiB, the target model 1.3 GiB more, and verification metadata 0.4 GiB. These allocations reduce the memory left for the KV and index pools. The index pool then holds 3.72 million rows per GPU against 5.37 million, enough for 36 complete histories of 100K input and 2,048 output tokens against 52. We configure 35, which gives 140 sessions on four GPUs against 208.

Configurations. We compare resident serving, HiSparse, and our adaptive configurations in SGLang (Zheng et al., 2024; Xie et al., 2026), with DSpark or DFlash2 where supported. For both models, HiSparse and the adaptive configurations use the same modelcomputation kernels. All adaptive configurations add the KV management of Section 3; Section 4.2.1 attributes the gain over HiSparse. GLM offloading 7

Resident

HiSparse

Adaptive HiSparse

Resident + DFlash2

Mean time per output token (ms)

c

300

800 600 400 200

100 30 10 3 1 0.3

0 0

Adaptive DHiSparse

b Mean time to first token (s)

Output throughput (tok/s/GPU)

a

32 64

128

256

Offered concurrency (requests)

0

200

400

600

800

Output throughput (tok/s/GPU)

60 50 40 30 20 10 0 0

200

400

600

800

Output throughput (tok/s/GPU)

Figure 4 | GLM-5.3 serving at 100K context. (a) Output throughput against offered concurrency, (b) mean TTFT and (c) mean TPOT against output throughput. Color gives the KV policy and filled markers add DFlash2. TTFT in (b) is on a logarithmic axis. Adaptive DHiSparse uses static blocks of 8 at C4 to C32, 6 at C64 and 4 at C128 and C256.

configurations use FP8 KV, 𝑘 = 2,048, and 𝐵 = 4,096, with cache settings matched across methods (SGLang contributors, 2026); Section 4.3.4 sweeps 𝐵 in this configuration: at C128 output peaks at 𝐵 = 2𝑘 for both systems. DeepSeek uses our optimized HiSparse port as its reference, with 𝑘 = 512 and hot caches of 8,192 rows for the KV source that five layers select from and 4,096 rows for each of the three sources that one layer selects from. Figure 4 specifies the DFlash2 block schedule; Figure 7 shows the block-size comparison.

1.13 to 1.16× non-speculative adaptive serving instead of 1.27×, and 1.29 to 1.31× HiSparse instead of 1.45×. Mean TTFT ends 2.1× lower than HiSparse and mean TPOT 29% lower. Why speculation stops paying at C256. With 𝑏 active sessions, output is proportional to 𝑏/TPOT, so the output 𝑄 of speculative serving (𝑠) against non-speculative adaptive serving (0) is 𝑄𝑠 𝑏𝑠 TPOT0 ≈ · . 𝑄0 𝑏0 TPOT𝑠

Measurement. Output throughput counts returned tokens per second per decode GPU, excluding prefill GPUs; Figure 1 reports aggregate output across four decode GPUs. Post-wave latency excludes the first 𝐶 submitted requests.

(13)

Speculation pays while it keeps a larger share of the sessions than its TPOT ratio. At C128 it keeps 99% of the sessions at 0.77 of the TPOT, and the relation gives 1.29× against 1.27× measured. At C256 the speculative configuration admits 140 sessions against 208. It keeps 69% of the sessions at 0.71 of the TPOT, which gives 0.96× against 1.00× measured (826 and 823 tokens/s per decode GPU). Speculation then serves fewer sessions, each faster, for the same output, and requests wait longer for a slot: mean TTFT is 1.9× higher. Both adaptive configurations remain about 13% above HiSparse.

4.2. Experimental Results 4.2.1. GLM-5.3 at 100K context Figure 4 shows the serving curves and Table 2 gives the C128 and C256 results. Where the 1.45× comes from. At C128, Adaptive DHiSparse delivers 793 tokens/s per decode GPU, 1.45× the output of HiSparse. The gain is the product of two steps. The first, from HiSparse to non-speculative adaptive serving, is 1.14×. The copy path launches over a grid that scales with the batch, 4 to 32 thread blocks against a fixed four, exposing more parallelism for row movement. Mean time per token is 7% lower. Residentfirst admission places incoming history directly in GPU storage and defers its host backup until memory pressure, which keeps that backup off the admission path: the prefill/KV handoff falls from 10.4 to 4.0 seconds, for 1.8 seconds of added allocation wait. In a closed loop a shorter wait keeps more of the 128 requests active, 126 against 121. The second step, speculation, is 1.27×. A round verifies four queries and emits 2.88 tokens in 2.2× the time of a non-speculative step, so each session runs 1.30× faster while 99% of the sessions stay active. The gain follows the tokens a round emits: with 2.67 tokens per round instead of 2.88, speculation delivered

Why the gap to resident serving grows with load. Resident serving requires 4.444 GiB of GPU memory per session at 100K, against 0.432 GiB for Adaptive HiSparse (Table 1). The memory of 52 offloaded sessions on a GPU therefore holds five resident ones, 20 on four GPUs. With DFlash2 the memory of its 35 sessions holds three, 12 on four GPUs. This is where the resident curves stop scaling: output stays near 178 tokens/s per decode GPU from C32 to C256, and near 220 with DFlash2 from C16, while mean TTFT without DFlash2 grows from 0.6 seconds at C16 to 30 seconds at C32. At C256, Adaptive DHiSparse delivers 4.6× the output of resident serving and 3.7× that of resident serving with DFlash2. Below the limit, resident serving has no rows to resolve and stays ahead: at C4 to C16, Adaptive HiSparse is 5 to 7% below it and 19 to 23% above HiSparse. A kernel profile at C8 records a shorter step with Adaptive HiSparse (16.6 versus 20.9 ms) and 8

Table 2 | GLM-5.3 at 100K context, offered concurrency 128 and 256. Output is in tokens/s per GPU. Post-wave latency excludes the first 𝐶 submitted requests. Active is the mean number of requests decoding at once over the central half of a run. Against 𝐶

Method

128

256

TTFT (s)

TPOT

Active

Output

HiSparse

mean

post-wave

post-wave p99

(ms)

/ slots

HiSparse Adaptive HiSparse Adaptive DHiSparse

548 622 793

– 1.14× 1.45×

10.68 6.13 5.13

5.11 2.48 2.25

26.97 11.85 9.67

50.3 46.6 35.9

121.1 / 208 125.6 / 208 124.7 / 140

HiSparse Adaptive HiSparse Adaptive DHiSparse

730 823 826

– 1.13× 1.13×

47.61 34.36 64.62

42.54 31.01 67.79

102.89 81.99 87.60

58.0 54.4 38.9

190.5 / 208 201.5 / 208 138.2 / 140

1.4 ms less cache-kernel work. In the same-build experiment of Section 4.3.1, optional residency alone makes each session 3.8% faster. At high load, where optional residency disappears, resident-first admission and the copy path remain active.

resident at 200K and 2.3× at 500K. At the shortest context every history fits in HBM and resident serving has nothing to resolve. As the history grows, fewer resident sessions fit and the hierarchy takes over. Speculation follows the same pattern. With DSpark, adaptive output is 1.85× resident serving with DSpark at C512 (5,054 tokens/s per decode GPU), with mean TTFT 3.2× lower. TPOT is 1.9× higher, because each request shares a step with many more active sessions. The return of speculation over non-speculative adaptive serving falls as load rises, from 1.54× at C128 to 1.20× at C512, as on GLM-5.3 above C32.

Where first-token time goes. Table A2 splits mean TTFT into allocation wait and handoff. At C128, the handoff is 97% of HiSparse’s TTFT. With nonspeculative adaptive serving it is 66%, and with DFlash2 43%. Below the session limit the handoff is thus the larger part for HiSparse and Adaptive HiSparse, and the adaptive configurations shorten it by 2.6× and 4.6×. With DFlash2, allocation wait, 49% of mean TTFT at C128, is already the larger part. At C256 the bottleneck moves to admission: allocation wait is 85 to 97% of mean TTFT for the adaptive configurations and 67% for HiSparse. Speculative TTFT stays as high after the first wave of requests, so the pressure is sustained.

4.3. Experimental Analysis 4.3.1. What residency buys and what it costs On GLM-5.3 at 100K context, without speculation, the same build runs with optional residency on and off under a C16–C256–C16 sequence, with five minutes at each offered load (Figure 6, Table A1).

Capacity under fixed latency bounds. Table A3 lists latency at every load. With post-wave p99 TTFT at most 10 s and mean TPOT at most 40 ms, HiSparse sustains C32 (212 tokens/s per decode GPU), Adaptive HiSparse C64 (393), and Adaptive DHiSparse C128 (793): 4× the concurrency and 3.7× the output of HiSparse. The TPOT bound is what holds HiSparse at C32: it reaches 42 ms at C64, while speculation holds 36 ms at C128. With the bound at 50 ms, HiSparse reaches C64 (348) and the output ratio is 2.3×. The ratios also depend on the first-token bound: Adaptive DHiSparse meets it at C128 with 9.67 s, and with a bound of 8 s it sustains C64, 2× the concurrency and 2.6× the output of HiSparse.

At low load. With residency on, 75 to 100% of the eligible history is on the GPU, and the share of selected rows fetched from host is 10.3× lower. In this first phase per-session speed is 3.8% higher and time to first token is 1.65 seconds against 3.22. When load rises. The pages have to be returned before new requests can use them. The runtime backs up and retires the resident history: coverage falls below 1% within 16 seconds, and the 208 slots fill after 53 seconds, against 3 seconds with residency off. In the C256 phase, allocation wait is 1.4× longer and per-session speed 3% lower, with the same host-fetched share in both arms. The prefill/KV handoff takes 8.5 seconds against 26.8, so time to first token in the phase is still 21% lower, and 11% more requests complete.

4.2.2. DeepSeek-V4.1-Flash at 500K context Figure 5 repeats the comparison on a second model, at five times the context and with DSpark as the drafter.

When load falls. In-flight requests drain first, then population refills the freed pages. Coverage is back above 50% after 142 seconds. The host-fetched share is again 5.8× lower and per-session speed 13.1% higher, while time to first token is the same in both arms (0.84 and 0.87 seconds). While memory is spare, optional residency thus removes 83 to 90% of host fetches, and sessions run 3.8% faster before the step and 13.1% faster after recovery. It costs 3% of per-session speed and slower slot filling for the minute that follows a load step.

The same limit, at a higher load. Resident serving stops scaling at C128: its output stays near 1,800 tokens/s per decode GPU up to C512 while mean TTFT grows from 11 to 115 seconds. Adaptive serving keeps scaling, from 1.16× resident at C128 to 2.3× at C512 (4,217 tokens/s per decode GPU), where its mean TTFT is 5.6× lower. Against HiSparse it is within 4% up to C256 and 9.4% higher at C512. Context length sets the crossover. At C512, adaptive output is 2.6% below resident at 32,736 tokens, 1.57× 9

Resident

HiSparse

Adaptive HiSparse

Resident + DSpark

3,000

1,500

c Mean time per output token (ms)

4,500

100 50 20 10 5 2 1 0.5

0 0

Adaptive DHiSparse

b Mean time to first token (s)

Output throughput (tok/s/GPU)

a

64 128

256

512

0

Offered concurrency (requests)

1,500

3,000

40

30

20

10

0

4,500

0

Output throughput (tok/s/GPU)

1,500

3,000

4,500

Output throughput (tok/s/GPU)

Figure 5 | DeepSeek-V4.1-Flash serving at 500K context. Panels and encoding as in Figure 4; filled markers add DSpark. HiSparse is our optimized port to DeepSeek-V4.1-Flash.

C256

C16

C16

Ready history coverage (%)

Allocated request slots

C16 200 150 100 50 0 0

Optional residency off

b

5

10

15

C256

C16

100 75 50 25 0 0

Time (minutes)

5

10

Time (minutes)

15

Selections fetched from host (%)

Optional residency on

a

c C16

C256

C16

15

10

5

0 0

5

10

15

Time (minutes)

Figure 6 | Residency under a load step. GLM-5.3 at 100K context, without speculation: the same build with optional residency on and off, offered C16, then C256, then C16 for five minutes each; in-flight requests drain when the load falls. (a) Allocated request slots. (b) Coverage: ready optional history over eligible history. (c) Share of selected rows fetched from host. One-second counters shown in five-second bins. Per-phase request statistics are in Table A1.

Time follows bytes. On GLM-5.3 (panel a), at verification width 8, batch 32 per GPU and 𝐵 = 4,096, resolver time is linear in the unique host-miss payload of a call. Six cases, from 0.031 ms with no miss to 6.23 ms for 302 MB, fit a rate of 49.4 GB/s ( 𝑅2 = 0.9992). Eight disjoint selections cost 7.9× eight identical ones. On GLM-5.3 the cost of resolution is the bytes it moves, and the two mechanisms that remove fetches act on it directly: optional residency, with 83 to 90% fewer host-fetched rows at low load, and shared retrieval, with 29% fewer during verification.

4.3.2. What a wider verification block returns Figure 7 compares static DFlash2 blocks of 4, 6 and 8 target queries at C64, C128 and C256. A wider block emits more tokens per round and takes longer per round. At C64 the two cancel: block 6 emits 18% more tokens than block 4 in a round that is 17% longer (Figure 11a), and the three blocks are within 3% in output. At higher load the wider blocks delivered less. At C128, blocks of 6 and 8 emit 17% and 25% more tokens per round than block 4 but deliver 6% and 9% less output, and at C256 they deliver 1% and 8% less. At these three loads the serving curve uses the block with the highest output, 6 at C64 and 4 at C128 and C256. Resolution and copying run alone for 17% of the block-4 round and 20% of the block-6 round. Shared retrieval keeps that share down: at C128 with four queries, verification counters show 1.41 missing selections per distinct host-fetched row, so fetching each row once removes 29% of the fetches the queries would otherwise issue.

The scan grows with the hot cache. On DeepSeekV4.1-Flash (panel b), a hot cache four times larger makes a call 1.9× slower at batch 8 and 2.2× slower at batch 160 with no miss at all, and at 𝐵 = 4,096 the direct path is 2.5× faster than the scan. Optional history adds rows without lengthening the scan. The scan is a small term in serving: on DeepSeek-V4.1-Flash at C128 the whole resolver takes 0.5 to 0.7 ms of a 25 ms step and does not grow with the budget, and on GLM-5.3 at C8 resolver work grows by 0.5 ms per step from 𝐵 = 𝑘 to 8𝑘 (Section 4.3.4). Misses add in proportion to the batch (panel c): 77 misses per request add a median of 1.3 𝜇 s per request at both batch sizes, which makes a

4.3.3. What a miss costs Figure 8 times the resolver apart from the model.

10

C64

C128

a Output throughput (tok/s/GPU)

3.8

Emitted tokens per round

C256

b

3.5

3.2

2.9

2.6 4

6

8

800

700

600

500 4

Block size (target queries per round)

6

8

Block size (target queries per round)

Figure 7 | Block size for DFlash2 at 100K context. (a) Emitted tokens per speculative round and (b) output throughput for static blocks of 4, 6 and 8 target queries at three offered loads.

cold cache retained cache

6

disjoint

4

49.4 GB/s

2

partial shared

0 0

100

batch 8 batch 160

60

40

20 direct path, batch 8

0 200

300

Unique host-miss payload (MB per call)

c DeepSeek-V4.1-Flash, B = 4,096 Resolver time (µs per call)

b DeepSeek-V4.1-Flash, no host miss Resolver time (µs per call)

Resolver time (ms per call)

a GLM-5.3, width 8, batch 32

300

4,096

8,192

no host miss 77 host misses

200

100 18.7

0 2,048

243.3

400

29.3

batch 8

36.9

batch 160

Hot-cache size B (rows)

Figure 8 | What resolution costs. (a) GLM-5.3, verification width 8, batch 32 per GPU, 𝐵 = 4,096: resolver time against the unique host-miss payload of a call, for synthetic shared, partial and disjoint selections across the eight queries, with a cold or a retained cache. The dashed line is the least-squares fit, 49.4 GB/s ( 𝑅2 = 0.9992). One query’s top-𝑘 over the 32 requests of a call is 37.75 MB, so disjoint selections fetch eight such payloads and shared selections one. (b) DeepSeek-V4.1-Flash at 500K, single-query resolver with no host miss: time against hot-cache size for two batch sizes. The dotted line is the direct path at batch 8. (c) The same resolver at 𝐵 = 4,096 with and without 77 host misses per request. Error bars span the minimum and maximum of nine timings in (a) and the 10th to 90th percentiles in (b) and (c). These are isolated kernel measurements with synthetic selections and real payload copies.

call 1.6× slower at batch 8 and 6.6× slower at batch 160, because the scan grows far less than the batch.

keep the size configured for 𝐵 = 2𝑘 and every other setting is as in Section 4.1, so a larger hot cache leaves room for fewer sessions. At C128, HiSparse’s host-miss rate falls from 33% at 𝐵 = 𝑘 to 3.9% at 𝐵 = 8𝑘, and its TPOT from 57 to 38 ms. Output nevertheless peaks at 𝐵 = 2𝑘 and then collapses: from 2𝑘 to 8𝑘 it loses 44%, and mean TTFT grows from 11 to 115 seconds. Adaptive HiSparse follows the same curve, 14% higher at the peak, and its profile shows the trade: from 2𝑘 to 8𝑘 a decode step shortens from 44.7 to 35.3 ms but holds 13 sessions per GPU instead of 28. A lower miss rate does not buy throughput once the cache takes the memory that sessions need. In these pools, 𝐵 = 2𝑘, the setting of our GLM-5.3 comparisons, gives the highest output of the four budgets for both systems. At C8 the picture changes. With optional residency the host-miss rate is 0.05% at every budget and output stays within 3%, so the smallest cache is enough. It is also the fastest: with the miss rate unchanged, resolver work per step grows from 0.35 to 0.85 ms between 𝐵 = 𝑘 and 8𝑘, and

Share of a step. On DeepSeek-V4.1-Flash, resolver kernel work is 0.4 to 1.4% of the step interval at C16 and 4.75% at C512 (Figure 11b). In a GLM-5.3 round with DFlash2 at C64, resolution and copying are the only kernels running for 17 to 20% of the time (Figure 11a). Figure 10 compares the two models at one load, without speculation. In the census of Section 3.2, about 70% of selected rows resolved directly and 30% through the hot cache. Rows generated during decoding go through the hot cache, so no lookup was wholly direct. 4.3.4. How large the hot cache should be We sweep the hot-cache budget from 𝐵 = 𝑘 to 8𝑘 on GLM-5.3 and from 2𝑘 to 16𝑘 on DeepSeek-V4.1-Flash, at C8 and C128, without speculation (Figure 9). On GLM-5.3 the budget trades step time against sessions. Here the sweep changes only 𝐵: the KV pools 11

120

Fetched from host (% of rows)

Output throughput (tok/s/GPU)

HiSparse

a GLM-5.3, C8

Adaptive HiSparse

b GLM-5.3, C128

400

c DeepSeek, C8

2,400

d DeepSeek, C128

600 300

80

400

40

200

0

40

2

4

0

8

e

1 40

800

100

0 1

1,600

200

2

4

0

8

2

f

40

4

8

16

g

40

30

30

30

30

20

20

20

20

10

10

10

10

0

0 1

2

4

8

Hot-cache budget B/k

0 1

2

4

8

4

8

16

2

4

8

16

h

0 2

Hot-cache budget B/k

2

4

8

16

Hot-cache budget B/k

Hot-cache budget B/k

Figure 9 | Output and host fetches against the hot-cache budget. (a–d) Output per decode GPU and (e–h) share of selected rows fetched from host, for HiSparse and Adaptive HiSparse as 𝐵/𝑘 varies, without speculation. GLM-5.3 runs at 100K context on one prepared prefix, and DeepSeek-V4.1-Flash at 500K on 16 prepared prefixes. Table 3 | Retrieval volume of the two models. Per generated token and request. Ranges run from the smallest to the largest budget of the sweep, for HiSparse at C128. The hot cache is that of Adaptive HiSparse at the budgets of Section 4.1. GLM-5.3

where it carries the rows of 3.7 payload layers on average. At the smallest budget, where both models miss about a third of their selections, HiSparse fetches 34 MB per generated token on GLM-5.3 and 0.42 MB on DeepSeek-V4.1-Flash, about 80× less. The DeepSeekV4.1-Flash hot cache is also about 6 MB per session at the budgets of Section 4.1, against 184 MB, so enlarging it costs no sessions.

DeepSeek

Selecting layers × 𝑘 21 × 2,048 8 × 512 Selections per token 43,008 4,096 KV sources 78, one per layer 4, shared Bytes per missed selection 2.4 to 2.5 KB 288 B Host bytes per token 34 to 4.2 MB 0.42 to 0.11 MB Hot cache per session 184 MB 6 MB Resolver calls per step 21 8 Follower copies per step 57 none

What the cache costs inside a step. Figure 10 profiles the kernels of Adaptive HiSparse at C128. On GLM-5.3 a step makes 21 resolver calls and 57 follower-layer copies. At 𝐵 = 2𝑘 they take 15.1 ms of kernel work, three quarters of it in the follower copies, and both shrink as the budget grows (panel a); per session, the work falls from 0.55 ms at 2𝑘 to 0.29 ms at 4𝑘 and 0.27 ms at 8𝑘. Most of this work runs under the model’s own kernels: 11.0 of the 15.1 ms overlap them, and the cache runs alone for 9% of the step (panel b). On DeepSeek-V4.1-Flash a step makes eight resolver calls and no follower copy. The resolver takes 0.5 to 0.7 ms of a 25 ms step at every budget, and the residency machinery itself (host backup and page bookkeeping) adds 0.05 ms. Expert matrix products run alone for 38% of the GLM-5.3 step and 53% of the DeepSeekV4.1-Flash step.

TPOT from 18.9 to 19.4 ms. Best against best, Adaptive HiSparse stays 21% ahead of HiSparse, the gap analyzed in Section 4.2.1. On DeepSeek-V4.1-Flash the budget does not matter. At C128, output stays within 2% across an eightfold range of 𝐵 for both systems, while the host-miss rate falls from 36% to 9.5% for HiSparse and from 18% to 5.9% for Adaptive HiSparse. Mean TPOT and mean TTFT are flat as well, the latter at 2.5 to 2.8 seconds. At C8, Adaptive HiSparse fetches nothing from host at any budget, and TPOT shows no trend with the budget for either system; the lower HiSparse output at 𝐵 = 2𝑘 comes from first-token time.

5. Discussion

Why the two models differ. Table 3 gives the reason. For each generated token, GLM-5.3 makes 43,008 selections through 21 index producers, and each selection feeds payload layers that keep their own bytes. DeepSeek-V4.1-Flash makes 4,096 selections through eight selecting layers that read four KV sources shared across its layers. A missed selection moves 288 bytes on DeepSeek-V4.1-Flash and about 2.4 KB on GLM-5.3,

When resident serving is enough. If every active history fits in HBM, resident serving has nothing to resolve: Adaptive HiSparse is 5 to 7% below it at C4 to C16 on GLM-5.3, and 2.6% below it on DeepSeek-V4.1Flash at 32K. The hierarchy pays once resident histories stop fitting, at about 20 sessions for 100K on our GPUs

12

Cache kernel work (ms per step)

30

a Cache work against the budget, C128

b Where a C128 step goes

follower-layer copy resolver

25.4

20

GLM-5.3 44.7 ms

38

7 6 8

30

9

15.1

10

7.1

DeepSeek 25.1 ms

3.4

0

1

2 4 GLM-5.3

8

0.7

0.5

2

4 8 16 DeepSeek

0.5

53

10 8

17

8

expert matrix products dense projections indexing and attention other model kernels several families at once cache kernels alone no kernel running

0.5

0

25

50

75

100

Share of step time (%)

Hot-cache budget B/k

Figure 10 | What the cache costs inside a step, Adaptive HiSparse at C128. (a) Summed duration of the cache kernels per GPU step as 𝐵/𝑘 varies, on one scale for both models: the fused resolver and, on GLM-5.3, the copies that load selected rows into the follower payload layers. A GLM-5.3 step holds 24 to 28 sessions per GPU up to 𝐵/𝑘 = 4 and 13 at 𝐵/𝑘 = 8; a DeepSeek-V4.1-Flash step holds 62 to 63. (b) Share of step time by kernel family at the budget used in the paper ( 𝐵/𝑘 = 2 for GLM-5.3, 16 for DeepSeek-V4.1-Flash), each family counted where it runs alone. “Several families at once” is time in which kernels of more than one family run; “no kernel running” is time between kernels, 1.5% and 1.9% of the step.

and 12 with DFlash2 (Table 1), and its advantage then grows with both load and context length.

makes draft KV independent of context: the ring adds 0.025 GiB per session at 32K and at 100K alike.

How to operate speculation. Equation (13) says when speculation pays: while the share of sessions it keeps exceeds its TPOT ratio, 0.77 at C128 and 0.71 at C256 in our runs. Once both configurations fill their slots, the share kept is the slot ratio, 140/208 = 0.67, below the 0.71 measured at C256, so speculation adds no output there and lengthens the wait for a slot. Our schedule narrows the block as load approaches that limit (8 up to C32, 6 at C64, 4 from C128); selecting the block online from load and memory is future work.

Scope. Our evaluation covers one GPU platform, two sparse-attention models, and closed-loop workloads at fixed input lengths over a fixed set of prepared prefixes, with cache settings matched across methods. Speculative gains depend on acceptance, which varied between runs of the same workload.

6. Related Work Hierarchical KV caching. HiSparse establishes LRUmanaged hierarchical storage, fused resolution with GPU-assisted host reads, and exact layer-wise prefetch from shared indices (Xie et al., 2026), keeping complete history available through a fixed device cache. HiSparse has no draft-model speculative path; we extend it with load-adaptive residency, shared verification retrieval, bounded draft KV, and reusable staging. Strata provides complementary hierarchical contextloading and GPU-assisted I/O mechanisms (Xie et al., 2025). InfiniGen and ShadowKV use prediction or reconstruction for offloaded execution, and DualDecoder predicts future retrieval (Lee et al., 2024; Sun et al., 2025; Liang et al., 2026).

What limits capacity next. With KV offloaded, GPU memory is still the binding resource. A session requires 0.432 GiB at 100K, or 0.492 GiB with DFlash2, and speculation adds 6.6 GiB of fixed allocations per GPU for the drafter, the target model and metadata; the configurations admit 208 and 140 sessions. Host memory is far from binding: 208 sessions back up 0.93 TB of a 4.3 TB host, which would hold about 950. At C256, allocation wait is 85 to 97% of time to first token, and after a load step slots take 53 seconds to fill. The next gains in capacity will come from a lower required state per session and faster reclaim. Implications for model design. The cost of offloading follows the bytes a token’s selections can pull from host. Against GLM-5.3, DeepSeek-V4.1-Flash has a 𝑘 four times smaller, 2.6× fewer selecting layers and rows about half as wide, and a selection reads one source row where GLM-5.3 reads 3.7 layer rows. Together these give about 80× fewer host bytes per token at a comparable miss rate. A DeepSeek-V4.1-Flash step needs eight resolver calls and no follower copy, and its output is insensitive to the cache budget. GLM5.3 keeps 78 payload layers with their own bytes: a step needs 57 follower copies, and output falls 44% when the budget grows from 2𝑘 to 8𝑘 in pools sized for 2𝑘 (Section 4.3.4). A drafter with a bounded window

Model-side reuse. Cross-Layer Attention and YOCO share KV state, while IndexCache reuses sparse selections (Brandon et al., 2024; Sun et al., 2024b; Bai et al., 2026). DeepSeek-V4.1-Flash combines global-KV and index reuse (DeepSeek-AI, 2026); GLM-5.3 illustrates shared indices with distinct payloads (Z.ai, 2026). We organize storage around those model-defined relationships, sharing recovered bytes where the model shares a source and preserving distinct bytes where it shares only positions.

13

Drafting and verification. Speculative decoding has moved from separate draft models (Leviathan et al., 2023) to drafters that reuse the target’s heads or features, as in Medusa (Cai et al., 2024) and the EAGLE series (Li et al., 2024a,b, 2025). DSpark develops semiautoregressive generation on a parallel DFlash backbone and schedules verification prefixes using confidence and engine cost (Chen et al., 2026; Cheng et al., 2026). DFlash2 retains parallel drafting with a conditional candidate-lattice selector (Inco AI, 2026); we support both through their existing proposals and acceptance procedures by supplying the KV retrieval and storage required for hierarchical verification. Two systems study speculative decoding at long context. MagicDec (Sadhukhan et al., 2025) shows that speculation can raise throughput at large batch sizes once sequences are long, using drafters with a sparse KV cache. TriForce (Sun et al., 2024a) combines retrieval-based drafting with hierarchical speculation to relieve the KV bottleneck of long sequences, including an offloading setting. Both give the drafter a sparse or partial KV cache; we keep the target’s sparse selections exact and manage where its full history lives.

ment and draft-model speculation compose. Each removes a different cost of long-context serving (what a query computes, where its history lives, how many target passes a token takes), and combining them leaves selections, proposals and acceptance unchanged. How cheaply they combine is decided in the model and the drafter. A model that selects few, narrow rows from KV sources shared across its layers leaves the hierarchy little to move (about 80× fewer host bytes per token on DeepSeek-V4.1-Flash than on GLM-5.3 at a comparable miss rate), and a drafter with a bounded window leaves speculation little to store at any context length.

References Yushi Bai, Qian Dong, Ting Jiang, Xin Lv, Zhengxiao Du, Aohan Zeng, Jie Tang, and Juanzi Li. IndexCache: Accelerating sparse attention via cross-layer index reuse. arXiv:2603.12201, 2026. URL https://arxiv.org/abs/2603.12201. William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, and Jonathan Ragan-Kelley. Reducing transformer key-value cache size with cross-layer attention. In Advances in Neural Information Processing Systems, volume 37, pages 86927–86957, 2024. URL https://arxiv.org/abs/2405.12981. Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 5209–5235. PMLR, 2024. URL https: //proceedings.mlr.press/v235/cai24b.html. Jian Chen, Yesheng Liang, and Zhijian Liu. DFlash: Block diffusion for flash speculative decoding. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pages 15343–15355. PMLR, 2026. URL https: //proceedings.mlr.press/v306/chen26bv.html. Xin Cheng, Xingkai Yu, Chenze Shao, Jiashi Li, Yunfan Xiong, Yi Qian, Jiaqi Zhu, Shirong Ma, Xiaokang Zhang, Jiasheng Ye, et al. DSpark: Confidence-scheduled speculative decoding with semi-autoregressive generation. arXiv:2607.05147, 2026. URL https://arxiv.org/abs/2607.05147. DeepSeek-AI. DeepSeek-V3.2: Pushing the frontier of open large language models. arXiv:2512.02556, 2025. URL https://arxiv.org/abs/2512.02556. DeepSeek-AI. DeepSeek-V4.1-Flash: Pushing the limits of KV cache compression. Technical report, DeepSeek-AI, 2026. URL https: //huggingface.co/deepseek-ai/DeepSeek-V4.1-F lash/blob/main/DeepSeek_V41_Tech_Report.pdf. Inco AI. DFlash 2: Keep drafting parallel. Technical article, 18 August 2026. URL https://inco.ai/blog/dflash2/. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th ACM Symposium on Operating Systems Principles, pages 611–626, 2023. doi: 10.1145/3600006.3613165. URL https://arxiv.org/abs/2309.06180. Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. InfiniGen: Efficient generative inference of large language models with dynamic KV cache management. In 18th USENIX Symposium on Operating Systems Design and Implementation, pages 155–172. USENIX Association, 2024. URL https://www.usenix.org/conference/osdi24 /presentation/lee. Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In

Serving infrastructure. PagedAttention, SGLang, and DistServe address paged allocation, efficient execution, and prefill–decode separation (Kwon et al., 2023; Zheng et al., 2024; Zhong et al., 2024). Active-request residency complements prefix caching: a prefix hit avoids repeated prefill computation, while the decode hierarchy manages the physical history used by ongoing requests. Our bounded draft and staging allocations operate alongside these serving mechanisms.

7. Conclusion Hierarchical KV caching lets long histories live in host memory, but with a fixed cache budget requests keep fetching from host while HBM sits idle below peak load, and with single-query retrieval draft-model speculation stays outside the hierarchy. Adaptive DHiSparse closes both gaps by bringing draft-model speculative decoding to hierarchical sparse KV serving and adapting history residency to load. A protected working set supports exact recovery, spare HBM retains additional history, and a source-aware resolver shares missing-row retrieval across verification queries. Window-bounded draft KV and reusable staging control the extra memory required by DSpark and DFlash2 while preserving their proposals and accepted-state updates. At 100K context and C128, GLM-5.3 with DFlash2 delivers 1.45× the output of HiSparse and, held to the same latency bounds, sustains up to 4× the concurrency. At 500K context and C512, DeepSeekV4.1-Flash with DSpark delivers 1.85× the output of resident serving with DSpark. With Adaptive DHiSparse, GPU residency becomes a runtime decision where it was a deployment constant. Optional history removes most host fetches while HBM is spare and returns its pages when sessions need them. More broadly, sparse attention, hierarchical KV place14

Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 19274–19286. PMLR, 2023. URL https://pr oceedings.mlr.press/v202/leviathan23a.html. Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE: Speculative sampling requires rethinking feature uncertainty. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 28935–28948. PMLR, 2024a. URL https: //proceedings.mlr.press/v235/li24bt.html. Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-2: Faster inference of language models with dynamic draft trees. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7421–7432. Association for Computational Linguistics, 2024b. URL https: //aclanthology.org/2024.emnlp-main.422/. Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-3: Scaling up inference acceleration of large language models via training-time test. In Advances in Neural Information Processing Systems, volume 38, 2025. URL https://proceedings.neurips.cc/paper_files/p aper/2025/hash/c7b5a35ea98b62512a869c19ea7b0 3cb-Abstract-Conference.html. Zuning Liang, Zhiyi Yao, Qi Chen, Yuedong Xu, Hao Dai, Zhiqiang Ding, Tongkai Yang, Jinlong Hou, and Yuan Cheng. DualDecoder: Accelerate long context LLM inference by predictive prefetch. arXiv:2607.26475, 2026. URL https://arxiv.org/abs/2607.26475. Ranajoy Sadhukhan, Jian Chen, Zhuoming Chen, Vashisth Tiwari, Ruihang Lai, Jinyuan Shi, Ian En-Hsu Yen, Avner May, Tianqi Chen, and Beidi Chen. MagicDec: Breaking the latency-throughput tradeoff for long context generation with speculative decoding. In International Conference on Learning Representations, 2025. URL https://arxiv.org/abs/2408.11049. SGLang contributors. HiSparse: Hierarchical sparse attention. SGLang documentation, 2026. URL https: //github.com/sgl-project/sglang/blob/main/do cs/docs/advanced_features/hisparse_guide.mdx. Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, and Beidi Chen. TriForce: Lossless acceleration of long sequence generation with hierarchical speculative decoding. In First Conference on Language Modeling (COLM), 2024a. URL https://arxiv.org/abs/2404.11912. Hanshi Sun, Li-Wen Chang, Wenlei Bao, Size Zheng, Ningxin Zheng, Xin Liu, Harry Dong, Yuejie Chi, and Beidi Chen. ShadowKV: KV cache in shadows for high-throughput long-context LLM inference. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 57355–57373. PMLR, 2025. URL https: //proceedings.mlr.press/v267/sun25b.html. Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei. You only cache once: Decoder-decoder architectures for language models. In Advances in Neural Information Processing Systems, volume 37, pages 7339–7361, 2024b. URL https://arxiv.org/abs/2405.05254. Zhiqiang Xie, Ziyi Xu, Mark Zhao, Yuwei An, Vikram Sharma Mailthody, Scott Mahlke, Michael Garland, and Christos Kozyrakis. Strata: Hierarchical context caching for long context language model serving. arXiv:2508.18572, 2025. URL https://arxiv.org/abs/2508.18572. Zhiqiang Xie, Zhangheng Huang, Tingwei Huang, Ziyi Xu, Ruiyang Ma, and Christos Kozyrakis. HiSparse: Scaling sparse-attention decoding with hierarchical KV cache management. arXiv:2608.07009, 2026. URL https://arxiv.org/abs/2608.07009. Z.ai. GLM-5.3. Model card, 2026. URL https://huggingface.co/zai-org/GLM-5.3.

Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark W. Barrett, and Ying Sheng. SGLang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems, 2024. URL https://arxiv.org/abs/2312.07104. Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation, 2024. URL https://arxiv.org/abs/2401.09670.

A. Additional Measurements This appendix gives the measurements behind the main results. Table A1 lists load-step statistics; Table A2 partitions first-token latency into adjacent request intervals; and Table A3 gives latency at every measured load. Figure 11 supplies the round and step breakdowns discussed in Sections 4.3.2 and 4.3.3.

15

a GLM-5.3 with DFlash2, C64

13.9

block 4

21.3

18.7

block 6

0

other model kernels several families at once

16.2

23.1

15.6

13.1

21.0

20

40

no kernel coverage

80.1

18.3

60

Resolver kernel work (% of step)

cache kernels alone expert matrix products

b DeepSeek-V4.1-Flash, no speculation

12.9

80

94.0

100

6 4.75%

4

2

1.36%

1.33%

C16 middle

C16 drain

0.41%

0

Profiled round time (ms)

C16 fill

C512 pressure

Figure 11 | Where step time goes. (a) GLM-5.3 with DFlash2 at C64: mean profiled round time for static blocks of 4 and 6, as an exclusive partition with a separate category for overlapping kernels. (b) DeepSeek-V4.1-Flash without speculation: resolver kernel work as a share of the step interval in serving profiles, at C16 (fill, middle, drain) and C512 under pressure. Table A1 | Load step, optional residency on / off. GLM-5.3 at 100K context, without speculation. Each cell gives the value with residency on, then off. The host-fetched share is counted over each phase; the other columns of the first two rows cover requests that start and finish inside the phase. The last row covers the time and the requests after the in-flight count returned to 16. Phase

Requests

From host (%)

TPOT (ms)

Tok/s per session

Gain

TTFT (s)

Alloc. wait (s)

C16 before C256 C16 after recovery

80 / 80 417 / 375 62 / 52

1.38 / 14.21 13.79 / 13.79 2.43 / 13.98

25.18 / 26.07 53.77 / 52.24 24.44 / 28.10

39.83 / 38.38 18.61 / 19.18 40.92 / 36.19

+3.8% −3.0% +13.1%

1.65 / 3.22 36.89 / 46.63 0.84 / 0.87

0.03 / 0.00 28.19 / 19.74 0.02 / 0.01

Table A2 | Where first-token latency is spent, in seconds. Allocation wait precedes slot allocation; handoff covers prefill readiness and KV transport. The right half excludes the first 𝐶 submitted requests. All requests 𝐶

Method

128

256

After the first wave

Allocation

Handoff

Other

TTFT

Allocation

Handoff

TTFT

HiSparse Adaptive HiSparse Adaptive DHiSparse

0.00 1.75 2.51

10.36 4.03 2.23

0.31 0.34 0.39

10.68 6.13 5.13

0.00 0.09 0.25

4.98 2.24 1.77

5.11 2.48 2.25

HiSparse Adaptive HiSparse Adaptive DHiSparse

32.10 29.15 62.50

14.97 4.70 1.50

0.53 0.51 0.61

47.61 34.36 64.62

33.70 27.09 66.46

8.71 3.77 1.09

42.54 31.01 67.79

Table A3 | First-token and per-token latency at every load, GLM-5.3 at 100K. The middle columns exclude the first 𝐶 submitted requests. TTFT (s)

TTFT after the first wave (s)

TPOT (ms)

Method

𝐶

mean

p99

mean

p99

mean

p99

HiSparse

4 8 16 32 64 128 256

0.79 1.05 1.85 3.82 5.09 10.68 47.61

1.50 2.81 5.49 10.81 21.35 43.53 158.86

0.70 0.75 1.20 2.49 2.68 5.11 42.54

0.95 1.82 3.49 7.40 9.63 26.97 102.89

19.4 23.4 28.4 34.8 42.2 50.3 58.0

19.5 23.6 28.7 36.4 43.4 51.8 59.1

Adaptive HiSparse

4 8 16 32 64 128 256

0.54 0.76 1.24 1.59 3.47 6.13 34.36

0.79 1.07 3.14 5.70 16.74 36.16 131.61

0.48 0.69 0.90 1.25 2.05 2.48 31.01

0.71 0.97 1.23 3.09 6.66 11.85 81.99

16.0 19.1 24.1 30.8 38.1 46.6 54.4

16.1 19.2 24.7 31.7 39.9 48.3 55.4

Adaptive DHiSparse

4 8 16 32 64 128 256

0.53 0.65 0.86 1.47 2.76 5.13 64.62

0.79 1.35 2.55 6.88 15.45 32.42 111.89

0.47 0.49 0.64 0.74 1.12 2.25 67.79

0.70 0.72 1.03 1.40 3.02 9.67 87.60

11.4 13.3 16.2 19.4 26.2 35.9 38.9

13.2 16.4 18.8 24.4 31.8 42.1 48.3

16

Related documents

• Adaptive DHiSparse: Speculative Decoding over a Load-Adaptive KV Hierarchy#1298204
• Resource-Efficient Speculative Decoding for Long-Context LLM Serving#1117896
• When Parallel Drafter Meets Parallel Speculative Decoding#1063607
• AdaFlash: Adaptive Speculative Decoding via On-Policy Distilled Diffusion Drafters#1257979
• Online Draft Co-Training for Speculative Decoding in Large-Scale, Long-Context RL Post-Training#975168