LeanSparse: Memory-Efficient Serving for Native Sparse-Attention LLMs

L EAN S PARSE : M EMORY-E FFICIENT S ERVING FOR N ATIVE S PARSE -ATTENTION LLM S

arXiv:2611.04671v1 [cs.DC] 3 Nov 2026

Zaifeng Pan * 1 Chris Wu * 1 Xinwei Qiang 1 Zhengding Hu 1 Zhongkai Yu 1 Yufei Ding 1

A BSTRACT Sparse attention limits the history read by each query, but serving long contexts still requires substantial GPU memory. Existing KV offloading systems retain the full history of indexer keys on GPU, limiting decode concurrency. Prefill’s temporary memory demand can further delay admission even when enough memory remains for the same request’s subsequent decode. We present LeanSparse, a serving system that reduces both decode’s persistent memory use and prefill’s temporary memory demand. Our design exploits two observations: prefill consumes historical data one layer at a time, and decode can identify promising candidates before comparing their original indexer scores. During prefill, LeanSparse streams layer histories through two reusable GPU buffers, overlapping next-layer loading with ongoing model computation. During decode, coarse-to-fine indexing searches compact representations and retrieves original indexer keys only for candidate refinement. Separate indexer key and KV caches reuse data across generation steps. We implement LeanSparse in SGLang and evaluate it on DeepSeek-V4-Flash. At 950K context length, LeanSparse supports 2.95× as many concurrent requests and achieves 2.14× decode throughput over HiSparse. On multi-turn Open-SWE traces, it reduces median time to first token from 21.6 to 0.6 s and median session completion time by 17.7% relative to ECHO. 1

I NTRODUCTION

Native sparse-attention LLMs incorporate sparse selection into training to reduce the cost of processing long contexts (Yuan et al., 2025; Lu et al., 2025; DeepSeek-AI et al., 2025; 2026; GLM-5-Team et al., 2026). This benefits multiturn conversations and agent workflows, whose histories accumulate messages, documents, and tool outputs. However, sparse access does not eliminate the need to retain history: tokens skipped by one query may be selected later. The key-value (KV) cache still grows with context length, so concurrent requests can exhaust GPU high-bandwidth memory (HBM) even when each query reads only a small fraction of the stored KV (Liu et al., 2026a; Xie et al., 2026). To address this memory pressure, ECHO (Liu et al., 2026a) and HiSparse (Xie et al., 2026) offload historical KV to host memory and retain a hot KV buffer on GPU. This design exploits the overlap between tokens selected by successive decode steps. Once a token’s KV has been fetched, later steps can reuse it from the buffer instead of transferring it again. Prefetching further reduces exposed transfer latency by loading likely selections in advance. These optimizations substantially reduce the GPU memory occupied by attention KV while allowing decoding to continue over a long history. *

Equal contribution 1 University of California, San Diego, CA, USA. Correspondence to: Zaifeng Pan <zapan@ucsd.edu>. Preprint.

With a learned indexer, however, these systems still retain the full history of indexer keys on GPU. The indexer scores these indexer keys to select tokens before attention. They are separate from the KV that attention consumes. Keeping all indexer keys resident allows the indexer to search the history at every decode step, but leaves a memory cost that grows linearly with context length. Figure 1 illustrates this cost for DeepSeek-V4-Flash (DeepSeek-AI et al., 2026). At 512K context tokens, its FP8 indexer keys require 7.2× the space of a 4K-entry hot KV buffer per indexed layer. Thus, reducing the attention-KV cache can leave the indexer as the larger allocation, limiting further concurrency growth. Besides indexer residency, existing offloading optimizations mainly target decode, where repeated accesses across generation steps make a hot buffer effective. Prefill has a different access pattern. It processes many input queries together, and the tokens they select can span a much larger part of the history than those selected in a single decode step. A newly admitted or resumed request also cannot rely on an already warm GPU buffer. Consequently, prefill cannot benefit from the hot buffer optimization used for decode. HiSparse (Xie et al., 2026) thus keeps the full KV history in HBM throughout prefill. When prefill and decode share a GPU, the remaining HBM may be enough for an arriving request’s decode state but insufficient for its temporary prefill demand. As Figure 2(a) shows, the request must then wait for active decode requests

LeanSparse: Memory-Efficient Serving for Native Sparse-Attention LLMs

Memory per request (MB)

Indexer keys

16

Hot KV

Indexer keys

R5 decode state

7.2×

4

Weights

Hot KV buffer

1 32K

64K

Prefill workspace

R5 prefill

128K 256K 512K Context length (tokens)

Weights

to release memory before prefill can start, increasing time to first token (TTFT). In contrast, reserving HBM for prefill avoids this wait but reduces decode concurrency even when no prefill is running, as shown in Figure 2(b). Existing systems (Liu et al., 2026a; Xie et al., 2026) thus suggest prefill-decode (PD) disaggregation (Zhong et al., 2024), placing the two phases on separate GPU pools. However, instance-level disaggregation duplicates model weights and restricts resource sharing between phases (Wu et al., 2026; Wang et al., 2025). When HBM is already scarce enough to require KV offloading, these extra weight copies further reduce the space available for request state under a fixed GPU budget. Sharing a model instance is therefore a more suitable choice in this setting, as both phases reuse the same weights and memory pool. Efficient memory sharing thus requires reducing both decode’s persistent indexer allocation and prefill’s temporary memory demand. To address these limitations, we present LeanSparse, a memory-efficient serving system for long-context sparse attention. LeanSparse keeps the full history of indexer keys and KV in host memory. It reduces prefill’s temporary memory demand through layer-wise prefill streaming and decode’s persistent indexer residency through coarse-to-fine indexing. Layer-wise prefill streaming. A prefill chunk can extend an already computed prefix, reused from a cached request or produced by earlier chunks of the same request. The prefix’s indexer keys and KV have already been computed for each layer, but the new queries consume them one layer at a time. LeanSparse streams this prefix state from host memory through two reusable GPU buffers. One serves the current layer while the other loads the next layer’s prefix state, without waiting for that layer’s new queries. Reusing the buffers reduces prefill’s temporary memory demand and leaves more HBM for concurrent decode requests. Coarse-to-fine indexing. Consecutive decode steps often select overlapping historical positions, yet the indexer still reads every original indexer key to find them. We use a com-

R2

R3

R4

free

R5 cannot start prefill: TTFT (a) Temporary prefill memory delays admission

1M

Figure 1. Indexer key storage grows with context length. Perrequest memory of one CSA layer in HiSparse: FP8 indexer keys versus the fixed hot KV buffer.

R1

R1

R2

R3

reserved for prefill

concurrent decodes (b) Reserved prefill space reduces throughput

Figure 2. Prefill admission competes with decode for GPU memory. A prefill request either waits for decode to release memory or uses a reservation that sits idle when no prefill runs.

pact approximation to screen the history before retrieving original indexer keys. This first pass retains more candidates than attention needs: positions that are difficult to rank approximately can all survive and be compared using their original scores. Only these candidates require original indexer keys on GPU, and cached indexer keys can serve repeated candidates across steps. Our implementation uses product quantization (PQ) (Jégou et al., 2011) for this coarse pass. We implement LeanSparse in SGLang (Zheng et al., 2024) and evaluate it on DeepSeek-V4-Flash (DeepSeek-AI et al., 2026). Across context lengths from 200K to 950K tokens, LeanSparse supports up to 2.95× as many concurrent requests and achieves up to 2.14× the decode throughput of HiSparse (Xie et al., 2026). On multi-turn OpenSWE replay, LeanSparse reduces median TTFT by 97.2% and median session completion time by 17.7% relative to ECHO (Liu et al., 2026a). Our contributions are: • We identify two memory bottlenecks that limit longcontext sparse-attention serving beyond attention-KV offloading: persistent indexer residency and temporary prefill demand. • We propose LeanSparse, a serving system that reduces GPU memory requirements across prefill and decode. It reuses GPU buffers across layers during prefill and retrieves original indexer keys only for candidate refinement during decode. • We implement LeanSparse in SGLang and demonstrate higher long-context serving concurrency and throughput, together with lower TTFT and session completion latency on multi-turn workloads.

LeanSparse: Memory-Efficient Serving for Native Sparse-Attention LLMs Layers

SWA

SWA

CSA

HCA

×20

GPU HBM

CSA

every position: grows with context

Indexer keys top-k lookup

SWA window

Context

Hot KV Query

4:1

Indexer keys

qI

Compressed KV

Host DRAM

Indexer top-k

Attention

4,096 entries

fetch misses

KV history qA

Attention

Figure 4. HiSparse data placement. Attention KV is offloaded to host memory, while all indexer keys remain on GPU. Figure 3. DeepSeek-V4-Flash attention architecture and the computation within a CSA layer.

local SWA window. 2.2

2

BACKGROUND

2.1

Sparse Attention in DeepSeek

Native sparse-attention LLMs reduce the cost of longcontext attention by restricting attention computation to a small, query-dependent subset of historical positions (Yuan et al., 2025; Lu et al., 2025; DeepSeek-AI et al., 2025; 2026; GLM-5-Team et al., 2026). These models are trained with this sparse access pattern so that they can adapt to the selected context and retain modeling quality under a reduced attention budget. We use DeepSeek-V4-Flash (DeepSeek-AI et al., 2026) as a concrete example for our serving optimizations. Figure 3 shows its attention architecture, where compressed sparse attention (CSA) layers alternate with heavily compressed attention (HCA) layers. Each Transformer block also contains a mixture-of-experts (MoE) feed-forward network (FFN). CSA applies DeepSeek Sparse Attention (DSA) (DeepSeekAI et al., 2025) to compressed history, using a learned indexer to select historical positions. Both layer types also use sliding-window attention (SWA) for recent tokens. SWA retains a fixed local window, while HCA compresses history at a stride of 128 and attends to all eligible compressed positions. CSA uses a stride of four, retaining far more historical entries while accessing only a small selected subset per query. For long contexts, CSA dominates historical KV storage, and its sparse reads make it suitable for KV offloading. Therefore, both HiSparse (Xie et al., 2026) and LeanSparse focus their memory optimizations on CSA in this architecture. Each CSA layer maintains indexer keys for selecting historical positions and separate attention KV for computing the output. Figure 3 illustrates this two-stage computation. For each query, the indexer scores all eligible historical indexer keys and selects up to k = 512 positions. Core attention then reads the selected compressed KV together with the

Offloading Historical KV for Sparse Attention

Although sparse attention reduces computation, sparse selection does not directly reduce KV storage. An entry skipped by one query may be selected later, so the complete KV history must be retained. If this history remains in HBM, GPU memory use still grows with context length (Xie et al., 2026; Liu et al., 2026a). HiSparse (Xie et al., 2026) reduces this GPU memory cost during decode by storing the complete attention-KV history in host memory and maintaining a fixed-capacity hot KV buffer on GPU, as shown in Figure 4. At each decode step, the GPU indexer selects Top-k positions. HiSparse reuses selected KV already in the buffer and fetches missing entries from host memory, replacing least recently used entries as needed. Attention then reads the selected KV from the GPU buffer. Because successive decode steps often select overlapping positions, the buffer reduces repeated host transfers while bounding the GPU allocation for attention KV. This design still leaves two memory limitations. First, HiSparse keeps all historical indexer keys on GPU so that the indexer can score them before each selection. This persistent allocation grows with context length and consumes substantial GPU memory, as shown in Figure 1. It therefore limits decode concurrency even when the attentionKV allocation is fixed. A straightforward solution would be to offload these indexer keys to host memory as well. However, the GPU indexer must still score all historical indexer keys before it can select the Top-k positions. It would therefore have to fetch the complete set at every decode step, introducing host-to-GPU traffic proportional to context length for every generated token. Second, HiSparse does not reduce the KV memory required during prefill. It keeps the full attention-KV history in HBM throughout prefill because input queries are processed together and can select KV from a much broader part of the history than a single decode query. A request’s temporary

LeanSparse: Memory-Efficient Serving for Native Sparse-Attention LLMs GPU HBM

Indexer keys

Layer-wise Prefill Streaming (§4) new chunk

Full layers → 2 buffers

Compressed KV

qI

queries

Indexer + Attention

Buffer A computing

Buffer B

loading

new indexer keys

Host DRAM

CSA layer ℓ

Coarse search

load next layer

Full indexer keys → PQ codes

qI

C candidates

PQ codes all positions

new KV (staging ring)

New tokens and state

Coarse-to-Fine Decode Indexing (§5) Query

A and B swap roles each layer

PQ codes

qA

top-k

Rerank

Attention

Hot indexer key fixed size

encode once

Hot KV fixed size

fetch misses

CSA layer j

CSA layer ℓ+1

all CSA layers

Figure 5. System overview of LeanSparse. Host memory stores the full history of indexer keys and compressed KV. GPU memory holds two reusable history buffers during prefill, and compact PQ codes with separate indexer key and KV caches during decode.

R1

3

time

decode

R2 arrival

R8

prefill

R9 Earlier prefill admission

S YSTEM OVERVIEW

Figure 5 gives an overview of LeanSparse. It addresses these two problems by keeping the full history of indexer keys and compressed KV in host memory and loading data onto GPU differently for prefill and decode. During prefill, each chunk extends a prefix whose indexer keys and compressed KV were computed by a previous request or earlier chunks of the same request. The runtime loads this prefix state one layer at a time and reuses the same GPU buffers across layers. During decode, it searches compact codes on GPU and retrieves original indexer keys only to refine promising candidates. Both paths use the same host copy of the history. As shown in Figure 6, reusing two GPU buffers reduces prefill’s temporary memory demand, allowing new requests to start while others are still decoding. Compact indexer representations reduce decode’s persistent memory use per request. Together, these changes support earlier prefill admission and more concurrent decoding under the same GPU memory budget. We build on SGLang’s radix cache (Zheng et al., 2024) to match incoming requests against stored prefixes. The cache maps logical positions to host pages, allowing requests to reuse a prefix without retaining its full GPU footprint. Before each prefill chunk, the runtime constructs indexer key and KV page plans for the already computed prefixes and

prefill

Higher concurrency

prefill demand can therefore exceed its GPU memory footprint after entering decode. When prefill and decode share a GPU, this mismatch can block a new request from starting prefill even though enough HBM remains for its subsequent decode state, as illustrated in Figure 2(a).

Figure 6. Earlier prefill admission and higher decode concurrency in LeanSparse.

deduplicates shared compressed KV pages. Logical attention positions are then mapped to their loaded copies or to locally produced state. Prefill’s buffer capacities must accommodate the active batch. The current chunk produces new indexer keys and KV, which remain on GPU until both their consumers and host writeback finish. These entries extend the prefix available to later chunks. Without a cached prefix, the first chunk starts with no prefix state to load. After prefill, the runtime encodes historical indexer keys and initializes the decode caches. Incomplete pages remain locally accessible as generation extends them. Completed pages join the host-backed history and retain only compact search codes on GPU. The following sections describe layer-wise prefill streaming and coarse-to-fine decode indexing.

LeanSparse: Memory-Efficient Serving for Native Sparse-Attention LLMs Uses buffer A

Uses buffer B

Write back new indexer keys

Load complete

Buffer released

Overlap window for Load 2

Compute stream

SWA SWA

Copy stream

Load 0 → A

CSA 0 · A

HCA

Load 1 → B

CSA 1 · B

HCA

Load 2 → A

CSA 2 · A

Load 3 → B

HCA

CSA 3 · B

…

…

Figure 7. Double-buffered prefill execution. Each compute segment includes its block’s MoE FFN. The next CSA layer’s prefix load overlaps the remaining work in the current CSA block and the intervening HCA block.

4

L AYER - WISE P REFILL S TREAMING

To lower prefill’s temporary memory demand, LeanSparse reuses GPU storage across CSA layers when processing a chunk that extends an already computed prefix. Whether reused from a cached request or produced by earlier chunks of the same request, this prefix has indexer keys and compressed KV already computed for every CSA layer. The current chunk consumes these states one layer at a time, so the GPU copy of a layer’s prefix state can be discarded once its consumers finish, while the host copy remains available. After any pending writeback completes, the same GPU storage can hold a later layer’s state, reducing temporary prefill demand by avoiding simultaneous residency across all CSA layers. Because the prefix state has already been computed, it can be loaded before the next layer’s queries are available. The indexer needs all prefix indexer keys, while the KV entries selected by attention depend on those queries. LeanSparse loads the full compressed KV together with the indexer keys, allowing both transfers to start without waiting for selection, at the cost of moving KV that attention may not use. To overlap these loads with computation, the runtime alternates two GPU buffers: one serves the current layer while the other receives the next layer’s prefix state. This pipeline keeps only two layers’ loaded prefix state on GPU, and each load serves all queries in the chunk. Section 4.2 analyzes when computation is long enough to hide the transfer latency. Layer-wise streaming also applies to HCA. For simplicity, our implementation focuses on CSA, which dominates historical state storage. 4.1

Streaming and Workspace Reuse

Before processing a chunk, the runtime resolves the prefix’s host pages and constructs indexer key and KV page plans for the batch. The compressed KV plan deduplicates pages shared by multiple requests, while the indexer key plan follows request order. Each layer load follows these plans. Without a cached prefix, the first chunk has no prefix state

to load.

Overlapping loading and computation. The runtime alternates two GPU buffers between computation and loading, as shown in Figure 7. While the current CSA layer uses buffer A, a transfer stream loads the next CSA layer’s prefix state into buffer B. Computation then switches to B, allowing A to receive the following layer’s prefix state once it is safe to reuse. The runtime starts loading the first CSA layer’s prefix state before the model’s two initial SWA layers execute. When CSA layer 0 acquires its buffer, the runtime submits the load for CSA layer 1 into B. This transfer can overlap the remaining work in the current Transformer block and the following HCA block, including both MoE FFNs. Each later load follows the same schedule. The compute stream waits for the corresponding load-complete event before consuming a buffer. The transfer stream waits for the buffer’s previous consumers and indexer key writeback before overwriting it.

Managing newly produced state. Processing the current chunk also produces new indexer keys and compressed KV. New indexer keys are written to host before their GPU buffer is reused. New compressed KV is kept in a separate GPU staging buffer, together with compressed KV from earlier chunks whose host writes are still pending. This buffer is organized as a ring whose slots become reusable only after their GPU consumers and host writeback finish. Attention reads selected KV from either the loaded prefix buffer or this local state. Compression state persists across chunks so that a later chunk can finish a compression window started by the previous one. As chunks complete and their state is written to host memory, their pages become available for later chunks to load through the same streaming path. At the transition to decode, the runtime builds compact codes from the completed history and retains the recent local tail, as described in Section 5.

Reducing peak memory. The runtime allocates both layer buffers at configured capacities. Let Icap be each buffer’s capacity for prefix and newly produced indexer keys, Vcap its capacity for compressed KV loaded from the prefix, and Rs the capacity of the separate staging buffer. The prefill allocation is Wp = 2 (Icap + Vcap ) + Rs + Wact + Wscratch ,

(1)

where Wact is the GPU memory occupied by activations for the current chunk, and Wscratch is the temporary workspace used internally by compute kernels while processing that chunk. Retaining an equally sized buffer for every CSA layer would multiply Icap +Vcap by the layer count. Streaming keeps this term at two buffers, reducing the HBM reservation needed to admit prefill alongside decode. Actual allocation follows the configured capacities, even when the batch uses fewer pages. Larger prefix batches require larger buffers, and admission must ensure that their page plans fit. 4.2

Break-even prefill chunk (tokens)

LeanSparse: Memory-Efficient Serving for Native Sparse-Attention LLMs 6K Tcomp ≥ Tload 17.6 GB/s

4K-token prefill chunk

4K

Load 2.9 ms < compute 5.2 ms

2K

η = 100% η = 50%

η = 25%

0

16

24

32

40

48

56

64

Effective host bandwidth per GPU, β (GB/s)

Figure 8. Modeled prefill chunk needed to cover the next-layer transfer for DeepSeek-V4-Flash with a 512K-token prefix on H100 SXM with TP4. The window includes both blocks’ MoE FFNs. Curves vary compute efficiency η with startup and reuse delay δ = 0.

Transfer-Compute Balance

Let ℓ index successive CSA layers, with an HCA layer between them. On the compute stream, the available overlap window consists of the work between submission of the load for layer ℓ + 1 and the first operation that requires its prefix state. As shown in Figure 7, we estimate this window from the remaining work in the current CSA block and the intervening HCA block, including both MoE FFNs. The estimate excludes current-CSA input projections completed before load submission and any preparation of the next CSA layer. Consider a request appending B tokens to an already computed prefix of length L. With CSA compression ratio r and sI , sV bytes per indexer key and compressed KV entry, the prefix payload for the next layer is L Dhist ≃ (sI + sV ). (2) r Let Dnew cover additional page traffic serialized with this load, including indexer key writeback. With effective host bandwidth β and buffer-reuse and startup delay δ, the time from load submission until the next buffer is ready is Dhist + Dnew . (3) β Here, bandwidth is the effective rate per GPU under concurrent traffic. Tload ≃ δ +

We apply the roofline model (Williams et al., 2009) to the serial operation groups in this window. Let Fg and Qg be a group’s FLOPs and HBM traffic, Pg its compute ceiling, η a common compute-efficiency factor, and γ the HBM bandwidth ceiling. Then   X Fg Qg Tcomp (η) ≃ max , . (4) ηPg γ g∈GCSA ∪GHCA

GCSA includes indexing, sparse attention, output projections, and the current block’s MoE FFN after load submission. GHCA includes the intervening block’s input and output projections, HCA attention, and MoE FFN. Counts follow the model dimensions and tensor-parallel placement. HBM traffic assumes ideal KV and weight reuse and a minimum seven-expert working set per block. Routing, selection, compression, normalization, launch, and collective overheads are omitted. Under these assumptions, the predicted wait for the next buffer is Twait = max(0, Tload − Tcomp (η)) .

(5)

Larger chunks increase the work sharing each prefix load, while the dominant prefix payload stays fixed. Figure 8 plots the modeled threshold with δ = 0 and dense FP8 compute ceilings. At η = 1, a 4K-token chunk provides a 5.2 ms window, including 0.36 ms from the two MoE FFNs. Loading and serialized writeback take 2.9 ms at 32 GB/s effective host bandwidth per GPU, leaving a modeled startup and reuse delay budget of about 2.4 ms. This margin depends on the effective transfer bandwidth under concurrent computation. WithP multiple requests, computation over P the prefixes scales with j Bj Lj , while loading scales with j Lj for disjoint prefixes. Total batch size alone therefore does not determine overlap. The first load has a different window because it overlaps the initial two Transformer blocks, each containing SWA and an MoE FFN.

LeanSparse: Memory-Efficient Serving for Native Sparse-Attention LLMs

5

C OARSE - TO -F INE D ECODE I NDEXING

HiSparse (Xie et al., 2026) observes that consecutive decode steps select many of the same historical positions. A small GPU cache can therefore serve much of the attention demand. The indexer, however, still reads every historical indexer key to find the next Top-k. Keeping these indexer keys on GPU consumes memory proportional to context length. Moving them to host would instead require a full-history transfer at every step. This repeated selection of historical positions across decode steps suggests that, over a short decode interval, attention repeatedly returns to a limited part of the history. This motivates using a compact approximation to identify likely candidates before reading their original indexer keys. Crucially, this approximation only needs to retain a useful candidate set. Even if it cannot distinguish two positions near the Top-k boundary, it can keep both and leave that decision to a second pass using their original scores. LeanSparse applies this coarse-to-fine approach to indexer key offloading. It searches compact representations of the full history, retrieves original indexer keys for the surviving candidates, and reranks them to select the final Top-k. Retrieved indexer keys are cached for later steps. The full history remains searchable, while only a small set of original indexer keys needs to reside on GPU or cross the host link at each step. At each decode step, the indexer searches compact representations of the history and retains C > k candidates. The extra candidates allow positions displaced by approximation error to remain available for refinement. Because this search covers the full history, it can discover positions that were neither selected nor cached at the previous step. Approximate scoring with PQ. Our implementation uses product quantization (PQ) (Jégou et al., 2011) for coarse scoring. We split each 128-dimensional indexer key into 32 four-dimensional subvectors and encode each with a onebyte index into a layer-specific codebook of 256 centroids trained offline. The resulting PQ32 code occupies 32 bytes, compared with 132 bytes for the original FP8 indexer key and its scale. The original representation remains in host memory for refinement. The coarse scorer substitutes codebook centroids for the original subvectors and applies the indexer’s weighted head aggregation:   X X T sei = wh ReLU qh,j Cj [ci,j ] , (6) h

j

where qh,j is head h’s query subvector, wh its weight, and Cj [ci,j ] the centroid selected by indexer key i in subspace j.

1

Approximate scoring top-C, C=4 compressed indexer keys

2

i=0

i=1

i=2

i=4

i=3

i=5

i=6

i=7

Exact rescoring indexer key cache

i=3

i=6

hit

miss

i=1

i=4

hit

miss

i=3

i=4

fetch new order top-k, k=2 3

i=1

i=6

Selected KV retrieval KV cache

i=1

hit

i=6

miss

Attention

fetch

Figure 9. Coarse-to-fine indexing with C = 4 and k = 2. Refinement changes the candidate ranking, so the indexer key cache and KV cache serve different positions.

As generation extends the history, the runtime keeps recent indexer keys in a GPU tail and encodes each completed page when writing it to host, allowing the tail storage to be reused. Retrieving and refining candidates. Consecutive decode steps often select many of the same historical positions, allowing refinement to reuse the corresponding indexer keys from earlier steps. Keeping these indexer keys in a GPU cache reduces host reads to candidates that are not already cached. This reuse also extends to positions that repeatedly enter the candidate set without surviving the final Top-k, whose indexer keys remain useful even when their KV is not needed. LeanSparse therefore maintains separate fixedcapacity caches for original indexer keys and compressed KV. Before fetching original indexer keys, LeanSparse retains all candidates that hit in the indexer key cache and filters low-scoring misses. In the evaluated configuration, a miss is discarded if its approximate score is below the candidates’ kth-largest score minus one sample standard deviation. Only the remaining misses require host transfers. LeanSparse computes original indexer scores for the retained candidates and the recent indexer keys in the GPU tail. It selects the final Top-k and retrieves their compressed KV from the KV cache or host memory for attention. Figure 9 shows how refinement changes the approximate ranking. Refinement cannot recover positions removed by coarse selection or miss filtering, but we observe that it recovers

LeanSparse: Memory-Efficient Serving for Native Sparse-Attention LLMs ECHO 1

1 p95 1.74 s

CDF

0.75

1

p95 28.19 s

p95 79.3 ms

p95 1472.9 s

0.75

p50 0.60 s

0.5

p95 100.3 ms

0.5

p50 21.58 s

0.25 0

LeanSparse

p50 66.4 ms

0.25

0

20

40

60

0

p50 1168.0 s

0.5

p50 81.7 ms

p95 1640.1 s

0.75 p50 1418.9 s

0.25

0

50

100

150

0

0

600

1200

1800

2400

TTFT (s)

TPOT (ms)

Session time (s)

(a) Time to first token

(b) Time per output token

(c) Session completion

Figure 10. Open-SWE replay latency distributions: (a) TTFT and (b) TPOT for the same 2,432 continuation calls, and (c) completion time for 128 full sessions, including tool waits. Vertical ticks and labels indicate p50 and p95.

nearly all positions selected by the original indexer and achieves substantially higher token-selection accuracy than selecting Top-k directly from PQ32 scores. Section 6.4 reports the selection accuracy and downstream model quality.

Overlapping transfers with computation. Indexer key retrieval must finish before refinement can determine the KV to retrieve. To overlap these transfers with computation, LeanSparse divides a decode batch into request groups. One group’s indexer key reads can overlap another group’s coarse scoring or refinement, while its KV reads can overlap later groups’ refinement.

6

E VALUATION

We evaluate how LeanSparse’s lower GPU memory use translates into serving performance. Multi-turn replay measures response latency and session completion time when prefill and decode share GPU resources. Long-context decode experiments examine concurrency and throughput as context length grows. Finally, we evaluate model accuracy on reasoning, long-context understanding, and retrieval tasks to assess the effect of approximate candidate filtering. 6.1

Experimental Setup

We run DeepSeek-V4-Flash (DeepSeek-AI et al., 2026) with four NVIDIA H100 GPUs and tensor parallelism in SGLang (Zheng et al., 2024). Multi-turn serving uses our implementation of ECHO (Liu et al., 2026a) because its open-source release does not support DeepSeek-V4. HiSparse (Xie et al., 2026) supports only disaggregated prefill and decode, so we exclude it from this experiment. Since the two baselines have nearly identical decode paths, decodeonly experiments compare against HiSparse.

LeanSparse uses 32-byte PQ codes and refines 2K candidates to select 512 entries. Its per-request GPU caches hold 4K indexer keys and 2K KV entries per CSA layer. HiSparse uses a 4K-entry KV buffer. ECHO shares 256K KV entries per layer and uses 512-entry per-request buffers. Replay enables prefix caching and mixed prefill-decode batches, with 4K-token prefill chunks and up to 128 active requests. 6.2

Multi-Turn Serving Performance

Workload. We replay NVIDIA Open-SWETraces (NVIDIA, 2026) using 128 long-context sessions, excluding codebook-training sessions. Each contributes a 20-call window ending at its longest prompt, preceded by a warmup prefill outside the measured interval. Sessions arrive as a Poisson process at 120 per minute. Calls remain sequential, with tool waits reconstructed from reported durations and command-based estimates. We replay recorded prompts and output lengths, capped at 8K output tokens and the context limit. Later prompts retain their recorded contents. TTFT and TPOT statistics cover the same 2,432 continuation calls after each session’s first measured call. Time per output token (TPOT) is the first-to-last-token interval divided by the number of inter-token gaps, excluding calls with fewer than two output tokens. Session time spans the first measured submission through the final completion, including tool waits. Throughput uses all 2,560 measured calls, dividing their total output tokens by the full replay duration. Results. LeanSparse substantially reduces response startup delays. Figure 10(a) shows median TTFT falling from 21.6 to 0.6 s, while the 95th percentile falls from 28.19 to 1.74 s. Shorter waits improve responsiveness across the

LeanSparse: Memory-Efficient Serving for Native Sparse-Attention LLMs

3,000

LeanSparse

3216 2371

2,000

2022 1474

1274 687

1,000 0

HiSparse

N=76 N=156

200K

N=36

N=96

512K

N=19

Step time (ms)

Decode throughput (tokens/s)

HiSparse 4,000

N=56

40

32.1 33.3

30

LeanSparse

28.3 26.4

27.6

22.2

20 10 0

N=76

N=36

N=19

200K

512K

950K

950K

Figure 12. Decode step time at equal concurrency. N denotes concurrency. Figure 11. Decode throughput at measured capacities. N denotes concurrency.

measured call distribution, including slow calls. These shorter waits also help complete sessions sooner. Figure 10(c) shows median session completion time decreasing by 17.7%. Both methods complete all calls. Because calls are sequential, faster response starts help sessions advance. Token generation and tool waits still contribute to completion time, so the session-level reduction is smaller than the TTFT reduction. Session time captures how these delays accumulate across dependent calls. Figure 10(b) shows median TPOT increasing from 66.4 to 81.7 ms. We attribute this primarily to larger decode batches and earlier prefill scheduling after memory pressure is reduced. More active requests and prefill work share each execution step, increasing the interval between output tokens. Sessions still finish sooner, and aggregate throughput rises by 10.2%. Indexer selection recall versus direct quantization. In an offline analysis of Open-SWE traces, we compare our coarse-to-fine approach with direct index scoring using PQ32. Recall measures the fraction of the exact indexer’s top-512 entries recovered by the selected set. Selecting 512 entries directly from PQ32 scores achieves 77.41% mean recall. Expanding the candidate set to 2,048 and reranking with exact scores raises mean recall to 99.49%. These results show that coarse-to-fine indexing combines compact GPU-resident representations with high selection recall.

concurrency reached in each stress test. For the step-time comparison, both methods use HiSparse’s maximum tested concurrency at each context length. Results. LeanSparse admits 2.05-2.95× as many concurrent requests across the three context lengths. Figure 11 shows that its throughput advantage grows from 1.36× at 200K to 2.14× at 950K. Compact indexer representations leave more GPU space for concurrent requests as histories grow. The throughput improvement is smaller than the concurrency increase because admitting more streams also lengthens each decode step. Figure 12 compares decode step time at equal concurrency. LeanSparse takes 3.8% longer per step at 200K but 19.6% less time at 950K. At shorter contexts, savings in exact indexer scoring are small relative to the added coarsescreening overhead. Longer histories increase HiSparse’s exact scoring work, while LeanSparse restricts exact scoring to 2K candidates after approximate filtering. The coarse pass still scans the compact history, so its cost grows with context length. The candidate budget bounds the subsequent reads of original indexer keys and exact scoring. Both systems admit fewer requests as context length grows. LeanSparse reduces the storage per historical position, but its compact representations still grow with the history. Its capacity advantage persists across the tested lengths, allowing more requests to share the GPU even as longer contexts increase each request’s memory demand. 6.4

6.3

Long-Context Decode Performance

Workload. We construct 200K-, 512K-, and 950K-token contexts from LongSWE-Bench (Rando et al., 2025), trimming repository files while preserving instructions and issue descriptions. Prefix reuse is disabled for these requests. All admitted requests finish prefill before decode begins, with outputs capped at 8K tokens. After a 60-second settling interval, we measure steady decode over 2K new tokens per stream. We report steady-state throughput at the maximum

Model Accuracy

Tasks. We compare LeanSparse with unmodified SGLang and HiSparse on GSM8K (OpenAI, 2021), multi-key needlein-a-haystack (NIAH) (Hsieh et al., 2024), LongBench v2 (Bai et al., 2025), and two-needle OpenAI MRCR (OpenAI, 2025). GSM8K uses 20 fixed demonstrations and all 1,319 test questions. NIAH contains 150 prompts at 32K, 128K, and 200K. LongBench v2 includes 403 questions fitting within 245K tokens, and MRCR contains 265 examples. For the comparison in Table 1, all methods use

LeanSparse: Memory-Efficient Serving for Native Sparse-Attention LLMs Table 1. Model quality with a 256K context limit. Scores are percentages. Higher is better.

Method SGLang HiSparse LeanSparse

GSM8K

NIAH

LBv2

MRCR

95.38 95.30 95.83

100.00 96.00 100.00

50.87 51.61 50.62

90.19 87.86 90.62

greedy non-thinking decoding, a 256K-token context limit, 4K-token prefill chunks, and up to 64 concurrent requests. Scores measure answer accuracy, recovered-needle fraction, or MRCR reference-text similarity. Results. Table 1 shows that LeanSparse largely preserves model quality across the tested tasks, with all scores within 0.5 percentage points of SGLang. NIAH recovery remains complete, and the other tasks show small changes in both directions. Paired McNemar tests find no significant difference on GSM8K or LongBench v2. This is consistent with our coarse-to-fine design, which uses approximate scores only to narrow the candidate set and computes exact scores from the original indexer keys for final selection. The results suggest that this filtering step reduces memory use without materially affecting accuracy on these workloads.

7

R ELATED W ORK

LLM serving. PagedAttention (Kwon et al., 2023) and SGLang (Zheng et al., 2024) improve KV allocation and prefix reuse. Sarathi-Serve (Agrawal et al., 2024) and DistServe (Zhong et al., 2024) address prefill-decode interference through chunked scheduling and phase separation. FlexGen (Sheng et al., 2023) distributes model state across GPU, CPU, and disk for throughput-oriented inference. CachedAttention (Gao et al., 2024) combines hierarchical KV reuse with layer-wise preloading, while Mooncake (Qin et al., 2025) extends KV reuse across distributed caches. For selective offloading, InfiniGen (Lee et al., 2024) predicts next-layer KV demand, while FreeKV (Liu et al., 2026b) combines speculative retrieval with correction and streamed transfers. LeanSparse reduces persistent decode state and temporary prefill storage for active requests. Sparse attention. Training-free methods reduce attention work without retraining: MInference (Jiang et al., 2024) exploits sparse prefill patterns, and Quest (Tang et al., 2024) selects query-dependent KV pages. ArkVale (Chen et al., 2024), PQCache (Zhang et al., 2025), and ShadowKV (Sun et al., 2025) use compact representations to retrieve offloaded KV selectively. Native Sparse Attention (Yuan et al., 2025), MoBA (Lu et al., 2025), and DeepSeek Sparse Attention (DeepSeek-AI et al., 2025) incorporate sparse selection into training. HISA (Xu et al., 2026) and DoSA (Meng &

Fang, 2026) accelerate indexing through candidate filtering. SparseServe (Zhou et al., 2025) combines KV offloading and batch control with layer-segmented prefill to reduce prefill KV residency, but does not discuss loading cached prefixes from host memory or overlapping these loads with prefill computation. ECHO (Liu et al., 2026a) and HiSparse (Xie et al., 2026) manage KV residency through caching and prefetching. LeanSparse combines coarse-tofine retrieval of separate indexer keys with double-buffered prefill streaming.

8

C ONCLUSION

Native sparse-attention serving remains constrained by full indexer histories on GPU and temporary prefill demand. These costs limit decode concurrency and delay prefill admission. We present LeanSparse to address both costs. Its prefill path streams layer histories through two reusable buffers, while coarse-to-fine decode indexing retrieves original indexer keys only for candidate refinement. Our evaluation shows that LeanSparse achieves higher decode throughput, shorter response delays, and faster session completion than state-of-the-art baselines.

R EFERENCES Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B., Tumanov, A., and Ramjee, R. Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 117–134. USENIX Association, 2024. URL https://www.usenix.org/conference/os di24/presentation/agrawal. Bai, Y., Tu, S., Zhang, J., Peng, H., Wang, X., Lv, X., Cao, S., Xu, J., Hou, L., Dong, Y., Tang, J., and Li, J. LongBench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3639– 3664. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.acl-long.183. URL https: //aclanthology.org/2025.acl-long.183/. Chen, R., Wang, Z., Cao, B., Wu, T., Zheng, S., Li, X., Wei, X., Yan, S., Li, M., and Liang, Y. ArkVale: Efficient generative LLM inference with recallable keyvalue eviction. In Advances in Neural Information Processing Systems, volume 37, 2024. URL https: //papers.nips.cc/paper_files/paper/2 024/hash/cd4b49379efac6e84186a3ffce1 08c37-Abstract-Conference.html. DeepSeek-AI, Liu, A., Mei, A., Lin, B., Xue, B., Wang,

LeanSparse: Memory-Efficient Serving for Native Sparse-Attention LLMs

B., Xu, B., Wu, B., Zhang, B., Lin, C., Dong, C., Lu, C., Zhao, C., Deng, C., Xu, C., Ruan, C., Dai, D., Guo, D., Yang, D., Chen, D., Li, E., Zhou, F., Lin, F., Dai, F., Hao, G., Chen, G., Li, G., Zhang, H., Xu, H., Li, H., Liang, H., Wei, H., Zhang, H., Luo, H., Ji, H., Ding, H., Tang, H., Cao, H., Gao, H., Qu, H., Zeng, H., Huang, J., Li, J., Xu, J., Hu, J., Chen, J., Xiang, J., Yuan, J., Cheng, J., Zhu, J., Ran, J., Jiang, J., Qiu, J., Li, J., Song, J., Dong, K., Gao, K., Guan, K., Huang, K., Zhou, K., Huang, K., Yu, K., Wang, L., Zhang, L., Wang, L., Zhao, L., Yin, L., Guo, L., Luo, L., Ma, L., Wang, L., Zhang, L., Di, M. S., Xu, M. Y., Zhang, M., Zhang, M., Tang, M., Zhou, M., Huang, P., Cong, P., Wang, P., Wang, Q., Zhu, Q., Li, Q., Chen, Q., Du, Q., Xu, R., Ge, R., Zhang, R., Pan, R., Wang, R., Yin, R., Xu, R., Shen, R., Zhang, R., Liu, S. H., Lu, S., Zhou, S., Chen, S., Cai, S., Chen, S., Hu, S., Liu, S., Hu, S., Ma, S., Wang, S., Yu, S., Zhou, S., Pan, S., Zhou, S., Ni, T., Yun, T., Pei, T., Ye, T., Yue, T., Zeng, W., Liu, W., Liang, W., Pang, W., Luo, W., Gao, W., Zhang, W., Gao, X., Wang, X., Bi, X., Liu, X., Wang, X., Chen, X., Zhang, X., Nie, X., Cheng, X., Liu, X., Xie, X., Liu, X., Yu, X., Li, X., Yang, X., Li, X., Chen, X., Su, X., Pan, X., Lin, X., Fu, X., Wang, Y. Q., Zhang, Y., Xu, Y., Ma, Y., Li, Y., Li, Y., Zhao, Y., Sun, Y., Wang, Y., Qian, Y., Yu, Y., Zhang, Y., Ding, Y., Shi, Y., Xiong, Y., He, Y., Zhou, Y., Zhong, Y., Piao, Y., Wang, Y., Chen, Y., Tan, Y., Wei, Y., Ma, Y., Liu, Y., Yang, Y., Guo, Y., Wu, Y., Wu, Y., Cheng, Y., Ou, Y., Xu, Y., Wang, Y., Gong, Y., Wu, Y., Zou, Y., Li, Y., Xiong, Y., Luo, Y., You, Y., Liu, Y., Zhou, Y., Wu, Z. F., Ren, Z. Z., Zhao, Z., Ren, Z., Sha, Z., Fu, Z., Xu, Z., Xie, Z., Zhang, Z., Hao, Z., Gou, Z., Ma, Z., Yan, Z., Shao, Z., Huang, Z., Wu, Z., Li, Z., Zhang, Z., Xu, Z., Wang, Z., Gu, Z., Zhu, Z., Li, Z., Zhang, Z., Xie, Z., Gao, Z., Pan, Z., Yao, Z., Feng, B., Li, H., Cai, J. L., Ni, J., Xu, L., Li, M., Tian, N., Chen, R. J., Jin, R. L., Li, S. S., Zhou, S., Sun, T., Li, X. Q., Jin, X., Shen, X., Chen, X., Song, X., Zhou, X., Zhu, Y. X., Huang, Y., Li, Y., Zheng, Y., Zhu, Y., Ma, Y., Huang, Z., Xu, Z., Zhang, Z., Ji, D., Liang, J., Guo, J., Chen, J., Xia, L., Wang, M., Li, M., Zhang, P., Chen, R., Sun, S., Wu, S., Ye, S., Wang, T., Xiao, W. L., An, W., Wang, X., Sun, X., Wang, X., Tang, Y., Zha, Y., Zhang, Z., Ju, Z., Zhang, Z., and Qu, Z. DeepSeek-V3.2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556, 2025. URL https://arxiv.org/abs/2512.02556. DeepSeek-AI, Xu, A., Lin, B., Xue, B., Wang, B., Xu, B., Wu, B., Zhang, B., Lin, C., Dong, C., Ling, C., Lu, C., Zhao, C., Deng, C., Hou, C., Xu, C., Shao, C., Ruan, C., Sun, C., Dai, D., Guo, D., Yang, D., Chen, D., Li, D., Ji, D., Li, E., Wei, F., Lin, F., Yuan, F., Xia, F., Dai, F., Hao, G., Chen, G., Cao, G., Meng, G., Li, G., Yu, H., Zhang, H., Xu, H., Li, H., Liang, H., Zhang, H., Luo, H., Wei, H., Yuan, H., Zhang, H., Luo, H., Chen, H., Ji, H., Zhang, H.,

Ding, H., Tang, H., Cao, H., Gao, H., Qu, H., Zeng, H., Yang, J., Zhu, J., Luo, J., Song, J., Yu, J., Huang, J., Cai, J., Liang, J., Zhou, J., Ye, J., Li, J., Xu, J., Hu, J., Yang, J., Chen, J., Yan, J., Chen, J., Zhou, J., Xiang, J., Yuan, J., Cheng, J., Zhou, J., Zhu, J., Yu, J., Sun, J., Ran, J., Jiang, J., Qiu, J., Li, J., Zheng, J., Song, J., Dong, K., Gao, K., Guan, K., Zhou, K., Huang, K., Yu, K., Wang, L., Zhang, L., Wang, L., Xia, L., Zhang, L., Zhao, L., Guo, L., Luo, L., Ma, L., Zhu, L., Wang, L., Cai, L., Zhang, L., Chen, L., Di, M., Xu, M., Mei, M., Wang, M., Zhang, M., Zhang, M., Tang, M., Li, M., Zhou, M., Han, M., Wang, N., Huang, P., Wang, P., Cong, P., Wang, P., Zhang, P., Wang, Q., Zhu, Q., Li, Q., Chen, Q., Du, Q., Jiang, Q., Tian, R., Xu, R., Lu, R., Xu, R., Ge, R., Zhang, R., Pan, R., Wang, R., Chen, R., Yin, R., Xu, R., Shen, R., Zhang, R., Chen, R., Liu, S., Lu, S., Sun, S., Zhou, S., Chen, S., Cai, S., Nie, S., Wu, S., Chen, S., Hu, S., Liu, S., Hu, S., Ma, S., Wang, S., Yu, S., Zhou, S., Pan, S., Yu, S., Zhou, S., Ni, T., Yun, T., Jin, T., Pei, T., Ye, T., Lin, T., Ji, T., Cui, T., Yue, T., Yu, T., Wang, T., Zhang, W., Xiao, W., Zeng, W., An, W., Zhao, W., Liu, W., Liang, W., Pang, W., Luo, W., Yao, W., Gao, W., Yang, W., Huang, W., Hou, W., Zhang, W., Ma, W., Gao, X., He, X., Wang, X., Wang, X., Bi, X., Liu, X., Wang, X., Chen, X., Zhang, X., Nie, X., Sun, X., Wang, X., Cheng, X., Liu, X., Xie, X., Liu, X., Liu, X., Yu, X., Li, X., Yang, X., Zhang, X., Chen, X., Wang, X., Su, X., Chen, X., Lin, X., Fu, X., Yan, Y., Wang, Y., Ma, Y., Luo, Y., Zhang, Y., Xu, Y., Ma, Y., Huang, Y., Li, Y., Li, Y., Xu, Y., Zhao, Y., Sun, Y., Wang, Y., Qian, Y., Shao, Y., Yu, Y., Zhang, Y., Ding, Y., Shi, Y., Wu, Y., Xiong, Y., Ma, Y., He, Y., Tang, Y., Zhou, Y., Luo, Y., Zhong, Y., Piao, Y., Wang, Y., Zhang, Y., Chen, Y., Tan, Y., Wei, Y., Ma, Y., Liu, Y., Yang, Y., Guo, Y., Wu, Y., Wu, Y., Li, Y., Cheng, Y., Ou, Y., Xu, Y., Li, Y., Wang, Y., Yang, Y., Xu, Y., Wu, Y., Meng, Y., Zou, Y., Zha, Y., Xiong, Y., Chen, Y., Lin, Y., Cao, Y., Wang, Y., Zhang, Y., Yan, Y., Lin, Y., Gu, Y., Luo, Y., You, Y., Liu, Y., Zhou, Y., Zhou, Y., Huang, Y., Wu, Z., Wang, Z., Zhao, Z., Ren, Z., Zhang, Z., Sha, Z., Fu, Z., Ju, Z., Xu, Z., Xie, Z., Zhang, Z., Gao, Z., Hao, Z., Gou, Z., Ma, Z., Yan, Z., Shao, Z., Huang, Z., Chen, Z., Wu, Z., Ren, Z., Wu, Z., Li, Z., Zhang, Z., Xu, Z., Wang, Z., Qu, Z., Gu, Z., Zhu, Z., Li, Z., Zhang, Z., Xie, Z., Gao, Z., Wan, Z., Pan, Z., and Yao, Z. DeepSeek-V4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026. URL https://ar xiv.org/abs/2606.19348. Gao, B., He, Z., Sharma, P., Kang, Q., Jevdjic, D., Deng, J., Yang, X., Yu, Z., and Zuo, P. Cost-efficient large language model serving for multi-turn conversations with CachedAttention. In 2024 USENIX Annual Technical Conference (USENIX ATC 24), pp. 111–126. USENIX Association, 2024. URL https://www.usenix.o

LeanSparse: Memory-Efficient Serving for Native Sparse-Attention LLMs

rg/conference/atc24/presentation/gao -bin-cost.

6f5671e82c76841ba687a8a9ecb-Abstrac t-Conference.html.

GLM-5-Team, Zeng, A., Lv, X., Hou, Z., Du, Z., Zheng, Q., Chen, B., Yin, D., Ge, C., Huang, C., Xie, C., Zhu, C., Yin, C., Wang, C., Pan, G., Zeng, H., Zhang, H., Wang, H., Chen, H., Zhang, J., Jiao, J., Guo, J., Wang, J., Du, J., Wu, J., Wang, K., Li, L., Fan, L., Zhong, L., Liu, M., Zhao, M., Du, P., Dong, Q., Lu, R., Shuang-Li, Cao, S., Liu, S., Jiang, T., Chen, X., Zhang, X., Huang, X., Dong, X., Xu, Y., Wei, Y., An, Y., Niu, Y., Zhu, Y., Wen, Y., Cen, Y., Bai, Y., Qiao, Z., Wang, Z., Wang, Z., Zhu, Z., Liu, Z., Li, Z., Wang, B., Wen, B., Huang, C., Cai, C., Yu, C., Li, C., Hu, C., Zhang, C., Zhang, D., Lin, D., Yang, D., Wang, D., Ai, D., Zhu, E., Yi, F., Chen, F., Wen, G., Sun, H., Zhao, H., Hu, H., Zhang, H., Liu, H., Zhang, H., Peng, H., Tai, H., Zhang, H., Liu, H., Wang, H., Yan, H., Ge, H., Liu, H., Chu, H., Zhao, J., Wang, J., Zhao, J., Ren, J., Wang, J., Zhang, J., Gui, J., Zhao, J., Li, J., An, J., Li, J., Yuan, J., Du, J., Liu, J., Zhi, J., Duan, J., Zhou, K., Wei, K., Wang, K., Luo, K., Zhang, L., Sha, L., Xu, L., Wu, L., Ding, L., Chen, L., Li, M., Lin, N., Ta, P., Zou, Q., Song, R., Yang, R., Tu, S., Yang, S., Wu, S., Zhang, S., Li, S., Li, S., Fan, S., Qin, W., Tian, W., Zhang, W., Yu, W., Liang, W., Kuang, X., Cheng, X., Li, X., Yan, X., Hu, X., Ling, X., Fan, X., Xia, X., Zhang, X., Zhang, X., Pan, X., Zou, X., Zhang, X., Liu, Y., Wu, Y., Li, Y., Wang, Y., Zhu, Y., Tan, Y., Zhou, Y., Pan, Y., Zhang, Y., Su, Y., Geng, Y., Yan, Y., Tan, Y., Bi, Y., Shen, Y., Yang, Y., Li, Y., Liu, Y., Wang, Y., Li, Y., Wu, Y., Zhang, Y., Duan, Y., Zhang, Y., Liu, Z., Jiang, Z., Yan, Z., Zhang, Z., Wei, Z., Chen, Z., Feng, Z., Yao, Z., Chai, Z., Wang, Z., Zhang, Z., Xu, B., Huang, M., Wang, H., Li, J., Dong, Y., and Tang, J. GLM-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026. URL https: //arxiv.org/abs/2602.15763.

Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles. Association for Computing Machinery, 2023. doi: 10.1145/3600006.3613165. URL https://arxiv. org/abs/2309.06180.

Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., and Ginsburg, B. RULER: What’s the real context size of your long-context language models? In Conference on Language Modeling, 2024. URL http s://arxiv.org/abs/2404.06654v3. Jégou, H., Douze, M., and Schmid, C. Product quantization for nearest neighbor search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(1):117–128, 2011. doi: 10.1109/TPAMI.2010.57. URL https: //inria.hal.science/inria-00514462v2. Jiang, H., Li, Y., Zhang, C., Wu, Q., Luo, X., Ahn, S., Han, Z., Abdi, A. H., Li, D., Lin, C.-Y., Yang, Y., and Qiu, L. MInference 1.0: Accelerating pre-filling for longcontext LLMs via dynamic sparse attention. In Advances in Neural Information Processing Systems, volume 37, 2024. URL https://proceedings.neurips. cc/paper_files/paper/2024/hash/5dfbe

Lee, W., Lee, J., Seo, J., and Sim, J. InfiniGen: Efficient generative inference of large language models with dynamic KV cache management. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 155–172. USENIX Association, 2024. URL https://www.usenix.org/conference/os di24/presentation/lee. Liu, G., Chen, W., Li, C., Ning, Z., Lin, J., Yao, Y., Chen, Q., Sun, S., Zhao, J., and Guo, M. ECHO: Efficient KV cache offloading with lossless prefetching for serving native sparse attention LLMs. In 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 26), pp. 17–37. USENIX Association, 2026a. URL ht tps://www.usenix.org/conference/osdi 26/presentation/liu-guangda. Liu, G., Li, C., Ning, Z., Lin, J., Yao, Y., Ke, D., Guo, M., and Zhao, J. FreeKV: Boosting KV cache retrieval for efficient LLM inference. In International Conference on Learning Representations, 2026b. URL https://op enreview.net/forum?id=wXAn7orB1H. Lu, E., Jiang, Z., Liu, J., Du, Y., Jiang, T., Hong, C., Liu, S., He, W., Yuan, E., Wang, Y., Huang, Z., Yuan, H., Xu, S., Xu, X., Lai, G., Chen, Y., Zheng, H., Yan, J., Su, J., Wu, Y., Zhang, N. Y., Yang, Z., Zhou, X., Zhang, M., and Qiu, J. MoBA: Mixture of block attention for long-context LLMs. arXiv preprint arXiv:2502.13189, 2025. URL https://arxiv.org/abs/2502.13189. Meng, F. and Fang, T. DoSA: Dots sparse attention. arXiv preprint arXiv:2610.13990, 2026. URL https://ar xiv.org/abs/2610.13990. NVIDIA. Open-SWE-Traces. Hugging Face dataset, 2026. URL https://huggingface.co/datasets/ nvidia/Open-SWE-Traces. OpenAI. GSM8K: Grade school math dataset, 2021. URL https://github.com/openai/grade-schoo l-math. OpenAI. MRCR: Multi-round coreference resolution dataset, 2025. URL https://huggingface.co /datasets/openai/mrcr.

LeanSparse: Memory-Efficient Serving for Native Sparse-Attention LLMs

Qin, R., Li, Z., He, W., Cui, J., Ren, F., Zhang, M., Wu, Y., Zheng, W., and Xu, X. Mooncake: Trading more storage for less computation—a KVCache-centric architecture for serving LLM chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST 25), pp. 155–170. USENIX Association, 2025. URL https://www.us enix.org/conference/fast25/presentat ion/qin. Rando, S., Romani, L., Sampieri, A., Franco, L., Yang, J., Kyuragi, Y., Galasso, F., and Hashimoto, T. LongCodeBench: Evaluating coding LLMs at 1M context windows. arXiv preprint arXiv:2505.07897, 2025. URL https://arxiv.org/abs/2505.07897v3. Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Chen, B., Liang, P., Ré, C., Stoica, I., and Zhang, C. FlexGen: High-throughput generative inference of large language models with a single GPU. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 31094–31116. PMLR, 2023. URL https://procee dings.mlr.press/v202/sheng23a.html. Sun, H., Chang, L.-W., Bao, W., Zheng, S., Zheng, N., Liu, X., Dong, H., Chi, Y., and Chen, B. ShadowKV: KV cache in shadows for high-throughput long-context LLM inference. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 57355–57373. PMLR, 2025. URL https://proceedings.mlr. press/v267/sun25b.html. Tang, J., Zhao, Y., Zhu, K., Xiao, G., Kasikci, B., and Han, S. QUEST: Query-aware sparsity for efficient longcontext LLM inference. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 47901–47911. PMLR, 2024. URL https://procee dings.mlr.press/v235/tang24l.html. Wang, C., Zuo, P., Chen, Z., Liang, Y., Yu, Z., and Yang, M.C. Prefill-decode aggregation or disaggregation? unifying both for goodput-optimized LLM serving. arXiv preprint arXiv:2508.01989, 2025. URL https://arxiv.or g/abs/2508.01989. Williams, S., Waterman, A., and Patterson, D. Roofline: An insightful visual performance model for multicore architectures. Communications of the ACM, 52(4):65–76, 2009. doi: 10.1145/1498765.1498785. Wu, B., Jin, C., Zhang, Z., Wei, X., Zhong, Y., Zhu, R., Yang, C., Jin, X., and Liu, Y. ExpertPlex: A high-goodput disaggregated serving system for MoE LLMs with adaptive persistent kernels. arXiv preprint arXiv:2607.18002,

2026. URL https://arxiv.org/abs/2607.1 8002v2. Xie, Z., Huang, Z., Huang, T., Xu, Z., Ma, R., and Kozyrakis, C. HiSparse: Scaling sparse-attention decoding with hierarchical KV cache management. arXiv preprint arXiv:2608.07009, 2026. URL https://ar xiv.org/abs/2608.07009v1. Xu, Y., Meng, F., Jiang, F., Wang, Y., Zhou, R., Wang, Z., Wu, J., Pan, Z., Tang, X., Pei, W., Liu, T., Yin, D., Sun, X., and Zhang, M. HISA: Efficient hierarchical indexing for fine-grained sparse attention. In Conference on Language Modeling, 2026. URL https://proceedings.co lm.cc/paper_files/paper/2026/hash/b3 b3e22dba07a71b0c0039af6af4d478-Abstr act-Conference.html. Yuan, J., Gao, H., Dai, D., Luo, J., Zhao, L., Zhang, Z., Xie, Z., Wei, Y., Wang, L., Xiao, Z., Wang, Y., Ruan, C., Zhang, M., Liang, W., and Zeng, W. Native sparse attention: Hardware-aligned and natively trainable sparse attention. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 23078–23097. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.acl-long.1126. URL https://ac lanthology.org/2025.acl-long.1126/. Zhang, H., Ji, X., Chen, Y., Fu, F., Miao, X., Nie, X., Chen, W., and Cui, B. PQCache: Product quantization-based KVCache for long context LLM inference. Proceedings of the ACM on Management of Data, 3(3):201:1–201:30, 2025. doi: 10.1145/3725338. URL https://arxiv. org/abs/2407.12820. Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., and Sheng, Y. SGLang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems, volume 37, 2024. URL https://proceedings.neurips.cc/paper _files/paper/2024/hash/724be4472168f 31ba1c9ac630f15dec8-Abstract-Confere nce.html. Zhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X., and Zhang, H. DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 193–210. USENIX Association, 2024. URL https://www.usenix.org/conference/os di24/presentation/zhong-yinmin. Zhou, Q., Yin, P., Zuo, P., and Cheng, J. SparseServe: Unlocking parallelism for dynamic sparse attention in long-

LeanSparse: Memory-Efficient Serving for Native Sparse-Attention LLMs

context LLM serving. arXiv preprint arXiv:2509.24626, 2025. URL https://arxiv.org/abs/2509.2 4626.

AI U SAGE D ISCLOSURE OpenAI Codex assisted with writing and revision and literature search. Anthropic Claude assisted with code implementation, experiments, and figure preparation. The authors take full responsibility for the correctness, originality, and integrity of the paper.

Related documents

• DASC: Decay-Aware State Compression for Hybrid Linear-Attention Serving#799590
• MemoryAthena: Adaptive Routing over Latent and Generated Memories#1061481
• Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Management#668000
• Memory-Efficient Expert Routing for Distributed MoE Training#1216708
• Memory-Efficient Expert Routing for Distributed MoE Training#1210249