FlatCake: A Flat KVCache Storage Architecture for LLM Serving

FlatCake: A Flat KVCache Storage Architecture for LLM Serving Xin Huang1

Teng Ma2 Wei Yang1

Zhuoran Li3 Yibo Zhao3 5 Chenxi Wang Tao Xie3,1

Zhaoshi Li4 Qianchao Zhu2 3, * Youwei Zhuo

1 Fudan Institute of Systems for Advanced Computing, Fudan University, Shanghai, China 2 Independent Researcher 4 MetaX Technology, Shanghai, China

3 School of Integrated Circuits, Peking University, Beijing, China

5 Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China

arXiv:2610.38355v1 [cs.DC] 28 Oct 2026

huangxin25@m.fudan.edu.cn stmatengss@gmail.com {zhuoranli, yibozhao}@stu.pku.edu.cn zhaoshi.li@metax-tech.com wangbo18@xiaohongshu.com yang_wei@fudan.edu.cn wangchenxi@ict.ac.cn {taoxie, youwei}@pku.edu.cn * Corresponding author

Abstract KV cache reuse accelerates LLM serving, but existing systems manage host DRAM and NVMe SSD as a tiered hierarchy where DRAM acts as an intermediate cache. Production LLM workloads break both premises of this hierarchy. First, bimodal reuse defeats DRAM temporal filtering, since workloads are dominated either by one-shot blocks with long reuse distances or by multi-gigabyte agentic sessions that exceed DRAM capacity, putting SSD on the critical read path. Second, DRAM staging and eviction-ordered layout bottleneck SSD restoration. Holding host staging buffers for in-flight prefixes causes contention under load that aborts prefetches, while incremental eviction scatters a session’s blocks across the SSD and prevents large sequential reads. We present FlatCake, a flat KV storage architecture that exposes host DRAM and NVMe SSD as peer media in a unified 64-bit address space. Exploiting the fact that KV blocks are write-once, immutable, and reused at session scope, FlatCake places each block directly onto its target medium in session-contiguous, layer-major order at creation time and never migrates it. FlatCake streams coalesced SSD reads directly into a bounded GPU window via GPUDirect Storage and decouples DRAM and SSD request queues to prevent head-of-line blocking. On a production Claude Code trace with DeepSeek-V4-Flash, FlatCake sustains a 94% cache hit rate as load increases (versus 64% for Mooncake), eliminates Mooncake’s 6.5×–6.7× SSD read amplification, and reduces peak-load mean TTFT by 1.6× while improving total token throughput by 1.6×. Across 8K–128K contexts on Llama-3.1-70B, FlatCake further reduces mean TTFT by up to 2.4× over Mooncake and 1.8× over Mooncake-GDS. Keywords: LLM serving, KV cache management, memory tiering, flat KV storage, GPUDirect Storage

1

Introduction

Prefix caching is now standard in large language model (LLM) serving, allowing requests that share a prefix to reuse previously computed key-value (KV) tensors and avoid redundant prefill computation [16, 27, 47]. As LLM workloads evolve

GPU 0 (Prefill)

GPU 1 (Decode)

GPU HBM (L1)

GPU 0 (Prefill)

GPU HBM (L1) Put/get

Put/get

Host DRAM (L2) LRU evict NVMe SSD (L3)

(a) KV Cache Tiered Storage

GPU 1 (Decode)

KV Cache Flat Storage Interface Unified Address Space

No eviction, direct placement Host DRAM (L2)

NVMe SSD (L2)

(b) KV Cache Flat Storage

Figure 1. Tiered and flat KV cache storage. (a) Tiering ranks DRAM above SSD, so a block reaches SSD only when LRU evicts it. (b) FlatCake places both media at one level behind a unified address space and writes each block straight to its target, so nothing migrates.

from short, globally shared system prompts [38] to multiturn conversations [12] and long-context agentic coding sessions that append tool outputs and retrieved files on every turn (averaging 219K input tokens per request in a production Claude Code trace [33]), active KV working sets routinely reach tens of gigabytes per session and span hours. Because these working sets far exceed GPU HBM and host DRAM capacity, while recomputing evicted history consumes up to 93% of prefill compute [12], NVMe SSDs must serve as a primary KV cache medium on the critical path [28]. Existing systems such as Mooncake [27], LMCache [7], and SGLang HiCache [42] manage GPU HBM, host DRAM, and NVMe SSD as a classical tiered hierarchy (Figure 1(a)), placing host DRAM in front of SSD as an intermediate L2 cache and staging layer. Borrowed from general-purpose memory and storage systems, this hierarchical design relies on two foundational premises: (1) temporal filtering, namely that recency-based caching in DRAM can capture the active working set so the slower SSD serves only cold misses off the critical path; and (2) staging and coalescing, namely that because incremental eviction scrambles on-disk layout relative to read order, staging SSD accesses through host DRAM buffers is an effective way to reassemble scattered

Xin Huang, Teng Ma, Zhuoran Li, Yibo Zhao, Zhaoshi Li, Qianchao Zhu, Wei Yang, Chenxi Wang, Tao Xie, and Youwei Zhuo

reads and hide device latency. Profiling eight production traces from Mooncake, Qwen Bailian, and Claude Code (§2.2) reveals that LLM KV cache workloads violate both premises simultaneously. First, bimodal reuse defeats DRAM temporal filtering, putting SSD on the critical read path. Across production traces, KV reuse falls into two regimes, and neither fits a DRAM cache filter. In conversational and API workloads (seven traces), 53–94% of KV blocks are accessed only once, while reusable prefixes span long reuse distances of hundreds to thousands of intervening requests; admitting every request into DRAM pollutes it with one-shot blocks that flush reusable prefixes to SSD before their next turn arrives. Conversely, in agentic coding (Claude Code), 83.4% of blocks are reused (98.3% ideal hit rate) with immediate turnto-turn locality, but multi-gigabyte session footprints quickly exceed host DRAM capacity across concurrent sessions. In both regimes, most cache hits must be served from NVMe SSD, so migrating blocks between DRAM and SSD adds data movement without keeping hot data in DRAM. Second, DRAM staging and eviction-ordered layout turn SSD restoration into a concurrency and I/O bottleneck. When SSD must serve the bulk of hits, the tiered mechanisms meant to smooth SSD access become the bottleneck itself. Restoring a long-context prefix through host DRAM requires holding host staging buffers for the entire in-flight hit, scaling memory demand with context length and concurrency. Even when SSD-to-DRAM and DRAM-to-GPU transfers are pipelined, concurrent restores contend for bounded staging memory, delaying or aborting prefetches under load and turning blocks already stored on SSD into recomputed misses. Worse, even when bypassing host DRAM via batched GPUDirect Storage (GDS), tiered storage populates and reclaims SSD space in eviction or FIFO-bucket order rather than session order. As a result, a session’s blocks are scattered across non-contiguous SSD offsets and coarse bucket reclamation invalidates active prefixes across sessions, forcing 𝑂 (layers × blocks) fragmented reads that underutilize sequential NVMe bandwidth. Why does tiered storage fall into this trap? Tiering is designed for mutable pages whose future access patterns and spatial grouping are unknown when written. Our key insight is that LLM KV cache blocks are write-once, immutable, and reused together at the granularity of a session scope. Because a block is never modified in place, there is no dirty state to write back; because blocks generated in a turn belong to a known session prefix and are always restored together across layers on subsequent turns, their ideal on-disk spatial grouping is already known at creation time. Consequently, if blocks are placed directly onto their target medium in session-contiguous, layer-major order at write time and never migrated, SSD restores become naturally sequential and no longer need host DRAM for either caching or reassembly.

Based on this insight, we propose FlatCake, a flat KV storage architecture that replaces the migratory hierarchy with peer storage media (§3). As shown in Figure 1(b), FlatCake exposes host DRAM and NVMe SSD at the same level behind a unified 64-bit Global Virtual Address (GVA) space, placing each KV block once at write time without cross-tier migration. FlatCake realizes this architecture through three coordinated components:

• FC-GVA: Global Virtual Address Manager (§3.2). FC-GVA decouples KV placement from eviction by binding each block to DRAM or SSD once at creation within a session-contiguous scope and encoding its medium, node, and offset in an immutable 64-bit GVA. This eliminates cross-tier migration, enables 𝑂 (1) media-aware routing without tier-by-tier lookup, and supports zero-writeback, session-granularity SSD reclamation. • FC-MAS: Medium-Aware Request Scheduler (§3.3). Using the media semantics embedded in each GVA, FC-MAS isolates fast DRAM-ready requests from long-latency SSD-preparing requests without DRAM pre-staging, schedules background SSD prefetches by size-aware response ratio, and re-admits prefetched requests by original arrival time to prevent both head-of-line blocking and starvation. • FC-CIO: Coalescing Direct I/O Engine (§3.4). FCCIO co-designs write-time layer-major scope layout with read-time request coalescing to collapse 𝑂 (layers×blocks) scattered page reads into 𝑂 (layers) sequential GPUDirect Storage transfers, streaming them through a bounded GPU HBM window overlapped with layer-wise computation to eliminate host staging contention.

We implement FlatCake in SGLang and evaluate it with DeepSeek-V4-Flash (284B hybrid MoE) on NVIDIA B200 GPUs and Llama-3.1-70B (dense GQA) on NVIDIA A100 GPUs (§5). Replaying a production Claude Code trace across increasing arrival rates, FlatCake sustains a 94.2–94.6% delivered cache hit rate, staying within 0.6 percentage points of the trace’s 94.7% prefix-reuse ceiling, whereas Mooncake’s hit rate drops from 87.9% at low load to 64.3% under staging contention. At peak load, FlatCake reduces mean and median TTFT by 1.6× and 1.8×, improves total token throughput by 1.6×, achieves 1.9× higher SSD read bandwidth with 5.8× lower prefetch latency, and eliminates Mooncake’s 6.5×–6.7× SSD read amplification. Across 8K–128K contexts on Llama-3.1-70B, FlatCake further reduces mean TTFT by 2.3×–4.9× over prefix recomputation, 1.5×–2.4× over Mooncake, and up to 1.8× over Mooncake-GDS.

FlatCake: A Flat KVCache Storage Architecture for LLM Serving

Table 1. KV cache reuse statistics across eight production LLM traces under an infinite-capacity cache. Unique

Once-

Requests

blocks

access

Mean

P99

rate

Mooncake Traces [27] Conversation 12,031 Synthetic 3,993 ToolAgent 23,608

182,790 43,924 183,300

75.8% 58.4% 78.5%

728 430 489

5,527 2,235 4,085

36.64% 63.96% 55.26%

Qwen Bailian Traces [38] to-C 43,058 to-B API 172,800 Thinking 10,812 Coder 43,011

2,530,337 3,830,390 1,735,015 5,202,517

53.2% 94.0% 85.7% 61.0%

1,450 2,550 155 1,095

13,973 54,728 2,357 16,224

59.4% 61.3% 46.2% 66.4%

Agentic Trace [33] Claude Code

5,717,185

16.6%

0.85

14

98.31%

Workload

98,827

Reuse dist

2

Background and Motivation

2.1

KV Cache Reuse and Tiered Storage

Hit

Once generated during prefill, KV cache [37] blocks are write-once and immutable, and their reuse across requests follows three primary patterns determined by how the shared prefix is formed. Prefix caching [46] reuses KV blocks across requests that share a static prefix, such as a system prompt or few-shot template, so only the uncached suffix requires prefill computation. Multi-turn conversations reuse KV blocks within a user session, where each turn appends a new prompt and model response to the accumulated history of prior turns. Agentic workloads also accumulate state within a session, but append tool invocations, retrieved files, and command outputs on every turn while spawning sub-agents that replay the shared session prefix in parallel. As a result, per-turn context grows by tens of thousands of tokens, and a single session’s KV footprint can reach tens of gigabytes. To accommodate working sets that exceed GPU HBM, existing systems such as Mooncake [27], LMCache [7], and SGLang HiCache [42] organize GPU HBM, host DRAM, and NVMe SSD as a three-level tiered hierarchy (Figure 1(a)). In this design, newly generated KV blocks are admitted into host DRAM first, evicted incrementally to NVMe SSD under LRU or FIFO policies when DRAM fills, and staged back through host DRAM buffers when restored to the GPU. This architecture relies on two premises: first, that host DRAM acts as an effective temporal filter that keeps the active working set in memory; and second, that staging and coalescing SSD reads in host DRAM hides on-disk fragmentation and device latency. 2.2

Why Tiered Storage Fails on LLM Workloads

Characterizing eight production traces from three providers (Table 1, evaluated at each trace’s native block size of 512, 16, and 64 tokens for Mooncake, Qwen Bailian, and Claude Code, with reuse distance measured in intervening

requests) shows that LLM KV workloads break both premises of tiered storage. Observation 1: Bimodal reuse defeats DRAM temporal filtering, putting SSD on the critical read path. Table 1 shows that ideal hit rates under an infinite-capacity cache span 36.6% to 98.3% across the eight traces. The traces fall into two distinct reuse regimes, and in neither regime can recency-based DRAM caching retain the active working set. Across the seven prefix-caching and conversation traces, 53.2–94.0% of blocks are accessed only once, peaking at 94.0% in Qwen to-B API and 78.5% in Mooncake ToolAgent. While the top 10% of blocks account for 34–65% of all accesses (Figure 2(a)), reuse distances exhibit heavy tails, reaching a mean of 2,550 intervening requests and a P99 of 54,728 in Qwen toB API (489 and 4,085 in Mooncake ToolAgent). When every newly generated block is admitted into host DRAM first, oneshot blocks continually flush reusable session prefixes to SSD before their next turn arrives. Consequently, replacing the DRAM eviction policy yields little benefit: even when the fast tier holds 40.1% of total blocks in multi-turn serving, LRU, FIFO, and queue-enhanced policies all achieve DRAM hit rates near 20% [12]. The agentic coding trace (Claude Code) exhibits the opposite reuse pattern but reaches the same storage outcome. Because each turn extends and replays the session prefix and sub-agent fan-out reuses prefixes almost immediately (mean reuse distance of 0.85 requests, P99 of 14), only 16.6% of blocks are one-shot and the infinite-capacity hit rate reaches 98.31%. However, each request spans an average of 3,421 blocks (218,922 tokens), compared with 23 blocks in Mooncake Conversation. Even under DeepSeek-V4-Flash’s compressed FP8 representation (Figure 2(b)), a single request requires a median of 0.52 GiB and a mean of 0.80 GiB of history KV state per GPU rank (4.1 GiB median and 6.4 GiB mean across eight ranks, reaching 16.0 GiB at P90). Across concurrent multi-turn sessions, the active working set rapidly exceeds host DRAM capacity. Thus, whether reuse is sparse with long reuse distances or dense across multi-gigabyte contexts, most cache hits must be served from NVMe SSD, making cross-tier migration between DRAM and SSD unproductive. Observation 2: DRAM staging and eviction-ordered layout bottleneck SSD restoration. Because most cache hits must be served from SSD (Observation 1), end-to-end prefill performance is governed by the SSD restoration path. Although storage I/O accounts for 51–98% of TTFT [6], raw NVMe bandwidth is not the primary limit: even when the SSD is emulated in DRAM at 16.9 GiB/s, KV offloading achieves only 2.0 GiB/s [31]. Instead, when SSD serves the bulk of hits, the tiered mechanisms meant to smooth SSD access become the bottleneck itself. First, staging SSD restores through host DRAM creates a concurrency bottleneck. In a tiered hierarchy, restoring an

Xin Huang, Teng Ma, Zhuoran Li, Yibo Zhao, Zhaoshi Li, Qianchao Zhu, Wei Yang, Chenxi Wang, Tao Xie, and Youwei Zhuo H2D Recompute

TTFT (s)

9

0.01

6 3

0.001

de Co

8K

16K

32K

64K

128K

Prefix length (tokens)

Cl

au

de

g

de r

kin

in Th

Co

AP I

-C

-B

to

to

nt

To o

Co

nv

er

sa t

Blocks by access rank (%)

lA ge

ic

0

100

(a) Trace access skew (Tab. 1)

SSD I/O Compute

128K ref.

0.1

n

50

CPU Ctrl GPU Stall

12

P50; P10–P90

1

io

0 10

Coder Claude Code (footprint)

he t

rm ifo un

50

0

FP8 hist./rank (GiB)

Accesses (%)

100

to-B API Thinking

nt

ToolAgent to-C

Sy

Conversation Synthetic

(b) DS-V4F KV footprint

(c) TTFT stage breakdown

Figure 2. Empirical bottlenecks of tiered KV cache storage: (a) cumulative block access skew across the seven non-agentic traces, (b) per-request FP8 history KV footprint per GPU rank across traces on DeepSeek-V4-Flash, and (c) TTFT stage breakdown of tiered SSD restoration versus full recomputation. SSD-resident prefix copies KV blocks from SSD into host staging buffers before transferring them layer-by-layer to GPU HBM, while background promotion separately populates the DRAM cache. Even when chunked pipelining overlaps the SSD → DRAM and DRAM → GPU hops to bound singlerequest transfer time near max(𝑇SSD→DRAM,𝑇DRAM→GPU ), the inference engine must hold host staging buffers for each in-flight prefix until H2D transfers complete. Staging memory demand therefore scales as 𝑂 (context × concurrency). Under concurrent load, competing writes and restores exhaust the bounded staging pool, delaying or aborting prefetches and forcing recomputation of KV blocks that are already stored on SSD (§5.2). Second, bypassing host DRAM via batched GPUDirect Storage (GDS) [3, 4, 7, 19, 29, 30] exposes eviction-ordered on-disk fragmentation directly to the drive. Because tiered storage populates and reclaims SSD space incrementally via DRAM eviction and coarse FIFO bucket recycling, blocks belonging to the same session prefix are scattered across non-contiguous SSD offsets, and bucket eviction invalidates active prefixes across sessions. Each NVMe read command requires a contiguous LBA range on disk, so restoring a scattered prefix issues 𝑂 (layers × blocks) small reads, explaining why a 3.3× increase in SSD bandwidth yields only a 1.33× throughput gain [12]. As prefix length grows, CPU submission overhead, fragmented SSD reads, and GPU stalls accumulate, pushing tiered restoration latency above the cost of full recomputation (Figure 2(c)). Takeaway. Tiered storage falls into this trap because it is designed for mutable pages whose future access patterns and spatial grouping are unknown when written. In contrast, LLM KV cache blocks are write-once, immutable, and reused together at the granularity of a session scope. Because a block is never modified in place and its session-prefix grouping

is known upon creation, placing blocks directly onto their target medium in session-contiguous, layer-major order at write time makes SSD restores naturally sequential and eliminates the need for host DRAM caching or reassembly.

3

The FlatCake Design

3.1

System Overview

Figure 3 gives an overview of FlatCake, which exposes host DRAM and NVMe SSD as peer storage media behind a unified 64-bit Global Virtual Address (GVA) space. In a traditional tiered hierarchy, the intermediate DRAM layer implicitly handles three tasks: selecting what spills to SSD via eviction, hiding SSD latency via pre-staging, and reassembling scattered SSD blocks in host buffers. Eliminating this intermediate layer removes cross-tier migration and staging contention, but requires FlatCake to answer three fundamental design questions along the request lifecycle (Figure 3): Q1: How to bind blocks to peer media at creation time with 𝑂 (1) routing?. On the write path, FlatCake’s Global Virtual Address Manager (FC-GVA, ❶, §3.2) assigns each newly generated KV block directly to DRAM or SSD once at creation time within a session-contiguous scope, encoding its medium, node, and byte offset in an immutable 64-bit GVA. On the read path, FC-GVA resolves block keys to GVAs in 𝑂 (1) time without tier-by-tier lookup and reclaims SSD space at session-extent granularity without writeback. Q2: How to schedule requests across heterogeneous media latencies without DRAM pre-staging?. On the read control path, FlatCake’s Medium-Aware Request Scheduler (FC-MAS, ❷, §3.3) inspects the media bits embedded in each GVA to decouple fast DRAM-ready requests from long-latency SSD-preparing requests. DRAM hits are

FlatCake: A Flat KVCache Storage Architecture for LLM Serving (a) Request control path

(b) Request data path

KV requests 1

3

key → 64-bit address write once, no migration 2

FC-CIO

SSD I/O coalescing

SSD read requests

FC-GVA

submit

Host DRAM (pinned)

FC-MAS

Table 2. FC-GVA write-time scope classification.

NVMe SSD

medium-aware scheduling cudaMemcpyAsync (H2D)

DRAM-ready

SSD-preparing

original ts

size + wait

GDS

Reuse Scope

Write-Time Signal

Target Medium

Shared prompt Short-turn chat Agentic / Long One-shot / Spill

Prefix refcount > 1 Turn > 1, prior reply < 𝜏 Context ≥ 𝐿agent or reply ≥ 𝜏 Turn = 1 or DRAM full

DRAM (m=0) DRAM (m=0) SSD (m=1) SSD (m=1)

GPU HBM Aligned GPU Staging

Ready Queue original ts

done

scatter

KV

KV

KV

KV

Dispatch to GPU KV data

Requests / completion

Figure 3. FlatCake architecture. (a) Control path: FCGVA maps block keys to 64-bit GVAs with write-once placement (❶), and FC-MAS separates DRAM-ready and SSD-preparing queues (❷). (b) Data path: DRAM hits use cudaMemcpyAsync, while FC-CIO streams coalesced SSD reads via GDS (❸) into a bounded GPU staging window.

dispatched immediately via cudaMemcpyAsync, while SSDbound requests are prefetched asynchronously under a sizeaware priority and re-admitted to the ready queue by their original arrival timestamps to prevent both head-of-line blocking and starvation. Q3: How to coalesce and stream SSD reads directly into a bounded GPU window?. On the read data path, FlatCake’s Coalescing Direct I/O Engine (FC-CIO, ❸, §3.4) co-designs write-time layer-major scope layout with read-time sort-merge coalescing to turn fragmented block requests into large sequential GPUDirect Storage (GDS) transfers, streaming them directly into a bounded GPU HBM window overlapped with layer-wise computation. 3.2

FC-GVA: Global Virtual Address Manager

In a tiered hierarchy, the storage layer does not need to decide a block’s long-term home at write time: every block enters host DRAM first, and LRU eviction reactively pushes cold blocks to SSD. However, as Observation 1 showed, this reactive filtering pollutes DRAM with one-shot blocks and scrambles session prefixes on disk. Eliminating cross-tier migration therefore raises Q1: Without reactive LRU eviction, how can FlatCake permanently bind KV blocks to DRAM or SSD at creation time and route accesses in 𝑂 (1) time? FC-GVA answers this question by exploiting two domain properties of LLM serving. First, reuse scope is observable at request arrival: whether a prefix belongs to a globally shared system prompt, a short-turn chat, or a multi-gigabyte agentic session is known when the blocks are generated. Second, flat storage exhibits asymmetric misprediction cost: because both DRAM and SSD are directly accessible peer media and SSD

scopes are laid out contiguously (§3.4), placing a reusable block on SSD does not cause a cache miss or cross-tier migration storm; it simply serves the hit via coalesced GDS at SSD bandwidth instead of DRAM bandwidth. Write-Once Scope-Direct Placement. Rather than funneling all writes through DRAM, FC-GVA binds each newly generated KV block directly to DRAM or SSD at creation time using write-time observable engine signals (Table 2). Scopes fall into two placement classes matching the bimodal reuse regimes in Observation 1. DRAM-bound scopes (MediaType=0) cover globally shared system prompts (identified by radix-tree reference count > 1) and short-reply interactive turns (session turn > 1 with prior output length < 𝜏 tokens [12]); because these prefixes have small footprints and short reuse distances, FC-GVA pins them in host DRAM. SSD-bound scopes (MediaType=1) cover multi-gigabyte agentic coding sessions (session context ≥ 𝐿agent ), long-reply turns whose expected reuse distance exceeds 𝜏 tokens, unshared single-turn API requests, and any writes arriving when DRAM is full. When a multi-turn or agentic session appends new tokens on a subsequent turn, FC-GVA binds the newly generated blocks to the same session scope so the session’s history remains grouped on its assigned medium. Across nodes, a lightweight coordinator tracks aggregate DRAM and SSD capacity and assigns GVAs on writes; because reads dominate KV traffic, capacity coordination stays off the critical read path. Unified 64-Bit Addressing and Routing. Because a block never migrates after creation, its physical location remains invariant throughout its lifetime and can be baked directly into its address handle. As shown in Figure 3(a), FC-GVA translates block keys in two steps without tier-by-tier lookup. At Level 1, a DRAM-resident hash table maps each block key (session ID, layer, and token range) to an immutable 64bit GVA in a single 𝑂 (1) lookup. To prevent read-after-write hazards without locking readers, FC-GVA allocates a block’s GVA before asynchronous write submission but publishes the key-to-GVA entry into the Level 1 index only after the DMA or GDS write callback completes. At Level 2, the 64-bit GVA encodes three fixed bit fields: MediaType [63:60] (4 bits: 0=DRAM, 1=SSD, with reserved encodings for future media), NodeID [59:48] (12 bits, up to 4,096 nodes), and Offset [47:0] (48-bit byte offset, up to 256 TiB per node).

Xin Huang, Teng Ma, Zhuoran Li, Yibo Zhao, Zhaoshi Li, Qianchao Zhu, Wei Yang, Chenxi Wang, Tao Xie, and Youwei Zhuo

Extracting MediaType and NodeID via register-level bit masks routes each access directly to its transport backend without per-tier page tables or lock-protected LRU chains: local DRAM via cudaMemcpyAsync, remote DRAM via RDMA, local SSD via GDS cuFile, and remote SSD via NVMe-oF. Placing MediaType in the most significant bits also partitions DRAM and SSD addresses into disjoint numeric ranges (GVA < 260 for DRAM vs. GVA ≥ 260 for SSD), allowing FC-MAS to classify a request’s medium with a single integer comparison. Session-Granularity SSD Reclamation. Write-once scope placement also simplifies storage reclamation. Because KV blocks are never modified in place, freeing SSD capacity requires zero dirty-block writeback. FC-GVA tracks active readers per session scope with an atomic reference count; when SSD utilization crosses a high watermark, FlatCake reclaims the oldest unreferenced session extents by returning their pre-allocated GPU Files to the free pool in 𝑂 (1) metadata operations. In contrast, tiered systems such as Mooncake allocate SSD pages incrementally across fixed-size buckets and recycle buckets in FIFO order, so evicting a single bucket slices across 𝑘 distinct sessions. If affected session 𝑖 has reaccess probability 𝑝𝑖 , a bucket eviction invalidates at least Î one active session prefix with probability 1 − 𝑘𝑖=1 (1 − 𝑝𝑖 ), which approaches certainty as 𝑘 grows and forces recomputation whenever an invalidated prefix is requested again. Reclaiming whole session extents eliminates cross-session prefix invalidation and preserves the contiguous on-disk layout required by FC-CIO (§3.4).

3.3

FC-MAS: Medium-Aware Request Scheduler

Tiered systems hide SSD latency from the inference scheduler by pre-staging every SSD hit into host DRAM before admission, which causes the 𝑂 (context × concurrency) host memory contention identified in Observation 2. Removing host DRAM pre-staging eliminates this memory bottleneck, but exposes a 5× bandwidth gap between local DRAM and NVMe SSD directly to the request scheduler (Table 3), raising Q2: How can FlatCake prevent slow SSD restores from stalling fast DRAM hits without starving long-context SSD sessions? Scheduling directly across peer media creates a two-sided dilemma. In a shared FCFS queue (Figure 4(a)), a multigigabyte SSD restore blocks subsequent DRAM-ready requests that need only a sub-millisecond cudaMemcpyAsync. Conversely, strict DRAM-first priority scheduling can starve large SSD-backed agentic sessions under continuous DRAM arrivals. FC-MAS resolves both hazards through dual-queue separation, size-aware SSD prefetching, and arrival-ordered re-admission (Figure 4(b)). Dual-Queue Separation. When a request arrives, FC-MAS stamps it with its arrival time (arrival_ts), resolves its prefix block GVAs (§3.2), and classifies the request by its slowest required medium. As illustrated in Figure 4(b), requests whose

Request Arrival Order

KV in DRAM

R1 R2 R3 R4 R5 R6

R2 R4 R6

KV in SSD R1 R3 R5

(a) FCFS SSD Read

R1

DRAM Read

R3 R2

R5 R4

R6 Time

(b) FC-MAS SSD Read (Preparing) DRAM Read (Ready)

R1

R3

R5

Earlier completion

R2 R4 R6 Time

Figure 4. Timeline comparison of (a) shared FCFS scheduling, which stalls fast DRAM hits (Requests 2, 4, 6) behind slow SSD restores (Requests 1, 3, 5), and (b) FC-MAS dualqueue scheduling, which dispatches DRAM-ready requests immediately while overlapping SSD prefetches in the background. cached blocks reside entirely in DRAM (max 𝑗 GVA 𝑗 < 260 , Requests 2, 4, and 6) enter the DRAM-Ready Queue and are dispatched immediately to the GPU. Requests that reference any SSD-resident blocks (max 𝑗 GVA 𝑗 ≥ 260 , Requests 1, 3, and 5, including hybrid requests that share a DRAM-pinned system prompt alongside an SSD-resident session history) enter the SSD-Preparing Queue. For these requests, FCMAS pins any DRAM-resident prefix blocks in place while FC-CIO asynchronously prefetches the SSD-resident blocks into GPU HBM, overlapping SSD I/O with GPU execution on DRAM-ready requests. Size-Aware Prefetching and Arrival-Ordered ReAdmission. FC-MAS enforces fairness and backpressure both within the SSD queue and across the two media queues. First, to prevent large agentic restores from blocking smaller SSD requests on the NVMe array without starving long-context sessions, the SSD-Preparing Queue orders pending prefetches using a size-aware highest-responseratio-next (HRRN) [5] priority that substitutes SSD KV transfer volume for service time: wait_time𝑖 Priority𝑖 = 1 + (1) KV_size𝑖 Smaller SSD restores receive higher initial priority to reduce mean queuing delay, while wait_time𝑖 grows linearly with waiting time so multi-gigabyte restores are guaranteed eventual dispatch. Second, once FC-CIO completes a request’s SSD prefetch, its KV blocks are resident in GPU HBM. Rather than appending the promoted request to the tail of the DRAMReady Queue, FC-MAS inserts it ordered by its original arrival timestamp (arrival_ts). Because the time spent in

FlatCake: A Flat KVCache Storage Architecture for LLM Serving

Read

GPU HBM KV Block

KV Block

KV Block

Paged and scattered, each block N tokens KV KV KV KV KV Block Block Block Block Block

Grouped by reuse scope, not LRU

GPU File 0 Layer -major kv layout L0

L1

…

L 2L-1

…

L2L-1

Preallocated LBA ranges

NVMe File Pool File 0

L1

File 1

Free

As the request arrives File 0: 0x00

SSD I/O cmd

File 0: 0x00

File 1: 0x00

SSD I/O cmd

File 0: 0x40

File 0: 0x40

SSD I/O cmd

File 0: 0x80

File 1: 0x40

SSD I/O cmd

File 1: 0x00

File 0: 0x80

SSD I/O cmd

5 SSD I/O Commands

Offload arena

Host buffer

GPU HBM

Re-read + staging

DRAM KV copy

SSD copy retained

L1

L2

L3

L1

L2

SSD → host

File 1: 0x40

SSD I/O cmd

Admit

Compute

L3

(ii) DRAM restore TE

H2D

Host buffer

DRAM → host

Read

After sorting by (file, offset)

Promo / admit

Same SSD copy

DRAM KV cache

(a) KV Cache Write Path: Layout Matching

Compute

SSD KV cache

Read

GPU File 1 L0

H2D / scatter

(i) SSD restore + promotion

Admit

L1

Compute

GPU HBM

L2

L3

L1

L2

L3

(iii) GDS SSD restore SSD KV cache

GPU buf. opt. scatter

GPU HBM

GDS

SSD I/O cmd

2 SSD I/O Commands

(b) KV Cache Read Path: Sort-Merge Coalescing I/O

L1

Read

L2

L3

S1

Scatter Compute

S2 L1

S3 L2

L3

(c) KV Cache Restore Paths and Layer Pipelines

Figure 5. Design and data path of FC-CIO: (a) write-time layer-major scope layout within a contiguous NVMe GPU File, (b) read-time sort-merge coalescing across adjacent block descriptors, and (c) restoration paths and layer pipelines: (i) SSD restoration with promotion, (ii) DRAM restoration, and (iii) direct GDS streaming into a bounded GPU HBM window. SSD prefetching already counts toward its queue seniority, the promoted request is dispatched ahead of newly arriving DRAM hits, preventing starvation across media. 3.4

FC-CIO: Coalescing Direct I/O Engine

Removing host DRAM staging eliminates host memory contention, but also removes the intermediate buffer where scattered SSD reads were previously reassembled, raising Q3: Without host DRAM buffers, how can FlatCake turn blocklevel prefix requests into sequential NVMe transfers and stream them into a bounded GPU HBM budget? As Observation 2 showed, simply enabling GPUDirect Storage (GDS) on a tiered storage backend fails because NVMe scatter-gather lists can scatter data into noncontiguous GPU pages only if the source bytes occupy a single contiguous LBA range on the SSD. When blocks are written in LRU eviction order, a session’s blocks are scattered across disk, forcing 𝑂 (layers × blocks) small reads. FC-CIO overcomes this hardware constraint by co-designing write-time disk layout, read-time descriptor merging, and layer-pipelined GPU streaming as a single producer-consumer pipeline (Figure 5). Write-Time Layer-Major Scope Layout. During prefill, attention at transformer layer 𝑙 consumes all cached prefix tokens for layer 𝑙 before execution advances to layer 𝑙+1. To match this consumption order on disk, when FC-GVA assigns a session reuse scope to SSD, FC-CIO groups its blocks into a

logical GPU File mapped one-to-one to a pre-allocated, singleextent file in the NVMe File Pool (Figure 5(a)). Within each turn’s write batch, FC-CIO lays out KV blocks in layer-major order so that all token blocks belonging to layer 𝑙 occupy one physically contiguous byte range on SSD, and subsequent turns of the same session append new layer-major segments to the same GPU File. Unlike incremental LRU eviction, which interleaves pages from concurrent sessions in eviction order, this write-time layout ensures that restoring a layer’s prefix targets contiguous LBA spans, reducing the ideal NVMe read count from 𝑂 (layers × blocks) to 𝑂 (layers). Read-Time Sort-Merge Coalescing. Even when blocks are contiguous on disk, the inference engine’s radix tree queries prefixes at page granularity, handing the storage backend hundreds of thousands of block descriptors that may span multiple turns or scopes and arrive out of offset order. To recover contiguous LBA ranges before issuing GDS calls, FC-CIO performs a lightweight sort-merge pass at the submission layer (Figure 5(b)). It first sorts block descriptors by (gpu_file_id, offset) to place physically adjacent blocks next to one another, and then scans linearly to coalesce adjacent ranges into single large read descriptors. When partial prefix hits or sub-block alignments leave small unrequested gaps between adjacent ranges, FC-CIO bridges gaps smaller than a configurable byte threshold (Δgap = 64 KiB), trading a bounded amount of extra sequential bytes to eliminate NVMe command split overhead. Merged descriptors are

Xin Huang, Teng Ma, Zhuoran Li, Yibo Zhao, Zhaoshi Li, Qianchao Zhu, Wei Yang, Chenxi Wang, Tao Xie, and Youwei Zhuo

striped round-robin across worker threads, each submitting batches via a private cuFileHandle, CUDA stream, and cuFileBatchIOSubmit. Layer-Pipelined Direct GPU Streaming. Because SGLang’s paged KV cache allocates 256-token HBM pages non-contiguously, issuing direct GDS reads into individual KV page slots would split each layer extent back into per-page DMA transfers. Conversely, staging an entire multi-gigabyte prefix in host DRAM causes 𝑂 (context × concurrency) memory contention. As shown in Figure 5(c)(iii), FC-CIO resolves both issues by streaming coalesced layer extents via GDS into a small, double-buffered GPU HBM staging window (sized to two transformer layers per active stream) before scattering blocks into paged HBM KV slots on the GPU: while the compute stream executes attention and FFN kernels for layer 𝑙, the I/O stream asynchronously executes cuFileBatchIOSubmit and a GPU scatter kernel for layer 𝑙+1. This pipeline bounds staging memory to a constant 𝑂 (1) HBM window per GPU. When concurrent restores saturate the window under heavy load, incoming prefetches wait on CUDA stream events rather than revoking in-flight prefetches and recomputing cached tokens (§5.2).

4

Implementation

We implement FlatCake in approximately 8.8K lines of C++17/CUDA and 7.1K lines of Python, integrating it into SGLang [47] behind the standard HiCacheStorage interface (exists, get, put) without modifying model runner or attention kernels. Pinned DRAM pool and hugepage allocation. At engine initialization, the local DRAM backend pre-allocates the host KV pool using mmap with MAP_POPULATE and 2 MB hugepages (MAP_HUGETLB), and registers the arena once with cudaHostRegister. Pre-faulting and pinning hugepages at startup eliminates page-fault traps, TLB thrashing, and pertransfer pinning overhead on the cudaMemcpyAsync path. Pre-allocated NVMe file pool and GDS alignment. To prevent filesystem metadata allocation and extent fragmentation on the write path, FC-CIO pre-allocates a pool of fixedsize single-extent files on the XFS/RAID-0 array at startup using fallocate and opens each file with O_DIRECT and cuFileHandleRegister. All logical block offsets within a GPU File and all GPU staging buffer slices are aligned to 4 KiB boundaries, satisfying libcufile DMA alignment requirements and preventing driver-internal bounce buffering. Double-buffered GPU staging and page scatter. Because calling cuFileBufRegister dynamically on per-request GPU buffers incurs millisecond-scale BAR1 and IOMMU mapping overhead, FC-CIO allocates and registers a fixed per-GPU HBM staging window once at initialization. Restores pipeline GDS reads across two ping-pong layer buffers within this registered window; as soon as a coalesced layer

Table 3. Hardware and transport specifications of the evaluation platform. Component

Configuration / Measured BW

GPU Interconnect CPU Host DRAM NVMe SSD SSD transport

8×NVIDIA B200 (180 GB HBM3e each) NVLink (18 links/GPU pair), PCIe Gen5 2×Xeon Platinum 8570 (112 cores, 4 NUMA) 3 TB (1 TiB KV); H2D 25–32 GB/s 8×3.84 TB PM9A3 RAID 0 (4 TiB KV) GDS cuFile, 5.5–6.5 GB/s per worker

read completes on the I/O CUDA stream, a lightweight CUDA scatter kernel copies each 256-token page from the contiguous staging buffer into its non-contiguous destination slot in SGLang’s paged KV pool, synchronizing with the model compute stream via cudaEventRecord and cudaStreamWaitEvent. Lock-free Level-1 GVA index. The Level 1 block-key-toGVA index is implemented as a concurrent open-addressed hash table using 64-bit atomic compare-and-swap (CAS) operations. Because GVAs are immutable once published and session reclamation frees entire scopes asynchronously, scheduler threads querying prefix hits execute wait-free reads without contending with background write-completion workers.

5

Evaluation

We evaluate FlatCake across a 284B-parameter hybrid MoE model on NVIDIA B200 GPUs and a 70B-parameter GQA model on NVIDIA A100 GPUs to answer three questions: • (E1) End-to-end scaling and hit-rate stability (§5.2): Does replacing tiered DRAM staging with flat peer storage prevent high-load cache hit-rate collapse, and how much does this improve TTFT and total token throughput as load scales? • (E2) Write-once peer placement and hit attribution (§5.3): Does FC-GVA’s write-once scope-direct placement offload the majority of reused prompt tokens to NVMe SSD while keeping DRAM unpolluted, increasing in-HBM turn reuse, and eliminating write amplification? • (E3) Scheduling, streaming, and cross-model generality (§5.3–§5.4): How much do FC-MAS and FCCIO reduce tail TTFT, SSD read amplification, and prefetch latency, and do these gains hold against direct-GDS baselines across model architectures and context lengths? 5.1

Experimental Setup

Testbed. Our primary evaluation runs on an 8-GPU NVIDIA B200 server with 3 TB of host DRAM and an 8-drive Samsung PM9A3 NVMe SSD RAID 0 array (Table 3). Every evaluated system is provisioned with an identical KV cache budget

FlatCake: A Flat KVCache Storage Architecture for LLM Serving GPU HBM + L2 Host DRAM

Source reuse reference Source reuse: 94.72%

10

12.24 7.57

5

0.1 0.25

0.5

0.75

1

18.39

15 10

11.45

5 0

6.26

0.1 0.25

0.5

0.75

100

98.3

80

61.6

60

43.2

40 20

1

0

0.1

0.25

0.5

0.75

1

100

Cache hit rate (%) ↑

17.51

Total tokens/s (10³) ↑

15

0

FlatCake

20

P50 TTFT (s) ↓

Mean TTFT (s) ↓

20

Mooncake

94.16% 75

64.26%

50

53.30%

25 0

0.1 0.25

0.5

0.75

1

QPS

QPS

QPS

QPS

(a) Mean TTFT

(b) P50 TTFT

(c) Total token throughput

(d) Cache hit rate

Figure 6. End-to-end serving performance under increasing request arrival rates on the Claude Code production trace (claude-code-weka-0621) with DeepSeek-V4-Flash-0731: (a) mean TTFT, (b) median (P50) TTFT, (c) total token throughput, and (d) delivered cache hit rate. of 1 TiB host DRAM and 4 TiB NVMe SSD. We build on SGLang v0.5.19, Mooncake v0.3.13, CUDA 13.1, and NVIDIA GPUDirect Storage (GDS) 1.16.1 (libcufile). To evaluate generality across model architectures and GPU generations, we also run a context-length sensitivity sweep with Llama3.1-70B (TP4) on a 4-GPU NVIDIA A100 server (§5.4). Model and workload. We serve DeepSeek-V4-Flash0731 [8], a 284B-parameter MoE model with 43 hybrid multi-head latent attention (MLA) layers and a 1Mtoken context window. The model uses an fp8_e4m3 KV cache with 256-token pages and a per-rank footprint of 7.5 KiB/token. We replay the production claude-codeweka-0621 agentic coding trace [33], comprising 98,827 requests and 21.6B input tokens across 393 multi-turn sessions (Table 1). Requests are replayed in recorded arrival order across arrival rates from 0.1 to 1.0 QPS without cache warm-up, following the SGLang HiCache benchmark methodology [42]. Baselines and metrics. We compare three systems under identical engine and storage budgets: • +L2 Host DRAM [42]: SGLang HiCache’s two-tier HBM–DRAM LRU cache without SSD offloading. • +L3 Mooncake [27]: A three-tier HBM–DRAM–SSD hierarchy where SSD-resident KV blocks are staged through host DRAM buffers before H2D transfer and promoted into the DRAM tier. • FlatCake: Our flat peer storage architecture combining FC-GVA, FC-MAS, and FC-CIO. In the Llama-3.1-70B context-length sweep, we additionally compare against Recompute-only (no KV offloading beyond GPU HBM) and Mooncake-GDS (which replaces Mooncake’s host-staged SSD read path with GPUDirect Storage while retaining tiered eviction-ordered SSD placement). We report mean and median (P50) Time-to-FirstToken (TTFT), total token throughput (tokens/s), delivered

cache hit rate (counting only blocks transferred to GPU HBM before prefill), SSD read bandwidth, SSD prefetch latency, and normalized SSD read/write amplification. 5.2

Overall Performance

To answer E1, Figure 6 compares end-to-end performance across request arrival rates on the Claude Code production trace. Time to First Token. As shown in Figures 6(a) and 6(b), FlatCake achieves the lowest mean and median TTFT across the entire load range. At the peak arrival rate of 1.0 QPS, FlatCake achieves a mean TTFT of 7.57 s, reducing latency by 1.6× over +L3 Mooncake (12.24 s) and 2.3× over +L2 Host DRAM (17.51 s). Median (P50) TTFT exhibits an even sharper separation under load: at 1.0 QPS, FlatCake achieves a P50 TTFT of 6.26 s, reducing median latency by 1.8× over +L3 Mooncake (11.45 s) and 2.9× over +L2 Host DRAM (18.39 s). Total Token Throughput. Figure 6(c) shows that all configurations track the offered load up to 0.25 QPS, beyond which tiered storage saturates. At 1.0 QPS, FlatCake sustains 98.3K tokens/s, outperforming +L3 Mooncake (61.6K tokens/s) by 1.6× and +L2 Host DRAM (43.2K tokens/s) by 2.3×. Why Tiered Storage Degrades: Delivered Cache Hit Rate. Figure 6(d) reveals the root cause of the latency and throughput gap. Across all arrival rates, FlatCake sustains a delivered cache hit rate of 94.16–94.57%, staying within 0.6 percentage points of the trace’s 94.72% prefix-reuse ceiling. In contrast, +L2 Host DRAM plateaus at 53.30% across all rates because multi-gigabyte agentic sessions exceed host DRAM capacity (Observation 1). More importantly, while +L3 Mooncake achieves an 87.9% hit rate at low load (0.1 QPS), its delivered hit rate drops to 64.26% at 0.5 QPS and above, recovering only 11 percentage points over DRAM-only caching despite having 4 TiB of NVMe SSD capacity.

Xin Huang, Teng Ma, Zhuoran Li, Yibo Zhao, Zhaoshi Li, Qianchao Zhu, Wei Yang, Chenxi Wang, Tao Xie, and Youwei Zhuo

94.6%

87.9%

Mooncake

Storage (L3/SSD)

94.2%

80 60 51.7% 38.1%

64.3% 22.4% 20.0% 18.4%

69.3% 65.0%

40 20 0

23.0%

8.5% 8.2% 8.0%

23.4% 27.8% 28.4% 30.5% 11.7%

10.8%

13.3% 16.6% 15.0% 15.9% 15.4%

13.6% 18.6%

0.1

0.1

0.25

0.5

0.75

1

55.2% 52.8% 52.6%

0.25

6

0.75

QPS

QPS

(a) Mooncake

(b) FlatCake

5.03 2.69

2

0.1 0.25

0.5

5.3

Effectiveness of Individual Mechanisms

We now evaluate E2 and E3 on DeepSeek-V4-Flash by breaking down how FC-GVA, FC-MAS, and FC-CIO contribute to end-to-end performance through per-medium hit attribution (Figure 7), SSD bandwidth and prefetch latency (Figure 8), and read/write amplification (Figure 9). FC-GVA: Per-medium hit attribution (E2). To answer E2, Figure 7 decomposes total delivered hits across GPU HBM, Host DRAM, and NVMe SSD. Three trends stand out. First, in FlatCake (Figure 7(b)), NVMe SSD supplies the majority of reused prompt tokens across the entire load range (69.3% at 0.1 QPS and 52.6% at 1.0 QPS), while Host DRAM serves a steady 8.0–11.7% share corresponding to short-horizon prefixes. This confirms that FC-GVA’s write-once scopedirect placement successfully offloads multi-gigabyte session histories straight to SSD while keeping DRAM unpolluted for small, high-turnover scopes. Second, as arrival rate rises from 0.1 to 1.0 QPS, FlatCake’s GPU HBM hit share grows

5 1.64

0.75

0.5

1

0.1 0.25

0.5

0.75

1

QPS

QPS

(a) SSD bandwidth

(b) Prefetch latency

Figure 8. SSD I/O efficiency across request arrival rates on DeepSeek-V4-Flash: (a) SSD read bandwidth per worker and (b) mean SSD prefetch latency.

Mooncake, cold Mooncake, retained

Amplification

FlatCake, cold FlatCake, retained

6.592×

7.50

This hit-rate collapse under +L3 Mooncake stems primarily from host staging contention (Observation 2) and coarse bucket reclamation (§3.2). Because a tiered restore must stage an entire in-flight prefix in host DRAM buffers before H2D transfer completes, concurrent long-context restores exhaust the engine’s bounded in-flight staging budget as QPS rises. When staging memory saturates, SGLang revokes the blocked prefetch and falls back to recomputing the prefix on the GPU, turning cached SSD blocks into expensive compute misses. In contrast, FlatCake streams coalesced SSD reads directly into a fixed-size GPU HBM staging window (§3.4); when the window is fully occupied under high load, incoming restores wait briefly on stream synchronization rather than aborting prefetches, preserving a 94.16% hit rate at peak load.

9.50

10

1 0

1

Figure 7. Delivered cache hit rate broken down by serving medium (GPU HBM, Host DRAM, and NVMe SSD) across request arrival rates for (a) +L3 Mooncake and (b) FlatCake.

20

4

30.6% 33.1% 33.6%

0.5

FlatCake

Latency (s)

Hit Rate Contribution (%)

100

Host DRAM

Peak read BW (GB/s)

GPU HBM

1.016 5.00

1.007×

1.008 2.50

1.000×

1.000×

1.000 0.00

0.1 0.25

0.5

0.75

1

0.1 0.25

0.5

0.75

1

QPS

QPS

(a) Read amplification

(b) Write amplification

Figure 9. Normalized SSD (a) read amplification and (b) write amplification across request arrival rates on DeepSeek-V4Flash (normalized to FlatCake = 1.0× under both cold-start and retained-cache states).

from 13.6% to 33.6%: because coalesced SSD restores complete quickly (§5.3), closely spaced turns and sub-agent fan-outs within an active session increasingly find their prefix still resident in GPU HBM, keeping total delivered hits at 94.2– 94.6%. Third, in +L3 Mooncake (Figure 7(a)), the storage (L3) contribution collapses by 2.8× from 51.7% at 0.1 QPS to 18.4% at 1.0 QPS as concurrent restores exhaust host staging buffers and are revoked, while long restore stalls prevent in-HBM turn overlap (keeping GPU HBM hits flat at 13.3–16.6%) and pull total delivered hits down from 87.9% to 64.3%. FC-GVA: Eliminating migration and write amplification (E2). Because FC-GVA binds each block to its target medium once at creation time and reclaims SSD capacity at session-extent granularity (§3.2), each SSD-admitted block is written exactly once (1.000× in Figure 9(b)). In contrast, in a tiered hierarchy where coarse FIFO bucket reclamation invalidates active prefixes across 𝑘 sessions and revoked prefetches trigger recomputation, recomputed prefixes must be re-persisted to storage, incurring 1.007×–1.018× normalized

FlatCake: A Flat KVCache Storage Architecture for LLM Serving

5.4

Cross-Model and Context-Length Scaling

To complete E3, we evaluate whether FlatCake’s layout and streaming gains generalize from hybrid compressed attention (DeepSeek-V4-Flash on B200) to standard groupedquery attention (GQA) and isolate FC-CIO’s layout coalescing from the GDS transport primitive itself. Because FC-CIO groups KV blocks by session scope in layer-major order (§3.4), each layer’s state forms a contiguous SSD extent regardless of per-layer tensor dimensions. Figure 10 evaluates Llama-3.1-70B (TP4 on four A100 GPUs) across input lengths from 8K to 128K tokens, comparing FlatCake against Recompute-only, +L2 Host DRAM, Mooncake, and Mooncake-GDS. Across all context lengths, FlatCake achieves the lowest mean TTFT and highest throughput, reducing mean TTFT by 2.3×–4.9× over Recompute-only and

Recompute-only +L2 Host DRAM

Mooncake Mooncake-GDS

FlatCake

Throughput (tok/s)

60

TTFT (s)

write amplification (Figure 9(b)) while disrupting on-disk contiguity. FC-MAS: Eliminating cross-medium head-of-line blocking (E3). To answer the first part of E3, we examine how FC-MAS handles the 5× bandwidth gap between local DRAM and SSD (Table 3). Comparing FlatCake against an ablation without dual-queue scheduling (FlatCake-w/oFC-MAS, which processes all requests in a single FCFS queue while retaining FC-GVA and FC-CIO) isolates the contribution of FC-MAS. Separating the DRAM-Ready Queue from the SSD-Preparing Queue and scheduling SSD prefetches via Eq. 1 reduces mean TTFT by 10–11% and P99 tail TTFT by 27%, while improving total token throughput by 12%. The larger improvement at P99 confirms that FC-MAS primarily eliminates cross-medium head-of-line blocking for fast DRAM hits that would otherwise stall behind multi-gigabyte SSD restores, while original-timestamp (arrival_ts) re-admission prevents DRAM bursts from starving SSD-bound sessions. FC-CIO: SSD bandwidth, prefetch latency, and read amplification (E3). To answer the second part of E3, Figure 8(a) shows that FC-CIO sustains 4.96–5.12 GB/s of SSD read bandwidth per worker across arrival rates (5.03 GB/s at 1.0 QPS), outperforming +L3 Mooncake (2.69 GB/s) by 1.9×. As a result, when request arrival rate increases by 10× from 0.1 to 1.0 QPS (Figure 8(b)), FlatCake’s mean SSD prefetch time remains nearly flat at 1.47–1.75 s (1.64 s at 1.0 QPS), whereas Mooncake’s SSD restoration latency climbs by 3.0×, from 3.16 s to 9.50 s (5.8× higher than FlatCake at peak load). Figure 9(a) explains why: across both cold-start and retained-cache states, Mooncake incurs 6.51×–6.72× higher SSD read amplification than FlatCake because evictionscattered blocks, background promotions into DRAM, and revoked prefetches consume NVMe read traffic without delivering completed restores to the GPU. By combining writetime layer-major scope layout with read-time sort-merge coalescing, FC-CIO converts sequential NVMe transfers directly into completed GPU restores.

40

20

0

8K

16K

32K

64K

128K

20K 15K 10K 5K 0K

8K

16K

32K

64K

Input length

Input length

(a) Mean TTFT

(b) Throughput

128K

Figure 10. Performance on Llama-3.1-70B (TP4 on 4×A100 GPUs) across input context lengths from 8K to 128K tokens: (a) mean TTFT and (b) token throughput.

1.5×–2.4× over Mooncake. Crucially, while Mooncake-GDS matches FlatCake at 16K tokens (2.31 s vs. 2.32 s), its TTFT degrades to 6.60 s at 64K (1.8× slower than FlatCake’s 3.74 s) and its throughput drops to 8.6K tokens/s (1.6× lower than FlatCake’s 14.0K tokens/s). This divergence confirms that enabling GDS alone is insufficient when blocks are written in LRU eviction order: FlatCake’s write-once session-scope placement (FC-GVA) and layer-major sort-merge coalescing (FC-CIO) are essential to sustain sequential SSD transfers on long contexts.

6

Related Work

Prior work has explored paged KV cache management [16, 26], prefix caching [27, 47], KV compression [22], and prefilldecode disaggregation [25]. We focus on KV cache placement, scheduling, and I/O across heterogeneous storage media. Tiered KV Cache Storage and Offloading. Existing LLM serving systems extend HBM capacity by offloading KV blocks to host DRAM and NVMe SSD through a tiered hierarchy [7, 11, 13, 14, 27, 34, 42, 45], building on generalpurpose tiered memory designs [1, 2, 9, 15, 17, 32, 35, 36, 39, 41, 43, 44]. Recent studies characterize production KV reuse [38] and improve tiered hierarchies via workloadaware eviction (Bidaw [12]), selective token or layer offloading [6, 18, 20, 49], joint compression and tier placement (AdaptCache [10]), or SSD-aware tier coordination (SolidAttention [48], KVDrive [21]). Despite these optimizations, all of these systems retain a migratory hierarchy in which blocks enter host DRAM first and reach SSD through incremental eviction or demotion. In contrast, FlatCake exploits the write-once, session-scoped immutability of KV blocks to bind each block directly to DRAM or SSD at creation time (FC-GVA) and never migrate it thereafter. Direct Storage I/O and Request Scheduling. Several systems accelerate the SSD-to-GPU read path using GPUDirect

Xin Huang, Teng Ma, Zhuoran Li, Yibo Zhao, Zhaoshi Li, Qianchao Zhu, Wei Yang, Chenxi Wang, Tao Xie, and Youwei Zhuo

Storage [3, 4, 19, 29, 30], pre-registered GPU buffers with parallel disk workers (LMCache [7]), GPU-initiated io_uring queues (Tutti [28]), or storage-near execution [23, 24]. However, optimizing the transport primitive alone does not resolve on-disk fragmentation: when SSD space is populated in LRU eviction order, a session’s prefix remains scattered across non-contiguous LBAs, preventing NVMe controllers from coalescing reads into single DMA transfers (Observation 2). FlatCake co-designs write-time layer-major scope layout within pre-allocated GPU Files and read-time sortmerge coalescing (FC-CIO) so that standard GDS transfers read sequential per-layer extents. Furthermore, unlike devicelevel I/O schedulers [40], FC-MAS uses the media identity encoded in each 64-bit GVA to decouple DRAM-ready and SSD-preparing queues while preserving arrival-order fairness across media.

7

Conclusion

We presented FlatCake, a flat KV storage architecture that replaces the migratory DRAM–SSD hierarchy with peer storage media behind a unified 64-bit address space. Exploiting the insight that LLM KV cache blocks are writeonce, immutable, and reused at session scope, FlatCake combines FC-GVA, FC-MAS, and FC-CIO to eliminate crosstier migration and host staging contention. On a production Claude Code trace with DeepSeek-V4-Flash, FlatCake sustains a 94% delivered cache hit rate under load, eliminates Mooncake’s 6.5×–6.7× SSD read amplification, and reduces peak-load mean TTFT by 1.6× while improving total token throughput by 1.6×. Across 8K–128K contexts on Llama-3.170B, FlatCake further reduces mean TTFT by up to 2.4× over Mooncake and 1.8× over Mooncake-GDS.

References [1] Ahmed H. M. O. Abulila, Vikram Sharma Mailthody, Zaid Qureshi, Jian Huang, Nam Sung Kim, Jinjun Xiong, and Wen-Mei W. Hwu. 2019. FlatFlash: Exploiting the Byte-Accessibility of SSDs within a Unified Memory-Storage Hierarchy. In Proceedings of the 24th International Conference on Architectural Support for Programming Languages and Operating Systems. Association for Computing Machinery, Providence, RI, USA, 971–985. doi:10.1145/3297858.3304061 [2] Hasan Al Maruf, Hao Wang, Abhishek Dhanotia, Johannes Weiner, Niket Agarwal, Pallab Bhattacharya, Chris Petersen, Mosharaf Chowdhury, Shobhit O. Kanaujia, and Prakash Chauhan. 2023. TPP: Transparent Page Placement for CXL-Enabled Tiered-Memory. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3. Association for Computing Machinery, Vancouver, BC, Canada, 742–755. doi:10.1145/3582016.3582063 [3] Jonghyun Bae, Jongsung Lee, Yunho Jin, Sam Son, Shine Kim, Hakbeom Jang, Tae Jun Ham, and Jae W. Lee. 2021. FlashNeuron: SSD-Enabled Large-Batch Training of Very Deep Neural Networks. In 19th USENIX Conference on File and Storage Technologies (FAST 21). USENIX Association, Santa Clara, CA, USA, 387–401. https: //www.usenix.org/conference/fast21/presentation/bae [4] Shai Bergman, Tanya Brokhman, Tzachi Cohen, and Mark Silberstein. 2017. SPIN: Seamless Operating System Integration of Peer-to-Peer

DMA Between SSDs and GPUs. In 2017 USENIX Annual Technical Conference (USENIX ATC 17). USENIX Association, Santa Clara, CA, USA, 167–179. https://www.usenix.org/conference/atc17/technicalsessions/presentation/bergman [5] Per Brinch Hansen. 1971. Short-term Scheduling in Multiprogramming Systems. In Proceedings of the 3rd ACM Symposium on Operating Systems Principles (SOSP ’71). Association for Computing Machinery, Palo Alto, CA, USA, 103–105. [6] Weijian Chen, Shuibing He, Haoyang Qu, Ruidong Zhang, Siling Yang, Ping Chen, Yi Zheng, Baoxing Huai, and Gang Chen. 2025. IMPRESS: An Importance-Informed Multi-Tier Prefix KV Storage System for Large Language Model Inference. In Proceedings of the 23rd USENIX Conference on File and Storage Technologies (FAST). USENIX Association, Santa Clara, CA, USA, 187–201. https://www.usenix.org/ conference/fast25/presentation/chen-weijian-impress [7] Cong Cheng et al. 2025. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference. arXiv preprint. arXiv:2510.09665 [8] DeepSeek-AI. 2026. DeepSeek-V4: Towards Highly Efficient MillionToken Context Intelligence. arXiv preprint. arXiv:2606.19348 §3.5.2 “On-Disk KV Cache Storage”. [9] Padmapriya Duraisamy, Wei Xu, Scott Hare, Ravi Rajwar, David E. Culler, Zhiyi Xu, Jianing Fan, Christopher Kennelly, Bill McCloskey, Danijela Mijailovic, Brian Morris, Chiranjit Mukherjee, Jingliang Ren, Greg Thelen, Paul Turner, Carlos Villavieja, Parthasarathy Ranganathan, and Amin Vahdat. 2023. Towards an Adaptable Systems Architecture for Memory Tiering at Warehouse-Scale. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3. Association for Computing Machinery, Vancouver, BC, Canada, 727–741. doi:10.1145/3582016.3582031 [10] Shaoting Feng, Hanchen Li, Kuntai Du, Zhuohan Gu, Yuhan Liu, Jiayi Yao, Siddhant Ray, Samuel Shen, Yihua Cheng, Ganesh Ananthanarayanan, and Junchen Jiang. 2025. AdaptCache: KV Cache Native Storage Hierarchy for Low-Delay and High-Quality Language Model Serving. ACM SOSP BigMem Workshop. https://www.microsoft.com/en-us/research/publication/adaptcachekv-cache-native-storage-hierarchy-for-low-delay-and-highquality-language-model-serving/ Non-archival workshop paper. [11] Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. 2024. CostEfficient Large Language Model Serving for Multi-turn Conversations with CachedAttention. In 2024 USENIX Annual Technical Conference (USENIX ATC 24). USENIX Association, Santa Clara, CA, USA, 111–126. https://www.usenix.org/conference/atc24/presentation/gao-bin-cost [12] Shipeng Hu, Guangyan Zhang, Yuqi Zhou, Yaya Wei, Ziyan Zhong, and Jike Chen. 2026. Bidaw: Enhancing Key-Value Caching for Interactive LLM Serving via Bidirectional Computation–Storage Awareness. In Proceedings of the 24th USENIX Conference on File and Storage Technologies (FAST). USENIX Association, Santa Clara, CA, USA, 101–116. [13] Inho Jeong, Sunghyeon Woo, Sol Namkung, and Dongsuk Jeon. 2025. HiFC: High-efficiency Flash-based KV Cache Swapping for Scaling LLM Inference. In Advances in Neural Information Processing Systems, Vol. 38. Curran Associates, Inc., San Diego, CA, USA, 47561–47590. doi:10.52202/085713-1587 [14] Jinwoo Jeong and Jeongseob Ahn. 2025. Accelerating LLM Serving for Multi-turn Dialogues with Efficient Resource Management. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. Association for Computing Machinery, Rotterdam, Netherlands, 1–15. doi:10.1145/3676641.3716245 [15] Jaehoon Jung, Jinpyo Kim, and Jaejin Lee. 2023. DeepUM: Tensor Migration and Prefetching in Unified Memory. In Proceedings of the 28th ACM International Conference on Architectural Support for

FlatCake: A Flat KVCache Storage Architecture for LLM Serving

Programming Languages and Operating Systems, Volume 2. Association for Computing Machinery, Vancouver, BC, Canada, 207–221. doi:10.1145/3575693.3575736 [16] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP). ACM, Koblenz, Germany, 611– 626. [17] Taehyung Lee, Sumit Kumar Monga, Changwoo Min, and Young Ik Eom. 2023. MEMTIS: Efficient Memory Tiering with Dynamic Page Classification and Page Size Determination. In Proceedings of the 29th Symposium on Operating Systems Principles. Association for Computing Machinery, Koblenz, Germany, 17–34. doi:10.1145/3600006. 3613167 [18] Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. 2024. InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management. In Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI). USENIX Association, Santa Clara, CA, USA, 155–172. https: //www.usenix.org/conference/osdi24/presentation/lee [19] Shaobo Li, Yirui Eric Zhou, Yuqi Xue, Yuan Xu, and Jian Huang. 2025. Managing Scalable Direct Storage Accesses for GPUs with GoFS. In Proceedings of the 31st ACM SIGOPS Symposium on Operating Systems Principles (SOSP). ACM, Seoul, Republic of Korea, 979–995. [20] Yongkun Li, Zhen Liu, Patrick P. C. Lee, Jiayu Wu, Yinlong Xu, Yi Wu, Liu Tang, Qi Liu, and Qiu Cui. 2021. Differentiated Key-Value Storage Management for Balanced I/O Performance. In Proceedings of the 2021 USENIX Annual Technical Conference (ATC). USENIX Association, Virtual, 673–687. https://www.usenix.org/conference/atc21/ presentation/li-yongkun [21] Jian Lin, Jiazhi Mi, Zicong Hong, Haodong Wang, Qianli Liu, Haodyue Zhang, Peng Li, and Song Guo. 2026. KVDrive: A Holistic MultiTier KV Cache Management System for Long-Context LLM Inference. arXiv preprint. arXiv:2605.18071 doi:10.48550/arXiv.2605.18071 [22] Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, and Junchen Jiang. 2024. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving. In Proceedings of the ACM SIGCOMM 2024 Conference. Association for Computing Machinery, Sydney, NSW, Australia, 38–56. doi:10.1145/3651890.3672274 [23] Shutian Luo and Haiying Shen. 2026. No Buffer, No Bottleneck: Efficient Zero-Copy KV Cache Offloading for Long-Context LLMs. In 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 26). USENIX Association, Seattle, WA, USA, 39–54. https://www.usenix.org/conference/osdi26/presentation/luo [24] Xiurui Pan, Endian Li, Qiao Li, Shengwen Liang, Yizhou Shan, Ke Zhou, Yingwei Luo, Xiaolin Wang, and Jie Zhang. 2025. InstAttention: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference. In 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, Las Vegas, NV, USA, 1510–1525. doi:10.1109/HPCA61900.2025.00113 [25] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, Buenos Aires, Argentina, 118–132. doi:10.1109/ISCA59077.2024. 00019 [26] Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ramjee, and Ashish Panwar. 2025. vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention. In Proceedings of the 30th ACM International Conference on Architectural Support for

Programming Languages and Operating Systems, Volume 1. Association for Computing Machinery, Rotterdam, Netherlands, 1133–1150. doi:10.1145/3669940.3707256 [27] Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading More Storage for Less Computation—A KVCache-centric Architecture for Serving LLM Chatbot. In Proceedings of the 23rd USENIX Conference on File and Storage Technologies (FAST). USENIX Association, Santa Clara, CA, USA, 155–170. https://www.usenix.org/ conference/fast25/presentation/qin [28] Shi Qiu, Yifan Hu, Xintao Wang, Wenhao Zhu, Jianqin Yan, Hao Chen, Kaiqiang Xu, Kai Chen, and Yiming Zhang. 2025. Tutti: Making SSDBacked KV Cache Practical for Long-Context LLM Serving. arXiv preprint. arXiv:2605.03375 [29] Shi Qiu, Weinan Liu, Yifan Hu, Jianqin Yan, Zhirong Shen, Xin Yao, Renhai Chen, Gong Zhang, and Yiming Zhang. 2025. GeminiFS: A Companion File System for GPUs. In 23rd USENIX Conference on File and Storage Technologies (FAST 25). USENIX Association, Santa Clara, CA, USA, 221–236. https://www.usenix.org/conference/fast25/ presentation/qiu [30] Zaid Qureshi, Vikram Sharma Mailthody, Isaac Gelado, Seung Won Min, Amna Masood, Jeongmin Park, Jinjun Xiong, C. J. Newburn, Dmitri Vainbrand, I-Hsin Chung, Michael Garland, William J. Dally, and Wen-Mei W. Hwu. 2023. GPU-Initiated On-Demand HighThroughput Storage Access in the BaM System Architecture. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. Association for Computing Machinery, Vancouver, BC, Canada, 325–339. doi:10.1145/3575693.3575748 [31] Zebin Ren, Christian Pinto, Krijn Doekemeijer, Tiziano De Matteis, Radu Stoica, and Animesh Trivedi. 2025. An I/O Characterizing Study of Offloading LLM Models and KV Caches to NVMe SSD. In Proceedings of the 5th Workshop on Challenges and Opportunities of Efficient and Performant Storage Systems (CHEOPS). ACM, Rotterdam, Netherlands, 23–33. [32] Zhenyuan Ruan, Malte Schwarzkopf, Marcos K. Aguilera, and Adam Belay. 2020. AIFM: High-Performance, Application-Integrated Far Memory. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20). USENIX Association, Virtual, 315–332. https://www.usenix.org/conference/osdi20/presentation/ruan [33] SemiAnalysis. 2026. CC Traces — Weka, With Subagents (v7). https://huggingface.co/datasets/semianalysisai/cc-traces-weka062126. Claude Code agentic coding traces; accessed 2026-07-28. [34] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU. In Proceedings of the 40th International Conference on Machine Learning (ICML). PMLR, Honolulu, HI, USA, 31094–31116. [35] Hyunkyun Shin, Seongtae Bang, Hyungwon Park, and Daehoon Kim. 2026. ARIADNE: Adaptive UVM Management for Efficient GPU Memory Oversubscription. In 2026 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, Sydney, Australia, 1–15. doi:10.1109/HPCA68181.2026.11408564 [36] Yan Sun, Jongyul Kim, Zeduo Yu, Jiyuan Zhang, Siyuan Chai, Michael Jaemin Kim, Hwayong Nam, Jaehyun Park, Eojin Na, Yifan Yuan, Ren Wang, Jung Ho Ahn, Tianyin Xu, and Nam Sung Kim. 2025. M5: Mastering Page Migration and Memory Management for CXLbased Tiered Memory Systems. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. Association for Computing Machinery, Rotterdam, Netherlands, 604–621. doi:10.1145/3676641.3711999 [37] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.

Xin Huang, Teng Ma, Zhuoran Li, Yibo Zhao, Zhaoshi Li, Qianchao Zhu, Wei Yang, Chenxi Wang, Tao Xie, and Youwei Zhuo

Attention Is All You Need. In Advances in Neural Information Processing Systems 30 (NIPS 2017). Curran Associates, Inc., Long Beach, CA, USA, 5998–6008. [38] Jiahao Wang, Jinbo Han, Xingda Wei, Sijie Shen, Dingyan Zhang, Chenguang Fang, Rong Chen, Wenyuan Yu, and Haibo Chen. 2025. KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider. In Proceedings of the 2025 USENIX Annual Technical Conference (ATC). USENIX Association, Boston, MA, 465–482. [39] Johannes Weiner, Niket Agarwal, Dan Schatzberg, Leon Yang, Hao Wang, Blaise Sanouillet, Bikash Sharma, Tejun Heo, Mayank Jain, Chunqiang Tang, and Dimitrios Skarlatos. 2022. TMO: Transparent Memory Offloading in Datacenters. In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems. Association for Computing Machinery, Lausanne, Switzerland, 609–621. doi:10.1145/3503222.3507731 [40] Jiwon Woo, Minwoo Ahn, Gyusun Lee, and Jinkyu Jeong. 2021. D2FQ: Device-Direct Fair Queueing for NVMe SSDs. In 19th USENIX Conference on File and Storage Technologies (FAST 21). USENIX Association, Santa Clara, CA, USA, 403–415. https://www.usenix.org/conference/ fast21/presentation/woo [41] Lingfeng Xiang, Zhen Lin, Weishu Deng, Hui Lu, Jia Rao, Yifan Yuan, and Ren Wang. 2024. NOMAD: Non-Exclusive Memory Tiering via Transactional Page Migration. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, USA, 19–35. https://www.usenix.org/conference/ osdi24/presentation/xiang [42] Zhiqiang Xie and SGLang Team. 2025. SGLang HiCache: Fast Hierarchical KV Caching with Your Favorite Storage Backends. https: //lmsys.org/blog/2025-09-10-sglang-hicache/. [43] Zhe Yang, Qing Wang, Xiaojian Liao, Youyou Lu, Keji Huang, and Jiwu Shu. 2024. TeRM: Extending RDMA-Attached Memory with SSD. In 22nd USENIX Conference on File and Storage Technologies (FAST 24). USENIX Association, Santa Clara, CA, USA, 1–16. https://www. usenix.org/conference/fast24/presentation/yang-zhe [44] Haoyang Zhang, Yirui Eric Zhou, Yuqi Xue, Yiqi Liu, and Jian Huang. 2023. G10: Enabling an Efficient Unified GPU Memory and Storage Architecture with Smart Tensor Migrations. In Proceedings of the 56th Annual IEEE/ACM International Symposium on Microarchitecture. Association for Computing Machinery, Toronto, ON, Canada, 395–410. doi:10.1145/3613424.3614309 [45] Youpeng Zhao, Di Wu, and Jun Wang. 2024. ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Caching. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, Buenos Aires, Argentina, 1005–1017. doi:10. 1109/ISCA59077.2024.00077 [46] Lianmin Zheng et al. 2024. RadixAttention: Efficient Prefix Caching for Large Language Model Serving. SGLang Technical Report, LMSYS. The mechanism is introduced in the SGLang paper. [47] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kober, Joseph E. Gonzalez, Clark Barrett, et al. 2024. SGLang: Efficient Execution of Structured Language Model Programs. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS, Vancouver, Canada, 62557–62583. [48] Xinrui Zheng, Dongliang Wei, Jianxiang Gao, Yixin Song, Zeyu Mi, and Haibo Chen. 2026. SolidAttention: Low-Latency SSD-based Serving on Memory-Constrained PCs. In 24th USENIX Conference on File and Storage Technologies (FAST 26). USENIX Association, Santa Clara, CA, USA, 67–82. https://www.usenix.org/conference/fast26/presentation/ zheng [49] Yuhong Zhong, Daniel S. Berger, Carl Waldspurger, Ryan Wee, Ishwar Agarwal, Rajat Agarwal, Frank Hady, Karthik Kumar, Mark D. Hill, Mosharaf Chowdhury, and Asaf Cidon. 2024. Managing Memory

Tiers with CXL in Virtualized Environments. In Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI). USENIX Association, Santa Clara, CA, USA, 37–56. https: //www.usenix.org/conference/osdi24/presentation/zhong-yuhong

Related documents

• FlatCake: A Flat KVCache Storage Architecture for LLM Serving#1489147
• PACT: NVFP4 Data-Flow LoRA Fine-Tuning on Consumer GPUs#1389994
• Dynamic HBM Repartitioning for Multi-Turn MoE Serving#1071318
• Budgeted Cache Repair for Cross-Context KV-Cache Reuse#1186547
• Same Request, Different Answer: Quantization Amplifies Cache-Induced Divergence in LLM Serving#971420