An Internet for the KV Cache: Rethinking Classical Infrastructure Boundaries in the LLM Inference Age

An Internet for the KV Cache: Rethinking Classical Infrastructure Boundaries in the LLM Inference Age Siddhant Ray, Nick Feamster, Junchen Jiang The University of Chicago

arXiv:2608.01526v1 [cs.NI] 2 Aug 2026

ABSTRACT

and academia are rapidly developing solutions to support an unprecedented scale of KV Cache management [14, 53, 59]. Currently, the model (LLM) side and system side optimizations for KV Cache are carried out in isolation. Model side optimizations independently aim to reduce the KV Cache computation costs by developing new model architectures, algorithms for efficient attention and harnesses to reduce the memory footprint (more in §2). The KV Cache size becomes smaller while retaining the same representation power. A concrete example: using optimizations like Compressed Sparse Attention, within one generation from Deepseek-V3.2 to Deepseek-V4-Pro, the KV Cache size for 100,000 tokens reduced by ∼9×, from 9 GB to 1 GB! [15] Systems are also optimizing for LLM workloads to reduce KV Cache computation costs with a) efficient GPUs to reduce prefill costs and b) newer interconnects to allow faster transfer of KV Caches (more in §2). NVIDIA’s Rubin GPU delivers 50 PFLOPS of NVFP4 inference compute [49], over the previous generation Blackwell Ultra with 15 PFLOPS [10] - a 3.3× increase. In parallel, storage and transfer bandwidth technology is also becoming more efficient. [6, 7, 45, 46] Both directions above evolve at different rates, with a common goal of reducing LLM inference costs. Unfortunately, all optimizations are being developed using the legacy connected systems architectures, with tight coupling and colocation of memory and compute, within existing datacenter and cloud boundaries (more in §4). Optimizing for overall lower inference cost requires more flexibility in infrastructure rules and movement of KV Caches, which often goes against the assumptions made by the underlying cloud and datacenter systems. With the gains possible from the (re)use of the KV Cache, the definitions of compute, storage and transfer boundaries for large-scale applications, inherited from these systems, may no longer be the correct in the new world of LLM inference. It now may be cheaper to store the KV Cache (to be reused within the same request) and load it over an ISP internet link, compared to recomputing it. For example, transferring a 1 GB KV Cache over a 10-Gbps Internet link takes about 0.8 seconds (an extremely small fraction of an agentic workload [1, 51]) and costs roughly $0.09 at AWS’s standard Internet-egress rate [5]. For DeepSeek-V4-Pro (1.6T-A49B), this equates to ∼100, 000 tokens with its full-precision, compressed cache format. Recomputing the KV Cache on an

LLM inference has become a global-scale, heterogeneous workload spanning agents, retrieval, tool-use, code execution and multi-modal reasoning. These workloads naturally enable context reuse from overlapping inputs, creating a major opportunity to store and reuse the contexts’ KV Caches instead of recomputing them. However, model-side advances that shrink the KV Cache and system-side advances that reduce compute, storage, and transfer costs are evolve independently within legacy cloud boundaries. We argue that future inference infrastructure should allow decoupling of compute and KV Cache storage across cloud and datacenters. The network becomes an active distribution channel; bandwidth, latency and pricing directly determines how the KV Cache should be managed. We propose a vision for an Internet for the KV Cache, with KV Cache management working as a content-distribution system. In this view, KV Cache storage and recompute decisions are driven by model, infrastructure, and application metrics, to enable adaptive, content-driven decisions for minimizing latency and cost.

1

INTRODUCTION

LLM inference has become a dominant workload in the world, with growing heterogeneity in its structure; it now has multistep agent-based workloads with LLMs, tool calls, code execution and more. A novel opportunity arises here: KV Cache reuse. LLM workloads repeatedly revisit overlapping contexts, allowing previously computed inference state of the model input (KV Cache) to be reused instead of being recomputed. The industry has already recognized the storage and reuse of KV Cache as the most effective tool to reduce inference cost and delay [9, 17, 17, 43, 52, 56, 60]. New model-side and system-side optimizations are emerging at a daily level to support both a) cheaper methods to compute the KV Cache and b) cheaper methods to store, manage and reuse the KV Cache. Given the growing scale of LLM inference, we pose the question - how should the architecture for compute, storage and distribution for KV Cache and LLM inference be designed and evolved over the next decade? Problem Formulation: The KV Cache is no longer just a storage vs recompute tradeoff: supporting the global scale of LLM inference efficiently has transformed it into an internetscale content management and delivery problem. Industry 1

Ray et al.

AWS 8×B200 GPU cluster takes ∼ 15𝑠 and costs $0.47 [4], making transferring the 1 GB KV Cache over a fully utilized 10-Gbps link ∼19× faster! and ∼5.2× cheaper. Our Vision: Building on emerging trends in LLM inference, we propose a vision of an Internet for the KV Cache, that places and connects compute and storage nodes beyond cloud and datacenter boundaries. The placement of compute and KV Cache nodes is determined by a function of a) model-specific metrics, like prefill FLOPs and KV Cache size; b) infrastructure-specific metrics, like GPU cost, storage cost, and network bandwidth; and c) application-specific metrics, like reuse frequency, tool-call duration, and number of turns in a request. We envision an adaptive internet infrastructure, where compute and storage placement can be selected and updated using real-time values of these inputs. Overall, this paper argues that KV Cache reuse has shifted LLM inference from being a compute-storage tradeoff problem to an internet scale content-management problem. In this world, LLM inference acts as an end point and the KV Cache movement is the content distribution system. We call on the community to treat networks and storage as first-class inference resources, and develop new metrics, abstractions, and architectures to optimize joint decisions on recompute, store and reuse for internet scale KV Cache management, further pushing the LLM inference efficiency boundaries.

2

Second, sparse-attention methods reduce the amount of historical state that must be recomputed or accessed: Native Sparse Attention[70] co-designs trainable sparsity with hardware-efficient execution, DuoAttention [62] applies full attention only to retrieval heads, FlexPrefill [36] adapts sparse patterns and computational budgets to each prompt and attention head, and TidalDecode [64] amortizes token selection across layers during decoding.

2.2

ADVANCEMENTS IN LLM INFERENCE

LLM inference is advanced by optimizations that reduce the cost of computation, memory, and data transfer. These advances span three knobs: model-side techniques to reduce the KV Cache computation cost, memory-side techniques to manage KV Cache and context, and system-side techniques to improve scheduling, placement, and transfer efficiency. Each knob targets a different bottleneck but exposes the same trend: inference is no longer limited to GPU computation. It is a distributed execution problem over compute, memory, storage, and network resources. We highlight how they create opportunities for an Internet for the KV Cache.

2.1

KV Cache Optimizations

Recent work reduces the KV Cache memory footprint along several complementary directions. First, cache blending and editing increases reuse of existing KV Caches opposed to regenerating request-specific copies: CacheBlend [66] selectively recomputes a subset of tokens when fusing cached chunks, Cache-Craft [2] repairs reusable RAG chunk caches, EPIC [24] enables position-independent modular cache reuse, and KVEraser [37] efficiently deletes localized information from KV Caches without recomputing the entire context. Second, KV Cache compression reduces the size of cached states: TurboQuant [71] applies online vector quantization, KVzip [35] learns query-agnostic eviction through context reconstruction, RocketKV [11] combines coarse-grained eviction with fine-grained sparse attention, and ChunkKV [40] preserves semantically coherent tokens during eviction. Finally, memory compaction and orchestration reduce the context instantiated as KV Cache and manage it efficiently over time. MemAgent [69] overwrites a bounded memory, Context Folding [58] compresses completed subtrajectories, HiAgent [25] hierarchically summarizes subgoals, and A-Mem [63] links and updates stored experiences. Beyond in-context compaction, ACE [72] evolves reusable agent playbooks, while RAGCache [33], METIS [54], and HedraRAG [26] manage retrieved state and generation tradeoffs to control underlying KV Cache states.

2.3

System Side Optimizations

System-side optimizations reduce KV Cache transfer overhead and improve GPU utilization along many complementary directions. First, DistServe [73] separates prefill and decode execution and independently provisions each phase, Sarathi-Serve [3] introduces chunked prefill and stall-free batching, Mooncake [53] extends disaggregation to a multitier KV Cache pool, and WindServe [19] applies stream-based dynamic scheduling to phase-disaggregated inference. Second, interconnect-aware transfer systems reduce the cost of moving KV Caches: KVDirect [13] uses tensor-centric GPU-RDMA and pull-based transfers, LMCache [14] batches and pipelines movement across GPU, CPU, storage, and network tiers while supporting NIXL backends, TraCT [68] replaces NIXL/UCX network transfers with direct GPU–CXL

Model (LLM) Side Optimizations

Recent work has pursued two complementary directions for reducing computation during LLM inference. First, mixtureof-experts architectures reduce per-token computation by activating only selected experts: Mixture of Lookup Experts [32] re-parameterizes experts as lookup tables, Fiddler [34] coordinates CPU and GPU execution, Klotski [18] overlaps expert loading with computation through an expert-aware multibatch pipeline, and K-Transformers [12] provides optimized heterogeneous kernels for CPU-GPU inference.

2

An Internet for the KV Cache: Rethinking Classical Infrastructure Boundaries in the LLM Inference Age

DMA to shared memory, and CXL-SpecKV [38] combines CXL-attached FPGA memory with speculative prefetching. Finally, scheduling systems adapt execution to workload and hardware conditions: SOLA performs state-aware periteration scheduling [23], ThunderServe [31] jointly plans placement and rescheduling across heterogeneous GPUs, Seesaw [57] dynamically re-shards models between inference phases, and Libra [55] partitions requests into micro-requests to construct SLO-aware batches.

determined KV Cache reuse pattern and the model architecture, multiple solutions can be correct. We consider the following formulation to understand the storage-recompute tradeoff for the KV Cache. We will look at cost of storage (including transfer cost over the network), calling it the CacheCost and the cost of fully recomputation the KV Cache, calling it the PrefillCost. We define CacheCost = cost of storing + transferring reusable KV Cache, PrefillCost = cost of recomputing the prompt prefill,

3

THE KV CACHE IS A FIRST-CLASS CITIZEN IN LLM INFERENCE 3.1 The KV Cache is not just another data structure

DeviceCosthr = storage device cost per hour per MB, GPUCosthr = GPU cost per hour, TransferCost = data transfer cost per MB, 𝐾 = frequency of KV Cache reuse, KVMB = KV cache size in MB,

A common view treats the KV Cache as yet another tensor data structure, often analogous to model weights in conventional machine learning systems. This view misses its broader role: the KV Cache encodes input knowledge into a unified representation that allows for auto-regressive token generation, conditioned on this knowledge. This representation can be shared and reused across knowledge bases, inputs, and, as new research suggest, even across different models [41]. It is a compact, persistent, steerable representation of contextual knowledge, converted from natural language into an expressive tool. This richness and expressiveness of the KV Cache enables LLMs to efficiently reuse context across environments, workflows, and tasks at global scale. The KV Cache is becoming a storage, communication, and scheduling primitive for LLM serving, not merely an serving optimization artifact. Several industry solutions and opensource projects such as LMCache [14],TensorRT [50],llmd [42] and NVIDIA Dynamo [48] focus on pushing the capabilities of the KV Cache. Popular innovations in this space include directly by exposing KV Caches across engines and queries, supporting offloading, prefill-decode disaggregation, KV Cache lookup, movement, compression, and orchestration across GPU, CPU, storage, and network layers.

3.2

PrefillTFLOPs = compute required for prefill in TFLOPs, PeakFlopRate = peak GPU throughput in TFLOPs/hour.

Using the above, we formulate CacheCost DeviceCosthr · 𝐾 · KVMB + TransferCost · KVMB   = PrefillCost GPUCost · PrefillTFLOPs hr

=

PeakFlopRate

(DeviceCosthr · 𝐾 + TransferCost) · KVMB   . PrefillTFLOPs GPUCosthr · PeakFlopRate

(1) 𝐶𝑎𝑐ℎ𝑒𝐶𝑜𝑠𝑡 Setting the left hand side ratio 𝑃𝑟𝑒 (for the break𝑓 𝑖𝑙𝑙𝐶𝑜𝑠𝑡 even condition), we obtain two new metrics DeviceCosthr · 𝐾 + TransferCost PrefillTFLOPs = . GPUCosthr /PeakFlopRate KVMB

(2)

TFLOPs From Equation 2, we have the term Prefill which repreKVMB

hr ·𝐾+TransferCost sents the model side ratio and the term DeviceCost GPUCosthr /PeakFlopRate which represents the infrastructure and application side ratio. These ratios can be independently calculated. Directly comparing the values of ratios helps choosing between recomputing or storing the KV Cache for optimal cost-saving. As established in Equations 1,2, the choice of any parameter can change the decision between storing and recomputing the KV Cache, which leads to the claim that multiple correct solutions are possible, conditioned on the setup. To demonstrate this further, we show in Table 1, the effect of changes in a single parameter changes the decision. Here, we only consider the extremes of full recomputation vs storage, by introducing partial recomputation and KV Cache blending, we can further push the decision boundary of the tradeoff-space.

(Re)Using the KV Cache allows for multiple correct solutions

The KV Cache introduces a fundamental tradeoff in LLM inference: it can either be recomputed or stored and reused. This tradeoff becomes richer at finer granularity, where systems may combine partial reuse with partial recomputation, or merge individual KV Caches from text segments to reconstruct the KV Cache of logically combined segments [66]. As a result, the answer to whether it is cheaper to recompute or store the KV Cache for future requests is not universal. Depending on the underlying infrastructure, the application

3

Ray et al.

# Model

Infrastructure

𝐾

1 2 3 4 5 6 7

AWS B200 + S3 Bucket AWS B200 + S3 Bucket GCP B200 + GCS Azure B200 + Blob Hot LRS GMI B200 + high storage AWS B200 + S3 Bucket AWS B200 + S3 Bucket

1 day 1 month 1 day 1 month 1 day 1 day 1 month

DeepSeek V4-Pro DeepSeek V4-Pro DeepSeek V4-Pro DeepSeek V4-Pro DeepSeek V4-Pro DeepSeek V4-Flash DeepSeek V4-Flash

Infra-side Ratio Model-side Ratio 0.436 13.083 0.670 5.512 20.250 0.436 13.083

Decision

11.410 Storing cheaper 11.410 Recompute cheaper 11.410 Storing cheaper 11.410 Storing cheaper 11.410 Recompute cheaper 4.325 Storing cheaper 4.325 Recompute cheaper

Table 1: Storage versus recomputation decision across models, infrastructure and reuse ratios.

3.3

Enabling KV Cache reuse to reduce the miss-rate should be the main goal

3.4 KV Cache innovations pave the way for next-generation LLM workloads

With LLM serving becoming a dominant industry workload, prefix caching [61] has emerged as a common KV cache management policy due to its simplicity and the natural occurrence of shared prefixes in applications such as multi-turn chat and coding agents. Industry systems find this attractive and therefore heavily optimize for increasing KV Cache hitrate, a metric that prefix caching can improve extensively when requests share the exact token prefixes. However, the hit rate alone is incomplete, as it does not capture the cost of failure. In LLM serving, a cache miss can be disproportionately expensive because it may require recomputing long-context KV Cache. For example, a hit-rate of 90% is not useful if recomputing the KV Cache in the remaining 10% cases becomes the computational bottleneck. Prefix caching is brittle to small input changes — even a single token difference will cause a cache miss and prevent KV Cache reuse. This brittleness becomes more limiting as workloads scale and context reuse requirements shift from exact matches to more semantic and approximate matches. The distinguishing factor in KV Cache reuse becomes reducing the miss-rate. The extent to which a system can push the tradeoff boundary between KV Cache reuse and recomputation is determined by reducing the cost and frequency of KV Cache misses. As recomputation is quadratic in the prefix length, smaller improvements in the KV Cache miss-rate compound and lead to significantly greater compute savings. Enabling non-prefix KV Cache reuse over partially matching contexts is one such way to expand this boundary. This can be achieved by methods such as blending KV Caches of individual RAG chunks or compacted agent memory, in different orders [2, 66]. This allows multiple versions and perturbations of several smaller contexts to reconstruct the overall KV Cache states, making reuse significantly more robust than exact-prefix matching. Reducing the KV Cache miss rate eventually unlocks the true potential which may be realized from KV reuse across requests and workloads.

It may be tempting to think that today’s LLM serving ecosystem does not require further optimizations via smarter KV Cache management, especially as many LLM and agent applications may already appear good enough. However, LLM inference is already a global-scale service, and both model capabilities and application demands have changed rapidly over the last two years. As adoption expands and the userbase grows, LLM workloads are very likely to evolve further from isolated prompts and sessions toward richer, longerrunning, and more interconnected applications. History in computer systems and networks suggests that infrastructure advances often precede the applications that eventually depend on them. Research and engineering on video streaming over HTTP(S), for example, helped make large-scale video delivery practical before video streaming became a dominant web workload [16, 20, 27, 30, 67]. Today, HTTP(S)-based delivery is the foundation for most contentbased Internet video streaming applications. Similarly, advancements built around KV Cache management and distribution can provide building blocks for future LLM applications, that need to share knowledge across silos, workflows, and environments. Investing in KV Cache research and infrastructure, can therefore push LLM inference towards lower cost, higher efficiency, and better quality at scale.

4 AN INTERNET FOR THE KV CACHE 4.1 How are compute-storage boundaries defined in today’s cloud infrastructure? Most modern cloud infrastructure is built around the principle that storage and compute should remain close to each other. This is encoded both in architecture and pricing: providers expose regions and availability-zone boundaries as decision knobs for placement. Co-locating storage and compute is recommended to reduce latency and data-transfer costs, and high costs are charged when traffic crosses many of these boundaries. For example, providers like AWS, GCP and Azure 4

An Internet for the KV Cache: Rethinking Classical Infrastructure Boundaries in the LLM Inference Age

Today: Compute-Storage Colocation

Vision: Store Anywhere, Use Everywhere

Cloud A – Region 1

Global KV Cache Control Plane

Compute Pool Storage Pool

KV Cache Directory

Expensive Transfer Cloud B – Region 2 Compute Pool Storage Pool

Lookup Service

Placement Planner

Prefetch Scheduler

Cloud A

Cloud B

Cloud C

Compute Pool

Compute Pool

Compute Pool

Hot KV Cache

Warm KV Cache

Cold KV Cache

Storage Follows Compute

Bandwidth-Aware KV Cache Movement

Reuse horizon : seconds -> store hot | minutes -> store warm | hours -> store cold KV Cache decisions: store / load / prefetch / partial fetch / recompute / evict

Figure 1: Our Vision: Building a global-scale Internet for KV Cache distribution and management

4.2

use these principles [8, 21, 47]. Data movement across region and zone boundaries has explicit higher pricing [5, 22]. As a result, today’s cloud economics favor tightly coupled placement: keeping data near compute reduces both tail latency and provider-imposed transfer cost. Moving data across zones, regions, or clouds can quickly become a dominant expense. Prior work on cloud transfers reaches the same conclusion, showing that wide-area cloud movement is often slow and that egress prices can dominate the cost of bulk data movement [29, 39]. Together, these constraints make storage-compute locality not just a performance optimization, but pose core restriction in the way applications deployed on top of cloud systems are priced, and controlled. Some recent systems have started to push the boundary of compute across clouds. SkyPilot introduces an inter-cloud broker that discovers feasible VM instances across zones, regions, and cloud providers, and places jobs based on price, performance, availability, and data-transfer overheads [65]. This model is especially attractive for spot-based workloads, where capacity is volatile. SkyServe extends this direction to model serving, by spreading spot and on-demand replicas across regions and clouds and using over-provisioning and on-demand fallback to mask spot unavailability [44]. The above systems make compute placement more flexible, while storage remains a dependency of the execution location. Other work is exploring cross-cloud disaggregated storage at the level of object placement, replication, and bulk transfer [39]. LLM inference and KV Caches need more: they need real-time KV Cache state management with finegrained placement, prefetching, and recomputation fallback.

Store Anywhere, Use Everywhere Model

Our vision (Figure 1) is an Internet for the KV Cache which completely dissolves cloud boundaries for KV Cache reuse and movement. Current clouds bind storage and compute, making proximity the default metric for efficiency. KV Caches challenge this model. Unlike other user or system data, a KV Cache can be recomputed or reused, based on the timesensitive inference state. Its optimal location depends not on administrative boundaries, but on the storage-networkrecompute tradeoff induced by future reuse. Our vision advocates that the Internet for the KV Cache should be fully distributed: it must allow KV Cached state to move across storage tiers, regions, and providers. A key insight behind our architecture is that KV Cache reuse is highly heterogeneous along the time domain. In agentic workflows, the same cached context may be reused after seconds, minutes, or even hours, creating reuse horizons that are much longer than the latency of a single generation [1, 51]. This temporal slack enriches the storagecompute tradeoff: KV Caches with near-term reuse can remain close to the GPU, while KV Caches with longer reuse horizons can be pushed to cheaper and farther storage tiers. Our vision builds on this principle: the placement of KV Cache and compute should be governed by jointly considering reuse rates and deadlines, available bandwidth, transfer and recomputation costs. KV Caches should not be viewed as bound to GPU compute or storage, but as mobile inference states that can be placed, replicated, and delivered wherever needed. We view this as a content-delivery principle: move useful KV Caches toward expected future demand, prefetch 5

Ray et al.

them when reuse is predictable, and relocate it when demand shifts. Unlike data in traditional CDNs, KV Caches can be regenerated. If delivery becomes too slow or expensive, the system can regenerate the missing KV Caches and reevaluate placement using updated reuse expectations, freeing the original storage slots for other KV Caches. For an Internet for the KV Cache, the underlying network is a first class citizen: its bandwidth should be used opportunistically to move and distribute KV Caches. This requires a KV Cache control plane with a lookup service that can discover reusable state, estimate transfer deadlines and available bandwidth, and decide when to transfer, replicate or avoid movement of the KV Cache altogether (Figure 1). In this world of LLM inference, a global distributed KV Cache Internet, where cached contexts serve as a content distribution system, will ultimately drive LLM performance.

5

An Internet for the KV Cache needs a control plane which needs to enable distributed lookup, discovery, and routing services for cached inference state. It must find where a KV Cache lives, analyze the costs of bandwidth and transfer deadlines, and select whether to move, replicate, partially fetch, or recompute it. Such a service may build on IP-prefixstyle routing primitives but its objective is significantly richer. Routing is no longer about reachability alone, but the lowestcost path through the storage-network-recompute tradeoff. The architecture should decide when a KV Cache must be recomputed based on cost and reliability targets, and expose fault-tolerance guarantees. Failures should also be first-class in the design. If transfer is delayed (e.g., link failure), or the KV Cache is no longer cost-effective to retrieve, the system should degrade gracefully to full or partial recomputation, and consider falling back to a lower-cost placement policy. The recomputed KV Cache should subsequently be propagated based on the updated placement policy. Finally, the architecture should jointly expose the independent optimizations emerging in LLM inference as coordinated control knobs rather than isolated mechanisms. Compression, offloading, prefetching, routing, recomputation, scheduling, and model-level reuse should be composed adaptively as workloads evolve. Coordination across regulators, providers, and users: KV Cache placement may now cross administrative and economic boundaries. If KV Cached inference state can be stored far and outside the compute provider, replicated across regions, or moved through third-party networks, we need explicit agreements about pricing, reliability, ownership, privacy, and failure handling and recovery. These agreements should define who controls the KV Cache, who pays for storage and transfer, who is responsible for correctness, and what happens when retrieval fails or violates a latency target. This is stark contrast from current cloud contracts, where compute, storage, and networking are usually bundled inside provider-defined boundaries. An Internet for the KV Cache will require creation of new coordinating bodies (like the IETF [28]) to standardize transfer, security, and interoperability protocols for KV Cached states. The Internet for the KV Cache must also allow finergrained user control. For example, if users can store reusable KV Caches cheaply on their own hardware, there should be support that choice. Privacy and security models must be reconsidered as the infrastructure becomes increasingly decoupled, as both providers and users establish control over compute and storage decisions.

CHALLENGES AND OPPORTUNITIES

Building an Internet for the KV Cache requires designing of new communication, policy and data-discovery abstractions . Existing systems provide some useful primitives: prefix caching shows that repeated contexts can be detected and reused; KV offloading shows that inference state can move across memory and storage tiers; prefetching and routing show that workload structure can guide placement decisions; and Internet-style routing shows that large distributed systems can evolve without fixed static paths. An Internet for the KV Cache requires rethinking how compute, storage, and the network are jointly exposed as inference resources. Network-aware KV Cache movement: KV Cache reuse makes the network a first-class part of inference execution. Today, the network is mainly treated as a transport layer between fixed compute and storage locations. In an Internet for the KV Cache, network bandwidth, latency, congestion, and transfer price directly determine whether a KV Cache should be fetched, partially reconstructed, or fully recomputed. This may require rethinking interconnect technologies, new transport protocols, and new cache-aware routing mechanisms that can move large KV Cache states predictably across datacenters, cloud regions, edge nodes, and even ISP paths. KV Cache movement should not be hidden behind a static storage abstraction, instead the system should expose data transfer costs and delays as explicit decision inputs. Adaptive infrastructure and control-plane design: Another challenge is to build a connected infrastructure that can adapt as models, workloads, and hardware evolve. An Internet for the KV Cache should provide freedom to add or remove compute and storage nodes and move cached state without binding execution to fixed storage. This requires treating the placement of KV Caches as a dynamic policy decision rather than a static allocation.

6

CONCLUSION

LLM inference has evolved from a compute problem into a global-scale content-distribution problem centered around 6

An Internet for the KV Cache: Rethinking Classical Infrastructure Boundaries in the LLM Inference Age

KV Cache management. This paper envisions an Internet for the KV Cache, where cached inference state is managed jointly using model, infrastructure, and application metrics, with networks and storage as active knobs for decisions. We position this work as a "call for arms", and hope it encourages the community to rethink and explore new directions and research around developing control planes, protocols, abstractions and metrics needed to support an internet-scale KV Cache management system.

[12] Hongtao Chen, Weiyu Xie, Boxin Zhang, Jingqi Tang, Jiahao Wang, Jianwei Dong, Shaoyuan Chen, Ziwei Yuan, Chen Lin, Chengyu Qiu, Yuening Zhu, Qingliang Ou, Jiaqi Liao, Xianglin Chen, Zhiyuan Ai, Yongwei Wu, and Mingxing Zhang. 2025. KTransformers: Unleashing the Full Potential of CPU/GPU Hybrid Inference for MoE Models. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles. Association for Computing Machinery, 1014–1029. https: //doi.org/10.1145/3731569.3764843 [13] Shiyang Chen, Rain Jiang, Dezhi Yu, Jinlai Xu, Mengyuan Chao, Fanlong Meng, Chenyu Jiang, Wei Xu, and Hang Liu. 2025. KVDirect: Distributed Disaggregated LLM Inference. arXiv preprint arXiv:2501.14743 (2025). https://doi.org/10.48550/arXiv.2501.14743 arXiv:2501.14743 [cs.DC] [14] Yihua Cheng, Yuhan Liu, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaoting Feng, Yuyang Huang, Samuel Shen, Kuntai Du, and Junchen Jiang. 2025. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference. arXiv preprint arXiv:2510.09665 (Oct. 2025). https: //doi.org/10.48550/arXiv.2510.09665 arXiv:2510.09665 [cs.DC] [15] DeepSeek-AI. 2026. DeepSeek-V4: Towards Highly Efficient MillionToken Context Intelligence. arXiv:2606.19348 [cs.CL] https://arxiv.or g/abs/2606.19348 [16] Florin Dobrian, Vyas Sekar, Asad Awan, Ion Stoica, Dilip Joseph, Aditya Ganjam, Jibin Zhan, and Hui Zhang. 2011. Understanding the Impact of Video Quality on User Engagement. In Proceedings of the ACM SIGCOMM 2011 Conference (SIGCOMM ’11). Association for Computing Machinery, 362–373. https://doi.org/10.1145/2043164.2018478 [17] Amr Elmeleegy and Akshatha Kamath. 2025. How to Reduce KV Cache Bottlenecks with NVIDIA Dynamo. https://developer.nvidia.com/b log/how-to-reduce-kv-cache-bottlenecks-with-nvidia-dynamo/. NVIDIA Technical Blog. [18] Zhiyuan Fang, Yuegui Huang, Zicong Hong, Yufeng Lyu, Wuhui Chen, Yue Yu, Fan Yu, and Zibin Zheng. 2025. Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Vol. 2. Association for Computing Machinery, 574–588. https: //doi.org/10.1145/3676641.3716261 [19] Jingqi Feng, Yukai Huang, Rui Zhang, Sicheng Liang, Ming Yan, and Jie Wu. 2025. WindServe: Efficient Phase-Disaggregated LLM Serving with Stream-Based Dynamic Scheduling. In Proceedings of the 52nd Annual International Symposium on Computer Architecture (ISCA ’25). Association for Computing Machinery, 1283–1295. https://doi.org/10 .1145/3695053.3730999 [20] Aditya Ganjam, Faisal Siddiqui, Jibin Zhan, Xi Liu, Ion Stoica, Junchen Jiang, Vyas Sekar, and Hui Zhang. 2015. C3: Internet-Scale Control Plane for Video Quality Optimization. In 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’15). USENIX Association, Oakland, CA, 131–144. https://www.usenix.org/confere nce/nsdi15/technical-sessions/presentation/ganjam [21] Google Cloud. 2026. Bucket Locations. https://cloud.google.com/sto rage/docs/locations. Accessed: 2026-07-09. [22] Google Cloud. 2026. Network Pricing. https://cloud.google.com/vpc /network-pricing. Accessed: 2026-07-09. [23] Ke Hong, Xiuhong Li, Lufang Chen, Qiuli Mao, Guohao Dai, Xuefei Ning, Shengen Yan, Yun Liang, and Yu Wang. 2025. SOLA: Optimizing SLO Attainment for Large Language Model Serving with State-Aware Scheduling. In Proceedings of Machine Learning and Systems, Vol. 7. https://proceedings.mlsys.org/paper_files/paper/2025/hash/bc82dbf bfa43232be85b8d9838f49c3e-Abstract-Conference.html [24] Junhao Hu, Wenrui Huang, Weidong Wang, Haoyi Wang, Tiancheng Hu, Zhang Qin, Hao Feng, Xusheng Chen, Yizhou Shan, and Tao Xie. 2025. EPIC: Efficient Position-Independent Caching for Serving Large

REFERENCES [1] Reyna Abhyankar, Qi Qi, and Yiying Zhang. 2026. OSWorldHuman: Benchmarking the Efficiency of Computer-Use Agents. arXiv:2506.16042 [cs.AI] https://arxiv.org/abs/2506.16042 [2] Shubham Agarwal, Sai Sundaresan, Subrata Mitra, Debabrata Mahapatra, Archit Gupta, Rounak Sharma, Nirmal Joshua Kapu, Tong Yu, and Shiv Saini. 2025. Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation. Proceedings of the ACM on Management of Data 3, 3 (2025), 136:1–136:28. https://doi.org/10.1145/3725273 [3] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 117–134. https://www.usenix.org/conference/osdi24/presentation/ag rawal [4] Amazon Web Services. [n. d.]. Amazon EC2 Capacity Blocks for ML Pricing. https://aws.amazon.com/ec2/capacityblocks/pricing/. Accessed: 2026-07-16. [5] Amazon Web Services. 2021. Overview of Data Transfer Costs for Common Architectures. https://aws.amazon.com/blogs/architect ure/overview-of-data-transfer-costs-for-common-architectures/. Accessed: 2026-07-09. [6] Amazon Web Services. 2026. Accelerate LLM Model Loading and Increase Context Windows with GPUDirect on Amazon FSx for Lustre and TurboQuant. https://aws.amazon.com/blogs/machine-learning/a ccelerate-llm-model-loading-and-increase-context-windows-withgpudirect-on-amazon-fsx-for-lustre-and-turboquant/. AWS Machine Learning Blog. [7] Amazon Web Services. 2026. Amazon EC2 P6e UltraServers and P6 Instances. https://aws.amazon.com/ec2/instance-types/p6/. AWS product documentation. [8] Amazon Web Services. 2026. Performance Guidelines for Amazon S3. https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizi ng-performance-guidelines.html. Accessed: 2026-07-09. [9] Anthropic. 2025. Prompt Caching with Claude. https://claude.com/b log/prompt-caching. Claude Blog. [10] Kyle Aubrey and Nick Stam. 2025. Inside NVIDIA Blackwell Ultra: The Chip Powering the AI Factory Era. https://developer.nvidia.c om/blog/inside-nvidia-blackwell-ultra-the-chip-powering-the-aifactory-era/. NVIDIA Technical Blog. [11] Payman Behnam, Yaosheng Fu, Ritchie Zhao, Po-An Tsai, Zhiding Yu, and Alexey Tumanov. 2025. RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression. In Proceedings of the 42nd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 267), Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (Eds.). PMLR, 3358–3392. https://procee dings.mlr.press/v267/behnam25a.html 7

Ray et al.

[35] Jang-Hyun Kim, Jinuk Kim, Sangwoo Kwon, Jae W. Lee, Sangdoo Yun, and Hyun Oh Song. 2025. KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction. In Advances in Neural Information Processing Systems, Vol. 38. https://proceedings.neurips.cc /paper_files/paper/2025/hash/f4eaa4b8f2d08edb3f0af990d56134eaAbstract-Conference.html [36] Xunhao Lai, Jianqiao Lu, Yao Luo, Yiyuan Ma, and Xun Zhou. 2025. FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=OfjIlb elrT [37] Mufei Li, Shikun Liu, Dongqi Fu, Haoyu Wang, Yinglong Xia, Hong Li, Hong Yan, and Pan Li. 2026. KVEraser: Learning to Steer KV Cache for Efficient Localized Context Erasing. arXiv preprint arXiv:2606.17034 (June 2026). htt ps:/ /do i.o rg/ 10.4 855 0/a rXiv .26 06. 170 34 arXiv:2606.17034 [cs.CL] [38] Dong Liu and Yanxuan Yu. 2026. CXL-SpecKV: A Disaggregated FPGA Speculative KV-Cache for Datacenter LLM Serving. In Proceedings of the 2026 ACM/SIGDA International Symposium on Field Programmable Gate Arrays (FPGA ’26). Association for Computing Machinery, 56–66. https://doi.org/10.1145/3748173.3779188 [39] Shu Liu, Xiangxi Mo, Moshik Hershcovitch, Henric Zhang, Audrey Cheng, Guy Girmonsky, Gil Vernik, Michael Factor, Tiemo Bang, Soujanya Ponnapalli, Natacha Crooks, Joseph E. Gonzalez, Danny Harnik, and Ion Stoica. 2025. SkyStore: Cost-Optimized Object Storage Across Regions and Clouds. arXiv preprint arXiv:2502.20818 (2025). arXiv:2502.20818 [cs.DC] https://arxiv.org/abs/2502.20818 [40] Xiang Liu, Zhenheng Tang, Peijie Dong, Zeyu Li, Liuyue, Bo Li, Xuming Hu, and Xiaowen Chu. 2025. ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference. In Advances in Neural Information Processing Systems, Vol. 38. https: //proceedings.neurips.cc/paper_files/paper/2025/hash/2987f911151 b39cd3a1761e212319e8e-Abstract-Conference.html [41] Yuhan Liu, Yuyang Huang, Jiayi Yao, Shaoting Feng, Zhuohan Gu, Kuntai Du, Hanchen Li, Yihua Cheng, Junchen Jiang, Shan Lu, Madan Musuvathi, and Esha Choukse. 2026. DroidSpeak: KV Cache Sharing Across Fine-tuned Model Variants. In Proceedings of the 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26) (Renton, WA, USA). USENIX Association. https://www.usenix.o rg/conference/nsdi26/presentation/liu-yuhan [42] llm-d Contributors. 2026. llm-d-kv-cache: Distributed KV Cache Scheduling and Offloading Libraries. https://github.com/llm-d/llm-d-kvcache. Accessed: 2026-07-08. [43] LMCache. 2026. Stop Calling It KV Cache: It’s Something Much Bigger. https://blog.lmcache.ai/en/2026/04/28/stop-calling-it-kv-cache-itssomething-much-bigger/. LMCache Blog. [44] Ziming Mao, Tian Xia, Zhanghao Wu, Wei-Lin Chiang, Tyler Griggs, Romil Bhardwaj, Zongheng Yang, Scott Shenker, and Ion Stoica. 2025. SkyServe: Serving AI Models across Regions and Clouds with Spot Instances. In Proceedings of the Twentieth European Conference on Computer Systems (EuroSys ’25). Association for Computing Machinery, 159–175. https://doi.org/10.1145/3689031.3717459 [45] Micron Technology. 2025. Micron Ships HBM4 to Key Customers to Power Next-Gen AI Platforms. https://investors.micron.com/newsreleases/news-release-details/micron-ships-hbm4-key-customerspower-next-gen-ai-platforms. Micron press release. [46] Micron Technology. 2025. Micron Unveils Portfolio of Industry-First SSDs to Power the AI Revolution. https://investors.micron.com/newsreleases/news-release-details/micron-unveils-portfolio-industryfirst-ssds-power-ai-revolution. Micron press release. [47] Microsoft Azure. 2026. Architecture Best Practices for Azure Files. https://learn.microsoft.com/en-us/azure/well-architected/service-

Language Models. In Proceedings of the 42nd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 267), Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (Eds.). PMLR, 24391–24402. https://proceedings.mlr.press/v267/hu25j.html [25] Mengkang Hu, Tianxing Chen, Qiguang Chen, Yao Mu, Wenqi Shao, and Ping Luo. 2025. HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Model. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 32779–32798. https://doi.org/10.18653/v1/2025.acl-long.1575 [26] Zhengding Hu, Vibha Murthy, Zaifeng Pan, Wanlu Li, Xiaoyi Fang, Yufei Ding, and Yuke Wang. 2025. HedraRAG: Co-Optimizing Generation and Retrieval for Heterogeneous RAG Workflows. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles. Association for Computing Machinery, 623–638. https: //doi.org/10.1145/3731569.3764806 [27] Te-Yuan Huang, Ramesh Johari, Nick McKeown, Matthew Trunnell, and Mark Watson. 2014. A Buffer-Based Approach to Rate Adaptation: Evidence from a Large Video Streaming Service. In Proceedings of the 2014 ACM Conference on SIGCOMM (SIGCOMM ’14). Association for Computing Machinery, 187–198. https://doi.org/10.1145/2619239.26 26296 [28] Internet Engineering Task Force. 2026. Introduction to the IETF. https://www.ietf.org/about/introduction/. Accessed July 2026. [29] Paras Jain, Sam Kumar, Sarah Wooders, Shishir G. Patil, Joseph E. Gonzalez, and Ion Stoica. 2022. Skyplane: Optimizing Transfer Cost and Throughput Using Cloud-Aware Overlays. arXiv preprint arXiv:2210.07259 (2022). arXiv:2210.07259 [cs.DC] https://arxiv.org/ abs/2210.07259 [30] Junchen Jiang, Vyas Sekar, and Hui Zhang. 2012. Improving Fairness, Efficiency, and Stability in HTTP-Based Adaptive Video Streaming with FESTIVE. In Proceedings of the 8th International Conference on Emerging Networking Experiments and Technologies (CoNEXT ’12). Association for Computing Machinery, 97–108. https://doi.org/10.1145/ 2413176.2413189 [31] Youhe Jiang, Fangcheng Fu, Xiaozhe Yao, Taiyi Wang, Bin Cui, Ana Klimovic, and Eiko Yoneki. 2025. ThunderServe: High-Performance and Cost-Efficient LLM Serving in Cloud Environments. In Proceedings of Machine Learning and Systems, Vol. 7. https://proceedings.mlsys.or g/paper_files/paper/2025/hash/c2a0e26dd9ee7d57e92bb1c24b39659aAbstract-Conference.html [32] Shibo Jie, Yehui Tang, Kai Han, Yitong Li, Duyu Tang, Zhi-Hong Deng, and Yunhe Wang. 2025. Mixture of Lookup Experts. In Proceedings of the 42nd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 267), Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (Eds.). PMLR, 27929–27940. https: //proceedings.mlr.press/v267/jie25b.html [33] Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Shufan Liu, Xuanzhe Liu, and Xin Jin. 2026. RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation. ACM Transactions on Computer Systems 44, 1 (2026), 2:1–2:27. https://doi.org/10.1145/3768628 [34] Keisuke Kamahori, Tian Tang, Yile Gu, Kan Zhu, and Baris Kasikci. 2025. Fiddler: CPU–GPU Orchestration for Fast Inference of Mixtureof-Experts Models. In The Thirteenth International Conference on Learning Representations. https://proceedings.iclr.cc/paper_files/p aper/2025/hash/8cd1ce03ea58b3d7dfd809e4d42f08ea-AbstractConference.html 8

An Internet for the KV Cache: Rethinking Classical Infrastructure Boundaries in the LLM Inference Age

guides/azure-files. Accessed: 2026-07-09. [48] NVIDIA. 2026. NVIDIA Dynamo: A Datacenter-Scale Distributed Inference Framework. https://docs.nvidia.com/dynamo/gettingstarted/introduction. Accessed: 2026-07-08. [49] NVIDIA. 2026. NVIDIA Kicks Off the Next Generation of AI With Rubin. https://nvidianews.nvidia.com/news/rubin-platform-aisupercomputer. NVIDIA Newsroom. [50] NVIDIA. 2026. TensorRT-LLM: KV Cache System. https://github.com /NVIDIA/TensorRT-LLM/blob/main/docs/source/features/kvcache .md. Accessed: 2026-07-08. [51] OpenAI. 2026. ChatGPT Agent. https://help.openai.com/en/articles/ 11752874-chatgpt-agent. OpenAI Help Center. [52] OpenAI. 2026. Prompt Caching 201. https://developers.openai.co m/cookbook/examples/prompt_caching_201. OpenAI Developers Cookbook. [53] Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading More Storage for Less Computation — A KVCache-Centric Architecture for Serving LLM Chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST 25). USENIX Association, Santa Clara, CA, 155–170. https://www.usenix.org/conference/fast25/prese ntation/qin [54] Siddhant Ray, Rui Pan, Zhuohan Gu, Kuntai Du, Shaoting Feng, Ganesh Ananthanarayanan, Ravi Netravali, and Junchen Jiang. 2025. METIS: Fast Quality-Aware RAG Systems with Configuration Adaptation. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles. Association for Computing Machinery, 606–622. https: //doi.org/10.1145/3731569.3764855 [55] Chaoyi Ruan, Yinhe Chen, Dongqi Tian, Yandong Shi, Yongji Wu, Jialin Li, and Cheng Li. 2026. Libra: Flexible Request Partitioning and Scheduling for Serving Unbalanced and Dynamic LLM Workloads. In 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26). USENIX Association, Renton, WA, 1243–1258. https://www.usenix.org/conference/nsdi26/presentation/ruan-libra [56] Kai Shen and Barak Epstein. 2025. Reducing TCO for AI Inferencing with External KV Cache on Managed Lustre. https://cloud.google.c om/blog/products/storage-data-transf er/choosing-google-cloudmanaged-lustre-for-your-external-kv-cache. Google Cloud Blog. [57] Qidong Su, Wei Zhao, Xin Li, Muralidhar Andoorveedu, Chenhao Jiang, Zhanda Zhu, Kevin Song, Christina Giannoula, and Gennady Pekhimenko. 2025. Seesaw: High-Throughput LLM Inference via Model Re-Sharding. In Proceedings of Machine Learning and Systems, Vol. 7. https://proceedings.mlsys.org/paper_files/paper/2025/hash/c bc4ab80cd77aa0eb87da062fbcddb46-Abstract-Conference.html [58] Weiwei Sun, Miao Lu, Zhan Ling, Kang Liu, Xuesong Yao, Yiming Yang, and Jiecao Chen. 2026. Scaling Long-Horizon Agent via Context Folding. In Proceedings of the 43rd International Conference on Machine Learning. https://openreview.net/forum?id=JaLXQnA2wi Forthcoming. [59] The AIBrix Team, Jiaxin Shan, Varun Gupta, Le Xu, Haiyang Shi, Jingyuan Zhang, Ning Wang, Linhui Xu, Rong Kang, Tongping Liu, Yifei Zhang, Yiqing Zhu, Shuowei Jin, Gangmuk Lim, Binbin Chen, Zuzhi Chen, Xiao Liu, Xin Chen, Kante Yin, Chak-Pong Chung, Chenyu Jiang, Yicheng Lu, Jianjun Chen, Caixue Lin, Wu Xiang, Rui Shi, and Liguang Xie. 2025. AIBrix: Towards Scalable, Cost-Effective Large Language Model Inference Infrastructure. arXiv:2504.03648 [cs.DC] https://arxiv.org/abs/2504.03648 [60] vLLM Project. 2026. Automatic Prefix Caching. https://docs.vllm.ai/e n/latest/features/automatic_prefix_caching/. vLLM Documentation. [61] vLLM Project. 2026. vLLM: Automatic Prefix Caching. https://docs.vll m.ai/en/stable/design/prefix_caching/. Accessed: 2026-07-08.

[62] Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. 2025. DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads. In The Thirteenth International Conference on Learning Representations. https://proceedings.iclr.cc/paper_files/paper/2025/hash/5 c1ddd2e59df46fd2aa85c833b1b36ed-Abstract-Conference.html [63] Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. 2025. A-Mem: Agentic Memory for LLM Agents. In Advances in Neural Information Processing Systems, Vol. 38. https://proceedings. neurips.cc/paper_files/paper/2025/hash/19909c36f51abc4856b4560 aff3d36d6-Abstract-Conference.html [64] Lijie Yang, Zhihao Zhang, Zhuofu Chen, Zikun Li, and Zhihao Jia. 2025. TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attention. In The Thirteenth International Conference on Learning Representations. https://proceedings.iclr.cc/paper_fil es/paper/2025/hash/11440c427f0f76f191ac06b50d7a2517-AbstractConference.html [65] Zongheng Yang, Zhanghao Wu, Michael Luo, Wei-Lin Chiang, Romil Bhardwaj, Woosuk Kwon, Siyuan Zhuang, Frank Sifei Luan, Gautam Mittal, Scott Shenker, and Ion Stoica. 2023. SkyPilot: An Intercloud Broker for Sky Computing. In Proceedings of the 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’23). USENIX Association, 437–455. https://www.usenix.org/conference/ nsdi23/presentation/yang-zongheng [66] Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. 2025. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion. In Proceedings of the Twentieth European Conference on Computer Systems. Association for Computing Machinery, 94–109. https://doi.org/10.1145/3689031.3696098 [67] Xiaoqi Yin, Abhishek Jindal, Vyas Sekar, and Bruno Sinopoli. 2015. A Control-Theoretic Approach for Dynamic Adaptive Video Streaming over HTTP. In Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication (SIGCOMM ’15). Association for Computing Machinery, 325–338. https://doi.org/10.1145/2785956. 2787486 [68] Dongha Yoon, Younghoon Min, Hoshik Kim, Sam H. Noh, and Jongryool Kim. 2025. TraCT: Disaggregated LLM Serving with CXL Shared Memory KV Cache at Rack-Scale. arXiv preprint arXiv:2512.18194 (Dec. 2025). https://doi.org/10.48550/arXiv.2512.18194 arXiv:2512.18194 [cs.DC] [69] Hongli Yu, Tinghong Chen, Jiangtao Feng, Jiangjie Chen, Weinan Dai, Qiying Yu, Ya-Qin Zhang, Wei-Ying Ma, Jingjing Liu, Mingxuan Wang, and Hao Zhou. 2026. MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-Based Memory Agent. In The Fourteenth International Conference on Learning Representations. https://openreview.net/for um?id=k5nIOvYGCL [70] Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Yuxing Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. 2025. Native Sparse Attention: HardwareAligned and Natively Trainable Sparse Attention. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 23078–23097. https://doi.org/10.18653/v1/2025.acl-long.1126 [71] Amir Zandieh, Majid Daliri, Majid Hadian, and Vahab Mirrokni. 2026. TurboQuant: Online Vector Quantization with Near-Optimal Distortion Rate. In The Fourteenth International Conference on Learning Representations. https://openreview.net/forum?id=tO3ASKZlok 9

Ray et al.

[72] Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Vamsidhar Kamanuru, Jay Rainton, Chen Wu, Mengmeng Ji, Hanchen Li, Urmish Thakker, James Zou, and Kunle Olukotun. 2026. Agentic Context Engineering: Evolving Contexts for SelfImproving Language Models. In The Fourteenth International Conference on Learning Representations. https://openreview.net/forum?id= eC4ygDs02R

[73] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized Large Language Model Serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 193– 210. https://www.usenix.org/conference/osdi24/presentation/zhongyinmin

10

Related documents

• Same Request, Different Answer: Quantization Amplifies Cache-Induced Divergence in LLM Serving#971420
• Hardware Acceleration of Block-Diffusion LLM for Edge Devices#821064
• Clairvoyant: Predictive SJF Scheduling to Mitigate Head-of-Line Blocking in Serial LLM Backends#266143
• PEEK: Predictive Queue-Informed KV Cache Management for LLM Serving#1070866
• Poseidon: DAG-Guided Parallelism Search for LLM Pre-Training on Heterogeneous Clusters#973693