arXiv CSopen access

Beyond Storage: State as a Runtime Control Problem in Parallel and Distributed Systems

Beyond Storage: State as a Runtime Control Problem in Parallel and Distributed Systems

arXiv:2607.16836v1 [cs.DC] 18 Jul 2026

SHUHAO ZHANG∗† , Huazhong University of Science and Technology, China HAORAN PENG, Huazhong University of Science and Technology, China CHUN HO MA, Huazhong University of Science and Technology, China YANCAN MAO, ByteDance, Singapore YUFENG DU, Huazhong University of Science and Technology, China SHIFENG LIU, Huazhong University of Science and Technology, China RUIJIE QIU, Huazhong University of Science and Technology, China XIAOFEI LIAO, Huazhong University of Science and Technology, China HAI JIN, Huazhong University of Science and Technology, China Shared state increasingly shapes both performance and failure behavior in streaming, serving, retrieval, and continual-learning systems. Existing studies, however, often isolate access control, hardware-aware execution, memory management, and long-horizon updates. The review organizes this literature around three coupled dimensions: state access and scheduling, state-aware execution, and state evolution and reuse. Across these dimensions, the literature is synthesized through a common scaffold: state object, control surface, coupling path, evaluation boundary, and unresolved contract. This comparison identifies recurring anti-patterns and informs a contract-oriented blueprint and disturbance-aware evaluation agenda. Taken together, the evidence characterizes state management as a runtime control problem. CCS Concepts: • Computer systems organization → Distributed architectures; • Software and its engineering → Middleware; • Information systems → Data management systems; • Computing methodologies → Machine learning. Additional Key Words and Phrases: state management, parallel and distributed systems, stream processing, LLM serving, retrievalaugmented generation, continual learning

1

Introduction

Systems have moved away from largely stateless throughput engines toward long-running services shaped by evolving state. Foundational streaming systems already treated state and time as runtime concerns [7, 8, 160, 244]; newer LLM serving and retrieval systems expose similar pressure through KV caches, scheduling metadata, vector memories, and retained update traces [6, 116, 262, 278, 284]. Across these settings, the central question is whether a runtime can keep shared, long-lived, performance-critical state governable without sacrificing service stability. That shift makes state management in parallel and distributed systems harder to reason about. Traditional decompositions examine operator placement, memory layout, or model adaptation in isolation, but deployed systems do not preserve those boundaries: access hotspots reshape queues and locality, kernel-level gains disappear once data movement or quality constraints dominate, and aggressive memory updates can destabilize downstream retention or retrieval. What matters, then, is not optimization within one phase, but control over a lifecycle in which a local gain often comes back as a later constraint. ∗ Corresponding author. † Supported by New Generation Artificial Intelligence-National Science and Technology Major Project (No. 2025ZD0123804).

.

1

2

Zhang et al. future reuse, freshness, and recovery pressure feed back into access access cost sets execution region

Access & Scheduling contention, locality, ownership

execution choices reshape update debt

State-Aware Execution placement, representation, quality

Evolution & Reuse refresh, retention, replay

Propagation view. A local state action is evaluated by the boundary it later stresses: access skew can destroy locality, execution shortcuts can create quality or movement debt, and aggressive updates can return as future contention or stale-memory exposure.

Fig. 1. A propagation-oriented view of state management. Access, execution, and evolution are coupled through feedback rather than separable pipeline stages.

We use state management to refer to the mechanisms that organize, observe, access, update, retain, and reuse shared state across execution boundaries. The underlying object may be a shared hash table, a compression dictionary, a KV cache, or retriever memory. Whatever its form, the runtime has to make the object visible, account for its access cost, tie it to execution policy, and preserve its future reuse value. We organize the literature around three dimensions that are analytically distinct but operationally coupled: access decisions reshape execution, execution choices change update cost, and evolution policies alter future access conditions. (1) State access and scheduling: how systems expose contention, model access cost, and schedule work around hotspots, topology, and recovery requirements. (2) State-aware execution optimization: how systems turn state layout and execution coupling into stable end-to-end gains under hardware, energy, latency, and quality constraints. (3) State evolution and reuse: how systems continuously write, retain, organize, and reuse state for dynamic learning and inference. Figure 1 summarizes this organizing perspective. Observation. Access, execution, and evolution form one control loop over long-lived state. A paper advances state management when it makes a decision in that loop observable, analyzable, or enforceable. The survey makes four contributions. It first introduces a vocabulary for comparing streaming, approximate execution, serving, retrieval, and memory-centric intelligent services. It then develops a three-axis taxonomy around access, execution, and evolution, identifies recurring anti-patterns together with a contract-oriented blueprint for stateful runtimes, and proposes disturbance-aware evaluation boundaries for mechanisms whose local gains may not survive composition. The emphasis is comparative rather than exhaustive. Section 2 defines the survey scope and analytical model. Section 3 develops the three dimensions of state management. Section 4 compares mechanisms across domains and identifies when they transfer beyond one workload family. Section 5 distills design principles, anti-patterns, and a contract-oriented blueprint. Section 6 gives an evaluation vocabulary and research agenda, and Section 7 concludes. 2

Foundations: Scope, Model, and Taxonomy

We focus on systems in which state is both shared and performance-critical: stream processing engines, transactional streaming systems, heterogeneous analytics runtimes, approximate execution frameworks with quality constraints, and

Beyond Storage: State as a Runtime Control Problem in Parallel and Distributed Systems

3

dynamic AI services such as continual learning and RAG. We exclude static storage engines, model-centric training papers that do not expose runtime state-management mechanisms, and applications in which state is incidental. Our source material includes representative systems papers, journal articles, and recent intelligent-service systems papers. Instead of assembling the corpus around a single lineage, we organize it around recurring control problems. Shared-state access and scheduling, hardware-conscious execution, and evolving memory services anchor the set [153, 227, 262, 263, 266]; streaming, serving, recovery, approximation, and dynamic retrieval systems then test whether those mechanisms transfer across domains [6–8, 116, 160, 244, 278]. Papers enter the analytical core when they externalize a governable runtime seam over state; otherwise, we treat them as contextual. We use three inclusion criteria: • State visibility. The system must surface state as an explicit runtime concern rather than a hidden storage detail. • Exposed mechanism. The paper must expose a systems mechanism such as scheduling, cost modeling, update control, migration, retention, or reuse orchestration. • End-to-end connection. The evaluation must connect that mechanism to end-to-end properties such as throughput, tail latency, recovery time, quality degradation, or long-horizon reuse quality. These criteria favor papers that make control decisions legible. In the retrieval and learning lines, model-centric papers enter the analytical core only when they expose managed memory objects, planner-visible controls, or maintenance boundaries. 2.1

Survey Protocol and Evidence Posture

The review protocol is comparative and mechanism-centered rather than narrowly bibliometric. We seed the corpus from canonical systems lines and adjacent surveys, then trace forward and backward citation paths around shared-state access, state-aware execution, and state evolution. The goal is to cover mechanism families, not to maximize paper counts. When a stable venue version exists, we use it rather than a moving preprint. Recent papers enter the analytical core only when they expose a new state object, runtime control seam, or evaluation boundary. We also distinguish foundational papers, transfer-oriented systems, and frontier systems, so local speedups, transfer demonstrations, and long-horizon governance claims are not all read as the same kind of evidence. The protocol also separates coverage from representativeness. Coverage asks whether the survey includes the mechanism families required by its taxonomy: visibility and progress, ownership transfer, hardware-conscious placement, approximation control, serving-memory lifecycle management, retrieval maintenance, and retention governance. Representativeness asks a different question, namely whether the selected papers expose enough design variation to support meaningful comparison. For example, a single KV-cache allocator would not justify claims about serving-memory lifecycle control, so the survey compares paging, prefix reuse, phase separation, tenant multiplexing, disaggregation, and restoration mechanisms under a common latency and memory boundary. By the same logic, one vector-index paper would not justify claims about retrieval evolution; the survey instead combines retriever invocation, mutable ANN maintenance, publication safety, freshness control, and placement. This distinction matters because the paper is not meant to function as an encyclopedic bibliography. A paper can be historically important yet remain contextual if it does not expose a runtime control surface over state; conversely, a recent system can enter the analytical core if it makes a previously hidden lifecycle boundary measurable. For borderline cases, we ask whether at least one field of the comparison tuple changes. A storage-engine paper, for instance, may remain a boundary case when it assumes stable object identities and clear invalidation semantics, but it becomes

4

Zhang et al.

analytically relevant once compaction, admission, eviction, or tiering is made visible to a scheduler or service-level controller. This rule keeps the scope broad enough to connect streaming, serving, retrieval, and retention without turning the survey into a list of systems that merely store data. The evidence classes used later in the paper follow from this rule. Foundational papers introduce state objects or control boundaries that later systems inherit, such as progress frontiers, checkpoint barriers, ownership metadata, bounded-error summaries, or reusable memory stores. Transfer-oriented papers show that a mechanism remains useful under a related systems constraint: a placement policy survives recovery, a cache policy survives tenant churn, or a retrieval policy survives update drift. Frontier papers expose a control boundary that is not yet mature but appears likely to matter as workloads change, such as disaggregated serving memory, planner-visible retrieval freshness, or budget-elastic retention. This classification helps avoid a familiar survey mistake: treating an early abstraction paper, a mature runtime system, and a speculative frontier system as if they made interchangeable claims. The comparison is strongest when it states which class a paper belongs to and what evidence would be needed to move the mechanism into a stronger class. Omitted measurements also delimit the available evidence. When a paper reports strong local performance but does not measure migration, recovery, freshness, restoration, or quality debt that its mechanism could plausibly affect, the result should not be extrapolated beyond that boundary. For example, an LLM serving paper that improves goodput through aggressive reclamation may still leave open whether the reclaimed state harms prefix reuse under follow-up bursts. A retrieval paper that improves recall-latency trade-offs on a static index may likewise leave open whether deletion repair and publication lag alter the service boundary. The survey’s unresolved-contract field records these limits explicitly. 2.2

Positioning Relative to Existing Surveys

Several prior surveys already cover important slices of this design space, including catalogs of stream-processing design choices [37, 79], transactional stream processing [263], hardware-conscious stream processing [266], and continual learning [39]. These are valuable adjacent references, but they adopt narrower units of analysis than the present manuscript. Our comparison unit is not one application class or one execution substrate; it is the runtime control problem exposed over state. That choice lets us place migration, checkpointing, KV-cache lifecycle control, vector-index maintenance, retrieval freshness governance, and retention budgeting into the same analytical frame without flattening them into a chronology of subfields. This manuscript does not replace specialized surveys. It asks which systems abstractions recur once state becomes the primary object of runtime control across streaming, serving, retrieval, and long-horizon memory services. Adjacent storage-engine and cache-management surveys provide boundary cases: they expose controls such as compaction, admission, eviction, and placement [9, 148], but usually assume stabler object models and clearer invalidation semantics. 2.3

Broader Literature Landscape

Several systems lines expose the same runtime seams under different names. Together they motivate the three-axis decomposition used in this survey: access and scheduling, execution optimization, and evolution and reuse. 2.3.1 Stream and Dataflow Systems. Stream and dataflow systems provide the historical root for the access-andscheduling axis. Aurora and Borealis made operator state and adaptation explicit inside continuous-query plans, while later systems showed that progress tracking, incremental maintenance, elasticity, and recovery all reshape the same

Beyond Storage: State as a Runtime Control Problem in Parallel and Distributed Systems

5

state boundary rather than living in isolated subsystems [1, 2, 12, 20, 21, 24, 69, 155]. Performance and correctness depend on when state is exposed, who owns it, and how safely it can move under disturbance. 2.3.2 Transactional, Replicated, and Memory-Resident Systems. Transactional, replicated, and memory-resident data systems supply the correctness boundary that the rest of the survey reuses. Systems such as H-Store, Spanner, Calvin, RAMCloud, and FaRM treat state as a jointly governed object spanning consistency, placement, logging, and remote access [34, 43, 103, 167, 207]; snapshot and replication results clarify which transitions may be migrated, replayed, or shared without violating service guarantees [25, 166]. This lineage explains why later mechanisms such as ownership transfer, shard exposure, and short-lived memory reuse must be evaluated against explicit authority and replay boundaries, not only against throughput. 2.3.3 Serving and Memory-Governance Systems. Serving and memory-governance systems motivate the executionoptimization axis. Clipper and Clockwork already treated caching, batching, predictability, and dispatch as runtime controls [36, 68]; modern LLM-serving systems make the state objects explicit through KV pages, adapter state, model residency, and prefill/decode separation [6, 116, 174, 196, 239, 251, 278, 282]. Tiered and disaggregated memory systems reach the same abstraction from another direction: extra capacity helps only when the runtime decides which state must stay near compute and which movement or maintenance debt can be deferred safely [26, 32, 84, 92, 128, 176, 181, 185, 218, 275, 283]. Stable execution gains require governed placement, representation, and movement, not faster kernels alone. 2.3.4 Retrieval, Retention, and Approximation Systems. Retrieval, long-horizon memory, continual retention, and approximation-aware systems motivate the evolution-and-reuse axis. Retrieval systems show that reusable memory is valuable only when refresh, lookup, and maintenance remain coordinated [13, 15, 74, 75, 91, 121, 192]. Continual-learning and bounded-retention systems expose the same problem under update pressure, where memory budgets, replay choices, and future reuse value must be co-governed [10, 16, 28, 39, 76, 112, 146, 177, 186, 188]. Approximate analytics adds a complementary constraint: once summaries or samples stand in for full state, the runtime must reason jointly about error, update cost, and future reuse [5, 64, 113, 183, 184]. Together these lines make state a lifecycle control problem: retention, compression, and refresh choices change future freshness, utility, and service stability. 2.3.5 Adjacent Traditions and Analytical Boundary. Several adjacent traditions fix the analytical boundary. Generalpurpose dataflow and storage systems show how large services externalize state into programmable, replicated layers with explicit update semantics [14, 23, 27, 40, 90, 117, 161, 175]. CRDT-style replicated objects, dynamic graph systems, vector indexes, and low-level attention kernels add further boundary cases: merge legality, mutable structure, index maintenance, and memory-traffic shape become runtime-visible once the state object persists long enough to constrain later control [38, 57, 102, 106, 108, 151, 194]. These cases ground the comparison in recurring runtime control problems rather than unrelated applications. 2.4

System Model: Where State Lives in a Runtime

The literature landscape motivates a common runtime abstraction: before comparing mechanisms, we need to specify where state lives and when it becomes manageable. Prior work on stream processing provides a methodological template [21, 22, 152, 157]: first describe computation, communication, deployment, and execution boundaries, then define which internal objects must be exposed as managed state. The modeling style is general: state becomes meaningful only after the runtime boundary around computation, communication, scheduling, and recovery has been made explicit.

6

Zhang et al. We view a parallel or distributed system as a runtime that executes computational units over data items, requests,

or events. These units may be streaming operators, transactions, serverless functions, tasks or actors, LLM workers, retrievers, or agentic stages; they communicate through shuffles, RPCs, remote memory transfers, object-store reads, collective communication, KV-cache transfers, or vector-index lookups. A scheduler assigns work, decides placement and batching, reacts to load changes, and may trigger migration, checkpointing, eviction, refresh, or recovery. State is any runtime object read, written, moved, retained, or reused across these execution boundaries. Formal sketch. A runtime can be abstracted as a tuple R = (W, P, Σ, Γ) where W is a set of computational workers, P is a set of communication paths, Σ is a set of state objects, and Γ is a scheduler mapping work to resources. An object 𝑠 ∈ Σ qualifies as managed state iff it satisfies three conditions: (1) Lifetime(𝑠) ≫ one invocation: 𝑠 persists beyond a single function call, request step, or operator firing; (2) Influence(𝑠) ≠ ∅: there exists at least one future scheduling, placement, recovery, quality, or reuse decision whose outcome depends on the value or location of 𝑠; (3) Controllable(𝑠): the runtime exposes a primitive 𝜋 such that applying 𝜋 to 𝑠 changes its placement, representation, ownership, or retention status. This definition covers both persistent state (database indexes, checkpoint logs) and semi-persistent state (KV-cache pages, retriever-update queues) as long as all three conditions hold. As summarized in Table 1, the concrete state object changes across runtime architectures, but its relationship to the runtime is stable. State constrains computation, communication, scheduling, memory, recovery, and long-horizon service quality because workers must read or update it, moving work often requires moving or reconstructing it, and future decisions depend on its size, ownership, hotness, freshness, and reuse potential. Table 1. State in representative runtime architectures. System class

Computational unit

Stream processing

Operators and tasks

Transactional dataflow and databases Distributed task/actor runtimes LLM serving RAG and agentic workflows Edge and approximate analytics

Communication path

Streams, shuffles, backpressure channels Transactions, execution Dependency exchange, log replicafragments, workers tion, remote reads RPCs, object-store transfers, lineage Tasks, actors, executors replay Prefill and decode workers KV transfer, model-parallel communication, request queues Retrievers, planners, tools, Vector-index queries, tool calls, memgenerators ory updates CPU/GPU operators, sensor Device transfer, edge–cloud transfer tasks

Representative state objects Keyed state, windows, operator state, watermarks, checkpoint metadata Records, indexes, versions, locks, dependency graphs, logs, recovery snapshots Objects, actor-local state, lineage metadata, placement state, scheduler queues KV caches, page tables, prefix trees, adapter state, model-residency metadata Vector indexes, document memories, workflow context, tool traces, retained memories Summaries, sketches, compression dictionaries, qualitycontrol metadata

This model also clarifies why state has multiple coupled forms: logical state captures the semantic object; physical state captures representation and placement; metadata state records ownership, routing, and progress; control state summarizes observations such as hotness or freshness; and evolution state records updates, compaction, retention, refresh, eviction, or reuse. A change in one form often forces movement, metadata updates, scheduler decisions, and new recovery constraints in the others. The model therefore does not force systems into one implementation template; it provides a vocabulary for deciding when an object should be treated as managed state and which lifecycle control problem dominates: access and scheduling, execution optimization, or evolution and reuse.

Beyond Storage: State as a Runtime Control Problem in Parallel and Distributed Systems 2.5

7

A Unifying View of Stateful Computing Systems

2.5.1 What Counts as State? In this survey, state includes any persistent or semi-persistent runtime object whose value influences future execution beyond a single operator invocation. It spans windows, shared indexes, KV caches, retriever memory, and retained samples. Such state is systems-hard because it is shared across tasks, requests, or time windows and temporally extended enough for today’s update decisions to reshape tomorrow’s scheduling, quality, or recovery space. 2.5.2 The Propagation Perspective. We organize state management in parallel and distributed systems using a propagation perspective. A request first encounters state access: it must find, read, or synchronize on shared state. The resulting data path then shapes state-aware execution: work is mapped to cores, sockets, accelerators, or approximate operators under latency and quality goals. The outputs in turn contribute to state evolution: some state is updated, some retained, and some exposed for future reuse. Failures in any one stage propagate. Uncontrolled contention in access can destroy locality and distort execution modeling; mis-modeled execution trade-offs can make update policies unsustainable; over-aggressive updates can increase future access pressure and destabilize inference. The literature repeatedly shows that these seams cross traditional layer boundaries [122, 249, 260, 262, 265, 284], so the propagation perspective provides a more faithful abstraction for long-running stateful systems. Propagation structure. Let the runtime state at logical time 𝑡 be denoted 𝑆𝑡 . Define three stage operators—access (A), execution (E), and evolution (V)—each producing the next state snapshot:

𝑆𝑡 +1 = V E A (𝑆𝑡 ; 𝑊𝑡 , Γ𝑡 ); H ; U𝑡 , where 𝑊𝑡 is the arriving workload, Γ𝑡 captures scheduling decisions, H denotes hardware constraints, and U𝑡 captures update and retention policies. The core claim is that these stages are not separable: 𝜕E ≠ 0, 𝜕A

𝜕V ≠ 0, 𝜕E

𝜕A ≠ 0. 𝜕V

In words, a local perturbation in one stage propagates and reappears as a constraint in the other two. State management is therefore a coupled control loop, not a pipeline of independent optimizations. Observation. Local state decisions do not stay local. Access contention, execution placement, and update policy propagate through the state lifecycle and reappear as constraints on the other two axes, so each mechanism must be read together with the downstream boundary it may quietly reshape. 2.6

Taxonomy of State Management Problems

Table 2 summarizes the taxonomy used throughout the survey. It decomposes runtime control problems rather than application communities: a serving stack, for example, can simultaneously expose access skew, execution-level memory pressure, and evolution-level reclamation debt. The taxonomy is intentionally operational and nonexclusive; a stream engine and a RAG service may both fall under evolution and reuse if the dominant challenge is continuous updates, while a compression engine and a transactional stream processor may both fall under execution optimization if the central problem is coordinating stateful execution with hardware constraints. 2.6.1 A Reusable Analysis Tuple. We apply the same five-field tuple whenever a system line is summarized: what is the state object; what control surface does the runtime expose; what coupling path makes that surface important; what

8

Zhang et al. Table 2. Taxonomy of state-management problems. Dimension

Core question

Representative concerns

Access and scheduling

How is shared state exposed, costed, and scheduled under concurrency?

Execution optimization

How does state interact with hardware and service objectives?

Evolution and reuse

How is state updated, retained, and reused over time?

hotspot diagnosis, conflict propagation, locality, topology, recovery, migration, prefix-aware sharing placement, compression, KV-cache paging, prefill/decode coupling, approximation, latencyenergy-quality trade-offs online updates, memory budgets, cache growth and reclamation, sample selection, retriever drift, knowledge maintenance

evaluation boundary is optimized; and what unresolved contract remains. The tuple keeps the comparison stable across domains and prevents the survey from becoming disconnected implementation summaries. Comparison unit. Each paper under review is normalized to the same five-field comparison tuple Q = (𝜎, 𝜅, 𝜒, 𝛽, 𝛾) where 𝜎 is the state object governed by the mechanism, 𝜅 is the control surface the runtime exposes over 𝜎, 𝜒 is the coupling path connecting local decisions on 𝜎 to system-wide behavior, 𝛽 is the evaluation boundary (the service-level property optimized or bounded), and 𝛾 is the unresolved contract left after the mechanism is applied. Two papers are directly comparable when their tuples differ in at most one field; they address fundamentally different control problems when they differ in 𝜎 or 𝛽. The tuple is meant to be reusable rather than merely descriptive. When reading a new paper, the first pass identifies the concrete state object and its lifetime: an object that exists only inside one kernel invocation does not create the same control problem as a KV page, mutable shard, progress frontier, replay buffer, or retriever memory that survives across requests or failures. The second pass identifies the control surface and coupling path: a mechanism is relevant only if the runtime can act on the object and if that action changes an end-to-end property through contention, locality, memory pressure, quality loss, replay, or future reuse. The final pass asks whether the evaluation boundary matches the coupling path. A scheduler that claims to manage lifecycle state but reports only steady-state throughput leaves a different gap from one that tests burst, drift, or recovery but omits ownership semantics. This procedure also clarifies how future work can extend the survey without rewriting the taxonomy. A new system can be inserted by changing one tuple field at a time. If the state object is familiar but the control surface is new, the contribution is a mechanism refinement; if the control surface is familiar but the evaluation boundary changes, the contribution is a transfer result; if the unresolved contract changes, the contribution is architectural. This discipline is especially useful for fast-moving LLM and retrieval systems, where papers often introduce new names for related cache, memory, or planner objects. Normalizing them through the tuple makes it harder to over-credit a local optimization and easier to see when a paper has actually exposed a new runtime boundary. 2.6.2 Representative System Classes. The state-management lens cuts across at least five recurring system classes: high-throughput streaming and transactional dataflow, hardware-conscious analytics, LLM serving and memory-bound inference, continual learning and adaptive services, and memory-centric retrieval or agentic inference. They matter

Beyond Storage: State as a Runtime Control Problem in Parallel and Distributed Systems

9

Table 3. Reusable analysis tuple for extending the survey. Question

What to extract from each paper

Typical answers in this survey

State object

Which runtime object persists across tasks, requests, or time? What decision does the system make over that object? Why does this state decision propagate to end-toend behavior? Which system-level property is optimized or bounded? What is still missing after the paper’s contribution?

windows, indexes, logs, dictionaries, KV caches, retriever memories, coresets placement, batching, migration, paging, admission, retention, reuse orchestration contention, locality, memory pressure, phase asymmetry, drift, quality-loss amplification throughput, p99 latency, recovery time, energy, bounded error, forgetting, reuse quality weak observability, no closed loop, limited portability, incomplete lifecycle control, no long-horizon evaluation

Control surface Coupling path Evaluation boundary Unresolved contract

because they expose structurally different failure modes, yet the same runtime ideas often reappear under different names: • Streaming and transactional dataflow fail by contention amplification, migration cost, or slow recovery when state ownership and progress semantics are not co-designed. • Hardware-conscious analytics fail by misplacing state relative to device topology or by violating quality boundaries when using approximate methods. • LLM serving systems fail when KV-cache growth, fragmentation, or prefix-sharing policy turns memory into the primary throughput bottleneck. • Continual-learning systems fail by exhausting retention budgets or forgetting useful history when admission and replay policies remain static. • Memory-centric retrieval and agentic systems fail by update drift, unstable retrieval, or uncontrolled growth in reusable state. 3

Core Dimensions of State Management

The recurring control problems can be grouped into three coupled dimensions: state access and scheduling, state-aware execution, and state evolution and reuse. A local gain in one dimension often returns as a downstream constraint on the other two. We keep the discussion grounded with three running examples: (A) a streaming fraud detector under hotspot skew, (B) a multi-tenant LLM assistant under KV-memory pressure, and (C) a RAG knowledge service under corpus drift. 3.1

State Access and Scheduling

3.1.1 From Invisible Contention to Observable State Access. In running example A, a fraud-detection pipeline suddenly receives a skewed burst after a campaign launch. The model logic has not changed, but a small set of hot keys now serializes queue and lock paths. The practical question is therefore not “can the operator run faster,” but “can the runtime detect and reshape shared access before the skew turns into replay and migration debt.” As multicore stream and event systems scaled, this issue became hard to ignore: operator logic was often not the main bottleneck, because hidden contention in shared access paths could dominate end-to-end behavior. Studies of multicore stream processing and complex event processing made that point concrete. They showed that shared queues, synchronization patterns,

10

Zhang et al.

fine-grained operator interactions, and sub-computation sharing can all limit scalability when the runtime cannot see where interference is forming [259, 264]. State management begins with observability, but access metrics are useful only if they preserve enough structural context for the runtime to decide whether the right response is repartitioning, migration, admission throttling, or a change in execution grain. Earlier stream systems exposed the same lesson at different boundaries. Aurora and Borealis made operator graphs and adaptation policies runtime concerns [1, 2]; MillWheel, Naiad, and Dataflow made time and completion visible [7, 8, 160]; Trill, Differential Dataflow, Flink, and StreamCloud connected progress tracking, incremental maintenance, elasticity, and fault handling to the same state boundary [21, 24, 69, 155]. Together they define a ladder of control surfaces: adaptation makes placement explicit, progress semantics determine when state may advance, and ownership transfer determines where it may move. 3.1.2 Cost Modeling Under Locality and Topology. Returning to Example A, once the fraud-detection runtime has identified a hotspot, it must estimate the cost of repartitioning or migrating the hot keys under the current NUMA topology, checkpoint state, and downstream dependencies. That is a cost-modeling problem rather than a detection problem alone. Once access patterns become visible, the harder task is to price them correctly under skew and topology. Topology-aware and concurrent stateful streaming systems suggest that hotspot placement, NUMA distance, synchronization design, state partitioning, and task assignment have to be considered together; locality decisions that ignore any one of these factors can wipe out the gains from parallelization [260, 265]. Taken together, these results point to a broader systems lesson: locality does not come from state layout alone, but from its interaction with scheduling and workload structure. Similar patterns appear in Spacker’s migration substrate [152], interval-join systems with shared indexes and ordering constraints [255], deployment planners such as MaveriQ [130], and intra-window join studies in which execution style, join method, and partitioning reshape skew and latency trade-offs [261]. Across these works, it is more accurate to think in terms of an access-cost surface shaped by contention and hardware placement than in terms of a static data structure. Partitioning and synchronization choices also shape later recovery behavior by determining checkpoint grain, replay skew, and migration debt under disturbance. Topology-aware placement, concurrency control, and state-transfer design should therefore be considered together [20, 152, 260, 265, 272]. Access management cannot stop at hotspot detection: a runtime that cannot account for the recovery consequences of a placement decision addresses only part of the state-management problem. 3.1.3 Runtime Control, Recovery, and Stateful Governance. Example A again: once the fraud detector’s access-cost model decides that migration is warranted, the runtime must execute the move without losing in-flight transactions or stalling downstream operators. That execution requires runtime control and stateful governance: the runtime must know who owns a state shard, where replay may safely resume, and when the old owner can be reclaimed. Effective access management therefore extends to runtime control. MorphStream and related transactional-stream systems treat execution as a scheduling problem over stateful dependencies; Flink’s asynchronous snapshots and fast parallel recovery show that checkpoint barriers, in-flight records, replay metadata, and access control belong to the same runtime path rather than to separate steady-state and failure-time subsystems [20, 153, 272, 274]. Earlier elasticity work and later migration substrates reach the same conclusion from another angle: SEEP, operator-state management, Megaphone, and Spacker all show that operator ownership, transfer granularity, and recovery semantics are inseparable parts of one state-governance problem [22, 80, 152, 157].

Beyond Storage: State as a Runtime Control Problem in Parallel and Distributed Systems

11

Across these systems, the natural abstraction is an ownership-transfer contract: (i) fence or freeze updates, (ii) transfer a consistent shard snapshot plus progress metadata, (iii) hand off authority with idempotent replay boundaries, and (iv) reopen updates under a new owner. Megaphone, Spacker, Flink’s asynchronous barriers, and fast-recovery designs each expose part of this contract [20, 80, 152, 272]. Adjacent transactional and replicated systems make the missing fields clearer: Spanner, Calvin, Raft, and RAMCloud show that a reusable handoff protocol needs current authority, replay boundary, and reclamation condition for the old owner [34, 166, 167, 207]. Most systems still encode these fields implicitly in separate migration and recovery paths. Comparable governance patterns appear in workflow schedulers, checkpointing systems, large-model execution stacks, and serving-oriented schedulers, all of which make warm state, snapshot timing, KV residency, or cross-workload handoff visible to the scheduler [83, 88, 131, 135, 142, 149, 150, 165, 212, 214, 273]. In these systems, recovery, rebalancing, serving admission, and utilization control can be viewed as closely coupled scheduling decisions over shared state. When migration, checkpointing, serving admission, and rescaling rely on separate ownership models, they can incur repeated coordination costs and brittle recovery semantics. Access governance would therefore benefit from a unified path linking observation, cost modeling, scheduling, recovery, and subsequent observation. 3.1.4 Open Challenges in Access Management. Despite the progress, the access dimension leaves five recurring control gaps: • Conflict observability. Most systems still rely on coarse metrics and do not expose state conflicts, ownership pressure, and migration debt as first-class runtime objects. • Explicit handoff contracts. Recovery, migration, and rescaling often maintain separate notions of authority, replay boundary, and reclamation condition. • Lifecycle-closed serving control. KV admission, eviction, transfer, and restoration are usually optimized separately even though they share one latency and memory boundary. • Semantic shared state. Intelligent-service runtimes increasingly create vector indexes, retriever memories, prefixshared KV caches, and workflow context whose conflict patterns are semantic, dynamic, and cross-request. • Compositional controllers. Local access policies rarely compose cleanly with topology changes, disaggregation, colocated training, or memory middleware. These gaps push access modeling beyond local contention management toward ownership and lifecycle contracts over dynamic state objects. 3.2

State-Aware Execution Optimization

3.2.1 Why Faster Kernels Do Not Guarantee Better Services. Running example B highlights the same point in serving form: a multi-tenant assistant with mixed prompt lengths may have excellent single-kernel speed, yet still miss SLOs once KV residency, prefill/decode phase asymmetry, and adapter multiplexing collide. The runtime wins only when it governs where short-lived state lives and when that state moves. A common limitation is to equate stateful execution with faster kernels or higher operator throughput. Once state interacts with heterogeneous hardware, data movement and coordination often dominate raw compute speed, while latency, energy, and quality objectives can overturn microbenchmark gains. Integrated CPU-GPU stream systems make this point concrete. Fine-grained window processing on integrated architectures showed that topology and data path design directly alter the feasible performance envelope, while co-running studies on the same class of hardware reached a similar conclusion from the broader angle of shared-resource

12

Zhang et al.

interference [205, 252, 253]. These systems emphasize that execution gains emerge only when state organization, data movement, and device mapping are optimized together. 3.2.2 Energy, Compression, and Stateful Data Paths. Example B surfaces a variant of this point: when the multi-tenant assistant offloads certain KV pages to CPU DRAM to relieve GPU memory pressure, the decision is not merely “which pages to evict” but which representation (full precision, quantized, or compressed) makes later restoration affordable without violating the p99 SLO. The system is navigating a stateful data path, not choosing a single compression level. The CStream line deepens this theme by examining stateful compression on edge and asymmetric multicore devices [247, 248]. Compression dictionaries are state: their access pattern, update granularity, and task decomposition shape energy, latency, and compression ratio. The observation generalizes beyond compression. Whenever intermediate state crosses operators or requests, device selection, data movement, and representation choice become one coupled decision rather than independent knobs. 3.2.3 KV-Cache Management as a State-Execution Problem. Large language model serving makes this point unusually concrete. In these systems, the KV cache is not a minor implementation artifact. It is the dominant short-lived runtime state that grows with prompt and output length, mediates prefix reuse, and constrains feasible batch size. Once the runtime hits memory pressure, almost every higher-level serving policy becomes a decision about KV ownership, residency, transfer cost, or reclamation timing. The serving literature can be read through six control surfaces: orchestration, memory layout, representation, tiering, disaggregation, and structured sharing. Orchestration systems such as Clipper, Clockwork, Nexus, Orca, InferLine, and INFaaS show that queue state, batching opportunity, provisioning state, and stage imbalance are themselves runtime-managed controls [35, 36, 68, 189, 195, 237]. vLLM and Sarathi-Serve then make memory layout and phase asymmetry explicit by treating KV memory as a paged state object whose footprint evolves differently in prefill and decode [6, 116]. DistServe and Splitwise extend the same logic across worker boundaries: once prefill and decode are disaggregated, the control problem shifts from local placement alone to queue isolation, inter-stage transfer, and goodput-aware admission [174, 282]. Recent systems refine this lifecycle by changing representation, tiering, or sharing semantics. QServe, Oaken, Keyformer, Q-Hitter, and VQ-LLM treat the cache as a representation-control problem, trading fidelity, memory footprint, and future reuse value inside one execution boundary [3, 110, 136, 145, 271]. FlexGen, Neo, AQA, LServe, LoongServe, and LeanAttention expose complementary tiering and long-context surfaces in which bytes may be reduced, moved, or avoided through sparse and tiled access [97, 191, 198, 209, 224, 234]. Punica, AlpaServe, SGLang, and FastGen add multi-tenant, model-parallel, structured-program, and long-prompt variants, showing that adapter sharing, partition multiplexing, and phase transition control all belong to the same short-lived state lifecycle [31, 81, 129, 278]. These mechanisms differ mainly in which boundary they protect first: queue stability, memory efficiency, or phase-local goodput. Their shared lesson is stronger than any single serving optimization: KV-cache management is a state-management problem because the cache persists across token steps, affects future scheduling, exposes reuse across requests, and demands explicit allocation, sharing, reclamation, placement, and transfer policies. Closely related work on compressed stream processing without decompression [268] and data-aware adaptive compression [269] extends this view to database-style execution. Here, state-aware execution means choosing representation, operator placement, and access path jointly. The system’s real unit of optimization becomes a stateful data path rather than an isolated operator.

Beyond Storage: State as a Runtime Control Problem in Parallel and Distributed Systems

13

3.2.4 Approximation and Quality Boundaries. Example B sharpens this lesson further: if the serving runtime applies token-level KV eviction to fit more concurrent requests, it must reason not only about memory savings but also about the quality floor at which downstream answer utility degrades faster than throughput improves. The same quality-boundary logic that governs approximate stream joins governs KV-budget management. Approximate execution adds output quality to the control boundary. PECJ [249] addresses disorder in stream window joins by compensating for missing or late data using explicit error modeling. LibAMM [246] shows that algorithm choice, dataset properties, and memory behavior jointly determine whether approximate matrix multiplication yields a stable efficiency-accuracy trade-off. FreeSAM [206] and LEAP [233] reach similar conclusions for joins and video databases: approximation works only when quantity, quality, and system cost are modeled together. When intermediate or retained state influences correctness, the runtime needs an execution boundary, not a single performance target. 3.2.5 Toward Closed-Loop Execution Control. Existing systems provide the ingredients for online execution control: topology-aware mapping [252], energy-aware decomposition [248], and quality-aware compensation [249]. The missing piece is an integrated runtime that combines state observability with online execution control across heterogeneous processors and service objectives. The need is acute in inference stacks, where accelerator choice, memory transfer, approximation, and retrieval quality interact. Observation. Execution gains emerge only when state organization, data movement, and device mapping are optimized together. The unit of optimization is a stateful data path, not an isolated operator or kernel. Stable improvements require representation, placement, and quality boundaries to be controlled together. 3.3

State Evolution and Reuse

3.3.1 From Updates to Long-Horizon Memory. Running example C makes the evolution problem tangible: a RAG service ingesting daily document churn can keep answering queries, but quality silently drifts unless refresh cadence, shard exposure, and compaction debt are co-managed. “Update more” and “serve faster” become conflicting objectives unless the runtime prices their future interaction. The third major dimension of state management in parallel and distributed systems therefore concerns what happens after execution produces new information. In many older data systems, state updates were treated as maintenance; in dynamic AI services, updates are themselves central because the system must decide what to absorb, what to retain, what to discard, and how to expose retained state for future reuse. Online adaptation and clustering systems make this shift visible. SentiStream [227] frames model update as part of a coupled co-training loop, while MOStream [221] and empirical stream-clustering studies [219] expose summarization, windowing, refinement, and outlier control as interacting maintenance decisions. Online update is therefore not one mechanism but a coupled control problem. 3.3.2 Memory Budgets, Retention, and Sample Selection. Example C illustrates the pressure directly: if the RAG service’s vector index grows by 50k documents per day while its memory budget remains fixed, the runtime must decide which older embeddings to compress, demote, or evict—and those decisions shape which queries can still retrieve relevant context tomorrow. Retention is not a background garbage-collection task; it is a forward-looking service-quality control. Once updates become continuous, retention becomes equally important: FERRET [284], StreamFP [122], and CANDOR-Bench [217] show that update rate, memory budget, pipeline structure, and vector-index churn must be coordinated so the runtime can preserve future reuse value while controlling immediate update cost.

14

Zhang et al.

3.3.3 Structured Memory and Dynamic Retrieval. Retrieval-augmented and knowledge-enhanced systems make reuse explicit by turning external knowledge into a managed runtime substrate. The relevant substrate has four layers: retriever-facing memory, retrieval policy, index structure, and maintenance. FlowRAG shows that reuse quality decays as corpora shift without controlled retriever adaptation [262]; KELDAR and Global Planning show that planner-visible knowledge structures can become runtime-managed memory objects [125, 126]. Recent memory-centric systems begin to make the middleware layer itself explicit. Neuromem decomposes memory into insertion, consolidation, retrieval, and integration stages, making latency cost and accuracy decay visible as lifecycle properties [258]. SAGE points in a workflow-native direction by exposing pluggable vector-index layers, asynchronous update queues, declarative window operators, and dual-stream joins as modular stateful services [139]. Placed alongside FlowRAG and Self-RAG, these lines suggest that long-horizon reasoning will eventually require memory middleware that externalizes both retrieval policy and memory-maintenance policy [13, 262]. The missing layer is a policy kernel that can decide when to consolidate, refresh, defer, or expose memory objects across heterogeneous reasoning workflows. Modern retrieval pipelines expose concrete control surfaces. REALM, RAG, RETRO, and Atlas make document memory explicit [15, 75, 91, 121]; Self-RAG, RAPTOR, Global Planning, and KELDAR add retrieval-policy and planning layers [13, 125, 126, 192]; DPR, ColBERT, PQ, HNSW, and HippoRAG expose vector-state representation, traversal, compression, and graph-structure choices [74, 102, 106, 108, 151]. CANDOR-Bench and FlowRAG then make the maintenance boundary concrete: insertion pressure, deletion churn, rebuild cadence, and freshness policy can move an index off its recall-latency operating point before the retriever architecture changes [217, 262]. The comparison separates four roles: memory systems make reuse visible, planning systems make traversal policy visible, vector-index systems make representation and maintenance debt visible, and dynamic-memory systems make freshness timing visible. What remains underdeveloped is middleware closure: few systems expose one policy kernel that governs retrieval policy, index maintenance, freshness exposure, and cross-workflow reuse together. KV caches are a useful boundary case because future decoding cost depends on whether short-lived state is retained, shared, compacted, or discarded at the right moment. 3.3.4 Bridging Evolution with Access and Execution. All three running examples converge here. In Example A, aggressive hot-key migration (an access decision) reshapes future checkpoint cost (an evolution concern). In Example B, token-level KV eviction (an execution decision) reduces future prefix-reuse value (an evolution loss). In Example C, deferred index compaction (an evolution deferral) concentrates future query traffic onto a shrinking set of fresh shards (an access hotspot). Each case shows that one-axis optimization silently shifts debt onto the other two. A central implication is that evolution cannot be managed independently: continuous updates create future access hotspots, retention decisions alter memory footprint and execution cost, and retrieval quality affects downstream approximation choices. Memory evolution should be an online control loop coupled to access and execution. 3.4

Quantitative Propagation Trace: A Worked Example

To make the propagation claim falsifiable rather than merely aspirational, we trace one concrete decision through all three axes using Example B (multi-tenant LLM serving) with representative numbers drawn from the vLLM [116] and Sarathi-Serve [6] literature. Under bursty multi-tenant load, an admission decision that looks locally beneficial can trigger KV eviction, destroy prefix reuse, increase TTFT, and create restoration debt that feeds back into future access pressure. In five-field terms, the state object is shared KV-cache state; the control surface is admission plus eviction; the coupling path is prefix-reuse destruction; the evaluation boundary is latency and goodput; and the unresolved contract

Beyond Storage: State as a Runtime Control Problem in Parallel and Distributed Systems

15

is a lifecycle-closed controller over admit, expose/transfer, evict/reclaim, and subsequent re-materialization costs. The example shows why eviction debt belongs on the future-reuse ledger, not only on the instantaneous-throughput ledger. Consider a multi-tenant LLM serving cluster with continuous batching, mixed prompt lengths, shared prefix reuse, and a fixed memory budget per decode pool. The runtime must decide whether to admit a new request whose prompt is long enough to threaten the current KV working set. At first glance, admitting the request looks locally beneficial: utilization stays high, and the prefill stage can begin immediately. The decision also changes the future legality and value of shared prefix state. The coupling path unfolds in five steps. First, admission pressure rises when a long-prompt request arrives while the decode pool is already populated with shorter ongoing sessions. Second, local memory slack shrinks: admitting the request widens the active prefill footprint and may force reclamation of KV pages belonging to lower-priority or seemingly colder sessions. Third, reuse structure is damaged because some reclaimed pages would have supported prefix reuse or cheaper continuation for later requests from the same conversation family or structured workflow. Fourth, restoration debt appears when the next burst or follow-up call must repopulate state that was previously available cheaply, competing with live decode work. Fifth, the original gain can reverse: an admission decision that improved instantaneous occupancy can reduce goodput or inflate p99 latency after the coupling path fully unfolds. Observation. Lifecycle debt is temporal before it is numerical. The runtime can spend future reuse value to improve present occupancy, and that trade is invisible unless admission, reclamation, and restoration are evaluated on the same boundary. The example isolates four serving-lifecycle boundaries. Admission-versus-restoration tradeoffs unfold over time, not at one instant. Prefix reuse is also a legality question: which shared state may be reclaimed? Burst-after-warmup and tenant-switch scenarios are necessary to expose lifecycle debt. The governed object is not one allocator event but a trajectory across admission, reclamation, and later restoration. 4

Comparative Synthesis Across Domains

The preceding sections established a mechanism inventory, but coverage alone does not show which lessons transfer. Normalizing the surveyed work to one comparison unit—state object, control surface, coupling path, evaluation boundary, and unresolved contract—separates transferable mechanisms from domain-local optimizations that look strong in isolation but do not compose across workloads and disturbance regimes. 4.1

Comparative Criteria and Integration Principles

Using that tuple, we group work by recurring control problem rather than by venue order or application label. A mature comparison contrasts multiple mechanisms under one service boundary and names the contract still missing. A paper that makes a seam visible is not yet equivalent to one that shows the seam remains governable under disturbance, multi-tenancy, and lifecycle interaction. 4.2

Cross-Domain Comparative Synthesis

Across the surveyed areas, four seams dominate: streaming centers on visibility and ownership, hardware-conscious execution on movement and quality boundaries, serving on short-lived lifecycle control, and retrieval plus retention on maintenance debt over longer horizons. Figure 2 and Table 4 summarize these seams as a visual index and a compact reading key. The domains do not need to share terminology for the same control pattern to recur: each must identify a

16

Zhang et al.

Domain state

Runtime control

Protected boundary

Open contract

Streaming windows, frontiers

visibility advancement ownership transfer

correctness pause-time stability recovery latency

one transfer contract across steady state and recovery

Hardware-aware encoded paths

format choice placement approximation

movement cost energy quality bound

portable policy that prices movement and error debt

LLM serving KV pages, queues

admit, share reclaim transfer, restore

TTFT / TPOT goodput tenant SLOs

lifecycle semantics for reuse and invalidation

Retrieval & retention shards, replay stores

refresh publication replay, demotion

freshness reuse value maintenance debt

exposure contract for stale or demoted memory

Fig. 2. Cross-domain comparison. The visual index summarizes the recurring seams discussed in the comparative synthesis. Table 4. Compact cross-domain reading key. Domain cluster

Dominant state object Mature control surfaces Boundary usually pro- Persistent open seam tected first

Streaming and transac- windows, frontiers, de- visibility, advancement, correctness plus pausetional dataflow pendency state, mov- migration, replay, own- time stability able shards ership transfer Hardware-conscious encoded paths, com- format choice, movement cost and execution pressed intermediates, placement, quality boundary tiered residency state decompression path, approximation steering LLM serving and short- KV pages, phase admission, paging, TTFT, TPOT, and goodlived lifecycle control queues, residency sharing, disaggregation, put under tight SLOs layers transfer, restoration Retrieval and retention mutable shards, invocation, refresh, freshness, reuse value, governance freshness metadata, publication, replay, maintenance stability replay stores, compression, demotion, demotable memory budget shrink objects

one ownership-transfer contract across steady state, elasticity, and recovery portable control that prices movement and error debt together lifecycle closure over reuse legality, isolation, restoration, ownership policy-visible exposure contract for stale or demoted memory

state object, expose a control surface, protect a service boundary, and name the contract that prevents local gains from composing cleanly. 4.2.1 Streaming and Transactional Dataflow Systems. Streaming and transactional dataflow systems are the clearest historical example of state management becoming a runtime control problem rather than a storage afterthought. Their dominant state objects are windows, progress metadata, dependency state, and movable operator shards; their mature

Beyond Storage: State as a Runtime Control Problem in Parallel and Distributed Systems

17

controls govern visibility, advancement, placement, and replay [7, 21, 24, 153, 160]. Progress semantics answer when a transition is safe, while ownership-transfer mechanisms answer where it may occur; strong systems need both. The tradeoff is between finer-grained control and stronger correctness obligations. Topology-aware placement, elasticity, and fast recovery reduce pause time or contention only when the runtime can preserve progress fences, dual-ownership bounds, and replay legality across disturbance [80, 152, 260, 272]. The more durable target is a reusable ownership-transfer contract that spans steady state, reconfiguration, and recovery under one disturbance-aware policy, not another isolated migration heuristic. 4.2.2 Hardware-Conscious and Approximation-Aware Stateful Execution. Hardware-conscious execution shows that stateful efficiency depends on movement and representation choices as much as arithmetic intensity. Here the dominant objects are encoded data paths, compressed intermediates, and tiered-residency state; the mature controls govern format, placement, decompression path, and approximation budget [248, 252, 268, 269]. Approximation-aware systems add the boundary model that makes this domain comparable to the rest of the survey. Once a runtime saves work by sampling, compressing, or summarizing state, it must also govern update debt, reuse value, and error stability over time [5, 64, 113, 184, 246, 249]. Read together, these papers expose three recurring families: state-path shaping, budgeted approximation, and online boundary steering. Quality budgets become scheduling inputs, not post-hoc validation checks. Portability across hardware and disturbance regimes remains weak. Capacity expansion, offload, and phase specialization help only when the runtime can also account for movement debt, metadata locality, and ownership overhead across devices and tiers [93, 118, 140, 162, 187, 256]. In comparative terms, this domain contributes a clear rule: state representation is a dynamic control variable, and its error budget belongs inside the protected service boundary. 4.2.3 LLM Serving and Short-Lived Memory Lifecycles. LLM serving is the clearest recent case where state management effectively is the runtime. KV pages, prefix-sharing metadata, phase-skewed queues, and multiplexed adapter or model-residency layers often define the service envelope more directly than the model graph alone [6, 31, 116, 129]. The local control problem is therefore not generic batching, but how a runtime governs short-lived state through the same canonical lifecycle used elsewhere in the survey: admit, place, mutate, expose or transfer, compact, and evict or reclaim. In serving, these stages take domain-specific forms: structural reuse and adapter-scoped sharing clarify when state may be exposed to a new request path; disaggregation and offload mechanisms govern transfer; and restoration debt appears as a downstream consequence of earlier evict or reclaim decisions. The literature is most comparable when grouped into a few control seams over one lifecycle: SLO-facing orchestration, KV allocation and sharing, prefill/decode phase control, and tenant or residency governance [6, 31, 36, 68, 116, 174, 196, 239, 251, 282]. Later predictive, structural-reuse, and temporal-lifecycle systems sharpen the same point from different angles: short-lived serving memory should be treated as a governed lifecycle rather than as a passive buffer pool [4, 50, 55, 62, 63, 65, 72, 98, 147, 158, 168, 180, 201, 202, 225, 235, 238, 241]. The first seam is predictive control. Systems such as WindServe, GLLM, Past-Future, HETIS, Seesaw, and ThunderServe do not merely predict load; they predict different future debts and bind those predictions to different actuation points [50, 65, 72, 98, 158, 201]. Some forecasts protect the prefill/decode balance, some anticipate future KV peaks, and others expose heterogeneous placement pressure before it becomes queue amplification. The systems lesson is therefore narrower and stronger than “prediction helps.” A prediction is useful only when the runtime can say which later cost it is avoiding, which action surface can still change that cost, and when a wrong prediction will be charged back to the same SLO boundary.

18

Zhang et al. The second seam is reuse legality. Structural-reuse systems such as PromptCache, Marconi, ICCache, CacheBlend,

and PIE make prompt segments, chunks, branches, or adapter-scoped state visible enough that the runtime can decide whether reuse is legal rather than merely cheap [62, 63, 168, 235, 241]. This differs from footprint reduction. Compression, paging, and offload mechanisms change the representation or location of state that has already been admitted as relevant; structural reuse decides whether the state object may be shared in the first place. Confusing these two control questions hides the isolation and invalidation contract behind memory-efficiency numbers. The third seam is temporal lifecycle control. Pensieve, Mooncake, and Symphony operate at the session-restoration or disaggregated-cache boundary by preserving reuse value across turns and devices [4, 180, 238]. FastServe and LLMnix operate during active contention, where preemption, migration, and priority redistribute latency debt over live request state [202, 225]. HydraServe and ServerlessLLM operate earlier at the bootstrap-residency boundary, where loading and tiered initialization determine which state can become reusable later [55, 147]. Grouped this way, the serving literature still fits the survey’s canonical lifecycle, but with serving-specific emphasis on reuse legality inside expose/transfer and on restoration debt after evict/reclaim, rather than on a separate domain-exclusive stage list. This distinction also clarifies why local serving metrics are often misleading. Allocator efficiency can improve while restoration cost grows; prefill/decode separation can improve goodput while transfer ordering becomes the next bottleneck; prefix reuse can reduce compute while weakening tenant isolation or invalidation semantics. A mature serving runtime therefore needs to state which reuse is legal, which state may be reclaimed, when restoration debt is charged, and whether the SLO boundary is request-local, session-local, or tenant-wide. The strongest serving papers expose at least part of this contract; the open problem is to make the whole lifecycle portable across batching, disaggregation, adapter multiplexing, and speculative execution. The unresolved issue is contractual. Current systems expose local interference more clearly than they reconcile prediction, reuse legality, tenant isolation, and restoration under one service boundary. Practical runtimes still lack portable semantics for when KV fragments may be shared, transferred, restored, or invalidated across continuous batching, adapter multiplexing, speculative branches, and disaggregated decode paths. 4.2.4 Retrieval Memory and Vector-Index State Layers. Retrieval systems are easiest to misread as a sequence of better retrievers, but the more durable systems lesson is about memory governance. Once retrieval moved from static lexical indexes to dense and hybrid memory substrates, invocation, refresh, exposure, and maintenance became coupled runtime decisions rather than offline preprocessing details [75, 106, 108, 121]. The local control problem is therefore how to keep a living memory layer queryable without hiding freshness debt, rebuild debt, or exposure risk. The literature is most comparable as a trigger hierarchy: invocation policy, index maintenance, safe publication, and placement or movement policy [13, 15, 29, 71, 75, 91, 102, 121, 151, 182, 199, 215, 231, 232, 242, 270]. Planner-mediated and continuous-maintenance systems make the missing coupling explicit: one controller can preserve short-run latency by querying stale memory while another silently accumulates rebuild debt [137, 217, 262]. The trigger hierarchy is worth making explicit. Request-time invocation decides whether external memory should be consulted at all; index-time maintenance decides whether insertion, deletion, repair, or compaction has pushed the memory substrate away from its intended recall-latency operating point; publication-time exposure decides whether a partially refreshed shard may serve internal traffic, restricted traffic, or general requests; placement-time residency decides whether movement is justified by downstream quality gain rather than lookup throughput alone. These triggers often live in different modules, but the user-facing answer quality depends on their ordering. A retriever can appear

Beyond Storage: State as a Runtime Control Problem in Parallel and Distributed Systems

19

accurate in a static benchmark while its publication lag, deletion debt, or rebuild backlog makes the live memory layer unsafe to expose. Retrieval systems still lack a compositional trigger policy that orders stale-memory tolerance, rebuild triggers, shard-exposure legality, and movement under one service boundary. Without that contract, reported gains remain hard to compare across request-time quality, maintenance stability, exposure safety, and placement efficiency. Retention systems arrive at the same abstraction from a different timescale. Retrieval asks when partially stale memory is still safe to expose; retention asks when compressed, demoted, or budget-constrained memory is still safe to preserve and replay. In both cases, the systems question is when the runtime can still treat memory as legally useful. 4.2.5 Continual Learning and Retention Governance. Continual-learning systems matter here not mainly because they report forgetting curves, but because they provide a runtime language for bounded retention. Once memory is scarce, the main question is which historical state deserves protection, admission, replay, compression, demotion, or eviction as future updates arrive [10, 16, 76, 112, 146, 177, 186]. The reusable comparison is therefore between lifecycle controls over one bounded store rather than between named replay heuristics alone. Protection methods decide what must not drift; admission methods decide what enters the scarce retained set; replay methods decide when stored state should re-enter the update path; compression methods decide whether a cheaper representation preserves enough future utility; and systems such as Ferret and StreamFP make budget elasticity and future-use ranking explicit runtime concerns [28, 76, 112, 122, 146, 177, 186, 284]. This same logic extends to broader long-horizon reuse settings such as federated adaptation and tiered training-state storage [107, 114, 250]. The missing abstraction is a retention-store contract that composes protect, admit, replay, compress, demote, and budget-shrink decisions under one service boundary, rather than treating end-task accuracy as the only judge [39]. The retrieval-retention comparison exposes a useful asymmetry. Retrieval systems usually know which memory object is queryable now but struggle to price the future cost of stale exposure and rebuild debt. Retention systems usually know the memory budget now but struggle to price the future value of a protected, compressed, or demoted item. In both cases, the runtime needs a demotion rule: when does memory remain fully usable, when does it become restricted, when is it only useful for replay or reconstruction, and when may it be dropped? Making those states explicit would let freshness, recall, forgetting, and budget pressure be evaluated as different views of the same lifecycle problem rather than as unrelated metrics. 4.2.6 Comparative Mechanism Matrices. The matrix makes the comparison unit explicit. It does not rank systems; it shows how the same five fields expose recurring seams across different state lifetimes and evaluation boundaries, and it keeps the comparison from collapsing into a list of domain-specific optimizations. 4.3

Analytical Maturity

Across access, execution, and evolution, local gains often disappear once effects propagate through the state lifecycle. Modern runtimes increasingly schedule around mutable state objects, but they still lack stable inter-layer contracts for observability, lifecycle coordination, policy composition, and long-horizon debt accounting [6, 116, 248, 260, 262, 284]. Evaluation should therefore treat state management as closed-loop control: every experiment should name its state object, control surface, and service boundary before reporting gains, then test burst, skew, topology, or semantic disturbance so stationary replay does not hide lifecycle debt [6, 116, 153, 260]. The next stage of progress is architectural: observability linked to service drift, state-centric scheduling, recovery-aware lifecycle control, and explicit semantics for composing overlapping policies [6, 116, 260].

20

Zhang et al. Table 5. Representative mechanism matrix for cross-domain comparison. Cluster

State object

Control surface

Evaluation boundary Remaining gap

Streaming visibility windows, frontiers, visibility, advancement, correctness plus la- progress and moveand progress [7, 8, 160] progress metadata buffering, replay timing tency ment remain weakly unified Streaming migration movable shards, replay migration trigger, trans- pause time, recovery ownership-transfer and recovery [20, 80, logs, ownership meta- fer grain, replay sched- time, overhead legality across 152, 272] data uling, barrier alignment steady state and disturbance Hardware-conscious encoded paths, local and format choice, speed, energy, portability across execution [248, 252, remote representations placement, latency, movement devices and tiers 268] decompression path, cost execution mapping Approximation samples, summaries, sample sizing, latency/error or service-level quality control [5, 184, 246, compensation state compensation, cost/quality contracts 249] approximation activation, quality steering Serving paging and KV pages, phase queues, paging, admission, TTFT, TPOT, good- lifecycle closure phase control [6, 116, transfer-ready state prefill/decode put, p99 latency over admit, 174, 282] separation, transfer expose/transfer, scheduling evict/reclaim, and restoration debt Serving tenant adapters, heterogeneous batching, eviction, co- throughput, SLO sta- lease and and model caches, residency win- placement, reactivation, bility, GPU memory invalidation multiplexing [31, dows pooled ownership efficiency semantics across 196, 239, 251] request, model, adapter, node boundaries Retrieval index main- mutable ANN shards, local repair, compaction, recall, freshness, cross-shard tenance [71, 199, 215, segment metadata, dele- sealing, publication, maintenance cost, exposure contracts 231, 232] tion and repair state routing latency and unified triggers Planner-mediated re- evolving retriever mem- query invocation, answer quality, fresh- explicit policy for trieval [13, 217, 262] ory, freshness and confi- refresh, rebuild trigger, ness, lifecycle cost stale memory dence state exposure control Table 6. Representative mechanism matrix for cross-domain comparison (continued). Cluster

State object

Control surface

Evaluation boundary Remaining gap

Retention protected parameters, protect, admit, replay, forgetting, reuse retention-store governance [10, exemplar stores, replay compress, demote, bud- value, store cost, contract with 76, 112, 122, 177, 284] buffers, budget traces get shrink disturbance response precedence

5

Design Implications for Stateful Runtimes

The comparative synthesis points to a bounded set of design implications. It does not imply one settled architecture; it identifies constraints that stateful mechanisms must satisfy if they are to remain composable under disturbance: make the boundary observable, schedule around state rather than passive work, close the loop between action and accounting, and model the service boundary explicitly.

Beyond Storage: State as a Runtime Control Problem in Parallel and Distributed Systems 5.1

21

A Design Space for Stateful Runtime Architectures

5.1.1 State Granularity: Record, Key, Segment, and Session. A first architectural choice is the granularity at which state is represented and controlled. Record, key, segment/page, and session/conversation granularities trade flexibility, locality, and metadata overhead differently [116, 260, 278]. The earlier migration, paging, and retrieval examples all exposed this same choice indirectly: fine-grained objects improve local placement and reuse decisions, while coarser objects simplify ownership and recovery semantics at the cost of wasted movement or weaker selectivity. 5.1.2 Control-Plane Coupling: Embedded, Sidecar, or Shared Service. A second choice is where control logic lives. Embedded control is fast but fragmented, sidecar-style control is modular but delayed, and shared services are consistent but can bottleneck if their APIs are too fine-grained [116, 262, 265]. This axis follows directly from the surveyed control surfaces: stream schedulers often embed legality into the operator path, serving systems increasingly split decision and actuation across allocators or disaggregated pools, and retrieval middleware starts to externalize policy into shared services that outlive any single request path. 5.1.3 Lifecycle Management: Admit, Place, Mutate, Expose/Transfer, Compact, Evict/Reclaim. A robust stateful runtime should make six lifecycle stages explicit: admit, place, mutate, expose or transfer, compact, and evict or reclaim. If any stage is implicit, local optimizations tend to create lifecycle debt under disturbance [196, 239, 251]. That lesson is visible across domains: migration and replay expose admit, place, and expose/transfer semantics in streaming; KV management exposes admit, expose/transfer, and evict/reclaim semantics in serving; and dynamic retrieval and retention expose mutate, compact, and evict/reclaim semantics once freshness or budget pressure is no longer negligible. Domain-specific actions such as sharing, refresh, demotion, or restoration are best read as specializations of these canonical stages rather than as parallel lifecycle vocabularies. 5.1.4 Service-Boundary Alignment: Local Wins vs End-to-End Wins. A final architectural axis is alignment between local optimization targets and the user-facing boundary. Without that contract, queue-local improvements can hurt p99 latency, retrieval-local freshness can hide publication debt, and retention-local compression can damage future reuse value [13, 68, 249]. The design question is therefore not only whether a local controller improves its own metric, but whether the metric is the boundary that should absorb the action’s future cost. 5.2

Failure Modes and Anti-Patterns

Many failures are repeated contract omissions rather than isolated bugs. Table 7 summarizes recurring anti-patterns and the repairs that make them reviewable. Read as missing contracts, they expose hidden invalid actions, conflicting objectives, and deferred debt; the repair names the state object, legal action, protected boundary, and charged cost. 5.3

A Contract-Oriented Blueprint for Stateful Runtimes

The literature supports a compact contract view with five layers: typed state catalog, observation and disturbance detection, boundary model and decision kernel, actuation interfaces, and safety and recovery envelope. Systems that made these layers explicit made state more governable; systems that left them implicit moved debt into disturbance handling, recovery, or later reuse. Stronger systems approximate a typed state catalog containing object name, granularity, mutability class, lifecycle stage, and quality sensitivity. Reliable migration, paging, refresh, or replay requires these fields to be explicit enough for multiple controllers to reason about the same object. The observation layer then combines steady metrics with

22

Zhang et al. Table 7. Recurring anti-patterns and the contract repairs that make them reviewable.

Anti-pattern

Missing contract

Primary blueprint layer

Expected design effect

Telemetry-action mismatch

signals must name a legal actuation target and its delay or confidence semantics hard constraints, soft objectives, and tie-breakers must be ordered explicitly disturbance phases and re-stabilization windows must be part of the contract mutation, compaction, deletion repair, and maintenance debt must be charged to the same boundary as read gains ownership transfer and no-eviction rules must survive migration, recovery, and restoration

observation and disturbance detection

dashboards become control inputs rather than passive monitoring

boundary model and decision kernel

overlapping controllers stop oscillating under burst or skew

observation plus safety and recovery envelope

reported gains remain meaningful outside stationary replay

state catalog plus actuation interfaces

deferred rebuild or compaction cost becomes visible during design review

actuation interfaces plus safety and recovery envelope

scale-out and fault handling share one transferable state model

Policy Layering Without Conflict Semantics One-shot benchmarking of non-stationary mechanisms Ignoring Write Amplification in Read-Optimized Designs

Treating State Ownership as Static

disturbance indicators such as hotspot flips, phase imbalance, retrieval drift, and fragmentation spikes, so the runtime can distinguish ordinary load variation from a change that threatens the service boundary. The decision kernel is where the state catalog and observation streams become boundary-aware choices. Whether heuristic, model-based, or learned, the comparative requirement is the same: decisions should remain analyzable and their precedence over competing objectives should be explicit. Actuation interfaces are most comparable when they are explicit and bounded; the surveyed mechanisms repeatedly distinguish between observation, decision, and actuation only when the actuation surface is narrow enough to describe. Finally, a stateful system remains analytically incomplete without a safety envelope: invariants that must hold during and after control actions. These guarantees depend on the fault model, whether crash-stop, crash-recovery with replay, network partition, or silent degradation [20, 25, 34, 153, 166, 217, 262, 272]. In retrieval and retention settings, the key invariants are freshness exposure and retention demotion. The blueprint is not intended to prescribe a single control-plane architecture. Its role is to make design reviews sharper. A runtime with an embedded scheduler can satisfy the blueprint if it names the objects it controls, exposes the signals that trigger action, and states the invariants that survive migration or recovery. A sidecar controller can also satisfy it, but only if the delay between observation and actuation is part of the contract. A shared memory service can satisfy it when its APIs expose not just allocation and release, but also lease duration, invalidation rules, restoration cost, and the boundary to which deferred debt is charged. The same review therefore applies to stream processors, serving stacks, and retrieval middleware even when their implementations look unrelated. The blueprint also helps separate mechanism novelty from integration maturity. A new eviction policy, approximation rule, or retrieval trigger may be technically clever, but it remains a local policy until the system states how it composes with neighboring controllers. Integration maturity increases when the mechanism has a typed state object, a measurable trigger, a bounded action surface, an explicit precedence relation, and a recovery or safety condition. This is why the survey repeatedly emphasizes contracts over algorithms. Algorithms decide what to do under a local objective; contracts decide when an action is legal, which boundary absorbs its cost, and how later controllers observe the debt it created.

Beyond Storage: State as a Runtime Control Problem in Parallel and Distributed Systems 5.4

23

Cross-Domain Case Studies

The three running scenarios expose the same rule from different directions. Multi-tenant LLM serving under burst arrival shows how admit, expose/transfer, evict/reclaim, and restoration debt share one p99-facing boundary. Stateful stream reconfiguration shows that migration, progress, and replay need one transfer protocol. Retrieval under corpus drift shows that freshness, rebuild debt, and exposure legality must be co-governed. Local gains are not stable unless the runtime also names the debt they push into neighboring phases. 6

Evaluation and Research Outlook

The remaining question is evidentiary: how should the field recognize progress once state-management mechanisms interact across layers and over time? We use a disturbance-oriented vocabulary for judging stability, composability, and long-horizon reuse quality, because stationary runs rarely expose the debt created by access reshaping, memory reclamation, delayed publication, or retention under shrinking budgets. 6.1

Evaluation Dimensions for Future Surveys and Systems

An enduring weakness is evaluation fragmentation. Access papers often report throughput or contention, execution papers report speedup or energy, and evolution papers report quality or forgetting; each often leaves one adjacent boundary weakly audited, so local gains look stronger than they remain under disturbance or lifecycle debt. Four dimensions should be tracked: • Steady-state efficiency: throughput, amortized cost per useful result, and locality-sensitive resource use. • Tail behavior: percentile latency, burst amplification, and recovery delay. • State quality: bounded approximation error, retrieval quality under drift, and retention value under memory budgets. • Operational sustainability: migration cost, update overhead, and how quickly a control loop re-stabilizes after workload or hardware changes. Systems that score well on only one dimension deliver local improvements rather than full state-management solutions. These dimensions form an evidence ladder. A strong paper makes its state object and control boundary explicit, demonstrates local gains, shows that they survive disturbance, and tests whether they remain meaningful when composed with neighboring controllers or maintenance tasks. The benchmark unit should shift from one-shot runs to multi-phase disturbance traces: warmup, burst, reconfiguration, drift, and long-horizon maintenance belong in one connected experiment. 6.2

Boundary Conditions and Threats to Transfer

The comparison above is deliberately mechanism-centered, but the mechanisms do not transfer automatically. A scheduler that works for keyed stream partitions may fail for KV-cache pages because the ownership interval is shorter, the reuse value is more speculative, and the protected boundary is tail latency rather than exactly-once progress. Conversely, a serving allocator that reacts well to phase imbalance may offer little guidance for retrieval maintenance, where the costly action is not immediate eviction but publication, repair, or rebuild under freshness constraints. The transferable unit is therefore not the policy itself, but the contract it implies: which object is governed, which actions are legal, which service boundary absorbs the cost, and which future debt is recorded. Three threats are especially easy to miss when reading across domains. First, state identity is often less stable than the paper’s terminology suggests. A stream shard, a KV page, an ANN segment, and an exemplar buffer can all be called

24

Zhang et al. Table 8. Representative disturbance-oriented evaluation matrix.

Cluster

Recommended disturbance tests

Common blind spot

Research extension

State access under skew

hotspot flips, burst tenant switch, rebalance under live joins mixed prompt lengths, burst after warmup, reclaim-and-restore stress, tenant churn continuous updates, deletion bursts, stale-data injection, compaction lag abrupt budget contraction, asynchronous updates, background compaction concurrent serving, retrieval, and update pressure

separating locality and recovery experiments

connect placement choices to future replay and migration debt

allocator-only or phase-only evaluation

measure admit, expose/transfer, evict/reclaim, and restoration debt on one SLO boundary add exposure-safety and maintenancedebt metrics

Serving lifecycle control

Retrieval under drift

Retention under budget elasticity Cross-domain middleware

one-shot index assumption

reporting accuracy alone

define retention-store metrics and disturbance protocols

layer-by-layer optimization

compare policy-composition and precedence models

“state,” but their mutability, ownership, and invalidation semantics differ sharply. Second, disturbance horizons differ. Stream recovery may unfold over seconds, serving reclamation over milliseconds, retriever rebuilds over minutes or hours, and continual-retention effects over many update rounds; an evaluation that is long enough for one domain can still be too short for another. Third, quality boundaries are not interchangeable. Bounded approximation error, answer freshness, p99 latency, and forgetting are all service properties, but they tolerate different kinds of temporary debt. These differences explain why the survey emphasizes tuples and contracts rather than mechanism names. Transfer claims are most convincing when they include at least one explicit non-transfer case. A mechanism that generalizes from stream migration to KV transfer, for example, should say which field of the tuple changes when progress fences become tenant SLOs, when shard ownership becomes page residency, or when replay legality becomes restoration priority. Likewise, a retrieval-maintenance policy that borrows from retention should identify whether it is transferring an admission rule, a demotion rule, or only an evaluation pattern. This negative boundary is not a weakness in the contribution; it is what makes the contribution reviewable. Without it, cross-domain synthesis can overstate similarity by reusing words such as locality, freshness, or memory while hiding the different legality conditions under which those words operate. For future systems papers, the practical implication is to state the transfer claim narrowly. A contribution is strongest when it names the state class it governs, reports the disturbance horizon over which the result holds, and exposes at least one adjacent boundary where the mechanism could fail. For example, a KV-cache policy should report not only memory savings and decode latency, but also whether reclamation harms later prefix reuse or restoration cost; a retrieval-maintenance policy should report not only recall and query latency, but also whether delayed compaction changes exposure safety or update backlog. Such reporting does not make every paper larger; it makes the claim reviewable. It also prevents the field from treating local throughput, local accuracy, and local freshness as interchangeable evidence for state-management progress. 6.3

An Integrated Research Agenda

The design implications above distill into four pressure points: cross-domain observability contracts, state-centric scheduling on heterogeneous hardware, memory middleware for dynamic AI services, and end-to-end evaluation

Beyond Storage: State as a Runtime Control Problem in Parallel and Distributed Systems

25

beyond local speedups [33, 116, 139, 198, 252, 258, 260]. Access mechanisms remain brittle when visibility and ownership semantics are implicit; execution mechanisms lose composability when movement cost, representation, and service boundary are optimized separately; evolution mechanisms remain unstable when refresh, retention, and exposure decisions hide future debt. The common research target is a shared control layer that makes state identity, movement cost, exposure legality, and lifecycle debt composable under one service boundary. 7

Conclusion

Efficient state management is best understood as one coupled control loop over access, execution, and evolution. Comparing streaming, serving, retrieval, and retention through the same five-field tuple shows that the recurring bottleneck is not the absence of clever local mechanisms, but the absence of contracts that keep those mechanisms compatible once state is shared across time, hardware, and service phases. Effective systems name the state object, bind local actions to a measurable service boundary, and charge deferred debt to the same boundary that received the immediate gain. Progress will therefore depend on runtimes that keep state identity, observability, and policy precedence stable under disturbance.

26

Zhang et al.

Appendix This appendix provides supporting material for the main paper, including literature-boundary context, design checklists, case walkthroughs, and additional domain figures. Its bibliography includes the works cited directly in the appendix. A

Literature Boundary and Historical Context

This appendix material clarifies which adjacent traditions contributed reusable state-management mechanisms and which mainly provide background or contrast. A.1

Streaming and Dataflow Context

A broader historical reading helps explain why visibility, ownership, and disturbance-aware control emerged as recurring seams. Early systems such as Aurora and Borealis already made continuous query plans, adaptation paths, and stateful operators explicit runtime artifacts rather than passive storage structures [1, 2]. Later systems such as MillWheel, Naiad, and the Dataflow model shifted the comparison unit from raw operator throughput to controlled advancement over time, frontiers, and externally visible progress [7, 8, 160]. That shift matters because it reveals an early version of the survey’s main thesis: performance-relevant state becomes governable only after the runtime exposes when it is safe to publish, buffer, replay, or move it. The same line then broadened toward elasticity, migration, and recovery. Trill and Differential Dataflow show that partially materialized state can still be updated incrementally when timestamps and frontier metadata make legality explicit [24, 155]. Flink, StreamCloud, Megaphone, and later migration/recovery work add the deployment-facing extension: once operator state is movable, the runtime must govern not just update cost but also transfer legality, replay scope, and disturbance-time pause behavior [21, 69, 80, 152, 272]. The broader landscape therefore helps justify why the survey treats visibility and ownership as coupled rather than separate concerns. A.2

Replicated, Transactional, and Memory-Resident Systems as Boundary References

Classic transactional or replicated systems are not treated as a full co-equal cluster, but they remain important boundary references because they make legality contracts more explicit than many newer runtime papers do. Spanner clarifies leaseholder continuity and writer legitimacy across distributed transactions [34]. Calvin clarifies how deterministic ordering can turn replay from conflict resolution into continued execution [207]. Raft and related replicated-log systems clarify term-scoped authority and old-owner retirement semantics [166]. FaRM and RAMCloud contribute the perspective that reconstruction speed and serving ownership after failure are themselves runtimemanaged state problems rather than mere recovery afterthoughts [43, 167]. These systems are not inserted into the survey to broaden venue coverage; they help specify what many modern stateful runtimes still leave implicit. A.3

Serving, Retrieval, and Memory-Governance Systems in Broader Perspective

The main text intentionally centers the serving literature on short-lived lifecycle control. A broader perspective shows why this concentration happened. Earlier serving systems such as Clipper, Clockwork, Nexus, InferLine, and INFaaS primarily exposed orchestration, provisioning, predictability, and admission as control surfaces around model invocation [35, 36, 68, 189, 195]. Later LLM-oriented systems internalized more of the memory substrate itself: vLLM and related paging systems govern KV state directly; Sarathi, DistServe, Splitwise, and FastGen govern phase asymmetry; Punica, S-LoRA, Jenga, and Prism

Appendix

27

govern tenant multiplexing, adapter residency, heterogeneous cache layouts, and whole-model activation windows [6, 31, 81, 116, 174, 196, 239, 251, 282]. This trajectory is exactly why the survey reads serving as a state-governance problem rather than as a mere systems-for-LLMs topic. Retrieval and memory-governance systems underwent a parallel shift. Dense retrieval and late-interaction systems such as DPR and ColBERT originally foregrounded query-time quality versus cost over mostly offline-built memory substrates [106, 108]. Dynamic ANN systems, vector stores, and planner-mediated retrievers later made freshness, repair, shard exposure, and long-horizon memory evolution first-class runtime concerns [13, 71, 199, 215, 217, 262]. The larger landscape therefore supports the claim that retrieval quality should increasingly be read as a lifecycle property rather than a static model property. A.4

Retention, Approximation, and Middleware as Adjacent Traditions

Approximation and bounded-retention lines also matter because they make evaluation boundaries explicit in ways many mainstream runtime papers still do not. BlinkDB, ApproxHadoop, ApproxJoin, StreamApprox, and IncApprox show that speedup claims over sampled or summarized state only become comparable after the runtime states which error boundary is being protected [5, 64, 113, 183, 184]. PECJ and LibAMM extend that logic toward online control and explicit quality compensation [246, 249]. Continualretention lines such as EWC, GEM, MIR, GDumb, REMIND, and Ferret similarly show that bounded memory is not just a model problem; it creates operational questions over protection, admission, replay, compression, and budget shrinkage [10, 76, 112, 146, 177, 284]. Finally, recent memory-middleware proposals such as Neuromem and SAGE are useful not because they already solve the full problem, but because they expose lifecycle decomposition and workflow-visible actuation as first-class abstractions [139, 258]. Together, these adjacent lines help motivate contract-oriented blueprint and middleware pressure-point arguments rather than one more domain-specific taxonomy. B

Design Checklist and Contract Skeleton

This appendix material keeps the design-space, anti-pattern, blueprint, and boundary evidence behind the design conclusions. The organization follows recurring principles, architecture axes, contract repairs, and concrete examples. B.1

Design Checklist

The checklist turns the comparative synthesis into concrete design questions. The table names recurring questions, and the following subsections unpack them into principles, axes, and repair patterns.

28

Zhang et al. Table 9. Appendix guide for translating synthesis into runtime design choices. Recurring synthesis result

Design question

Concrete artifact to look for

Observability gaps dominate fail-

Which signal names the state object, lifecycle stage, delay

typed telemetry and distur-

ure analysis

semantics, and legal action?

bance indicators

Local speedups disappear under

Which boundary is hard, which objective is soft, and who

precedence-aware decision

composition

wins on conflict?

kernel

Lifecycle debt accumulates across

Where are admit, place, mutate, expose/transfer, compact,

lifecycle

updates and movement

and evict/reclaim made explicit?

bounded actuation interfaces

Recovery reuses the same state

Which invariants survive migration, replay, rebuild, or

safety

objects as steady state

restore?

ownership-transfer rules

B.2

pipeline

with

envelope

and

Contract Principles

Observation before optimization means the relevant state boundary must already be named in telemetry. State as a first-class scheduling object means runtimes should schedule around mutable state rather than passive tasks. Closed-loop rather than open-loop policies means action must feed back into accounting. Stable gains require boundary modeling means every gain must declare what is held fixed, what debt is deferred, and what disturbance invalidates the claim. B.3

Architecture Axes

The architecture axes are state granularity, control-plane coupling, lifecycle management, and service-boundary alignment. Granularity ranges from record to session. Control logic may be embedded, sidecar-based, or shared. Lifecycle management should make admit, place, mutate, expose or transfer, compact, and evict or reclaim explicit. Boundary alignment asks whether local wins survive end-to-end service constraints. B.4

Anti-Pattern Repairs

Telemetry-action mismatch is repaired by naming a legal control action for each signal. Policy Layering Without Conflict Semantics is repaired by ordering hard constraints, soft objectives, and tie-breakers. One-shot benchmarking is repaired by adding warmup, disturbance, adaptation, and re-stabilization. Ignoring Write Amplification in Read-Optimized Designs is repaired by charging mutation and repair debt to the same service boundary as read gains. Treating State Ownership as Static is repaired by transferable ownership and lease semantics. B.5

Blueprint Contract Skeleton

The blueprint is a conceptual reference model, not a monolithic runtime. Its five layers are typed state catalog, observation and disturbance detection, boundary model and decision kernel, actuation interfaces, and safety and recovery envelope. The main paper includes the compact anti-pattern repair table. The appendix keeps the abstract policy skeleton and additional evidence anchors for readers who want the contract view in a more operational form. B.6

Boundary Case Bridges

These case bridges highlight three recurring boundary failures. Multi-tenant LLM serving under burst arrival shows how admit, expose/transfer, evict/reclaim, and restoration debt share one p99-facing boundary. Stateful stream reconfiguration

Appendix

29 Implicit anti-pattern

Missing boundary

Explicit contract repair

Telemetry-action mismatch signals do not name legal actions

signal → state object

Typed observation layer metrics feed bounded actions

Policy layering objectives lack precedence

ordering and tie-break

Precedence-aware kernel hard rules precede soft goals

One-shot benchmark disturbance is omitted

disturbance phases

Multi-phase envelope warmup, burst, drift, recovery

Hidden write path read gains defer mutation debt

mutation accounting

Auditable state catalog writes charged to same boundary

Static ownership handoff legality is implicit

transfer survival

Transferable state model authority, replay, reclamation

Fig. 3. Anti-pattern to contract repair map. Table 10. Abstract policy skeleton for unified state governance. Step

Policy skeleton

1

observe_signals() and update telemetry confidence for contention, queueing, local freshness, exposure freshness, and retention pressure. identify_state_objects() whose boundary is currently active: hot operator shards, KV pages, retrieval indexes, replay buffers, or demotion candidates. check_hard_constraints() for SLA, memory limit, recovery envelope, shard-exposure safety, and quality floor before any optimization action. rank_actions() over admit, place, mutate, expose/transfer, compact, and evict/reclaim; domain-specific actions such as migrate, refresh, or demote are evaluated as specialized instances of those lifecycle stages using expected utility and maintenance debt. apply_bounded_action() with explicit rollback or retry semantics when telemetry is delayed or conflicting. account_outcome() by charging the decision against tail latency, quality drift, effective budget loss, and long-horizon reuse value.

2 3 4

5 6

shows that migration, progress, and replay need one transfer protocol. Retrieval under corpus drift shows that freshness, rebuild debt, and exposure legality must be co-governed. B.7

Hardware-Conscious Evidence Anchors

The hardware-conscious cluster is most useful when read through three debts that the main paper could only summarize briefly. The first is representation debt: approximate execution and compressed-state systems such as Anda, TurboAttention, QServe, ATOM, VAttention, FlashInfer, SampleAttention, QuantLLM, and related low-bit cache work all show that encoding decisions reshape memory traffic, kernel regularity, and quality loss simultaneously [47, 105, 136, 178, 228, 236, 271, 276, 286]. Even systems that make representation look cheaper, such as FuseMax-style lines, do so by shifting complexity into calibration, selection, or error accounting rather than eliminating it [94, 115, 163].

30

Zhang et al. The second is movement debt. Systems such as Chimera, HeterInfer, LIA, HETIS, Mercury, WaferLLM, DRAMCache,

Tiertune, MoE-lightweight offloading, PhoenixOS, JIT-Serve, MLP offload, Colloid, BeyondHotness, Optimus, Tigon, and MemStrata all show that placement helps only when the runtime can jointly price compute skew, cache residency, interconnect bandwidth, and cross-device movement [18, 30, 51, 67, 77, 82, 85, 86, 109, 119, 138, 154, 158, 210, 222, 267, 281]. Storage- and I/O-adjacent systems such as AutoScratch, PFSCK, PolarDB, CXL-based designs, Toleo, and VM-control make the same point from another boundary: once state crosses devices or tiers, metadata handling, legality, and transport cost become first-order control variables rather than implementation details [19, 41, 42, 53, 95, 204, 279]. The third is resilience and repair debt. Save, ChecknRun, CXL memory-protection work, Concealing, TurboAttentionstyle repair-aware approximation, SampleAttention, and PFSCK show that movement and representation cannot be priced independently from failure handling, degradation tolerance, or hidden repair cost [41, 45, 100, 105, 275, 280, 286]. Heterogeneity-aware execution and server-placement lines such as GMLake, Flex-MoE, Kamath-style overlap, Clap, DecDec, Coruscant, Zico, MEPipe, Clone, PowerInfer, PIMBA, QFactory, and MoC reinforce the same lesson from scheduling: overlap, offload, or recomputation only help if the control overhead and recovery debt stay below the service debt they are meant to avoid [17, 18, 70, 101, 104, 111, 132, 172, 173, 200, 203, 208, 257]. Dynamic input pruning with cache-aware masking, near-core decompression, and CXL-centered inference expose different actuators, yet each still ties placement and movement to an explicit runtime choice over state: what may be skipped, decompressed, kept near compute, or shifted into the memory hierarchy [49, 61, 66]. Placement, movement, and state transformation therefore remain one coupled control problem even when the local actuator changes. Grouped this way, the hardware cluster reads less like a long annex and more like a precise extension of the main claim. The exact mechanisms differ, but the comparative result is stable: once state crosses a hardware boundary, representation, movement, and resilience have to be priced together or the local optimization stops being trustworthy. C

Cross-Domain Case Walkthroughs

These walkthroughs preserve the same five-field structure while making the deployment-shaped reasoning more concrete. C.1

Multi-Tenant LLM Serving Under Burst Arrival

State object. The governed objects are paged KV state, prefix-sharing metadata, tenant-specific adapters, modelresidency windows, and the restoration debt inherited by subsequent requests. Control surface. The runtime controls admission, batch shaping, paging, prefill/decode separation, expose/transfer decisions, evict/reclaim decisions, and tenant-sensitive placement [6, 31, 116, 164, 174, 282]. Coupling path. A locally beneficial admission may force evict/reclaim decisions that destroy future reuse, causing later re-materialization traffic and queue amplification. Once phase disaggregation is enabled, the next bottleneck may shift from raw memory capacity to expose/transfer ordering and the downstream cost of restoring discarded state. Evaluation boundary. The meaningful boundary is TTFT, TPOT, p99 latency, and goodput over a multi-phase window rather than isolated allocator efficiency. Remaining gap. The unresolved problem is still a unified contract that ranks admit, expose/transfer, evict/reclaim, offload, tenant isolation, and restoration debt against the same service boundary instead of letting each sub-policy optimize its own phase locally.

Appendix C.2

31

Stateful Stream Processing Under Reconfiguration

State object. The runtime governs keyed windows, operator shards, progress metadata, checkpoint boundaries, replay buffers, and migration ownership metadata. Control surface. The active controls are migration grain, trigger policy, checkpoint cadence, replay scheduling, and ownership-transfer timing [7, 21, 80, 152, 272]. Coupling path. Reconfiguration changes locality and dual-ownership windows; recovery policy changes replay pressure and convergence cost. These effects feed back into future contention and scheduling stability rather than remaining isolated maintenance events. Evaluation boundary. The right boundary combines pause time, throughput stability, correctness, and post-failure recovery behavior. Measuring migration and recovery separately hides the control loop that actually governs the state object. Remaining gap. The missing piece is a disturbance-invariant ownership protocol that unifies visibility, transfer legality, and replay semantics across scale-out and recovery rather than re-deriving them per subsystem. C.3

Retrieval Memory Under Continuous Corpus Drift

State object. The relevant objects are mutable ANN shards, segment-sealing metadata, shard-exposure state, plannervisible freshness signals, and long-horizon graph or hierarchy memory. Control surface. The runtime controls incremental indexing, local repair, compaction, rebuild trigger, publication, stale-memory eviction, and planner-mediated retrieval invocation [13, 71, 199, 215, 217, 262]. Coupling path. Aggressive refresh improves freshness in the short term but can increase maintenance debt, publication lag, and answer instability if exposure, rebuild, and planner behavior are not coordinated. Evaluation boundary. The relevant boundary is answer quality under drift plus freshness lag, query latency, maintenance cost, and exposure safety. Remaining gap. The missing abstraction is an exposure-aware accounting layer that can combine recall drift, rebuild debt, planner confidence, and publication legality into one explicit trigger policy. D

Domain-Specific Figures and Evidence Anchors

This appendix material gathers domain evidence under the same analytical frame as the main survey. It is organized around the pressure regions where additional mechanism detail most improves comparative clarity. D.1

Access-Scheduling Design Space

The access-scheduling design-space visual matters because state access becomes a control problem when the runtime must decide what is visible, when a transition is legal, who owns the object now, and how disturbance changes the cost of the next action. The figure is a compact reminder that observability, locality, ownership transfer, and recovery are not separate subtopics but one coupled design space.

32

Zhang et al.

Observability

queue exposure conflict density progress frontiers

Aurora, MillWheel, Naiad, Trill, Differential Dataflow

Cost and scheduling

topology, skew NUMA, ordering admission, migration

BriskStream, MorphStream, Megaphone, SOLA, ExeGPT

Recovery and elasticity

checkpoint alignment ownership transfer replay boundary

Spacker, Flink, SONIC, ORION, fast recovery

Fig. 4. Access-scheduling design space for locality, legality, transfer, and recovery.

The comparative lesson behind the figure should remain explicit. Visibility mechanisms tell the runtime when a state transition is safe to expose; locality mechanisms tell it which placement keeps the next access affordable; ownership-transfer mechanisms tell it when relocation may occur without dual writers or replay ambiguity; recovery mechanisms tell it how those same promises survive disturbance. These are not four optional implementation concerns. They are the four places where access control becomes system-wide behavior rather than local bookkeeping. The broader access-scheduling lineage is equally important for keeping the survey’s boundary crisp. Classical consistency and concurrency systems such as TicToc, Percolator, and RACE show that timestamping, metadata layout, and remote coordination are already control surfaces over state, even before modern serving runtimes made the same pattern explicit [175, 240, 287]. Systems such as CheckFreq, ChecknRun, PipeFill, and the newer runtime-monitoring lines show that the access problem is not just placement, but also when the runtime may safely observe, checkpoint, and resume without invalidating later actions [11, 45, 159, 285]. Tooling for dynamic reconfiguration and transfer, including Caerus, MORPH-style recovery, and queue-aware stream control, matters because it keeps the state-ownership question visible instead of treating movement as a mere implementation detail [52, 60, 153, 254]. The same lineage also includes systems that turn placement pressure into explicit runtime policy rather than hidden implementation cost. Resource-multiplexing and scheduler-centric systems such as G10, resource multiplexing, PFS-style reconfiguration, and workload-sensitive monitoring show that queueing, hot-state exposure, and placement are inseparable once the runtime manages shared state across stages or tenants [41, 78, 96, 99]. Transactional dataflow and distributed-storage systems such as Flash-style state flow, Cloud-style buffering, and multi-stage transfer control reinforce the same point: the access problem is always also a visibility problem and a handoff problem [52, 54, 60, 123]. That is why the survey keeps access and scheduling as a first-class axis rather than folding it into generic runtime management. Additional boundary references preserve the historical path. Transaction-processing systems such as TicToc, Percolator, and TiDB-style concurrency control show that control surfaces over mutable state have long existed in distributed data systems, even if they were not described in the same vocabulary as modern state management [175, 240]. Stream-processing systems such as Corun, Elastic stream systems, and BriskStream-style monitoring show that scheduling, state exposure, and reconfiguration were already intertwined before serving and retrieval brought those issues to the foreground [205, 253, 260]. More recent monitoring and admission lines such as CheckFreq, ChecknRun, and PipeFill show that the runtime often needs an explicit contract for when observations can be trusted after disturbance [11, 45, 159].

Appendix

33

The tail of the access line also includes a few systems that are easy to overlook because they sit between streaming, scheduling, and memory management. More recent scheduling and observability papers such as MOC, Optimum-style optimization, FS-MoE runtime control, and tree-aware transfer paths show that access control is still the main lever whenever a runtime has to reason about hidden contention and state movement together [17, 51, 170, 220]. Likewise, Tigon, VM-control, and the updated memory-aware runtime lines keep the same lesson visible: once state crosses a device or tenancy boundary, legality and placement become one problem [86, 204, 279]. Even storage-to-compute bridges such as Snowflake-style disaggregation and mixed-access control belong in this lineage because they expose how latency, placement, and handoff interact [211, 285]. Two compact boundary references keep the access story complete. DRTMH-style transaction scheduling and the state-machine tradition remind us that legality over mutable state was a control problem long before modern runtimes named it that way [193, 223]. EndMyth-style stream repair work shows the same issue from the failure side: access, recovery, and trust in observations are inseparable once the system can replay or resume [245]. D.2

Serving Lifecycle Control

The main paper now carries the core predictive, structural-reuse, and temporal-lifecycle comparison. The appendix keeps the supporting bridge references and the lifecycle figure so readers can trace how the broader serving literature maps onto the same control loop without repeating the main argument. Additional serving papers refine the same boundary rather than forming a separate taxonomy. SplitZip, CacheFlow, and ThunderServe make transfer cost and cloud heterogeneity explicit; ChandGPT-style reuse systems, Jenga, Prism, and Aegaeon make multi-model or multi-tenant residency explicit; Spinfer, FlexGen, and related offload designs make memory pressure and phase placement explicit; HeterInfer, MoE-Lightning, and resource multiplexing make the hardware/topology asymmetry explicit [18, 30, 46, 59, 73, 78, 98, 164, 198, 239, 251]. Together, they keep the lifecycle argument from shrinking into a single KV-cache story. Bridge references connect older and newer state-governance lines. PagedAttention, Orca, and AlpaServe explain why batching and scheduling were already state-sensitive before the latest disaggregation wave; SlimPipe, What-If Stragglers, and Pumped trace-based systems explain why scheduling debt becomes visible only after phase or hardware asymmetry is modeled; output-length prediction and cache-aware routing systems show why admission and future-memory forecasting belong in the same discussion as restoration and transfer [65, 96, 116, 127, 129, 134, 237]. That bridge keeps the survey’s lineage coherent across older and newer serving lines. CacheSlide and SpecInfer-style speculative reuse make legality and verification a precondition for cross-position or tree-structured reuse; CachedAttention and HCache expose restoration cost across conversation turns; and MELL, Libra, fairness-aware serving, LayerKV, and Prefill-Only show that GPU residency, request partitioning, fairness, layer-wise allocation, and prefill admission determine which cached state remains reusable later [44, 56, 58, 141, 143, 156, 190, 197, 230]. These systems make reuse, restoration, contention, and initialization part of the same serving control problem rather than separate local optimizations. Three more serving families sharpen the boundary. Application-centric serving such as Parrot shows that workflowlevel state and prompt DAGs can dominate placement decisions even before the cache story starts [133]. Tree-shaped speculative and group-wise systems such as FastTree and InstAttention show that reuse and scheduling depend on structure in the request stream, not just on raw throughput targets [169, 171]. Decentralized and overlay-based systems such as PlanetServe and GLLM-adjacent coordination lines show that cache reuse, routing, and trust become intertwined once the scheduler is no longer centralized [48]. The same cluster also includes systems such as Infinity-style eviction,

34

Zhang et al.

Admit request

Place KV pages

Mutate token state

Expose / Transfer

Compact fragments

Evict / Reclaim

SLO gate admission

residency plan

growth and sharing edits

reuse legality transfer

fragmentation debt

restore priority release

Memory pressure is not a terminal event: reclaiming KV state changes later restoration cost, prefix reuse, and tenant-visible SLO stability. Representative systems: vLLM, Sarathi-Serve, SGLang, DistServe, Splitwise, Jenga, CacheFlow, Pensieve, Mooncake, FastServe.

Fig. 5. KV-cache lifecycle.

Duplex-style cross-turn reuse, and world-load balancing variants, which are best understood as refinements of lifecycle control rather than new top-level subfields [120, 220, 243]. Two final serving references are useful as compact examples of the same lifecycle logic. Prefill-optimized trainingserving crossover systems such as MEPipe show that the serving boundary is often governed by how pipeline slack is distributed, not just by the cache itself [203]. Copy-aware or clone-style state movement lines show the same thing from a different angle: once request state is shareable, the runtime has to decide whether cloning, relocating, or reusing is cheaper than recomputing [208]. The remaining serving papers that have not yet been mentioned directly still fit the same lifecycle view. Multi-model and adapter-aware systems such as Aegaeon, DLoRA, Chameleon, and Toppings expose tenant-locality and residency pressure; cache and reuse systems such as PromptCache, CacheWild, and Droidspeak expose the boundary between prompt overlap and safe reuse; inference-control systems such as Alise, Atom, and Prism-like lineage expose the boundary between active scheduling and future debt; and cloud-disaggregation systems such as Mooncake and PRFaaS expose the difference between local phase control and cross-cluster ownership [62, 89, 124, 144, 179, 213, 226, 229, 239, 276, 277]. Those distinctions keep the lifecycle account tied to concrete control boundaries. Across these systems, the serving cluster still fits the survey’s canonical short-lived lifecycle: admit -> place -> mutate -> expose/transfer -> compact -> evict/reclaim. The family-level differences matter because serving papers attach to different parts of that loop: structural reuse clarifies what may be exposed safely, disaggregation governs transfer, and restoration debt appears after evict/reclaim decisions have already been made. The lifecycle figure makes the control seams concrete. Admit decides whether a request is allowed to create new short-lived state at all. Place decides where that state lives and therefore which queues or buses it will burden next. Mutate covers decode-time growth, prefix edits, and other in-place evolution of live serving state. Expose/transfer covers whether prefix, adapter, or session-local state may be reused safely and when it may move across workers or phases. Compact covers fragmentation control and structural cleanup. Evict/reclaim covers what can be released without hiding future restoration debt and how that debt is later charged back to the original SLO. The following evidence is especially useful in this subsection: • predictive-serving families grouped by the debt they forecast; • legality conditions for prefix, branch, and multi-call reuse; • temporal-lifecycle policies over expose/transfer, evict/reclaim, warm-start, and fairness debt; • clearer distinctions between allocator realism, transfer scheduling, and ownership semantics.

Appendix D.3

35

Retrieval and Retention Governance

The unifying point is that retrieval and retention both expose a longer-lived memory layer whose value decays gradually and whose maintenance debt is easy to hide if the runtime only reports steady-state quality. In retrieval systems, the state object is not just embeddings or an ANN graph in the abstract, but mutable shards, publication metadata, freshness signals, deletion debt, and planner-visible exposure state [13, 71, 199, 215, 217, 262]. In retention systems, the state object is not just “memory” generically, but a bounded store whose protected parameters, exemplars, replay entries, compressed features, and effective budget must be co-governed over time [10, 76, 112, 122, 177, 186, 284]. Retrieval systems expose at least four trigger layers: request-time invocation, index-time maintenance, publicationtime exposure, and placement-time residency. Self-RAG changes whether external memory should be consulted at all; FlowRAG changes when retriever state should be refreshed before reuse quality decays too far; CANDOR-Bench and FreshDiskANN expose when update churn, deletion debt, or local repair have already pushed the index away from its intended operating region; Milvus and Manu expose when partially refreshed state may be published safely under concurrent updates; Rummy and composable-memory search expose when movement or placement is justified by downstream quality gain instead of raw lookup throughput [13, 71, 182, 199, 215, 217, 262, 270]. The comparison shows that retrieval quality is a lifecycle property, not merely an encoder property. Retention systems expose an analogous trigger hierarchy over one bounded store. EWC protects already-committed parameter state; iCaRL, GDumb, and StreamFP govern what enters the scarce retained set; GEM, A-GEM, MIR, and DER govern when stored state should re-enter the update path; REMIND governs whether retained signal is compressed and later reconstructed; Ferret governs what happens when the budget itself contracts under live update pressure [10, 16, 28, 76, 112, 122, 146, 177, 186, 284]. Read this way, the continual-learning literature matters to the survey not because it supplies another application area, but because it provides one of the clearest examples of how protect, admit, replay, compress, and budget-shrink decisions compose over time. The comparative lesson across these two clusters should remain explicit. Retrieval runtimes lack a stable rule for when stale-but-local memory remains queryable; retention runtimes lack a stable rule for when compressed or demoted memory remains good enough to preserve. In both cases, the missing abstraction is a policy-visible exposure contract that turns hidden maintenance debt into an explicit control surface. Several remaining papers fit this same long-lived-memory picture. Dynamic ANN and index-maintenance lines such as HNSW, DiskANN, Milvus, and graph- or shard-centric systems make clear that freshness, shard publication, and rebuild debt are all part of one operational budget [71, 151, 199, 215]. Retrieval-augmented and memory-augmented systems such as DPR, ColBERT, REALM, RAG, ATLAS, HippoRAG, and Raptor show the same phenomenon at the model boundary: the runtime must manage what gets exposed, refreshed, abstracted, and reused rather than treating memory as a static auxiliary index [74, 75, 91, 106, 108, 121, 192]. Those references preserve the lifecycle logic behind the retrieval chapter. Retention has a similar tail of boundary papers. Experience replay, memory budget, and continual-update lines such as MIR, GDumb, REMIND, Ferret, and StreamFP establish that protect/admit/replay/demote decisions are all budgeted governance acts over one bounded store [10, 76, 122, 177, 284]. The resulting comparison connects long-horizon retention with short-lived serving reuse without treating the two as the same problem. The taxonomy figure prevents a recurring misreading: retention is not one replay heuristic with a few engineering variants. It is a store-level governance problem over multiple actions with different temporal meanings. Protect decisions stabilize already-committed state; admission decisions determine future coverage; replay decisions determine when

36

Zhang et al. Strategy family

Mechanism

Representative systems

Protection constrain drift

importance weights gradient regularization

EWC, Online EWC, SI, MAS

Replay absorb interference

gradient projection reservoir sampling memory replay

GEM, A-GEM, MIR, ER, DER++

Admission bound the store

exemplar anchoring utility scoring coreset selection

iCaRL, GDumb, StreamFP, BiC

Compression contract budget

feature distillation elastic budgets compressed replay

REMIND, Ferret, PackNet, HAT

Open contract. Admission, replay, compression, and demotion need one retention-store API so budget pressure can be compared with future reuse value.

Fig. 6. Retention-governance taxonomy.

retained state becomes active again; compression and demotion decisions decide how much of that future value survives under tighter budgets; budget-shrink rules decide which guarantees fail first when the store contract is stressed. D.4

Long-Horizon Evolution Anchors

The longest-horizon evidence keeps the survey from collapsing “evolution” into a narrow continual-learning story. SuperNeurons and related memory-reuse systems show that activation liveness, recomputation, and offload policy already form a managed state loop in deep-learning training, not just in inference-side serving [216]. Bubble-filling and recomputation-aware training systems such as Obscura show the same thing from another angle: when state is expensive to keep live, the runtime must decide which debt to shift into pipeline slack and which debt to pay immediately [87]. That is the same governance pattern that later appears in retention stores, but with a different temporal horizon and a different type of reuse value. D.5

Blueprint and Fault-Model Boundary Conditions

The first piece that should remain visible here is the distinction between typed state objects, observation contracts, decision precedence, actuation scope, and safety-envelope invariants. The blueprint is not a proposal for one monolithic runtime; its role is to make recurring hidden assumptions explicit enough that neighboring mechanisms can compose without silently violating one another’s boundaries. One implication is that recovery should not be treated as a separate semantic regime. If steady-state control reasons over typed KV pages, refreshable vector shards, replay buffers, or demotable retention objects, then recovery must restore and validate the same objects under the same exposure and action boundaries. Otherwise a system effectively runs one state model in steady state and another during failure. That inconsistency matters because it explains why fault-model notes are part of the mechanism, not optional decoration.

Appendix

37

The fault-model distinction also deserves fuller treatment here. A safety claim that holds under crash-stop may fail under crash-recovery with replay, may fail differently under network partition, and may fail silently under degradation modes such as stale telemetry, compaction lag, or sudden budget contraction [25, 34, 153, 166, 217, 262, 272]. These are not small implementation variants. They determine whether a controller may expose partially refreshed retrieval state, whether ownership may transfer across disaggregated serving pools, and whether a retention tier may demote memory before replaying it. The blueprint’s value lies in precisely these boundary conditions. Two invariant families deserve to be carried here explicitly. The first is freshness exposure invariants: a retrieval shard or index segment should specify when it is queryable for internal warmup, restricted traffic, or general external service after rebuild, deletion repair, or compaction. The second is retention demotion invariants: a retention object should specify whether it is protected, compressible, summarizable, demotable, or droppable when effective budget shrinks. These invariants preserve the blueprint’s operational meaning. D.6

Foundations Figures and Boundary Material

The inclusion rule is not venue prestige or application relevance. A paper is central when it turns state into an explicit runtime-managed object and exposes a reusable control surface over that state. That is why transactional streaming, migration, KV-cache lifecycle control, retrieval maintenance, and retention governance sit in the core, while many application-facing LLM, agent, or prediction papers stay outside it even when they are technically strong. That definition also has a formal lineage. The state-machine tradition matters not because the survey is about formal methods, but because it shows that a system only becomes governable when the runtime can name, observe, and transform its state explicitly [193]. That conceptual boundary is the reason the survey keeps the access, serving, retrieval, and retention lines together instead of splitting them into unrelated application silos. System domains

Streaming and transactional Hardware-conscious execution

Core dimensions

Survey outputs

Access and scheduling

Comparative synthesis

Execution optimization

Runtime blueprint

Evolution and reuse

Research agenda

LLM serving and inference Retrieval and agentic memory

access, execution, and evolution form one feedback loop

Fig. 7. Poster-style survey overview. It serves as a compact map of the survey’s scope and argumentative flow.

The core-versus-context distinction can also be stated operationally. Core material changes the survey’s comparative result by sharpening the state object, control surface, coupling path, evaluation boundary, or remaining gap. Context material instead clarifies neighboring traditions, records boundary-setting decisions, or preserves mappings between

38

Zhang et al.

triage labels and the five-field frame. Keeping that distinction explicit prevents the survey from drifting back into a broad but weakly comparative literature tour. Two overview visuals also belong conceptually with this foundations rationale. The first is the poster-style overview, which summarizes the survey spine and clarifies how the three axes and cross-domain synthesis fit together. The second is the running-example triptych, which explains why the paper reuses one access case, one serving case, and one retrieval case across sections instead of multiplying disconnected examples. State object

Stress event

Runtime decision

Service effect

A

streaming fraud hot keys

campaign burst flips popularity

repartition hotspots adjust snapshots

tail latency recovers replay debt bounded

B

LLM assistant KV pages, adapters

long prompts create memory pressure

phase-aware admit selective transfer

TTFT stabilizes goodput survives

C

RAG knowledge index, refresh queue

corpus drift and delete backlog

gate stale shards stage refresh

freshness improves compaction controlled

Fig. 8. Running examples that recur across access, execution, and evolution.

References [1] Daniel J. Abadi, Yanif Ahmad, Magdalena Balazinska, Ugur Cetintemel, Mitch Cherniack, Jeong-Hyon Hwang, Wolfgang Lindner, Anurag Maskey, Alex Rasin, Esther Ryvkina, Nesime Tatbul, Ying Xing, and Stan Zdonik. 2005. The Design of the Borealis Stream Processing Engine. In Proceedings of the Second Biennial Conference on Innovative Data Systems Research (CIDR). 277–289. https://www.cidrdb.org/cidr2005/papers/P23.pdf [2] Daniel J. Abadi, Don Carney, Ugur Çetintemel, Mitch Cherniack, Christian Convey, Sangdon Lee, Michael Stonebraker, Nesime Tatbul, and Stan Zdonik. 2003. Aurora: a new model and architecture for data stream management. The VLDB Journal 12, 2 (Aug. 2003), 120–139. doi:10.1007/s00778003-0095-z [3] Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant J. Nair, Ilya Soloveychik, and Purushotham Kamath. 2024. Keyformer: KV Cache reduction through key tokens selection for Efficient Generative Inference. In Proceedings of Machine Learning and Systems, P. Gibbons, G. Pekhimenko, and C. De Sa (Eds.), Vol. 6. 114–127. https://proceedings.mlsys.org/paper_files/paper/2024/file/48fecef47b19fe501d27d338b6d52582-PaperConference.pdf [4] Saurabh Agarwal, Bodun Hu, Anyong Mao, Aditya Akella, and Shivaram Venkataraman. 2026. SYMPHONY: Enabling Compute-Memory Disaggregation in LLM Serving Systems. In 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26). USENIX Association, Renton, WA, 2027–2041. https://www.usenix.org/conference/nsdi26/presentation/agarwal [5] Sameer Agarwal, Barzan Mozafari, Aurojit Panda, Henry Milner, Samuel Madden, and Ion Stoica. 2013. BlinkDB: queries with bounded errors and bounded response times on very large data. In Proceedings of the 8th ACM European Conference on Computer Systems (Prague, Czech Republic) (EuroSys ’13). Association for Computing Machinery, New York, NY, USA, 29–42. doi:10.1145/2465351.2465355 [6] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming throughput-latency tradeoff in LLM inference with sarathi-serve. In Proceedings of the 18th USENIX Conference on Operating Systems Design and Implementation (Santa Clara, CA, USA) (OSDI’24). USENIX Association, USA, Article 7, 18 pages. [7] Tyler Akidau, Alex Balikov, Kaya Bekiroğlu, Slava Chernyak, Josh Haberman, Reuven Lax, Sam McVeety, Daniel Mills, Paul Nordstrom, and Sam Whittle. 2013. MillWheel: fault-tolerant stream processing at internet scale. Proc. VLDB Endow. 6, 11 (Aug. 2013), 1033–1044. doi:10.14778/2536222. 2536229 [8] Tyler Akidau, Robert Bradshaw, Craig Chambers, Slava Chernyak, Rafael J. Fernández-Moctezuma, Reuven Lax, Sam McVeety, Daniel Mills, Frances Perry, Eric Schmidt, and Sam Whittle. 2015. The dataflow model: a practical approach to balancing correctness, latency, and cost in massive-scale, unbounded, out-of-order data processing. Proc. VLDB Endow. 8, 12 (Aug. 2015), 1792–1803. doi:10.14778/2824032.2824076 [9] Waleed Ali, Siti Mariyam Shamsuddin, and Abdul Samad Ismail. 2011. A Survey of Web Caching and Prefetching A Survey of Web Caching and Prefetching. International Journal of Advances in Soft Computing and its Applications 3 (03 2011). [10] Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. 2019. Online Continual Learning with Maximal Interfered Retrieval. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer,

Appendix

39

F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2019/file/ 15825aee15eb335cc13f9b559f166ee8-Paper.pdf [11] Daiyaan Arfeen, Zhen Zhang, Xinwei Fu, Gregory Ganger, and Yida Wang. 2025. PipeFill: Using GPUs During Bubbles in Pipeline-parallel LLM Training. In Proceedings of Machine Learning and Systems, M. Zaharia, G. Joshi, and Y. Lin (Eds.), Vol. 7. MLSys. https://proceedings.mlsys.org/ paper_files/paper/2025/file/53d3f45797970d323bd8a0d379c525aa-Paper-Conference.pdf [12] Michael Armbrust, Tathagata Das, Joseph Torres, Burak Yavuz, Shixiong Zhu, Reynold Xin, Ali Ghodsi, Ion Stoica, and Matei Zaharia. 2018. Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark. In Proceedings of the 2018 International Conference on Management of Data (Houston, TX, USA) (SIGMOD ’18). Association for Computing Machinery, New York, NY, USA, 601–613. doi:10.1145/3183713. 3190664 [13] Akari Asai, Zeqiu Wu, Yizhong Wang, Avi Sil, and Hannaneh Hajishirzi. 2024. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. In International Conference on Learning Representations, B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (Eds.), Vol. 2024. 9112–9141. https://proceedings.iclr.cc/paper_files/paper/2024/file/25f7be9694d7b32d5cc670927b8091e1-Paper-Conference.pdf [14] Jason Baker, Chris Bond, James C. Corbett, JJ Furman, Andrey Khorlin, James Larson, Jean-Michel Leon, Yawei Li, Alexander Lloyd, and Vadim Yushprakh. 2011. Megastore: Providing Scalable, Highly Available Storage for Interactive Services. In Proceedings of the Conference on Innovative Data system Research (CIDR). 223–234. http://www.cidrdb.org/cidr2011/Papers/CIDR11_Paper32.pdf [15] Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre. 2022. Improving Language Models by Retrieving from Trillions of Tokens. In Proceedings of the 39th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 162), Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (Eds.). PMLR, 2206–2240. https://proceedings.mlr.press/v162/borgeaud22a.html [16] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and SIMONE CALDERARA. 2020. Dark Experience for General Continual Learning: a Strong, Simple Baseline. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 15920–15930. https://proceedings.neurips.cc/paper_files/paper/2020/file/b704ea2c39778f07c617f6b7ce480e9ePaper.pdf [17] Weilin Cai, Le Qin, and Jiayi Huang. 2025. MoC-System: Efficient Fault Tolerance for Sparse Mixture-of-Experts Model Training. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (Rotterdam, Netherlands) (ASPLOS ’25). Association for Computing Machinery, New York, NY, USA, 655–671. doi:10.1145/3676641.3716006 [18] Shiyi Cao, Shu Liu, Tyler Griggs, Peter Schafhalter, Xiaoxuan Liu, Ying Sheng, Joseph E. Gonzalez, Matei Zaharia, and Ion Stoica. 2025. MoELightning: High-Throughput MoE Inference on Memory-constrained GPUs. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 (Rotterdam, Netherlands) (ASPLOS ’25). Association for Computing Machinery, New York, NY, USA, 715–730. doi:10.1145/3669940.3707267 [19] Wei Cao, Yang Liu, Zhushi Cheng, Ning Zheng, Wei Li, Wenjie Wu, Linqiang Ouyang, Peng Wang, Yijing Wang, Ray Kuan, Zhenjun Liu, Feng Zhu, and Tong Zhang. 2020. POLARDB Meets Computational Storage: Efficiently Support Analytical Workloads in Cloud-Native Relational Database. In 18th USENIX Conference on File and Storage Technologies (FAST 20). USENIX Association, Santa Clara, CA, 29–41. https://www.usenix.org/ conference/fast20/presentation/cao-wei [20] Paris Carbone, Gyula Fóra, Stephan Ewen, Seif Haridi, and Kostas Tzoumas. 2015. Lightweight Asynchronous Snapshots for Distributed Dataflows. (2015). doi:10.48550/arXiv.1506.08603 [21] Paris Carbone, Asterios Katsifodimos, Stephan Ewen, Volker Markl, Seif Haridi, and Kostas Tzoumas. 2015. Apache flink: Stream and batch processing in a single engine. The Bulletin of the Technical Committee on Data Engineering 38, 4 (2015). [22] Raul Castro Fernandez, Matteo Migliavacca, Evangelia Kalyvianaki, and Peter Pietzuch. 2013. Integrating scale out and fault tolerance in stream processing using operator state management. In Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data (New York, New York, USA) (SIGMOD ’13). Association for Computing Machinery, New York, NY, USA, 725–736. doi:10.1145/2463676.2465282 [23] Craig Chambers, Ashish Raniwala, Frances Perry, Stephen Adams, Robert R. Henry, Robert Bradshaw, and Nathan Weizenbaum. 2010. FlumeJava: easy, efficient data-parallel pipelines. In Proceedings of the 31st ACM SIGPLAN Conference on Programming Language Design and Implementation (Toronto, Ontario, Canada) (PLDI ’10). Association for Computing Machinery, New York, NY, USA, 363–375. doi:10.1145/1806596.1806638 [24] Badrish Chandramouli, Jonathan Goldstein, Mike Barnett, Robert DeLine, Danyel Fisher, John C. Platt, James F. Terwilliger, and John Wernsing. 2014. Trill: a high-performance incremental query processor for diverse analytics. Proc. VLDB Endow. 8, 4 (Dec. 2014), 401–412. doi:10.14778/ 2735496.2735503 [25] K. Mani Chandy and Leslie Lamport. 1985. Distributed snapshots: determining global states of distributed systems. ACM Trans. Comput. Syst. 3, 1 (Feb. 1985), 63–75. doi:10.1145/214451.214456 [26] Chia-Hao Chang, Jihoon Han, Anand Sivasubramaniam, Vikram Sharma Mailthody, Zaid Qureshi, and Wen mei Hwu. 2024. GMT: GPU Orchestrated Memory Tiering for the Big Data Era. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, ASPLOS 2024, La Jolla, CA, USA, 27 April 2024- 1 May 2024, Rajiv Gupta 0001, Nael B. Abu-Ghazaleh, Madan Musuvathi, and Dan Tsafrir (Eds.). ACM, 464–478. doi:10.1145/3620666.3651353

40

Zhang et al.

[27] Fay Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C. Hsieh, Deborah A. Wallach, Mike Burrows, Tushar Chandra, Andrew Fikes, and Robert E. Gruber. 2008. Bigtable: A Distributed Storage System for Structured Data. ACM Trans. Comput. Syst. 26, 2, Article 4 (June 2008), 26 pages. doi:10.1145/1365815.1365816 [28] Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. 2019. Efficient Lifelong Learning with A-GEM. In International Conference on Learning Representations. https://openreview.net/forum?id=Hkf2_sC5FX [29] Cheng Chen, Chenzhe Jin, Yunan Zhang, Sasha Podolsky, Chun Wu, Szu-Po Wang, Eric Hanson, Zhou Sun, Robert Walzer, and Jianguo Wang. 2024. SingleStore-V: An Integrated Vector Database System in SingleStore. Proc. VLDB Endow. 17, 12 (Aug. 2024), 3772–3785. doi:10.14778/3685800.3685805 [30] Le Chen, Dahu Feng, Erhu Feng, Yingrui Wang, Rong Zhao, Yubin Xia, Pinjie Xu, and Haibo Chen. 2025. Characterizing Mobile SoC for Accelerating Heterogeneous LLM Inference. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles (Lotte Hotel World, Seoul, Republic of Korea) (SOSP ’25). Association for Computing Machinery, New York, NY, USA, 359–374. doi:10.1145/3731569.3764808 [31] Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo, Luis Ceze, and Arvind Krishnamurthy. 2024. Punica: Multi-Tenant LoRA Serving. In Proceedings of Machine Learning and Systems, P. Gibbons, G. Pekhimenko, and C. De Sa (Eds.), Vol. 6. 1–13. https://proceedings.mlsys.org/paper_files/paper/ 2024/file/054de805fcceb78a201f5e9d53c85908-Paper-Conference.pdf [32] Albert Cho, Anish Saxena, Moinuddin Qureshi, and Alexandros Daglis. 2024. COAXIAL: A CXL-Centric Memory System for Scalable Servers. In Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis (Atlanta, GA, USA) (SC ’24). IEEE Press, Article 95, 15 pages. doi:10.1109/SC41406.2024.00101 [33] Peizhuang Cong, Tong Yang, Yuchao Zhang, Wendong Wang, and Ke Xu. 2026. MICO: efficient query scheduling for multi-cloud deployed LLM inference service. Science China Information Sciences 69, 3 (2026), 132102. [34] James C. Corbett, Jeffrey Dean, Michael Epstein, Andrew Fikes, Christopher Frost, J. J. Furman, Sanjay Ghemawat, Andrey Gubarev, Christopher Heiser, Peter Hochschild, Wilson Hsieh, Sebastian Kanthak, Eugene Kogan, Hongyi Li, Alexander Lloyd, Sergey Melnik, David Mwaura, David Nagle, Sean Quinlan, Rajesh Rao, Lindsay Rolig, Yasushi Saito, Michal Szymaniak, Christopher Taylor, Ruth Wang, and Dale Woodford. 2013. Spanner: Google’s Globally Distributed Database. ACM Trans. Comput. Syst. 31, 3, Article 8 (Aug. 2013), 22 pages. doi:10.1145/2491245 [35] Daniel Crankshaw, Gur-Eyal Sela, Xiangxi Mo, Corey Zumar, Ion Stoica, Joseph Gonzalez, and Alexey Tumanov. 2020. InferLine: latency-aware provisioning and scaling for prediction serving pipelines. In Proceedings of the 11th ACM Symposium on Cloud Computing (Virtual Event, USA) (SoCC ’20). Association for Computing Machinery, New York, NY, USA, 477–491. doi:10.1145/3419111.3421285 [36] Daniel Crankshaw, Xin Wang, Giulio Zhou, Michael J. Franklin, Joseph E. Gonzalez, and Ion Stoica. 2017. Clipper: a low-latency online prediction serving system. In Proceedings of the 14th USENIX Conference on Networked Systems Design and Implementation (Boston, MA, USA) (NSDI’17). USENIX Association, USA, 613–627. [37] Gianpaolo Cugola and Alessandro Margara. 2012. Processing flows of information: From data stream to complex event processing. ACM Comput. Surv. 44, 3, Article 15 (June 2012), 62 pages. doi:10.1145/2187671.2187677 [38] Tri Dao. 2024. Flashattention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations, Vol. 2024. 35549–35562. [39] Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. 2022. A Continual Learning Survey: Defying Forgetting in Classification Tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 7 (2022), 3366–3385. doi:10.1109/TPAMI.2021.3057446 [40] Giuseppe DeCandia, Deniz Hastorun, Madan Jampani, Gunavardhan Kakulapati, Avinash Lakshman, Alex Pilchin, Swaminathan Sivasubramanian, Peter Vosshall, and Werner Vogels. 2007. Dynamo: amazon’s highly available key-value store. In Proceedings of Twenty-First ACM SIGOPS Symposium on Operating Systems Principles (Stevenson, Washington, USA) (SOSP ’07). Association for Computing Machinery, New York, NY, USA, 205–220. doi:10.1145/1294261.1294281 [41] David Domingo and Sudarsun Kannan. 2021. pFSCK: Accelerating File System Checking and Repair for Modern Storage. In 19th USENIX Conference on File and Storage Technologies (FAST 21). USENIX Association, 113–126. https://www.usenix.org/conference/fast21/presentation/domingo [42] Juechu Dong, Jonah Rosenblum, and Satish Narayanasamy. 2025. Toleo: Scaling Freshness to Tera-scale Memory Using CXL and PIM. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 4 (Hilton La Jolla Torrey Pines, La Jolla, CA, USA) (ASPLOS ’24). Association for Computing Machinery, New York, NY, USA, 313–328. doi:10.1145/3622781.3674180 [43] Aleksandar Dragojević, Dushyanth Narayanan, Orion Hodson, and Miguel Castro. 2014. FaRM: fast remote memory. In Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation (Seattle, WA) (NSDI’14). USENIX Association, USA, 401–414. [44] Kuntai Du, Bowen Wang, Chen Zhang, Yiming Cheng, Qing Lan, Hejian Sang, Yihua Cheng, Jiayi Yao, Xiaoxuan Liu, Yifan Qiao, Ion Stoica, and Junchen Jiang. 2025. PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applications. (2025). doi:10.48550/ ARXIV.2505.07203 [45] Assaf Eisenman, Kiran Kumar Matam, Steven Ingram, Dheevatsa Mudigere, Raghuraman Krishnamoorthi, Krishnakumar Nair, Misha Smelyanskiy, and Murali Annavaram. 2022. Check-N-Run: a Checkpointing System for Training Deep Learning Recommendation Models. In 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22). USENIX Association, Renton, WA, 929–943. https://www.usenix.org/ conference/nsdi22/presentation/eisenman [46] Ruibo Fan, Xiangrui Yu, Peijie Dong, Zeyu Li, Gu Gong, Qiang Wang, Wei Wang, and Xiaowen Chu. 2025. SpInfer: Leveraging Low-Level Sparsity for Efficient Large Language Model Inference on GPUs. In Proceedings of the Twentieth European Conference on Computer Systems (Rotterdam, Netherlands) (EuroSys ’25). Association for Computing Machinery, New York, NY, USA, 243–260. doi:10.1145/3689031.3717481

Appendix

41

[47] Chao Fang, Man Shi, Robin Geens, Arne Symons, Zhongfeng Wang, and Marian Verhelst. 2025. Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Format. In 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA). 1467–1481. doi:10.1109/HPCA61900.2025.00110 [48] Fei Fang, Yifan Hua, Shengze Wang, Ruilin Zhou, Yi Liu, Chen Qian, and Xiaoxue Zhang. 2026. PlanetServe: A Decentralized, Scalable, and Privacy-Preserving Overlay for Democratizing Large Language Model Serving. In 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26). USENIX Association, Renton, WA, 2111–2129. https://www.usenix.org/conference/nsdi26/presentation/fang [49] Marco Federici, Davide Belli, Mart Van Baalen, Amir Jalalirad, Andrii Skliar, Bence Major, Markus Nagel, and Paul Whatmough. 2025. Efficient LLM Inference using Dynamic Input Pruning and Cache-Aware Masking. In Proceedings of Machine Learning and Systems, M. Zaharia, G. Joshi, and Y. Lin (Eds.), Vol. 7. MLSys. https://proceedings.mlsys.org/paper_files/paper/2025/file/afd6374c7f2839cba22f537f15f4f760-Paper-Conference.pdf [50] Jingqi Feng, Yukai Huang, Rui Zhang, Sicheng Liang, Ming Yan, and Jie Wu. 2025. WindServe: Efficient Phase-Disaggregated LLM Serving with Stream-based Dynamic Scheduling. In Proceedings of the 52nd Annual International Symposium on Computer Architecture (ISCA ’25). Association for Computing Machinery, New York, NY, USA, 1283–1295. doi:10.1145/3695053.3730999 [51] Weiqi Feng, Yangrui Chen, Shaoyu Wang, Yanghua Peng, Haibin Lin, and Minlan Yu. 2025. Optimus: accelerating large-scale multi-modal LLM training by bubble exploitation. In Proceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference (Boston, MA, USA) (USENIX ATC ’25). USENIX Association, USA, Article 10, 17 pages. [52] Xiang Fu, Weiping Zhang, Shiman Meng, Xin Huang, Wubiao Xu, Luanzheng Guo, and Kento Sato. 2024. AutoCheck: Automatically Identifying Variables for Checkpointing by Data Dependency Analysis. In Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis (Atlanta, GA, USA) (SC ’24). IEEE Press, Article 99, 16 pages. doi:10.1109/SC41406.2024.00105 [53] Yaosheng Fu, Evgeny Bolotin, Aamer Jaleel, Gal Dalal, Shie Mannor, Jacob Subag, Noam Korem, Michael Behar, and David Nellans. 2023. AutoScratch: ML-Optimized Cache Management for Inference-Oriented GPUs. In Proceedings of Machine Learning and Systems, D. Song, M. Carbin, and T. Chen (Eds.), Vol. 5. Curan, 495–512. https://proceedings.mlsys.org/paper_files/paper/2023/file/9d32b9324a89001520ae456b9e5ec73b-Paper-mlsys2023.pdf [54] Yuqi Fu, Li Liu, Haoliang Wang, Yue Cheng, and Songqing Chen. 2022. SFS: smart OS scheduling for serverless functions. In Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis (Dallas, Texas) (SC ’22). IEEE Press, Article 42, 16 pages. [55] Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yuvraj Patel, and Luo Mai. 2024. ServerlessLLM: low-latency serverless inference for large language models. In Proceedings of the 18th USENIX Conference on Operating Systems Design and Implementation (Santa Clara, CA, USA) (OSDI’24). USENIX Association, USA, Article 8, 19 pages. [56] Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. 2024. Cost-efficient large language model serving for multi-turn conversations with CachedAttention. In Proceedings of the 2024 USENIX Conference on Usenix Annual Technical Conference (Santa Clara, CA, USA) (USENIX ATC’24). USENIX Association, USA, Article 7, 16 pages. [57] Hongru Gao, Shuhao Zhang, Xiaofei Liao, and Hai Jin. 2026. GRACE: Alleviating Reconstruction Cost in Dynamic Graph Processing Systems. In Proceedings of the IEEE International Conference on Data Engineering. [58] Shiwei Gao, Youmin Chen, and Jiwu Shu. 2025. Fast State Restoration in LLM Serving with HCache. In Proceedings of the Twentieth European Conference on Computer Systems (Rotterdam, Netherlands) (EuroSys ’25). Association for Computing Machinery, New York, NY, USA, 128–143. doi:10.1145/3689031.3696072 [59] Shiwei Gao, Qing Wang, Shaoxun Zeng, Youyou Lu, and Jiwu Shu. 2025. Weaver: Efficient Multi-LLM Serving with Attention Offloading. In 2025 USENIX Annual Technical Conference (USENIX ATC 25). USENIX Association, Boston, MA, 587–595. https://www.usenix.org/conference/atc25/ presentation/gao [60] Xin Gao, Sibasish Acharya, Sihui Han, Yongxiong Ren, Yanli Zhao, Liang Luo, Chucheng Wang, Pradeep Fernando, Saurabh Mishra, Siqi Yan, Yicong Du, Elzbieta Krepska, Intaik Park, Min Ni, Qunshu Zhang, and Shen Li. 2025. DECK: Experiences on Delta Checkpointing for Industrial Recommendation Systems. Proc. VLDB Endow. 18, 12 (Aug. 2025), 4978–4990. doi:10.14778/3750601.3750621 [61] Gerasimos Gerogiannis, Stijn Eyerman, Evangelos Georganas, Wim Heirman, and Josep Torrellas. 2025. DECA: A Near-Core LLM Decompression Accelerator Grounded on a 3D Roofline Model. In Proceedings of the 58th IEEE/ACM International Symposium on Microarchitecture (MICRO ’25). Association for Computing Machinery, New York, NY, USA, 184–200. doi:10.1145/3725843.3756073 [62] In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. 2024. Prompt Cache: Modular Attention Reuse for Low-Latency Inference. In Proceedings of Machine Learning and Systems, P. Gibbons, G. Pekhimenko, and C. De Sa (Eds.), Vol. 6. 325–338. https://proceedings.mlsys.org/paper_files/paper/2024/file/a66caa1703fe34705a4368c3014c1966-Paper-Conference.pdf [63] In Gim, Zhiyao Ma, Seung-seob Lee, and Lin Zhong. 2025. Pie: A Programmable Serving System for Emerging LLM Applications. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles (Lotte Hotel World, Seoul, Republic of Korea) (SOSP ’25). Association for Computing Machinery, New York, NY, USA, 415–430. doi:10.1145/3731569.3764814 [64] Inigo Goiri, Ricardo Bianchini, Santosh Nagarakatte, and Thu D. Nguyen. 2015. ApproxHadoop: Bringing Approximations to MapReduce Frameworks. In Proceedings of the Twentieth International Conference on Architectural Support for Programming Languages and Operating Systems (Istanbul, Turkey) (ASPLOS ’15). Association for Computing Machinery, New York, NY, USA, 383–397. doi:10.1145/2694344.2694351 [65] Ruihao Gong, Shihao Bai, Siyu Wu, Yunqian Fan, Zaijun Wang, Xiuhong Li, Hailong Yang, and Xianglong Liu. 2025. Past-Future Scheduler for LLM Serving under SLA Guarantees. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (Rotterdam, Netherlands) (ASPLOS ’25). Association for Computing Machinery, New York, NY, USA, 798–813. doi:10.1145/3676641.3716011

42

Zhang et al.

[66] Yufeng Gu, Alireza Khadem, Sumanth Umesh, Ning Liang, Xavier Servot, Onur Mutlu, Ravi Iyer, and Reetuparna Das. 2025. PIM Is All You Need: A CXL-Enabled GPU-Free System for Large Language Model Inference. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (Rotterdam, Netherlands) (ASPLOS ’25). Association for Computing Machinery, New York, NY, USA, 862–881. doi:10.1145/3676641.3716267 [67] Yue Guan, Xinwei Qiang, Zaifeng Pan, Daniels Johnson, Yuanwei Fang, Keren Zhou, Yuke Wang, Wanlu Li, Yufei Ding, and Adnan Aziz. 2025. Mercury: Unlocking Multi-GPU Operator Optimization for LLMs via Remote Memory Scheduling. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles (Lotte Hotel World, Seoul, Republic of Korea) (SOSP ’25). Association for Computing Machinery, New York, NY, USA, 1046–1061. doi:10.1145/3731569.3764798 [68] Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, Antoine Kaufmann, Ymir Vigfusson, and Jonathan Mace. 2020. Serving DNNs like clockwork: performance predictability from the bottom up. In Proceedings of the 14th USENIX Conference on Operating Systems Design and Implementation (OSDI’20). USENIX Association, USA, Article 25, 20 pages. [69] Vincenzo Gulisano, Ricardo Jiménez-Peris, Marta Patiño-Martínez, Claudio Soriente, and Patrick Valduriez. 2012. StreamCloud: An Elastic and Scalable Data Streaming System. IEEE Transactions on Parallel and Distributed Systems 23, 12 (2012), 2351–2365. doi:10.1109/TPDS.2012.24 [70] Cong Guo, Rui Zhang, Jiale Xu, Jingwen Leng, Zihan Liu, Ziyu Huang, Minyi Guo, Hao Wu, Shouren Zhao, Junping Zhao, and Ke Zhang. 2024. GMLake: Efficient and Transparent GPU Memory Defragmentation for Large-scale DNN Training with Virtual Memory Stitching. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (La Jolla, CA, USA) (ASPLOS ’24). Association for Computing Machinery, New York, NY, USA, 450–466. doi:10.1145/3620665.3640423 [71] Rentong Guo, Xiaofan Luan, Long Xiang, Xiao Yan, Xiaomeng Yi, Jigao Luo, Qianya Cheng, Weizhi Xu, Jiarui Luo, Frank Liu, Zhenshan Cao, Yanliang Qiao, Ting Wang, Bo Tang, and Charles Xie. 2022. Manu: a cloud native vector database management system. Proc. VLDB Endow. 15, 12 (Aug. 2022), 3548–3561. doi:10.14778/3554821.3554843 [72] Tianyu Guo, Xianwei Zhang, Jiangsu Du, Zhiguang Chen, Nong Xiao, and Yutong Lu. 2025. gLLM: Global Balanced Pipeline Parallelism Systems for Distributed LLMs Serving with Token Throttling. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’25). Association for Computing Machinery, New York, NY, USA, 1725–1741. doi:10.1145/3712285.3759823 [73] Yipin Guo and Siddharth Joshi. 2026. SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving. (2026). doi:10.48550/ARXIV. 2605.01708 [74] Bernal Jiménez Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. 2024. HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37. Curran Associates, Inc., 59532–59569. doi:10.52202/079017-1902 [75] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. REALM: retrieval-augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning (ICML’20). JMLR.org, Article 368, 10 pages. [76] Tyler L. Hayes, Kushal Kafle, Robik Shrestha, Manoj Acharya, and Christopher Kanan. 2020. REMIND Your Neural Network to Prevent Catastrophic Forgetting. In Computer Vision – ECCV 2020, Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer International Publishing, Cham, 466–483. [77] Congjie He, Yeqi Huang, Pei Mu, Ziming Miao, Jilong Xue, Lingxiao Ma, Fan Yang, and Luo Mai. 2025. WaferLLM: large language model inference at wafer scale. In Proceedings of the 19th USENIX Conference on Operating Systems Design and Implementation (Boston, MA, USA) (OSDI ’25). USENIX Association, USA, Article 15, 17 pages. [78] Yongjun He, Haofeng Yang, Yao Lu, Ana Klimović, and Gustavo Alonso. 2025. Resource multiplexing in tuning and serving large language models. In Proceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference (Boston, MA, USA) (USENIX ATC ’25). USENIX Association, USA, Article 97, 17 pages. [79] Martin Hirzel, Robert Soulé, Scott Schneider, Buğra Gedik, and Robert Grimm. 2014. A catalog of stream processing optimizations. ACM Comput. Surv. 46, 4, Article 46 (March 2014), 34 pages. doi:10.1145/2528412 [80] Moritz Hoffmann, Andrea Lattuada, and Frank McSherry. 2019. Megaphone: latency-conscious state migration for distributed streaming dataflows. Proc. VLDB Endow. 12, 9 (May 2019), 1002–1015. doi:10.14778/3329772.3329777 [81] Connor Holmes, Masahiro Tanaka, Michael Wyatt, Ammar Ahmad Awan, Jeff Rasley, Samyam Rajbhandari, Reza Yazdani Aminabadi, Heyang Qin, Arash Bakhtiari, Lev Kurilenko, and Yuxiong He. 2024. DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference. (2024). doi:10.48550/ARXIV.2401.08671 [82] Jeongmin Hong, Sungjun Cho, Geonwoo Park, Wonhyuk Yang, Young-Ho Gong, and Gwangsun Kim. 2024. Bandwidth-Effective DRAM Cache for GPU s with Storage-Class Memory. In 2024 IEEE International Symposium on High-Performance Computer Architecture (HPCA). 139–155. doi:10.1109/HPCA57654.2024.00021 [83] Ke Hong, Xiuhong Li, Lufang Chen, Qiuli Mao, Guohao Dai, Xuefei Ning, Shengen Yan, Yun Liang, and Yu Wang. 2025. SOLA: Optimizing SLO Attainment for Large Language Model Serving with State-Aware Scheduling. In Proceedings of Machine Learning and Systems, M. Zaharia, G. Joshi, and Y. Lin (Eds.), Vol. 7. MLSys. https://proceedings.mlsys.org/paper_files/paper/2025/file/bc82dbfbfa43232be85b8d9838f49c3e-PaperConference.pdf [84] Zhisheng Hu, Pengfei Zuo, Yizou Chen, Chao Wang, Junliang Hu, and Ming-Chang Yang. 2024. Aceso: Achieving Efficient Fault Tolerance in Memory-Disaggregated Key-Value Stores. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles (Austin, TX, USA) (SOSP ’24). Association for Computing Machinery, New York, NY, USA, 127–143. doi:10.1145/3694715.3695951

Appendix

43

[85] Chunyue Huang, Shuang Liu, Xinyi Zhang, Wenhao Li, Wei Lu, and Xiaoyong Du. 2025. Chimera: Mitigating Ownership Transfers in Multi-Primary Shared-Storage Cloud-Native Databases. Proc. VLDB Endow. 18, 10 (June 2025), 3368–3381. doi:10.14778/3748191.3748201 [86] Yibo Huang, Haowei Chen, Newton Ni, Yan Sun, Vijay Chidambaram, Dixin Tang, and Emmett Witchel. 2025. Tigon: a distributed database for a CXL pod. In Proceedings of the 19th USENIX Conference on Operating Systems Design and Implementation (Boston, MA, USA) (OSDI ’25). USENIX Association, USA, Article 7, 20 pages. [87] Yuzhou Huang, Yapeng Jiang, Zicong Hong, Wuhui Chen, Bin Wang, Weixi Zhu, Yue Yu, and Zibin Zheng. 2025. Obscura: concealing recomputation overhead in training of large language models with bubble-filling pipeline transformation. In Proceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference (Boston, MA, USA) (USENIX ATC ’25). USENIX Association, USA, Article 40, 14 pages. [88] Zimeng Huang, Hao Nie, Haonan Jia, Bo Jiang, Junchen Guo, Jianyuan Lu, Rong Wen, Biao Lyu, Shunmin Zhu, and Xinbing Wang. 2025. FlowCheck: Decoupling Checkpointing and Training of Large-Scale Models. In Proceedings of the Twentieth European Conference on Computer Systems (Rotterdam, Netherlands) (EuroSys ’25). Association for Computing Machinery, New York, NY, USA, 1334–1349. doi:10.1145/3689031.3696088 [89] Nikoleta Iliakopoulou, Jovan Stojkovic, Chloe Alverti, Tianyin Xu, Hubertus Franke, and Josep Torrellas. 2025. Chameleon: Adaptive Caching and Scheduling for Many-Adapter LLM Inference Environments. In Proceedings of the 58th IEEE/ACM International Symposium on Microarchitecture (MICRO ’25). Association for Computing Machinery, New York, NY, USA, 217–231. doi:10.1145/3725843.3756083 [90] Michael Isard, Mihai Budiu, Yuan Yu, Andrew Birrell, and Dennis Fetterly. 2007. Dryad: distributed data-parallel programs from sequential building blocks. In Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007 (Lisbon, Portugal) (EuroSys ’07). Association for Computing Machinery, New York, NY, USA, 59–72. doi:10.1145/1272996.1273005 [91] Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2023. Atlas: few-shot learning with retrieval augmented language models. J. Mach. Learn. Res. 24, 1, Article 251 (Jan. 2023), 43 pages. [92] Sepehr Jalalian, Shaurya Patel, Milad Rezaei Hajidehi, Margo Seltzer, and Alexandra Fedorova. 2024. EXTMEM: enabling application-aware virtual memory management for data-intensive applications. In Proceedings of the 2024 USENIX Conference on Usenix Annual Technical Conference (Santa Clara, CA, USA) (USENIX ATC’24). USENIX Association, USA, Article 25, 12 pages. [93] Hongsun Jang, Jaeyong Song, Jaewon Jung, Jaeyoung Park, Youngsok Kim, and Jinho Lee. 2024. Smart-Infinity: Fast Large Language Model Training using Near-Storage Processing on a Real System. In 2024 IEEE International Symposium on High-Performance Computer Architecture (HPCA). 345–360. doi:10.1109/HPCA57654.2024.00034 [94] Keshav Vinayak Jha, Shweta Pandey, Murali Annavaram, and Arkaprava Basu. 2025. HyCache: hybrid caching for accelerating DNN input preprocessing pipelines. In Proceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference (Boston, MA, USA) (USENIX ATC ’25). USENIX Association, USA, Article 26, 16 pages. [95] Houxiang Ji, Yifan Yuan, Yang Zhou, Ipoom Jeong, Ren Wang, Saksham Agarwal, and Nam Sung Kim. 2025. Re-architecting End-host Networking with CXL: Coherence, Memory, and Offloading. In Proceedings of the 58th IEEE/ACM International Symposium on Microarchitecture (MICRO ’25). Association for Computing Machinery, New York, NY, USA, 1809–1823. doi:10.1145/3725843.3756102 [96] Sheng Jiang and Ming Liu. 2025. Building an Elastic Block Storage over EBOFs Using Shadow Views. In 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25). USENIX Association, Philadelphia, PA, 1137–1153. https://www.usenix.org/conference/nsdi25/ presentation/jiang [97] Xuanlin Jiang, Yang Zhou, Shiyi Cao, Ion Stoica, and Minlan Yu. 2025. NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference. In Proceedings of Machine Learning and Systems, M. Zaharia, G. Joshi, and Y. Lin (Eds.), Vol. 7. MLSys. https://proceedings.mlsys.org/ paper_files/paper/2025/file/66a026c0d17040889b50f0dfa650e5e0-Paper-Conference.pdf [98] YOUHE JIANG, Fangcheng Fu, Xiaozhe Yao, Taiyi Wang, Bin CUI, Ana Klimovic, and Eiko Yoneki. 2025. ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments. In Proceedings of Machine Learning and Systems, M. Zaharia, G. Joshi, and Y. Lin (Eds.), Vol. 7. MLSys. https://proceedings.mlsys.org/paper_files/paper/2025/file/c2a0e26dd9ee7d57e92bb1c24b39659a-Paper-Conference.pdf [99] Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, Yulu Jia, Sun He, Hongmin Chen, Zhihao Bai, Qi Hou, Shipeng Yan, Ding Zhou, Yiyao Sheng, Zhuo Jiang, Haohan Xu, Haoran Wei, Zhang Zhang, Pengfei Nie, Leqi Zou, Sida Zhao, Liang Xiang, Zherui Liu, Zhe Li, Xiaoying Jia, Jianxi Ye, Xin Jin, and Xin Liu. 2024. MegaScale: scaling large language model training to more than 10,000 GPUs. In Proceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation (Santa Clara, CA, USA) (NSDI’24). USENIX Association, USA, Article 41, 16 pages. [100] Sian Jin, Sheng Di, Frédéric Vivien, Daoce Wang, Yves Robert, Dingwen Tao, and Franck Cappello. 2024. Concealing Compression-accelerated I/O for HPC Applications through In Situ Task Scheduling. In Proceedings of the Nineteenth European Conference on Computer Systems (Athens, Greece) (EuroSys ’24). Association for Computing Machinery, New York, NY, USA, 981–998. doi:10.1145/3627703.3629573 [101] Donghyeon Joo, Helya Hosseini, Ramyad Hadidi, and Bahar Asgari. 2025. Coruscant: Co-Designing GPU Kernel and Sparse Tensor Core to Advocate Unstructured Sparsity in Efficient LLM Inference. In Proceedings of the 58th IEEE/ACM International Symposium on Microarchitecture (MICRO ’25). Association for Computing Machinery, New York, NY, USA, 232–245. doi:10.1145/3725843.3756065 [102] Herve Jégou, Matthijs Douze, and Cordelia Schmid. 2011. Product Quantization for Nearest Neighbor Search. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 1 (2011), 117–128. doi:10.1109/TPAMI.2010.57 [103] Robert Kallman, Hideaki Kimura, Jonathan Natkins, Andrew Pavlo, Alexander Rasin, Stanley Zdonik, Evan P. C. Jones, Samuel Madden, Michael Stonebraker, Yang Zhang, John Hugg, and Daniel J. Abadi. 2008. H-store: a high-performance, distributed main memory transaction processing

44

Zhang et al.

system. Proc. VLDB Endow. 1, 2 (Aug. 2008), 1496–1499. doi:10.14778/1454159.1454211 [104] Aditya K. Kamath, Ramya Prabhu, Jayashree Mohan, Simon Peter, Ramachandran Ramjee, and Ashish Panwar. 2025. POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (Rotterdam, Netherlands) (ASPLOS ’25). Association for Computing Machinery, New York, NY, USA, 897–912. doi:10.1145/3676641.3715996 [105] Hao Kang, Srikant Bharadwaj, James Hensman, Tushar Krishna, Victor Rühle, and Saravan Rajmohan. 2025. TurboAttention: Efficient attention approximation for high throughputs llm. In Proceedings of Machine Learning and Systems, M. Zaharia, G. Joshi, and Y. Lin (Eds.), Vol. 7. MLSys. https://proceedings.mlsys.org/paper_files/paper/2025/file/f4f55846501f3336f293fd8b6de10770-Paper-Conference.pdf [106] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, Online, 6769–6781. doi:10.18653/v1/2020.emnlpmain.550 [107] Ahmad Faraz Khan, Samuel Fountain, Ahmed M. Abdelmoniem, Ali R. Butt, and Ali Anwar. 2025. FLStore: Efficient Federated Learning Storage for non-training workloads. In Proceedings of Machine Learning and Systems, M. Zaharia, G. Joshi, and Y. Lin (Eds.), Vol. 7. MLSys. https://proceedings.mlsys.org/paper_files/paper/2025/file/f37347375d8b54e3203e5d24aeb6c58c-Paper-Conference.pdf [108] Omar Khattab and Matei Zaharia. 2020. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (Virtual Event, China) (SIGIR ’20). Association for Computing Machinery, New York, NY, USA, 39–48. doi:10.1145/3397271.3401075 [109] Hyungyo Kim, Nachuan Wang, Qirong Xia, Jinghan Huang, Amir Yazdanbakhsh, and Nam Sung Kim. 2025. LIA: A Single-GPU LLM Inference Acceleration with Cooperative AMX-Enabled CPU-GPU Computation and CXL Offloading. In Proceedings of the 52nd Annual International Symposium on Computer Architecture (ISCA ’25). Association for Computing Machinery, New York, NY, USA, 544–558. doi:10.1145/3695053.3731092 [110] Minsu Kim, Seongmin Hong, RyeoWook Ko, Soongyu Choi, Hunjong Lee, Junsoo Kim, Joo-Young Kim, and Jongse Park. 2025. Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantization. In Proceedings of the 52nd Annual International Symposium on Computer Architecture (ISCA ’25). Association for Computing Machinery, New York, NY, USA, 482–497. doi:10.1145/3695053.3731019 [111] Wonung Kim, Yubin Lee, Yoonsung Kim, Jinwoo Hwang, Seongryong Oh, Jiyong Jung, Aziz Huseynov, Woong Gyu Park, Chang Hyun Park, Divya Mahajan, and Jongse Park. 2025. Pimba: A Processing-in-Memory Acceleration for Post-Transformer Large Language Model Serving. In Proceedings of the 58th IEEE/ACM International Symposium on Microarchitecture (MICRO ’25). Association for Computing Machinery, New York, NY, USA, 292–307. doi:10.1145/3725843.3756121 [112] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences 114, 13 (2017), 3521–3526. arXiv:https://www.pnas.org/doi/pdf/10.1073/pnas.1611835114 doi:10.1073/pnas.1611835114 [113] Dhanya R. Krishnan, Do Le Quoc, Pramod Bhatotia, Christof Fetzer, and Rodrigo Rodrigues. 2016. IncApprox: A Data Analytics System for Incremental Approximate Computing. In Proceedings of the 25th International Conference on World Wide Web (Montréal, Québec, Canada) (WWW ’16). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 1133–1144. doi:10.1145/2872427.2883026 [114] Abhishek Vijaya Kumar and Muthian Sivathanu. 2020. Quiver: an informed storage cache for deep learning. In Proceedings of the 18th USENIX Conference on File and Storage Technologies (Santa Clara, CA, USA) (FAST’20). USENIX Association, USA, 283–296. [115] Dongup Kwon, Junehyuk Boo, Dongryeong Kim, and Jangwoo Kim. 2020. FVM: FPGA-assisted virtual device emulation for fast, scalable, and flexible storage virtualization. In Proceedings of the 14th USENIX Conference on Operating Systems Design and Implementation (OSDI’20). USENIX Association, USA, Article 54, 17 pages. [116] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (Koblenz, Germany) (SOSP ’23). Association for Computing Machinery, New York, NY, USA, 611–626. doi:10.1145/3600006.3613165 [117] Avinash Lakshman and Prashant Malik. 2010. Cassandra: a decentralized structured storage system. SIGOPS Oper. Syst. Rev. 44, 2 (April 2010), 35–40. doi:10.1145/1773912.1773922 [118] Hyojung Lee, Daehyeon Baek, Jimyoung Son, Jieun Choi, Kihyo Moon, and Minsung Jang. 2025. PAISE: PIM-Accelerated Inference Scheduling Engine for Transformer-based LLM. In 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA). 1707–1719. doi:10.1109/HPCA61900.2025.00126 [119] Hwanjun Lee, Minho Kim, Yeji Jung, Seonmu Oh, Ki-Dong Kang, Seunghak Lee, and Daehoon Kim. 2025. Beyond Page Migration: Enhancing Tiered Memory Performance via Integrated Last-Level Cache Management and Page Migration. In Proceedings of the 58th IEEE/ACM International Symposium on Microarchitecture (MICRO ’25). Association for Computing Machinery, New York, NY, USA, 1763–1776. doi:10.1145/3725843.3756063 [120] Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. 2024. InfiniGen: efficient generative inference of large language models with dynamic KV cache management. In Proceedings of the 18th USENIX Conference on Operating Systems Design and Implementation (Santa Clara, CA, USA) (OSDI’24). USENIX Association, USA, Article 9, 18 pages. [121] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances

Appendix

45

in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 9459–9474. https://proceedings.neurips.cc/paper_files/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf [122] Changwu Li, Tongjun Shi, Shuhao Zhang, Binbin Chen, Bingsheng He, Xiaofei Liao, and Hai Jin. 2026. StreamFP: Fingerprint-guided Data Selection for Efficient Stream Learning. In Proceedings of the ACM Web Conference 2026 (United Arab Emirates) (WWW ’26). Association for Computing Machinery, New York, NY, USA, 7474–7484. doi:10.1145/3774904.3792584 [123] Jialin Li, Jacob Nelson, Ellis Michael, Xin Jin, and Dan R. K. Ports. 2020. Pegasus: Tolerating skewed workloads in distributed storage with in-network coherence directories. In Proceedings of the 14th USENIX Conference on Operating Systems Design and Implementation (OSDI’20). USENIX Association, USA, Article 22, 20 pages. [124] Suyi Li, Hanfeng Lu, Tianyuan Wu, Minchen Yu, Qizhen Weng, Xusheng Chen, Yizhou Shan, Binhang Yuan, and Wei Wang. 2025. TOPPINGS: CPU-assisted, rank-aware adapter serving for LLM inference. In Proceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference (Boston, MA, USA) (USENIX ATC ’25). USENIX Association, USA, Article 37, 17 pages. [125] Yading Li, Dandan Song, Yuhang Tian, Hao Wang, Changzhi Zhou, and Shuhao Zhang. 2026. A Framework of Knowledge Graph-Enhanced Large Language Model Based on Global Planning. IEEE Transactions on Knowledge and Data Engineering 38, 2 (2026), 736–748. doi:10.1109/TKDE.2025. 3639599 [126] Yading Li, Dandan Song, Changzhi Zhou, Yuhang Tian, Hao Wang, Ziyi Yang, and Shuhao Zhang. 2024. A Framework of Knowledge Graph-Enhanced Large Language Model Based on Question Decomposition and Atomic Retrieval. In Findings of the Association for Computational Linguistics: EMNLP 2024, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 11472–11485. doi:10.18653/v1/2024.findings-emnlp.670 [127] Zhouyang Li, Yuliang Liu, Wei Zhang, Tailing Yuan, Bin Chen, and Chengru Song. 2025. SlimPipe: Memory-Thrifty and Efficient Pipeline Parallelism for Long-Context LLM Training. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’25). Association for Computing Machinery, New York, NY, USA, 1409–1428. doi:10.1145/3712285.3759855 [128] Zhiyue Li and Guangyan Zhang. 2024. StreamCache: revisiting page cache for file scanning on fast storage devices. In Proceedings of the 2024 USENIX Conference on Usenix Annual Technical Conference (Santa Clara, CA, USA) (USENIX ATC’24). USENIX Association, USA, Article 68, 16 pages. [129] Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23). USENIX Association, Boston, MA, 663–679. https://www.usenix.org/conference/osdi23/presentation/lizhouhan [130] Dimitrios Liakopoulos, Prasoon Sinha, Tianrui Hu, Myungjin Lee, and Neeraja J. Yadwadkar. 2025. MaverIQ: Fingerprint-Guided Extrapolation and Fragmentation-Aware Layering for Intent-Based LLM Serving. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’25). Association for Computing Machinery, New York, NY, USA, 1676–1696. doi:10.1145/3712285.3759867 [131] Xinyu Lian, Sam Ade Jacobs, Lev Kurilenko, Masahiro Tanaka, Stas Bekman, Olatunji Ruwase, and Minjia Zhang. 2025. Universal checkpointing: a flexible and efficient distributed checkpointing system for large-scale DNN training with reconfigurable parallelism. In Proceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference (Boston, MA, USA) (USENIX ATC ’25). USENIX Association, USA, Article 90, 16 pages. [132] Gangmuk Lim, Jeongseob Ahn, Wencong Xiao, Youngjin Kwon, and Myeongjae Jeon. 2021. Zico: Efficient GPU Memory Sharing for Concurrent DNN Training. In 2021 USENIX Annual Technical Conference (USENIX ATC 21). USENIX Association, 161–175. https://www.usenix.org/conference/ atc21/presentation/lim [133] Chaofan Lin, Zhenhua Han, Chengruidong Zhang, Yuqing Yang, Fan Yang, Chen Chen, and Lili Qiu. 2024. Parrot: efficient serving of LLM-based applications with semantic variable. In Proceedings of the 18th USENIX Conference on Operating Systems Design and Implementation (Santa Clara, CA, USA) (OSDI’24). USENIX Association, USA, Article 50, 17 pages. [134] Jinkun Lin, Ziheng Jiang, Zuquan Song, Sida Zhao, Menghan Yu, Zhanghan Wang, Chenyuan Wang, Zuocheng Shi, Xiang Shi, Wei Jia, Zherui Liu, Shuguang Wang, Haibin Lin, Xin Liu, Aurojit Panda, and Jinyang Li. 2025. Understanding stragglers in large model training using what-if analysis. In Proceedings of the 19th USENIX Conference on Operating Systems Design and Implementation (Boston, MA, USA) (OSDI ’25). USENIX Association, USA, Article 27, 16 pages. [135] Junfeng Lin, Ziming Liu, Yang You, Jun Wang, Weihao Zhang, and Rong Zhao. 2025. WeiPipe: Weight Pipeline Parallelism for CommunicationEffective Long-Context Large Model Training. In Proceedings of the 30th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming (Las Vegas, NV, USA) (PPoPP ’25). Association for Computing Machinery, New York, NY, USA, 225–238. doi:10.1145/3710848.3710869 [136] Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. 2025. QServe:W4A8KV4 Quantization and System Co-design for Efficient LLM Serving. In Proceedings of Machine Learning and Systems, M. Zaharia, G. Joshi, and Y. Lin (Eds.), Vol. 7. MLSys. https://proceedings.mlsys.org/paper_files/paper/2025/file/fbe2b2f74a2ece8070d8fb073717bda6-Paper-Conference.pdf [137] Chaoqiang Liu, Haifeng Liu, Dan Chen, Yu Huang, Yi Zhang, Wenjing Xiao, Xiaofei Liao, and Hai Jin. 2025. HeterRAG: Heterogeneous Processing-in-Memory Acceleration for Retrieval-augmented Generation. In Proceedings of the 52nd Annual International Symposium on Computer Architecture (ISCA ’25). Association for Computing Machinery, New York, NY, USA, 884–898. doi:10.1145/3695053.3731089 [138] Jinshu Liu, Hamid Hadian, Hanchen Xu, and Huaicheng Li. 2025. Tiered memory management beyond hotness. In Proceedings of the 19th USENIX Conference on Operating Systems Design and Implementation (Boston, MA, USA) (OSDI ’25). USENIX Association, USA, Article 40, 17 pages. [139] Jun Liu, Peilin Liu, Ruicheng Zhang, Zhang Senlei, Yanbo Chen, Ziao Wang, Jinyun Yang, mingqi wang, Shuhao Zhang, Xiaofei Liao, and Hai Jin. 2026. SAGE: A Dataflow-Native Framework for Modular, Controllable, and Transparent LLM-Augmented Reasoning. In Forty-third International

46

Zhang et al.

Conference on Machine Learning. https://openreview.net/forum?id=TXcFJdT7at [140] Lian Liu, Shixin Zhao, Bing Li, Haimeng Ren, Zhaohui Xu, Mengdi Wang, Xiaowei Li, Yinhe Han, and Ying Wang. 2025. Make LLM Inference Affordable to Everyone: Augmenting GPU Memory with NDP-DIMM. In 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA). 1751–1765. doi:10.1109/HPCA61900.2025.00129 [141] Qianli Liu, Zicong Hong, Peng Li, Fahao Chen, and Song Guo. 2025. Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Management. In IEEE INFOCOM 2025 - IEEE Conference on Computer Communications. 1–10. doi:10.1109/INFOCOM55648.2025.11044533 [142] Weijian Liu, Mingzhen Li, Guangming Tan, and Weile Jia. 2025. Mario: Near Zero-cost Activation Checkpointing in Pipeline Parallelism. In Proceedings of the 30th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming (Las Vegas, NV, USA) (PPoPP ’25). Association for Computing Machinery, New York, NY, USA, 197–211. doi:10.1145/3710848.3710878 [143] Yang Liu, Yunfei Gu, Liqiang Zhang, Chentao Wu, Guangtao Xue, Jie Li, Minyi Guo, Junhao Hu, and Jie Meng. 2026. CacheSlide: Unlocking Cross Position-Aware KV Cache Reuse for Accelerating LLM Serving. In 24th USENIX Conference on File and Storage Technologies (FAST 26). USENIX Association, Santa Clara, CA, 83–99. https://www.usenix.org/conference/fast26/presentation/liu-yang [144] Yuhan Liu, Yuyang Huang, Jiayi Yao, Shaoting Feng, Zhuohan Gu, Kuntai Du, Hanchen Li, Yihua Cheng, Junchen Jiang, Shan Lu, Madan Musuvathi, and Esha Choukse. 2026. DroidSpeak: KV Cache Sharing Across Fine-tuned Model Variants. In 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26). USENIX Association, Renton, WA, 319–338. https://www.usenix.org/conference/nsdi26/presentation/liu-yuhan [145] Zihan Liu, Xinhao Luo, Junxian Guo, Wentao Ni, Yangjie Zhou, Yue Guan, Cong Guo, Weihao Cui, Yu Feng, Minyi Guo, Yuhao Zhu, Minjia Zhang, Chen Jin, and Jingwen Leng. 2025. VQ-LLM: High-performance Code Generation for Vector Quantization Augmented LLM Inference. In 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA). 1496–1509. doi:10.1109/HPCA61900.2025.00112 [146] David Lopez-Paz and Marc’Aurelio Ranzato. 2017. Gradient episodic memory for continual learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems (Long Beach, California, USA) (NIPS’17). Curran Associates Inc., Red Hook, NY, USA, 6470–6479. [147] Chiheng Lou, Sheng Qi, Chao Jin, Dapeng Nie, Haoran Yang, Yu Ding, Xuanzhe Liu, and Xin Jin. 2026. HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds. In 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26). USENIX Association, Renton, WA, 415–430. https://www.usenix.org/conference/nsdi26/presentation/lou [148] Chen Luo and Michael J. Carey. 2019. LSM-based storage techniques: a survey. The VLDB Journal 29, 1 (2019), 393–418. doi:10.1007/s00778-01900555-y [149] Ashraf Mahgoub, Karthick Shankar, Subrata Mitra, Ana Klimovic, Somali Chaterji, and Saurabh Bagchi. 2021. SONIC: Application-aware Data Passing for Chained Serverless Applications. In 2021 USENIX Annual Technical Conference (USENIX ATC 21). USENIX Association, 285–301. https://www.usenix.org/conference/atc21/presentation/mahgoub [150] Ashraf Mahgoub, Edgardo Barsallo Yi, Karthick Shankar, Sameh Elnikety, Somali Chaterji, and Saurabh Bagchi. 2022. ORION and the Three Rights: Sizing, Bundling, and Prewarming for Serverless DAGs. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, Carlsbad, CA, 303–320. https://www.usenix.org/conference/osdi22/presentation/mahgoub [151] Yu A. Malkov and D. A. Yashunin. 2020. Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence 42, 4 (2020), 824–836. doi:10.1109/TPAMI.2018.2889473 [152] Yancan Mao, Shuhao Zhang, and Richard T. B. Ma. 2025. Spacker: Unified State Migration for Distributed Streaming. In 2025 IEEE 45th International Conference on Distributed Computing Systems (ICDCS). 1000–1010. doi:10.1109/ICDCS63083.2025.00101 [153] Yancan Mao, Jianjun Zhao, Shuhao Zhang, Haikun Liu, and Volker Markl. 2023. MorphStream: Adaptive Scheduling for Scalable Transactional Stream Processing on Multicores. Proc. ACM Manag. Data 1, 1, Article 59 (May 2023), 26 pages. doi:10.1145/3588913 [154] Avinash Kumar Maurya, M. Mustafa Rafique, Franck Cappello, and Bogdan Nicolae. 2025. MLP-Offload: Multi-Level, Multi-Path Offloading for LLM Pre-training to Break the GPU Memory Wall. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’25). Association for Computing Machinery, New York, NY, USA, 1381–1394. doi:10.1145/3712285.3759864 [155] Frank McSherry, Derek Gordon Murray, Rebecca Isaacs, and Michael Isard. 2013. Differential dataflow.. In Proceedings of the Sixth Biennial Conference on Innovative Data Systems Research (CIDR). https://www.cidrdb.org/cidr2013/Papers/CIDR13_Paper111.pdf [156] Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. 2024. SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3 (La Jolla, CA, USA) (ASPLOS ’24). Association for Computing Machinery, New York, NY, USA, 932–949. doi:10.1145/3620666.3651335 [157] Matteo Migliavacca, David Eyers, Jean Bacon, Yiannis Papagiannis, Brian Shand, and Peter Pietzuch. 2010. SEEP: scalable and elastic event processing. In Middleware ’10 Posters and Demos Track (Bangalore, India) (Middleware Posters ’10). Association for Computing Machinery, New York, NY, USA, Article 4, 2 pages. doi:10.1145/1930028.1930032 [158] Zizhao Mo, Jianxiong Liao, Huanle Xu, Zhi Zhou, and ChengZhong Xu. 2025. Hetis: Serving LLMs in Heterogeneous GPU Clusters with Fine-grained and Dynamic Parallelism. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’25). Association for Computing Machinery, New York, NY, USA, 1710–1724. doi:10.1145/3712285.3759784 [159] Jayashree Mohan, Amar Phanishayee, and Vijay Chidambaram. 2021. CheckFreq: Frequent, Fine-Grained DNN Checkpointing. In 19th USENIX Conference on File and Storage Technologies (FAST 21). USENIX Association, 203–216. https://www.usenix.org/conference/fast21/presentation/mohan

Appendix

47

[160] Derek G. Murray, Frank McSherry, Rebecca Isaacs, Michael Isard, Paul Barham, and Martín Abadi. 2013. Naiad: a timely dataflow system. In Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles (Farminton, Pennsylvania) (SOSP ’13). Association for Computing Machinery, New York, NY, USA, 439–455. doi:10.1145/2517349.2522738 [161] Derek G. Murray, Malte Schwarzkopf, Christopher Smowton, Steven Smith, Anil Madhavapeddy, and Steven Hand. 2011. CIEL: a universal execution engine for distributed data-flow computing. In Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation (Boston, MA) (NSDI’11). USENIX Association, USA, 113–126. [162] Seonjin Na, Geonhwa Jeong, Byung Hoon Ahn, Aaron Jezghani, Jeffrey Young, Christopher J. Hughes, Tushar Krishna, and Hyesoon Kim. 2025. FlexInfer: Flexible LLM Inference with CPU Computations. In Proceedings of Machine Learning and Systems, M. Zaharia, G. Joshi, and Y. Lin (Eds.), Vol. 7. MLSys. https://proceedings.mlsys.org/paper_files/paper/2025/file/698cfaf72a208aef2e78bcac55b74328-Paper-Conference.pdf [163] Nandeeka Nayak, Xinrui Wu, Toluwanimi O. Odemuyiwa, Michael Pellauer, Joel S. Emer, and Christopher W. Fletcher. 2024. FuseMax: Leveraging Extended Einsums to Optimize Attention Accelerator Design. In 2024 57th IEEE/ACM International Symposium on Microarchitecture (MICRO). 1458–1473. doi:10.1109/MICRO61859.2024.00107 [164] Sean Nian, Jiahao Fang, Qilong Feng, Zhiyu Wu, and Fan Lai. 2026. CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restoration. (2026). doi:10.48550/ARXIV.2604.25080 [165] Hyungjun Oh, Kihong Kim, Jaemin Kim, Sungkyun Kim, Junyeol Lee, Du-seong Chang, and Jiwon Seo. 2024. ExeGPT: Constraint-Aware Resource Scheduling for LLM Inference. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (La Jolla, CA, USA) (ASPLOS ’24). Association for Computing Machinery, New York, NY, USA, 369–384. doi:10.1145/3620665.3640383 [166] Diego Ongaro and John Ousterhout. 2014. In search of an understandable consensus algorithm. In Proceedings of the 2014 USENIX Conference on USENIX Annual Technical Conference (Philadelphia, PA) (USENIX ATC’14). USENIX Association, USA, 305–320. [167] John Ousterhout, Parag Agrawal, David Erickson, Christos Kozyrakis, Jacob Leverich, David Mazières, Subhasish Mitra, Aravind Narayanan, Guru Parulkar, Mendel Rosenblum, Stephen M. Rumble, Eric Stratmann, and Ryan Stutsman. 2010. The case for RAMClouds: scalable high-performance storage entirely in DRAM. SIGOPS Oper. Syst. Rev. 43, 4 (Jan. 2010), 92–105. doi:10.1145/1713254.1713276 [168] Rui Pan, Zhuang Wang, Zhen Jia, Can Karakus, Luca Zancato, Tri Dao, Yida Wang, and Ravi Netravali. 2025. Marconi: Prefix Caching for the Era of Hybrid LLMs. In Proceedings of Machine Learning and Systems, M. Zaharia, G. Joshi, and Y. Lin (Eds.), Vol. 7. MLSys. https://proceedings.mlsys. org/paper_files/paper/2025/file/7c180af017258d239bac6248d1eb26ac-Paper-Conference.pdf [169] Xiurui Pan, Endian Li, Qiao Li, Shengwen Liang, Yizhou Shan, Ke Zhou, Yingwei Luo, Xiaolin Wang, and Jie Zhang. 2025. InstAttention: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference. In 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA). 1510–1525. doi:10.1109/HPCA61900.2025.00113 [170] Xinglin Pan, Wenxiang Lin, Lin Zhang, Shaohuai Shi, Zhenheng Tang, Rui Wang, Bo Li, and Xiaowen Chu. 2025. FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 (Rotterdam, Netherlands) (ASPLOS ’25). Association for Computing Machinery, New York, NY, USA, 524–539. doi:10.1145/3669940.3707272 [171] Zaifeng Pan, Yitong Ding, Yue Guan, Zheng Wang, Zhongkai Yu, Xulong Tang, Yida Wang, and Yufei Ding. 2025. FastTree: Optimizing Attention Kernel and Runtime for Tree-Structured LLM Inference. In Proceedings of Machine Learning and Systems. [172] Junhyeok Park, Sungbin Jang, Osang Kwon, Yongho Lee, and Seokin Hong. 2025. Leveraging Chiplet-Locality for Efficient Memory Mapping in Multi-Chip Module GPUs. In Proceedings of the 58th IEEE/ACM International Symposium on Microarchitecture (MICRO ’25). Association for Computing Machinery, New York, NY, USA, 1040–1057. doi:10.1145/3725843.3756090 [173] Yeonhong Park, Jake Hyun, Hojoon Kim, and Jae W. Lee. 2025. DecDEC: A Systems Approach to Advancing Low-Bit LLM Quantization. In 19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25). USENIX Association, Boston, MA, 803–819. https: //www.usenix.org/conference/osdi25/presentation/park-yeonhong [174] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). 118–132. doi:10.1109/ISCA59077.2024.00019 [175] Daniel Peng and Frank Dabek. 2010. Large-scale Incremental Processing Using Distributed Transactions and Notifications. In 9th USENIX Symposium on Operating Systems Design and Implementation (OSDI 10). USENIX Association, Vancouver, BC. https://www.usenix.org/conference/osdi10/largescale-incremental-processing-using-distributed-transactions-and [176] Li Peng, Yuda An, You Zhou, Chenxi Wang, Qiao Li, Chuanning Cheng, and Jie Zhang. 2024. ScalaCache: scalable user-space page cache management with software-hardware coordination. In Proceedings of the 2024 USENIX Conference on Usenix Annual Technical Conference (Santa Clara, CA, USA) (USENIX ATC’24). USENIX Association, USA, Article 72, 18 pages. [177] Ameya Prabhu, Philip H. S. Torr, and Puneet K. Dokania. 2020. GDumb: A Simple Approach that Questions Our Progress in Continual Learning. In Computer Vision – ECCV 2020, Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer International Publishing, Cham, 524–540. [178] Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ramjee, and Ashish Panwar. 2025. vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 (Rotterdam, Netherlands) (ASPLOS ’25). Association for Computing Machinery, New York, NY, USA,

48

Zhang et al.

1133–1150. doi:10.1145/3669940.3707256 [179] Ruoyu Qin, Weiran He, Yaoyu Wang, Zheming Li, Xinran Xu, Yongwei Wu, Weimin Zheng, and Mingxing Zhang. 2026. Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacenter. (2026). doi:10.48550/ARXIV.2604.15039 [180] Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Heyi Tang, Feng Ren, Teng Ma, Shangming Cai, Yineng Zhang, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. ACM Trans. Storage (Nov. 2025). doi:10.1145/3773772 Just Accepted. [181] Jiansheng Qiu, Fangzhou Yuan, Mingyu Gao, and Huanchen Zhang. 2025. HotRAP: hot record retention and promotion for LSM-trees with tiered storage. In Proceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference (Boston, MA, USA) (USENIX ATC ’25). USENIX Association, USA, Article 30, 15 pages. [182] Derrick Quinn, Mohammad Nouri, Neel Patel, John Salihu, Alireza Salemi, Sukhan Lee, Hamed Zamani, and Mohammad Alian. 2025. Accelerating Retrieval-Augmented Generation. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 (Rotterdam, Netherlands) (ASPLOS ’25). Association for Computing Machinery, New York, NY, USA, 15–32. doi:10.1145/3669940.3707264 [183] Do Le Quoc, Istemi Ekin Akkus, Pramod Bhatotia, Spyros Blanas, Ruichuan Chen, Christof Fetzer, and Thorsten Strufe. 2018. ApproxJoin: Approximate Distributed Joins. In Proceedings of the ACM Symposium on Cloud Computing (Carlsbad, CA, USA) (SoCC ’18). Association for Computing Machinery, New York, NY, USA, 426–438. doi:10.1145/3267809.3267834 [184] Do Le Quoc, Ruichuan Chen, Pramod Bhatotia, Christof Fetzer, Volker Hilt, and Thorsten Strufe. 2017. StreamApprox: approximate computing for stream analytics. In Proceedings of the 18th ACM/IFIP/USENIX Middleware Conference (Las Vegas, Nevada) (Middleware ’17). Association for Computing Machinery, New York, NY, USA, 185–197. doi:10.1145/3135974.3135989 [185] Amanda Raybuck, Tim Stamler, Wei Zhang, Mattan Erez, and Simon Peter. 2021. HeMem: Scalable Tiered Memory Management for Big Data Applications and Real NVM. In Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles (Virtual Event, Germany) (SOSP ’21). Association for Computing Machinery, New York, NY, USA, 392–407. doi:10.1145/3477132.3483550 [186] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. 2017. iCaRL: Incremental Classifier and Representation Learning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 5533–5542. doi:10.1109/cvpr.2017.587 [187] Xiaowei Ren, Daniel Lustig, Evgeny Bolotin, Aamer Jaleel, Oreste Villa, and David Nellans. 2020. HMG: Extending Cache Coherence Protocols Across Modern Hierarchical Multi-GPU Systems. In 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA). 582–595. doi:10.1109/HPCA47549.2020.00054 [188] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. 2019. Experience Replay for Continual Learning. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2019/file/fa7cdfad1a5aaf8370ebeda47a1ff1c3-Paper.pdf [189] Francisco Romero, Qian Li, Neeraja J. Yadwadkar, and Christos Kozyrakis. 2021. INFaaS: Automated Model-less Inference Serving. In 2021 USENIX Annual Technical Conference (USENIX ATC 21). USENIX Association, 397–411. https://www.usenix.org/conference/atc21/presentation/romero [190] Chaoyi Ruan, Yinhe Chen, Dongqi Tian, Yandong Shi, Yongji Wu, Jialin Li, and Cheng Li. 2026. Libra: Flexible Request Partitioning and Scheduling for Serving Unbalanced and Dynamic LLM Workloads. In 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26). USENIX Association, Renton, WA, 1243–1258. https://www.usenix.org/conference/nsdi26/presentation/ruan-libra [191] Rya Sanovar, Srikant Bharadwaj, Renée St. Amant, Victor Rühle, and Saravan Rajmohan. 2025. LeanAttention: Hardware-Aware Scalable Attention Mechanism for the Decode-Phase of Transformers. In Proceedings of Machine Learning and Systems, M. Zaharia, G. Joshi, and Y. Lin (Eds.), Vol. 7. MLSys. https://proceedings.mlsys.org/paper_files/paper/2025/file/16ec6494e9b5a4138de7238761d715b4-Paper-Conference.pdf [192] Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher Manning. 2024. RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval. In International Conference on Learning Representations, B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (Eds.), Vol. 2024. 32628–32649. https://proceedings.iclr.cc/paper_files/paper/2024/file/8a2acd174940dbca361a6398a4f9df91-PaperConference.pdf [193] Fred B. Schneider. 1990. Implementing fault-tolerant services using the state machine approach: a tutorial. ACM Comput. Surv. 22, 4 (Dec. 1990), 299–319. doi:10.1145/98163.98167 [194] Marc Shapiro, Nuno Preguiça, Carlos Baquero, and Marek Zawirski. 2011. Conflict-Free Replicated Data Types. In Stabilization, Safety, and Security of Distributed Systems, Xavier Défago, Franck Petit, and Vincent Villain (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 386–400. [195] Haichen Shen, Lequn Chen, Yuchen Jin, Liangyu Zhao, Bingyu Kong, Matthai Philipose, Arvind Krishnamurthy, and Ravi Sundaram. 2019. Nexus: a GPU cluster engine for accelerating DNN-based video analysis. In Proceedings of the 27th ACM Symposium on Operating Systems Principles (Huntsville, Ontario, Canada) (SOSP ’19). Association for Computing Machinery, New York, NY, USA, 322–337. doi:10.1145/3341301.3359658 [196] Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E. Gonzalez, and Ion Stoica. 2023. S-LoRA: Serving Thousands of Concurrent LoRA Adapters. (2023). doi:10.48550/ARXIV.2311.03285 [197] Ying Sheng, Shiyi Cao, Dacheng Li, Banghua Zhu, Zhuohan Li, Danyang Zhuo, Joseph E. Gonzalez, and Ion Stoica. 2024. Fairness in serving large language models. In Proceedings of the 18th USENIX Conference on Operating Systems Design and Implementation (Santa Clara, CA, USA) (OSDI’24). USENIX Association, USA, Article 52, 24 pages. [198] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. FlexGen: high-throughput generative inference of large language models with a single GPU. In Proceedings of the 40th International Conference on

Appendix

49

Machine Learning (Honolulu, Hawaii, USA) (ICML’23). JMLR.org, Article 1288, 23 pages. [199] Aditi Singh, Suhas Jayaram Subramanya, Ravishankar Krishnaswamy, and Harsha Vardhan Simhadri. 2021. FreshDiskANN: A Fast and Accurate Graph-Based ANN Index for Streaming Similarity Search. arXiv preprint arXiv:2105.09613 (2021). doi:10.48550/arXiv.2105.09613 [200] Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. 2024. PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles (Austin, TX, USA) (SOSP ’24). Association for Computing Machinery, New York, NY, USA, 590–606. doi:10.1145/3694715.3695964 [201] Qidong Su, Wei Zhao, Xin Li, Muralidhar Andoorveedu, Chenhao Jiang, Zhanda Zhu, Kevin Song, Christina Giannoula, and Gennady Pekhimenko. 2025. Seesaw: High-throughput LLM Inference via Model Re-sharding. In Proceedings of Machine Learning and Systems, M. Zaharia, G. Joshi, and Y. Lin (Eds.), Vol. 7. MLSys. https://proceedings.mlsys.org/paper_files/paper/2025/file/cbc4ab80cd77aa0eb87da062fbcddb46-Paper-Conference.pdf [202] Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. 2024. Llumnix: dynamic scheduling for large language model serving. In Proceedings of the 18th USENIX Conference on Operating Systems Design and Implementation (Santa Clara, CA, USA) (OSDI’24). USENIX Association, USA, Article 10, 19 pages. [203] Zhenbo Sun, Shengqi Chen, Yuanwei Wang, Jian Sha, Guanyu Feng, and Wenguang Chen. 2025. MEPipe: Democratizing LLM Training with Memory-Efficient Slice-Level Pipeline Scheduling on Cost-Effective Accelerators. In Proceedings of the Twentieth European Conference on Computer Systems (Rotterdam, Netherlands) (EuroSys ’25). Association for Computing Machinery, New York, NY, USA, 1263–1278. doi:10.1145/3689031.3717469 [204] Bijan Tabatabai, James Sorenson, and Michael M. Swift. 2024. FBMM: making memory management extensible with filesystems. In Proceedings of the 2024 USENIX Conference on Usenix Annual Technical Conference (Santa Clara, CA, USA) (USENIX ATC’24). USENIX Association, USA, Article 48, 14 pages. [205] Shanjiang Tang, BingSheng He, Shuhao Zhang, and Zhaojie Niu. 2016. Elastic Multi-resource Fairness: Balancing Fairness and Efficiency in Coupled CPU-GPU Architectures. In SC ’16: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. 875–886. doi:10.1109/SC.2016.74 [206] Xilin Tang, Feng Zhang, Shuhao Zhang, Yani Liu, Bingsheng He, and Xiaoyong Du. 2024. Enabling Adaptive Sampling for Intra-Window Join: Simultaneously Optimizing Quantity and Quality. Proc. ACM Manag. Data 2, 4, Article 198 (Sept. 2024), 31 pages. doi:10.1145/3677134 [207] Alexander Thomson, Thaddeus Diamond, Shu-Chun Weng, Kun Ren, Philip Shao, and Daniel J. Abadi. 2012. Calvin: fast distributed transactions for partitioned database systems. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data (Scottsdale, Arizona, USA) (SIGMOD ’12). Association for Computing Machinery, New York, NY, USA, 1–12. doi:10.1145/2213836.2213838 [208] Chunlin Tian, Xinpeng Qin, Kahou Tam, Li Li, Zijian Wang, Yuanzhe Zhao, Minglei Zhang, and Chengzhong Xu. 2025. CLONE: customizing LLMs for efficient latency-aware inference at the edge. In Proceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference (Boston, MA, USA) (USENIX ATC ’25). USENIX Association, USA, Article 34, 23 pages. [209] Abhishek Vijaya Kumar, Gianni Antichi, and Rachee Singh. 2025. Aqua: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (Rotterdam, Netherlands) (ASPLOS ’25). Association for Computing Machinery, New York, NY, USA, 48–62. doi:10.1145/3676641.3715983 [210] Midhul Vuppalapati and Rachit Agarwal. 2024. Tiered Memory Management: Access Latency is the Key!. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles (Austin, TX, USA) (SOSP ’24). Association for Computing Machinery, New York, NY, USA, 79–94. doi:10.1145/3694715.3695968 [211] Midhul Vuppalapati, Justin Miron, Rachit Agarwal, Dan Truong, Ashish Motivala, and Thierry Cruanes. 2020. Building an elastic query engine on disaggregated storage. In Proceedings of the 17th Usenix Conference on Networked Systems Design and Implementation (Santa Clara, CA, USA) (NSDI’20). USENIX Association, USA, 449–462. [212] Borui Wan, Mingji Han, Yiyao Sheng, Yanghua Peng, Haibin Lin, Mofan Zhang, Zhichao Lai, Menghan Yu, Junda Zhang, Zuquan Song, Xin Liu, and Chuan Wu. 2025. ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Development. In 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25). USENIX Association, Philadelphia, PA, 559–578. https://www.usenix.org/conference/ nsdi25/presentation/wan-borui [213] Jiahao Wang, Jinbo Han, Xingda Wei, Sijie Shen, Dingyan Zhang, Chenguang Fang, Rong Chen, Wenyuan Yu, and Haibo Chen. 2025. KVCache cache in the wild: characterizing and optimizing KVCache cache at a large cloud provider. In Proceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference (Boston, MA, USA) (USENIX ATC ’25). USENIX Association, USA, Article 28, 18 pages. [214] Jiali Wang, Yankui Wang, Mingcong Han, and Rong Chen. 2025. Colocating ML Inference and Training with Fast GPU Memory Handover. In 2025 USENIX Annual Technical Conference (USENIX ATC 25). USENIX Association, Boston, MA, 1657–1675. https://www.usenix.org/conference/atc25/ presentation/wang-jiali [215] Jianguo Wang, Xiaomeng Yi, Rentong Guo, Hai Jin, Peng Xu, Shengjun Li, Xiangyu Wang, Xiangzhou Guo, Chengming Li, Xiaohai Xu, Kun Yu, Yuxing Yuan, Yinghao Zou, Jiquan Long, Yudong Cai, Zhenxiang Li, Zhifeng Zhang, Yihua Mo, Jun Gu, Ruiyi Jiang, Yi Wei, and Charles Xie. 2021. Milvus: A Purpose-Built Vector Data Management System. In Proceedings of the 2021 International Conference on Management of Data (Virtual Event, China) (SIGMOD ’21). Association for Computing Machinery, New York, NY, USA, 2614–2627. doi:10.1145/3448016.3457550 [216] Linnan Wang, Jinmian Ye, Yiyang Zhao, Wei Wu, Ang Li, Shuaiwen Leon Song, Zenglin Xu, and Tim Kraska. 2018. Superneurons: dynamic GPU memory management for training deep neural networks. In Proceedings of the 23rd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (Vienna, Austria) (PPoPP ’18). Association for Computing Machinery, New York, NY, USA, 41–53. doi:10.1145/3178487.3178491

50

Zhang et al.

[217] Mingqi Wang, Junyao Dong, Zhuoyan Wu, Jun Liu, Ruicheng Zhang, Jianjun Zhao, Ruipeng Wan, Xinyan Lei, Shuhao Zhang, Bolong Zheng, Haikun Liu, Xiaofei Liao, and Hai Jin. 2026. CANDOR-Bench: Benchmarking In-Memory Continuous ANNS under Dynamic Open-World Streams [Experiments & Analysis]. Proc. ACM Manag. Data 4, 1, Article 16 (April 2026), 27 pages. doi:10.1145/3786630 [218] Xiaoyang Wang, Yongkun Li, Kan Wu, Wenzhe Zhu, Yuqi Li, and Yinlong Xu. 2025. FineMem: breaking the allocation overhead vs. memory waste dilemma in fine-grained disaggregated memory management. In Proceedings of the 19th USENIX Conference on Operating Systems Design and Implementation (Boston, MA, USA) (OSDI ’25). USENIX Association, USA, Article 4, 18 pages. [219] Xin Wang, Zhengru Wang, Zhenyu Wu, Shuhao Zhang, Xuanhua Shi, and Li Lu. 2023. Data Stream Clustering: An In-depth Empirical Study. Proc. ACM Manag. Data 1, 2, Article 162 (June 2023), 26 pages. doi:10.1145/3589307 [220] Zheng Wang, Anna Cai, Xinfeng Xie, Zaifeng Pan, Yue Guan, Weiwei Chu, Jie Wang, Shikai Li, Jianyu Huang, Chris Cai, Yuchen Hao, and Yufei Ding. 2025. WLB-LLM: workload-balanced 4D parallelism for large language model training. In Proceedings of the 19th USENIX Conference on Operating Systems Design and Implementation (Boston, MA, USA) (OSDI ’25). USENIX Association, USA, Article 43, 17 pages. [221] Zhengru Wang, Xin Wang, and Shuhao Zhang. 2024. MOStream: A Modular and Self-Optimizing Data Stream Clustering Algorithm. In 2024 IEEE International Conference on Data Mining (ICDM). 500–509. doi:10.1109/ICDM59182.2024.00057 [222] Xingda Wei, Zhuobin Huang, Tianle Sun, Yingyi Hao, Rong Chen, Mingcong Han, Jinyu Gu, and Haibo Chen. 2025. PhoenixOS: Concurrent OS-level GPU Checkpoint and Restore with Validated Speculation. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles (Lotte Hotel World, Seoul, Republic of Korea) (SOSP ’25). Association for Computing Machinery, New York, NY, USA, 996–1013. doi:10.1145/3731569.3764813 [223] Xingda Wei, Jiaxin Shi, Yanzhe Chen, Rong Chen, and Haibo Chen. 2015. Fast in-memory transaction processing using RDMA and HTM. In Proceedings of the 25th Symposium on Operating Systems Principles (Monterey, California) (SOSP ’15). Association for Computing Machinery, New York, NY, USA, 87–104. doi:10.1145/2815400.2815419 [224] Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. 2024. LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles (Austin, TX, USA) (SOSP ’24). Association for Computing Machinery, New York, NY, USA, 640–654. doi:10.1145/3694715.3695948 [225] Bingyang Wu, Yinmin Zhong, Zili Zhang, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xuanzhe Liu, and Xin Jin. 2026. FastServe: Iteration-Level Preemptive Scheduling for Large Language Model Inference. In 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26). USENIX Association, Renton, WA, 57–74. https://www.usenix.org/conference/nsdi26/presentation/wu-bingyang [226] Bingyang Wu, Ruidong Zhu, Zili Zhang, Peng Sun, Xuanzhe Liu, and Xin Jin. 2024. dLoRA: dynamically orchestrating requests and adapters for LoRA LLM serving. In Proceedings of the 18th USENIX Conference on Operating Systems Design and Implementation (Santa Clara, CA, USA) (OSDI’24). USENIX Association, USA, Article 49, 17 pages. [227] Yuhao Wu, Karthick Sharma, Chun Seah, and Shuhao Zhang. 2023. SentiStream: A Co-Training Framework for Adaptive Online Sentiment Analysis in Evolving Data Streams. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 6198–6212. doi:10.18653/v1/2023.emnlp-main.380 [228] Haojun Xia, Zhen Zheng, Xiaoxia Wu, Shiyang Chen, Zhewei Yao, Stephen Youn, Arash Bakhtiari, Michael Wyatt, Donglin Zhuang, Zhongzhu Zhou, Olatunji Ruwase, Yuxiong He, and Shuaiwen Leon Song. 2024. Quant-LLM: accelerating the serving of large language models via FP6-centric algorithm-system co-design on modern GPUs. In Proceedings of the 2024 USENIX Conference on Usenix Annual Technical Conference (Santa Clara, CA, USA) (USENIX ATC’24). USENIX Association, USA, Article 43, 15 pages. [229] Yuxing Xiang, Xue Li, Kun Qian, Yufan Yang, Diwen Zhu, Wenyuan Yu, Ennan Zhai, Xuanzhe Liu, Xin Jin, and Jingren Zhou. 2025. Aegaeon: Effective GPU Pooling for Concurrent LLM Serving on the Market. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles (Lotte Hotel World, Seoul, Republic of Korea) (SOSP ’25). Association for Computing Machinery, New York, NY, USA, 1030–1045. doi:10.1145/3731569.3764815 [230] Yi Xiong, Hao Wu, Changxu Shao, Ziqing Wang, Rui Zhang, Yuhong Guo, Junping Zhao, Ke Zhang, and Zhenxuan Pan. 2024. LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Management. (2024). doi:10.48550/ARXIV.2410.00428 [231] Haike Xu, Magdalen Dobson Manohar, Philip A. Bernstein, Badrish Chandramouli, Richard Wen, and Harsha Vardhan Simhadri. 2025. In-Place Updates of a Graph Index for Streaming Approximate Nearest Neighbor Search. (2025). doi:10.48550/ARXIV.2502.13826 [232] Yuming Xu, Hengyu Liang, Jin Li, Shuotao Xu, Qi Chen, Qianxi Zhang, Cheng Li, Ziyue Yang, Fan Yang, Yuqing Yang, Peng Cheng, and Mao Yang. 2023. SPFresh: Incremental In-Place Update for Billion-Scale Vector Search. In Proceedings of the 29th Symposium on Operating Systems Principles (Koblenz, Germany) (SOSP ’23). Association for Computing Machinery, New York, NY, USA, 545–561. doi:10.1145/3600006.3613166 [233] Yanchao Xu, Dongxiang Zhang, Shuhao Zhang, Sai Wu, Zexu Feng, and Gang Chen. 2024. Predictive and Near-Optimal Sampling for View Materialization in Video Databases. Proc. ACM Manag. Data 2, 1, Article 19 (March 2024), 27 pages. doi:10.1145/3639274 [234] Shang Yang, Junxian Guo, Haotian Tang, Qinghao Hu, Guangxuan Xiao, Jiaming Tang, Yujun Lin, Zhijian Liu, Yao Lu, and Song Han. 2025. LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention. In Proceedings of Machine Learning and Systems, M. Zaharia, G. Joshi, and Y. Lin (Eds.), Vol. 7. MLSys. https://proceedings.mlsys.org/paper_files/paper/2025/file/cc8c6b9d89f7a898a29f58869b238e46-Paper-Conference.pdf [235] Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. 2025. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion. In Proceedings of the Twentieth European Conference on Computer Systems (Rotterdam, Netherlands) (EuroSys ’25). Association for Computing Machinery, New York, NY, USA, 94–109. doi:10.1145/3689031.3696098

Appendix

51

[236] Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, and Luis Ceze. 2025. FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving. In Proceedings of Machine Learning and Systems, M. Zaharia, G. Joshi, and Y. Lin (Eds.), Vol. 7. MLSys. https://proceedings.mlsys.org/paper_files/paper/2025/file/ dbf02b21d77409a2db30e56866a8ab3a-Paper-Conference.pdf [237] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for TransformerBased Generative Models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, Carlsbad, CA, 521–538. https://www.usenix.org/conference/osdi22/presentation/yu [238] Lingfan Yu, Jinkun Lin, and Jinyang Li. 2025. Stateful Large Language Model Serving with Pensieve. In Proceedings of the Twentieth European Conference on Computer Systems (Rotterdam, Netherlands) (EuroSys ’25). Association for Computing Machinery, New York, NY, USA, 144–158. doi:10.1145/3689031.3696086 [239] Shan Yu, Yifan Qiao, Mingyuan Ma, Yangmin Li, Shuo Yang, Xinyuan Tong, Yang Wang, Zhiqiang Xie, Yuwei An, Shiyi Cao, Ke Bao, Deepak Vij, Xiaoning Ding, Yichen Wang, Qingda Lu, Zhong Wang, Gao Gao, Harry Xu, Junyi Shu, Jiarong Xing, and Ying Sheng. 2025. Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooning. (2025). doi:10.48550/ARXIV.2505.04021 [240] Xiangyao Yu, Andrew Pavlo, Daniel Sanchez, and Srinivas Devadas. 2016. TicToc: Time Traveling Optimistic Concurrency Control. In Proceedings of the 2016 International Conference on Management of Data (San Francisco, California, USA) (SIGMOD ’16). Association for Computing Machinery, New York, NY, USA, 1629–1642. doi:10.1145/2882903.2882935 [241] Yifan Yu, Yu Gan, Nikhil Sarda, Lillian Tsai, Jiaming Shen, Yanqi Zhou, Arvind Krishnamurthy, Fan Lai, Hank Levy, and David Culler. 2025. IC-Cache: Efficient Large Language Model Serving via In-context Caching. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles (Lotte Hotel World, Seoul, Republic of Korea) (SOSP ’25). Association for Computing Machinery, New York, NY, USA, 375–398. doi:10.1145/3731569.3764829 [242] Ziyang Yue, Bolong Zheng, Ling Xu, Kanru Xu, Shuhao Zhang, Yajuan Du, Yunjun Gao, Xiaofang Zhou, and Christian S. Jensen. 2025. Select Edges Wisely: Monotonic Path Aware Graph Layout Optimization for Disk-Based ANN Search. Proc. VLDB Endow. 18, 11 (July 2025), 4337–4349. doi:10.14778/3749646.3749697 [243] Sungmin Yun, Kwanhee Kyung, Juhwan Cho, Jaewan Choi, Jongmin Kim, Byeongho Kim, Sukhan Lee, Kyomin Sohn, and Jung Ho Ahn. 2024. Duplex: A Device for Large Language Models with Mixture of Experts, Grouped Query Attention, and Continuous Batching. In Proceedings of the 2024 57th IEEE/ACM International Symposium on Microarchitecture (Austin, TX, USA) (MICRO ’24). IEEE Press, 1429–1443. doi:10.1109/MICRO61859.2024.00105 [244] Matei Zaharia, Tathagata Das, Haoyuan Li, Timothy Hunter, Scott Shenker, and Ion Stoica. 2013. Discretized streams: fault-tolerant streaming computation at scale. In Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles (Farminton, Pennsylvania) (SOSP ’13). Association for Computing Machinery, New York, NY, USA, 423–438. doi:10.1145/2517349.2522737 [245] Erfan Zamanian, Carsten Binnig, Tim Harris, and Tim Kraska. 2017. The end of a myth: distributed transactions can scale. Proc. VLDB Endow. 10, 6 (Feb. 2017), 685–696. doi:10.14778/3055330.3055335 [246] Xianzhi Zeng, Wenchao Jiang, and Shuhao Zhang. 2024. LibAMM: Empirical Insights into Approximate Computing for Accelerating Matrix Multiplication. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37. Curran Associates, Inc., 60517–60530. doi:10.52202/079017-1935 [247] Xianzhi Zeng and Shuhao Zhang. 2023. Parallelizing Stream Compression for IoT Applications on Asymmetric Multicores. In 2023 IEEE 39th International Conference on Data Engineering (ICDE). 950–964. doi:10.1109/ICDE55515.2023.00078 [248] Xianzhi Zeng and Shuhao Zhang. 2024. CStream: Parallel Data Stream Compression on Multicore Edge Devices. IEEE Transactions on Knowledge and Data Engineering 36, 11 (2024), 5889–5904. doi:10.1109/TKDE.2024.3386862 [249] Xianzhi Zeng, Shuhao Zhang, Hongbin Zhong, Hao Zhang, Mian Lu, Zhao Zheng, and Yuqiang Chen. 2024. PECJ: Stream Window Join on Disorder Data Streams with Proactive Error Compensation. Proc. ACM Manag. Data 2, 1, Article 13 (March 2024), 24 pages. doi:10.1145/3639268 [250] Shichen Zhan, Li Li, and Chengzhong Xu. 2025. AssyLLM: efficient federated fine-tuning of LLMs via assembling pre-trained blocks. In Proceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference (Boston, MA, USA) (USENIX ATC ’25). USENIX Association, USA, Article 99, 15 pages. [251] Chen Zhang, Kuntai Du, Shu Liu, Woosuk Kwon, Xiangxi Mo, Yufeng Wang, Xiaoxuan Liu, Kaichao You, Zhuohan Li, Mingsheng Long, Jidong Zhai, Joseph Gonzalez, and Ion Stoica. 2025. Jenga: Effective Memory Management for Serving LLM with Heterogeneity. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles (Lotte Hotel World, Seoul, Republic of Korea) (SOSP ’25). Association for Computing Machinery, New York, NY, USA, 446–461. doi:10.1145/3731569.3764823 [252] Feng Zhang, Lin Yang, Shuhao Zhang, Bingsheng He, Wei Lu, and Xiaoyong Du. 2020. FineStream: fine-grained window-based stream processing on CPU-GPU integrated architectures. In Proceedings of the 2020 USENIX Conference on Usenix Annual Technical Conference (USENIX ATC’20). USENIX Association, USA, Article 43, 15 pages. [253] Feng Zhang, Jidong Zhai, Bingsheng He, Shuhao Zhang, and Wenguang Chen. 2017. Understanding Co-Running Behaviors on Integrated CPU/GPU Architectures. IEEE Transactions on Parallel and Distributed Systems 28, 3 (2017), 905–918. doi:10.1109/TPDS.2016.2586074 [254] Hong Zhang, Yupeng Tang, Anurag Khandelwal, Jingrong Chen, and Ion Stoica. 2021. Caerus: NIMBLE Task Scheduling for Serverless Analytics. In 18th USENIX Symposium on Networked Systems Design and Implementation (NSDI 21). USENIX Association, 653–669. https://www.usenix.org/ conference/nsdi21/presentation/zhang-hong

52

Zhang et al.

[255] Hao Zhang, Xianzhi Zeng, Shuhao Zhang, Xinyi Liu, Mian Lu, and Zhao Zheng. 2023. Scalable Online Interval Join on Modern Multicore Processors in OpenMLDB. In 2023 IEEE 39th International Conference on Data Engineering (ICDE). 3031–3042. doi:10.1109/ICDE55515.2023.00232 [256] Haoyang Zhang, Yirui Zhou, Yuqi Xue, Yiqi Liu, and Jian Huang. 2023. G10: Enabling An Efficient Unified GPU Memory and Storage Architecture with Smart Tensor Migrations. In Proceedings of the 56th Annual IEEE/ACM International Symposium on Microarchitecture (Toronto, ON, Canada) (MICRO ’23). Association for Computing Machinery, New York, NY, USA, 395–410. doi:10.1145/3613424.3614309 [257] Qihao Zhang, Mingshu Zhai, Rui Sun, and Jidong Zhai. 2025. QFactory: accelerating quantized large language model serving with Qtile graphs. In Proceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference (Boston, MA, USA) (USENIX ATC ’25). USENIX Association, USA, Article 38, 16 pages. [258] Ruicheng Zhang, Xinyi Li, Tianyi Xu, Shuhao Zhang, Xiaofei Liao, and Hai Jin. 2026. Neuromem: A Granular Decomposition of the Streaming Lifecycle in External Memory for LLMs. In Forty-third International Conference on Machine Learning. https://openreview.net/forum?id= mO7DgwFFVe [259] Shuhao Zhang, Bingsheng He, Daniel Dahlmeier, Amelie Chi Zhou, and Thomas Heinze. 2017. Revisiting the Design of Data Stream Processing Systems on Multi-Core Processors. In 2017 IEEE 33rd International Conference on Data Engineering (ICDE). 659–670. doi:10.1109/ICDE.2017.119 [260] Shuhao Zhang, Jiong He, Amelie Chi Zhou, and Bingsheng He. 2019. BriskStream: Scaling Data Stream Processing on Shared-Memory Multicore Architectures. In Proceedings of the 2019 International Conference on Management of Data (Amsterdam, Netherlands) (SIGMOD ’19). Association for Computing Machinery, New York, NY, USA, 705–722. doi:10.1145/3299869.3300067 [261] Shuhao Zhang, Yancan Mao, Jiong He, Philipp M. Grulich, Steffen Zeuch, Bingsheng He, Richard T. B. Ma, and Volker Markl. 2021. Parallelizing Intra-Window Join on Multicores: An Experimental Study. In Proceedings of the 2021 International Conference on Management of Data (Virtual Event, China) (SIGMOD ’21). Association for Computing Machinery, New York, NY, USA, 2089–2101. doi:10.1145/3448016.3452793 [262] Senlei Zhang, Tongjun Shi, Dandan Song, Luan Zhang, Shuhao Zhang, Xiaofei Liao, and Hai Jin. 2026. FlowRAG: Continual Learning for Dynamic Retriever in Retrieval-Augmented Generation. In Proceedings of the ACM Web Conference 2026 (United Arab Emirates) (WWW ’26). Association for Computing Machinery, New York, NY, USA, 2160–2170. doi:10.1145/3774904.3792361 [263] Shuhao Zhang, Juan Soto, and Volker Markl. 2023. A survey on transactional stream processing. The VLDB Journal 33, 2 (2023), 451–479. doi:10.1007/s00778-023-00814-z [264] Shuhao Zhang, Hoang Tam Vo, Daniel Dahlmeier, and Bingsheng He. 2017. Multi-Query Optimization for Complex Event Processing in SAP ESP. In 2017 IEEE 33rd International Conference on Data Engineering (ICDE). 1213–1224. doi:10.1109/ICDE.2017.166 [265] Shuhao Zhang, Yingjun Wu, Feng Zhang, and Bingsheng He. 2020. Towards Concurrent Stateful Stream Processing on Multicore Processors. In 2020 IEEE 36th International Conference on Data Engineering (ICDE). 1537–1548. doi:10.1109/ICDE48307.2020.00136 [266] Shuhao Zhang, Feng Zhang, Yingjun Wu, Bingsheng He, and Paul Johns. 2020. Hardware-Conscious Stream Processing: A Survey. SIGMOD Rec. 48, 4 (Feb. 2020), 18–29. doi:10.1145/3385658.3385662 [267] Wei Zhang, Zhiyu Wu, Yi Mu, Rui Ning, Banruo Liu, Nikhil Sarda, Myungjin Lee, and Fan Lai. 2026. JITServe: SLO-aware LLM Serving with Imprecise Request Information. In 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26). USENIX Association, Renton, WA, 825–848. https://www.usenix.org/conference/nsdi26/presentation/zhang-wei [268] Yu Zhang, Feng Zhang, Hourun Li, Shuhao Zhang, and Xiaoyong Du. 2023. CompressStreamDB: Fine-Grained Adaptive Stream Processing without Decompression. In 2023 IEEE 39th International Conference on Data Engineering (ICDE). 408–422. doi:10.1109/ICDE55515.2023.00038 [269] Yu Zhang, Feng Zhang, Hourun Li, Shuhao Zhang, Xiaoguang Guo, Yuxing Chen, Anqun Pan, and Xiaoyong Du. 2024. Data-Aware Adaptive Compression for Stream Processing. IEEE Transactions on Knowledge and Data Engineering 36, 9 (2024), 4531–4549. doi:10.1109/TKDE.2024.3377710 [270] Zili Zhang, Fangyue Liu, Gang Huang, Xuanzhe Liu, and Xin Jin. 2024. Fast vector query processing for large datasets beyond GPU memory with reordered pipelining. In Proceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation (Santa Clara, CA, USA) (NSDI’24). USENIX Association, USA, Article 2, 18 pages. [271] Zhenyu Zhang, Shiwei Liu, Runjin Chen, Bhavya Kailkhura, Beidi Chen, and Zhangyang Wang. 2024. Q-Hitter: A Better Token Oracle for Efficient LLM Inference via Sparse-Quantized KV Cache. In Proceedings of Machine Learning and Systems, P. Gibbons, G. Pekhimenko, and C. De Sa (Eds.), Vol. 6. 381–394. https://proceedings.mlsys.org/paper_files/paper/2024/file/bbb7506579431a85861a05fff048d3e1-Paper-Conference.pdf [272] Jianjun Zhao, Haikun Liu, Shuhao Zhang, Zhuohui Duan, Xiaofei Liao, Hai Jin, and Yu Zhang. 2024. Fast Parallel Recovery for Transactional Stream Processing on Multicores. In 2024 IEEE 40th International Conference on Data Engineering (ICDE). 1478–1491. doi:10.1109/ICDE60146.2024.00122 [273] Jianjun Zhao, Haikun Liu, Shuhao Zhang, Haodi Lu, Yancan Mao, Zhuohui Duan, Xiaofei Liao, and Hai Jin. 2025. Towards high-performance transactional stateful serverless workflows with affinity-aware leasing. In Proceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference (Boston, MA, USA) (USENIX ATC ’25). USENIX Association, USA, Article 91, 17 pages. [274] Jianjun Zhao, Yancan Mao, Zhonghao Yang, Haikun Liu, and Shuhao Zhang. 2025. Scalable Transactional Stream Processing on Multicore Processors. IEEE Transactions on Knowledge and Data Engineering 37, 7 (2025), 4254–4269. doi:10.1109/TKDE.2025.3556741 [275] Kaiyang Zhao, Neha Gholkar, Hasan Maruf, Abhishek Dhanotia, Johannes Weiner, Gregory Price, Ning Sun, Bhavya Dwivedi, Stuart Clark, and Dimitrios Skarlatos. 2026. Equilibria: Fair Multi-Tenant CXL Memory Tiering At Scale. arXiv preprint arXiv:2602.08800 (2026). doi:10.48550/arXiv. 2602.08800 [276] Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. 2024. Atom: Low-Bit Quantization for Efficient and Accurate LLM Serving. In Proceedings of Machine Learning and Systems, P. Gibbons, G. Pekhimenko, and C. De Sa (Eds.), Vol. 6. 196–209. https://proceedings.mlsys.org/paper_files/paper/2024/file/5edb57c05c81d04beb716ef1d542fe9e-Paper-

Appendix

53

Conference.pdf [277] Youpeng Zhao and Jun Wang. 2024. ALISE: Accelerating Large Language Model Serving with Speculative Scheduling. In 2024 ACM/IEEE International Conference On Computer Aided Design (ICCAD). 1–9. doi:10.1145/3676536.3676659 [278] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Structured Language Model Programs. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37. Curran Associates, Inc., 62557–62583. doi:10.52202/079017-2000 [279] Size Zheng, Renze Chen, Meng Li, Zihao Ye, Luis Ceze, and Yun Liang. 2024. vMCU: Coordinated Memory Management and Kernel Optimization for DNN Inference on MCUs. In Proceedings of Machine Learning and Systems, P. Gibbons, G. Pekhimenko, and C. De Sa (Eds.), Vol. 6. 452–464. https://proceedings.mlsys.org/paper_files/paper/2024/file/d5a655b8b373737b4f2aea8f78e5e754-Paper-Conference.pdf [280] Wenxin Zheng, Bin Xu, Jinyu Gu, and Haibo Chen. 2025. SAVE: software-implemented fault tolerance for model inference against GPU memory bit flips. In Proceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference (Boston, MA, USA) (USENIX ATC ’25). USENIX Association, USA, Article 94, 20 pages. [281] Yuhong Zhong, Daniel S. Berger, Carl Waldspurger, Ryan Wee, Ishwar Agarwal, Rajat Agarwal, Frank Hady, Karthik Kumar, Mark D. Hill, Mosharaf Chowdhury, and Asaf Cidon. 2024. Managing Memory Tiers with CXL in Virtualized Environments. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 37–56. https://www.usenix.org/conference/osdi24/presentation/zhongyuhong [282] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: disaggregating prefill and decoding for goodput-optimized large language model serving. In Proceedings of the 18th USENIX Conference on Operating Systems Design and Implementation (Santa Clara, CA, USA) (OSDI’24). USENIX Association, USA, Article 11, 18 pages. [283] Yijie Zhong, Minqiang Zhou, Zhirong Shen, and Jiwu Shu. 2024. UniMem: redesigning disaggregated memory within a unified local-remote memory hierarchy. In Proceedings of the 2024 USENIX Conference on Usenix Annual Technical Conference (Santa Clara, CA, USA) (USENIX ATC’24). USENIX Association, USA, Article 29, 15 pages. [284] Yuhao Zhou, Yuxin Tian, Jindi Lv, Mingjia Shi, Yuanxi Li, Qing Ye, Shuhao Zhang, and Jiancheng Lv. 2025. Ferret: An Efficient Online Continual Learning Framework under Varying Memory Constraints. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 4850–4861. doi:10.1109/CVPR52734.2025.00457 [285] Zhe Zhou, Yiqi Chen, Tao Zhang, Yang Wang, Ran Shu, Shuotao Xu, Peng Cheng, Lei Qu, Yongqiang Xiong, Jie Zhang, and Guangyu Sun. 2024. NeoMem: Hardware/Software Co-Design for CXL-Native Memory Tiering. In Proceedings of the 2024 57th IEEE/ACM International Symposium on Microarchitecture (Austin, TX, USA) (MICRO ’24). IEEE Press, 1518–1531. doi:10.1109/MICRO61859.2024.00111 [286] Qianchao Zhu, Jiangfei Duan, Chang Chen, Siran Liu, Xiuhong Li, Guanyu Feng, Xin Lv, Xiao Chuanfu, Dahua Lin, and Chao Yang. 2025. SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attention. In Proceedings of Machine Learning and Systems, M. Zaharia, G. Joshi, and Y. Lin (Eds.), Vol. 7. MLSys. https://proceedings.mlsys.org/paper_files/paper/2025/file/ 2d04d97593c8c33d415337f408ed0e1b-Paper-Conference.pdf [287] Pengfei Zuo, Jiazhao Sun, Liu Yang, Shuangwu Zhang, and Yu Hua. 2021. One-sided RDMA-Conscious Extendible Hashing for Disaggregated Memory. In 2021 USENIX Annual Technical Conference (USENIX ATC 21). USENIX Association, 15–29. https://www.usenix.org/conference/atc21/ presentation/zuo