Understanding KV Residency in Production-Scale Agentic LLM Workloads

arXiv:2610.22863v1 [cs.DC] 16 Oct 2026

Understanding KV Residency in Production-Scale Agentic LLM Workloads Mayank Arya

Agrim Bari

Indian Institute of Science (IISc)

Microsoft M365 Research

Anjaly Parayil

Yogesh Simmhan

Microsoft M365 Research

Indian Institute of Science (IISc)

Renee St. Amant

Victor Rühle

Saravan Rajmohan

Microsoft M365 Research

Microsoft M365 Research

Microsoft M365 Research

Email: mayankarya@iisc.ac.in; agrimbari@microsoft.com; aparayil@microsoft.com; simmhan@iisc.ac.in Abstract Agentic large language model (LLM) applications turn a single user request into a sequence of model and tool calls over a growing context. Retaining the key-value (KV) state computed for this context in GPU memory allows subsequent calls to reuse it rather than recompute it. However, retaining KV state across a serving fleet can require substantial GPU memory. We characterize four production Microsoft M365 workload traces, namely Deep Research, Enterprise Assistant, Coding Agent, and Enterprise Chat, spanning more than 121 million model calls. Agentic applications issue most calls within a session, and each call re-reads the growing context. In Deep Research, the application issues 89% of model calls. Sessions have a median of 13 model calls, while Enterprise Chat, our lower-agency reference, has 333. The median context grows from approximately 10K tokens to 68K tokens by the 30th call. About 80% of context tokens are served from the prefix cache, and a predictable class of models accounts for 67% of call-context KV. This reusable KV state is large and remains idle for most of its lifetime. Since the production models are confidential, we use the public Llama-2-70B KV layout, with grouped-query attention and 16-bit values, to estimate the memory demand consistently across models. Under this assumption, the KV state of a single 99th-percentile Deep Research session exceeds the 80 GB high-bandwidth memory (HBM) capacity of an NVIDIA H100 GPU. Retaining the KV state of all live sessions at their peak, which we term the hold-all-live ceiling, requires an estimated 501.8 TB of GPU HBM. However, model inference occupies only 5.6% of the time for which this state is retained. Enterprise Chat has smaller sessions but approximately 349K concurrent live sessions, resulting in a similar capacity challenge. Trace-driven replay shows that an idle cap of one hour reduces the hold-all-live ceiling by 3.6×, while increasing the fraction of calls that recompute their context from 4.07% to 4.53%. However, eviction alone is insufficient under tighter memory budgets. With a pooled budget equal to one tenth of the busiest hour’s no-eviction demand, least recently used (LRU) eviction causes 63% of Deep Research calls to recompute their context. Furthest-next-use (FNU), a clairvoyant policy based on Belady’s rule, reduces this to 31%. Multi-resolution retention (MRR) instead demotes the LRU session’s KV state to an int4 representation within the same memory budget. MRR reduces the recomputation rate to 13.6%, 2.3× lower than FNU, despite selecting sessions using recency alone.

1

Placement imposes a separate limit on reuse. When the next call to the same model reaches a different deployment within the region, cached-context reuse drops from 89% to 29%. A Deep Research session accesses 4.5 deployments on average. Treating KV state as local to a deployment increases the first-touch floor, the fraction of calls that recompute their context even with unlimited memory, from 5.4% to 20.2%. Additional memory and better eviction cannot reduce this floor. These findings motivate retention mechanisms that change the representation of KV state, together with deployment-aware routing or migration. Since the production traces cannot be shared, we plan to release AgenticTraceSim, a parametric trace generator, and a residency replay framework for reproducible counterfactual studies.

Keywords: Large language model serving, agentic AI, KV cache, prefix caching, KV reuse, KV working set, KV cache residency, GPU memory capacity, fleet provisioning, deployment-aware routing, workload characterization

1

Introduction

Production LLM serving is shifting from individual requests to persistent, tool-interleaved sessions. In conventional chat, a user request opens a turn: the serving system performs a short sequence of model and optional tool calls, and control returns to the user. In an agentic workload, the request that opens a turn initiates a longer internal loop in which an orchestrator (e.g., one following the ReAct or Magentic pattern [58, 11]) repeatedly invokes models and tools, appends their results, and decides when the task is complete [66, 60, 28]. A session is a user’s ongoing interaction with one application: a sequence of one or more turns that share a session identifier, together with the context they accumulate (Section 2.2 defines calls, turns, and sessions precisely). This distinction changes the state that an LLM serving system must manage. Within a turn, a chat application typically returns control to the user after a short orchestration path, whereas an agentic application may re-enter the model many times without human input (Figure 1); a session repeats this loop once per turn. Each model call first prefills its input context, computing the attention keys and values of every context token in a Key-Value (KV) cache [20], and then decodes output tokens one at a time. Keeping reusable KV resident in GPU memory lets a later call skip prefill over a context prefix that an earlier call already processed, so the retained state remains valuable across otherwise idle intervals. Serving must therefore manage the size, lifetime, and placement of a session’s evolving KV working set, in addition to scheduling the prefill and decode phases of individual model calls. Agentic applications are emerging as a substantial component of datacenter LLM workloads, but their long-running control loops create KV residency and placement demands that request-level measurements do not capture. Recent studies (TraceLab [66], the GitHub Copilot study [28], and the ReAct study [60]) show that agents repeatedly invoke models and tools, carry growing contexts, and produce comparatively short outputs. None exposes the serving deployment selected for each call, and only the Copilot study operates at production scale, for a single coding workload and at the granularity of logical model endpoints. At hyperscaler scale, even modest errors in estimating prefix reuse, KV residency lifetime, or placement locality can translate into terabytes of excess GPU memory or extensive context reconstruction. We address these gaps using four production traces from Microsoft M365. Deep Research

2

(Research), an agent that answers a request through a long loop of model and enterprise-search tool calls; Enterprise Assistant (Assist), a single-model productivity assistant; and Coding Agent (Code). The fourth, Enterprise Chat (Chat), also invokes tools, but its internal loops are much shorter, so it serves as the lower-agency reference (Section 2.4). Together, the traces contain more than 121 million model calls. The primary agentic trace, Research, contains 17M model calls in 691k sessions over eight days in June 2026; Chat contains 104M model calls in 13.2M sessions over 24 hours in July 2026 (Table 2); a session averages 2.62 user turns in Research and 2.29 in Chat. Unlike prior traces, our telemetry combines production-scale enterprise workloads with session structure, token accounting, model identity, and, for three workloads, the serving region and deployment selected for each call. We follow an agentic session’s KV state through five questions, from how the state accumulates and is reused, to how memory capacity and deployment placement affect its residency. Q1. How agentic is each workload? Orchestration metrics place the traces on a spectrum of agency (Section 2). Q2. Why does KV matter? The model–tool loop makes agentic sessions deep: Research has a median session depth of 13 model calls and a P99 of 160, compared with 3 and 52 for Chat. Context keeps growing across these calls, and the three agentic workloads achieve aggregate KV reuse of 0.50–0.80, compared with 0.27 for Chat, although all four have median within-session idle gaps of 6–10 s (Code: tool-execution cadence; Figure 2; Sections 3.5 and 4.3). These measurements show that the agentic workloads carry substantially more reusable KV state across idle intervals with similar median durations (Section 3). Q3. Why not keep all of it? Under the reference fp16 grouped-query-attention (GQA) KV representation (Section 2.3), a single session’s KV can approach the memory of an entire GPU; holding every live Research session at its peak working set would require an estimated 501.8 TB; the state sits idle across heavy-tailed gaps; and its reuse is associated with staying on the serving endpoint, which we call a deployment, that built it (Section 4). Q4. What can be done? Reclaiming state after one idle hour lowers the Research holdall-live ceiling from 501.8 to 140.8 TB (Section 5.1), and replaying residency under constrained memory separates recomputation caused by insufficient GPU memory from that caused by routing when KV is deployment-local, and quantifies the potential value of affinity-aware routing [45, 38, 29] and multi-resolution KV retention [32, 17, 30, 57] (Section 5). Q5. How can others study this? Because the traces cannot be released, we plan to release a parametric trace generator (Section 6). Contributions. This operational systems article makes five contributions, one per question. 1. We present a production-scale characterization of four Microsoft M365 workloads spanning three agentic applications and one enterprise-chat workload, and compare their agency where the telemetry permits. The traces contain more than 121 million model calls and expose session, token, and model structure and, for three workloads, deployment structure (Section 2). 2. We characterize how persistent model-tool loops shape session depth, context growth, and KV reuse. Context continues growing across deep sessions, reuse is bimodal at the call level, and in Research call-context KV concentrates in a predictable model class: the heavyweight Spine class, which carries the long-running context, holds 67% of call-context KV bytes but only 32% of uncached-context tokens and has a self-transition probability of 0.97 (Section 3). 3. We show why retaining every session’s KV is a fleet-wide GPU-memory capacity problem. 3

User

Model

Tool

Model

median idle gap (s)

Short orchestration path

Response

Mean: 2.49 model calls/turn

(a) Chat turn Repeated model-tool loop

User

Model

Tool

Model

Tool

Model

12 10

Research (0.80, 8.4 s)

8 6

Chat (0.27, 7.9 s)

4

Code (0.50, 6.3 s)

2 0 0.0

Response

Assist (0.51, 9.9 s)

0.2

0.4 0.6 0.8 aggregate KV reuse R

1.0

Mean: 9.38 model calls/turn

(b) Agentic turn

Figure 1: Control flow within one user turn. A turn runs from a user request until control returns to the user; a session is the sequence of turns that share a session identifier, so this loop repeats once per turn. In-figure means are model calls per user turn (total model calls divided by total user turns), as in Table 2.

Figure 2: Aggregate KV reuse R (Eq. 2) versus the median within-session idle gap (time between consecutive model calls, intra- and inter-turn gaps pooled) for the four workloads; the shaded band spans 6–10 s. Code’s R uses its session-cumulative cache_read accounting, and its idle value is its median tool-execution cadence; the other traces use provider-reported cached tokens.

Under the reference KV representation, a single P99 Research session requires 81.6 GiB, exceeding the 80 GB high-bandwidth memory (HBM) of an NVIDIA H100 GPU, and holding every live Research session at its peak lifetime working set requires an estimated 501.8 TB at the exact peak. In Research, mean same-model reuse is also associated with deployment continuity: 88.9% when consecutive calls remain on one deployment versus 29.0% after a same-model deployment change within the region (Section 4). 4. We identify two distinct sources of KV recomputation: eviction under limited memory and unavailability under deployment-local placement. At a pooled memory budget of 10% of the busiest-hour current-context peak (the KV that hour needs with no eviction), even clairvoyant eviction by Belady’s furthest-next-use rule (FNU) leaves 31.1% of Research calls to recompute their context, while int4 multi-resolution retention (MRR), which keeps an evicted session’s KV as a 4-bit copy instead of discarding it, reduces this call-level miss rate to 13.6%. Separately, modeling KV as deployment-local raises the first-touch floor, the share of calls that miss even with unlimited memory, from 5.4% to 20.2%, motivating both recoverable KV retention and affinity-aware routing (Section 5). 5. Because the production telemetry cannot be released without risking client confidentiality, we plan to release AgenticTraceSim, a parametric trace generator, together with a residency replay; the generator reproduces the principal Research distributions, their dependencies, and the downstream policy ordering without exposing production events (Section 6).

2

Agentic Workloads and How We Measure Them

Before examining KV behavior, we establish what prior characterizations leave unmeasured, describe the four production workloads and how we measure them, and show how agentic each

4

workload is.

2.1

What Prior Characterizations Miss

Existing LLM serving systems primarily optimize request-level prefill, decode, and latency [20, 3, 65]. Agentic applications instead compose repeated model and tool calls into persistent sessions whose KV cache remains valuable between model calls [26]. The closest agentic workload characterizations cover coding-agent traces and benchmark-driven agents [66, 28, 60]. They establish that agentic workloads contain repeated model and tool calls, growing contexts, short outputs, and substantial prefix reuse. None jointly examines diverse production enterprise workloads, the serving deployment of each call, byte-valued KV residency, and capacity-constrained residency replay at fleet scale (Table 1). These studies record the logical model of each call but not the serving deployment selected for it. Consequently, they cannot measure how KV reuse changes when consecutive calls to the same model are served by different deployments. Our telemetry includes the serving region and deployment for three traces, allowing us to compare same-deployment continuations with same-model calls routed to another deployment (Section 4.4.2). Prior characterizations also report demand primarily in tokens. We convert peak session context to KV bytes using an explicit, linearly rescalable model assumption and aggregate these estimates across concurrently live sessions (Sections 2.3 and 4.2). This enables comparisons with GPU memory capacity and exposes the sensitivity of fleet estimates to how long a session counts as live and to temporal resolution. TraceLab evaluates cache timeouts against hit rate and relative storage, but prior studies do not compare destructive eviction policies, which discard an evicted session’s KV, using byte-valued session state. We therefore develop a trace-driven residency replay that consumes the observed production calls under fixed memory budgets (Sections 5.2–5.5). It separates recomputation caused by limited memory from that caused by deployment-local placement, and evaluates retaining displaced KV at lower precision.

2.2

Four Production M365 Workloads

Table 2 summarizes the four traces, collected between late June and early August 2026. We call a workload agentic when a user request can initiate an extended internal loop of model and tool calls without human input between calls; agency is a matter of degree, which Section 2.4 measures. For each model call i, the available telemetry includes context tokens xi , cached tokens ci , output tokens oi , timestamp ti and model identity; additional timing and tool-activity fields are available for a subset of traces. The analyzed telemetry excludes user request/response content and contains only the operational metadata required for the measurements reported in this paper. Calls, turns and sessions. A model call is one inference request to an LLM. A turn pairs one user request with the application’s complete response: its first model call is user-initiated, and each later call is internal, issued by the orchestrator (for example, after a tool returns) without new human input. A session is the ordered sequence of model calls that share a session identifier; it spans one or more turns, and its depth counts its model calls. The idle gap before a call is the time since the preceding call of its session; it is inter-turn if the call opens a turn

5

Table 1: Features of closest agentic workload characterizations. TraceLab [66]

Feature

Workload domain

Trace scale Fleet-scale production trace Multiple agent/chat workloads Per-call cache accounting Serving deployment per call KV residency in bytes Capacity-constrained replay Released trace or generator

Copilot [28]

ReAct study [60] This work (M365)

Coding

Coding

Benchmarks

Enterprise research, assistant, coding, chat

4,265 sessions

761M calls/13.5M sessions

5 benchmarks

121M calls/13.9M sessions

–

•

–

•

–

–

•

•

•

•

•

•

–

–

–

•

–

–

–

•

–

–

–

•

•

–

–

Planned

and intra-turn otherwise. Research and Chat log a usable turn counter; Assist’s counter is empty, and Code logs tool and session events rather than model calls. For Research, Assist, and Chat, the telemetry also identifies the serving region and endpoint instance selected for each model call. We call this endpoint instance a deployment; multiple deployments may serve the same model. Code does not expose region or deployment information. The traces differ in granularity and cache accounting. Research, Assist, and Chat report model calls with per-call context and provider-cached token counts. Code contains tool executions, command-line runs, and other session events rather than model calls; its token counts are cumulative over the session, and it has an Anthropic-style cache_read field. We therefore measure Code depth in events, report its reuse separately, and exclude it from the byte-valued working-set and placement analyses.

2.3

Measurement Methodology

Unless stated otherwise, descriptive measurements use the complete traces. Reuse and sessionlevel analyses exclude calls without a valid session identifier, retaining essentially all Research calls but only 75.22M of the 104M Chat calls; the Assist trace in Table 2 already contains only calls with a session identifier. The capacity-policy experiment replays the busiest one-hour Research interval (176k calls), while the fleet-residency and idle-cap analyses (Sections 4.2 and 5.1) use the complete eligible Research trace. We report the applicable population whenever it differs from the trace total. We parse ISO-8601 timestamps directly into elapsed seconds, order calls by timestamp within each session, and remove exact duplicate Code events before analysis. Code samples sessions selected on June 30, 2026, and retains their complete histories, so its events span 34 days. We distinguish three kinds of results. Measured quantities are computed directly from pro6

Table 2: Four production Microsoft M365 traces. Turns and internal calls are defined in Section 2.2; “–” marks traces without a usable turn counter. R is aggregate reuse (Eq. 2) over calls with a session identifier; Code reports session-cumulative cache_read tokens instead. Research Deep Research Window Model Calls Sessions Models User turns per session Internal calls Model calls per user turn Median session depth Realized reuse R

Assist Enterprise Assistant

Code

Chat

Coding Agent

Enterprise Chat

8 days 17,008,715 691,293 Multiple models

8 days 217,106 23,538 Single model

34-day event span 1,119,432 events 8,157 Multiple models

24 hours 104,013,216 13,184,080 Multiple models

2.62

–

–

2.29

89.3%

–

–

59.9%

9.38

–

–

2.49

13 calls

1 call

14 events

3 calls

0.803

0.506

0.496

0.270

duction telemetry, including token counts, reuse, timestamps, and placement transitions. Estimated quantities convert token counts to KV bytes using an explicit KV representation assumption. Replayed quantities report outcomes from applying counterfactual residency policies to the observed call stream. KV representation assumption. To avoid disclosing confidential details of the deployed models, we convert tokens to bytes using a public stand-in: the KV geometry of the released Llama-2-70B model [49], a Grouped-Query Attention (GQA) model [4] with 80 layers, eight KV heads, and a head dimension of 128. Separate key and value tensors in fp16 give β = 327,680 bytes/token, which we apply to every model. Absolute byte estimates (working sets, fleet ceilings, budgets, and transfer volumes) scale linearly with β, while reuse, context share, temporal ratios, and reconstruction rates at budgets set as fractions of a measured peak are independent of β. Reuse. The serving system keeps a prefix cache: KV computed by earlier calls, indexed by the token prefix that produced it [50]; ci counts the leading context tokens of call i found there. For a call with xi > 0, per-call reuse is the provider-reported cached fraction of its context, ri = ci /xi , P P Eq. (1), and aggregate reuse is the token-weighted fraction R = i ci i xi , Eq. (2). Because ci records cache behavior realized by the deployed infrastructure, reuse is a serving outcome rather than an intrinsic workload property. We define every other metric where it is first used.

2.4

How Agentic Is Each Workload?

In Research, over 89% of classified model calls are issued internally by the application, not the user. For Research and Chat, the logged turn counter lets us measure this directly (Table 2): a Research user turn triggers far more model calls than a Chat turn, and a larger share of Research calls occur without a new user turn. Research depth is therefore generated

7

primarily by the application’s internal orchestration rather than by additional user turns. Chat shows the same behavior to a lesser degree: it invokes tools, routes turns through multiple models, and a majority of its calls occur without a new user turn. Its loops are, however, much shorter (fewer model calls per user turn), and its sessions are shallower (Section 3.1). We therefore treat Chat as the lower-agency reference, not as a non-agentic, single-response chat baseline. The event-level Code trace provides complementary evidence: 84.2% of its events are explicit tool executions, and the median session contains 10 tool calls. Neither figure is a model-to-tool call ratio. Assist lacks a usable turn counter and tool field, so its internal orchestration is not measured directly; its depth distribution indicates a mixture of single-call sessions and longer multi-call sessions (Section 3.1), and as a single-model workload it later serves as a placement control. Figure 2 adds a serving outcome: the three agentic workloads realize much higher reuse than Chat, while all four idle for similar median intervals between calls (Code: tool-execution cadence). The next section asks what the agentic loop does to a session’s KV state.

2.5

Use of generative AI

Generative AI tools were used to refine language and presentation, and to help implement code for trace-analysis, figure-generation, claim-verification, trace-generator and residency-replay. They were not used to generate the study’s ideas, telemetry or conclusions. The authors reviewed all assisted edits and source code, and accept responsibility for the manuscript and its artifacts.

2.6

Privacy Considerations

We analyze only aggregate operational metadata, excluding prompts, responses, retrieved content and other message content, and report no production identifiers. Confidential production traces are not released, and the public artifact uses only aggregate fitted parameters and synthetic traces.

3

The Agentic Loop Makes KV Retention Valuable

This section measures how persistent orchestration shapes session depth, context growth, token volume, model-class structure, and KV reuse. Within each turn, agentic sessions re-enter the model repeatedly without human input and accumulate context across these calls, which creates sustained demand for reusable KV state.

3.1

Agentic Sessions Are Deep

Figure 3(a,b) shows the distribution of session depth, the number of model calls in a session, summed over all of its turns; in Research, this depth comes primarily from internal orchestration (Section 2.4). Research sessions have a median depth of 13 calls and a P99 of 160, compared with 3 and 52 for Chat. Assist has a median of one call but a P90 of 28, a mixture of single-call sessions and longer multi-call sessions. Code counts logged events, mostly tool executions, so we report it separately and do not plot it: its median session has 14 events and its P99 session 963.

8

0.5

0.0 0

50

calls per session

(a) Session-depth CCDF, linear axes.

100 10−1 10−2 10−3 100 101 102

context (K tokens)

Research Assist Chat

P(depth > x)

P(depth > x)

1.0

50 depth P50

0 1

calls per session

(b) Session-depth CCDF, log–log axes.

Q1–Q3 median

100

10

20

30

call index

(c) Research context by call index.

Figure 3: Agentic sessions are deep and keep accumulating context.

3.2

Context Accumulates Across the Loop

Deeper sessions tend to require more KV memory because context continues growing across model calls. Figure 3(c) shows that the median Research context grows from 9,764 tokens on the first call to 41,045 by call 11 and 68,118 by call 30, and first reaches 95% of its call-30 value at call 27. Thus, estimating session demand from its first few calls would systematically understate the KV required by deep sessions. Individual sessions, however, do not grow monotonically. In Research, 90.9% of multi-call sessions contain at least one context reduction, a call with less context than its predecessor, and the median reduction leaves 23.7% of the preceding call’s context. The rising median in Figure 3(c) therefore reflects net growth across repeated growth-and-reduction cycles. A reduction may reflect compaction, summarization, truncation, or a transition between model roles; the telemetry does not record its cause. Compaction near the model’s context-window limit is a plausible cause, but we do not equate every reduction with an explicit compaction event.

3.3

Context, Not Generation, Dominates Token Volume

We measure the context share, the fraction of input-plus-output token volume contributed by P P context, as ρ = i xi i (xi + oi ), Eq. (3). Across all four traces, context tokens constitute at least 98% of combined input and output token volume processed by model calls, with generated output tokens accounting for less than 2%. The context shares are ρ = 0.98 for Research, 0.99 for Assist and Chat, and 0.997 for Code. Equivalently, aggregate context-to-output ratios range from 52:1 for Research to 357:1 for Code. Context share counts cached tokens, so it measures the KV state carried by model calls, not the prefill compute spent on uncached context; the phase comparison of the ReAct study [60] instead counts recomputed work, so the two are not interchangeable. Memory demand depends on how much context remains resident, whereas recomputation depends on how much of it has been evicted when the session resumes.

3.4

The Loop Is Multi-model, and One Class Holds Most Reusable State

A single Research session may invoke several models. We group them into four functional classes: a heavyweight Spine class that carries the long-running context, a Mid class for or-

9

chestration and general reasoning, a Scoped class for bounded subtasks, and a Fanout class containing all remaining models. The classes describe model roles within a session; they do not imply that every lighter-model call comes from a distinct subagent. The telemetry carries no class labels, so we assign each call to a class by matching substrings of its model name; we do not disclose the model names. The Spine–Mid boundary is a choice: our default conservative mapping places only the deep-research models in Spine and the reasoning models in Mid, whereas a broader mapping also places the reasoning models in Spine. We report both. 3.4.1

Call-context KV concentrates on the Spine.

We measure each class’s share of call-context KV : every call’s context, summed over the class’s calls, so a context carried by ten calls counts ten times; because one β applies to every model, byte and token shares coincide. Figure 4(a) shows that call-context KV is highly concentrated. Under the conservative mapping, the Spine carries 67% of call-context KV bytes, followed by Mid at 31%; Scoped and Fanout together carry less than 2%. The broader mapping raises its share to 90%. The class boundary is definition-dependent, but one heavyweight class dominates Research call-context KV under both mappings. 3.4.2

Call-context bytes and uncached prefill have different class distributions.

The Spine dominates call-context bytes, not uncached prefill. Under the conservative mapping, P it carries 67% of call-context KV bytes but only 32% of uncached-context tokens, i max(xi − ci , 0), the context the serving system had to prefill (Figure 4(b)), so most Spine context is served from the prefix cache rather than recomputed. This asymmetry makes the Spine a high-value target for residency management: it occupies substantial GPU memory, and losing its cached prefix would turn a large reused context into costly recomputation. 3.4.3

Model-class transitions are predictable.

Consecutive calls within a session also follow a strong class-transition structure (Figure 4(c)). The Spine and Fanout self-transition with probabilities 0.97 and 0.95, Scoped transitions to Mid with probability 0.86, and Mid either remains in its class or transitions to Scoped or Spine. Predicting the next class as the most frequent successor of the current class is correct for ≈ 88% of observed Research transitions. This low-cost signal indicates which model, and hence which KV state, a session is likely to access next. The opportunity is workload-dependent: it exists in multi-class Research but not in single-model Assist. 3.4.4

Reuse declines after model-class excursions.

Model-class control flow is also associated with whether retained Spine state remains reusable. For each Spine call i after the first in its session, let k be the number of non-Spine calls since the preceding Spine call (the excursion length), and define h(k) = E[ri | i ∈ Spine, k], Eq. (4). Mean reuse is 92.5% for consecutive Spine calls (k = 0), falls to 45.1% after one intervening call, and is 30.1% after three or more (Figure 5). Most of the decline comes with the first intervening call. Model-class history is therefore a candidate signal, alongside wall-clock age (the idle time examined in Section 4.3), for estimating the future value of resident KV state. Because the analysis does not control for elapsed time or deployment placement, the decline is 10

100 60 40

31.1%

20 0

0.7% Spine

Mid

75

67.1%

50 31.9% 25 0

1.1%

Scoped Fanout

(a) Call-context KV byte share.

current class

Spine share (%)

call-context KV byte share (%)

67.1%

Spine 0.973

0.027

0.001

0.000

Mid 0.135

0.636

0.226

0.002

Scoped 0.009

0.860

0.112

0.019

Fanout 0.003

0.020

0.026

0.951

Spine

Mid

Scoped Fanout

call-context uncached KV context

(b) Spine call-context uncached-context shares.

next class

and (c) Model-class transition probabilities.

50

92.5%

100

% of calls

Spine reuse h(k) (%)

Figure 4: Call-context KV concentrates on a predictable model class. (a) Under the conservative mapping, Spine carries 67.1% of call-context KV bytes; the broader mapping raises this share to 90.0%. (b) Spine accounts for only 31.9% of uncached-context tokens. (c) Spine is strongly selftransitioning, with PSpineSpine = 0.97; selecting the most frequent outgoing transition predicts the next class for approximately 88% of observed transitions.

75 50 45.1%

25 0

0

1

40.3%

2

30.1%

40 30 20 10 0 0.0

3

intervening non-Spine calls k

Figure 5: Spine reuse declines after an excursion. Mean reuse falls from 92.5% for consecutive Spine calls to 45.1% after one intervening non-Spine call. The analysis does not independently control for elapsed time or deployment placement.

Research Assist Chat

0.2

0.4

0.6

0.8

per-call reuse ri = ci/xi

1.0

Figure 6: Per-call KV reuse is bimodal, but differently weighted. Research is warmweighted, with 51.9% of calls achieving at least 95% reuse, whereas Chat is cold-weighted, with 47.6% of calls below 20% reuse. Assist has the largest intermediate-reuse share.

an association, not evidence of a causal effect of model-class interference; Section 4.4.2 examines how deployment continuity relates to reuse.

3.5

Reuse Is Bimodal at the Call Level

KV reuse lets a model call skip recomputing context that earlier calls in its session already processed, whether the call continues a turn or opens a new one. Figure 6 shows that per-call reuse ri is bimodal: calls concentrate near low or high reuse, but the weights of the cold and warm modes differ across workloads. Research is warm-weighted, with 51.9% of calls having at least 95% reuse and only 11.4% having reuse below 20%. Chat is cold-weighted: 47.6% of calls have reuse below 20%, while only 21.1% reach ≥ 95%. Assist is more evenly distributed, with 29.9% of calls in the intermediate 20–80% range. The bimodality is a per-call property: only 40.7% of Research sessions have a session-level aggregate reuse of at least 0.8, and 52.1% lie between 0.2 and 0.8, because sessions mix cold calls with warm continuations. Prefix-level reuse 11

is therefore a useful residency signal, but a policy must combine it with session continuation, working-set size, and fleet memory pressure. Aggregate reuse R is 0.27 for Chat, 0.50 for Code, 0.51 for Assist, and 0.80 for Research. The high Research value reflects repeated access to accumulated context, but realized reuse also depends on the serving system’s ability to retain and recover that context; Section 4.4.2 shows that reuse is strongly associated with whether consecutive calls remain on the same deployment. Equivalently, Research has a context-carry tax τ = 1/(1 − R) of 5.1: it processes 5.1 context tokens for every uncached context token it must prefill, and the rest is served from the prefix cache. Losing that cached context would multiply aggregate context prefill work by up to 5.1×, which is why KV retention matters. The tax is close to TraceLab’s prefill amplification [66] (5.1× versus 5.3×), although the denominators differ. Retained KV is therefore valuable: agentic sessions are deep, their context keeps growing, much of it is served from the prefix cache, and in Research most call-context KV bytes sit in one large, predictable Spine state. Why, then, not keep every session’s KV resident for the session’s lifetime? Section 4 examines the hurdles this policy faces.

4

Why Not Keep Every Session’s KV Resident?

The simplest residency policy retains each session’s KV state from its first to its last call. We now show four hurdles to this policy: a single session can fill a GPU’s memory, live sessions add up across the fleet, the retained state sits idle across heavy-tailed gaps, and its reuse is associated with staying on the deployment that built it.

4.1

A Single Session Can Fill a GPU

Under the reference KV representation (Section 2.3), the estimated per-session KV working set is the peak KV state of session s, Ws = β · maxi∈s (xi ), Eq. (5). This is a byte estimate, not a measurement of observed physical HBM allocation. A Research session requires an estimated 25.7 GiB at P50, 59.2 GiB at P90, and 81.6 GiB at P99 (Figure 7). These estimates include KV state alone: model weights, activations, and runtime state require additional GPU memory. For scale, the P90 working set is comparable to most of the 80 GB HBM of one H100, while the P99 estimate exceeds this capacity. In practice, a model with the reference attention configuration may be served by a tensor-parallel group of GPUs, each holding the KV of a subset of the attention heads [20]; the tensor-parallel model replica, rather than one GPU, must collectively hold the session’s KV. The workingset distributions also differ across workloads (Figure 7): a median Research session carries approximately 2.5× the KV state of a median Chat session.

4.2

Across the Fleet, Live Sessions Add Up

Across the serving fleet, the large working sets of concurrent, long-running agentic sessions place sustained pressure on scarce and costly HBM. We bound this pressure with a deliberately pessimistic hold-all-live model in which each session retains its peak lifetime KV working set from its first through its last observed call. Let S(t) = {s | tfirst ≤ t ≤ tlast } denote the sessions s P s live at time t. The estimated fleet-wide KV residency is Klive (t) = s∈S(t) Ws , Eq. (6), and the ⋆ = max (K full-trace residency ceiling is Klive t live (t)). 12

1.0

1.0

Research Assist Chat

CDF

0.6 H100 80 GB

CDF

0.8

0.8

0.4 0.2 0.0 1

10

0.6

Research Assist Chat

0.4 0.2 0.0 10−1

101

103

105

within-session idle gap (s, log)

100

per-session peak KV working set (GiB, log)

Figure 8: Gaps between consecutive model calls Figure 7: Per-session KV working sets differ of a session (intra- and inter-turn pooled) are across workloads. The dashed line marks the heavy-tailed: medians are 7.9–9.9 s, but the tail 80 GB (74.5 GiB) HBM of an H100. extends to minutes or days. Treating all regions as one logical pool, an exact event sweep of Equation 6 over the eight-day ⋆ = 501.8 TB, and the number of concurrently live sessions peaks at Research trace gives Klive 11,358. This is the estimated footprint of KV state alone under the reference representation; it excludes model weights and other runtime memory. Temporal resolution matters for this ceiling. One-minute bins give 504.9 TB, 0.6% above the exact value, whereas hourly bins, which count a session as resident for every hour its lifetime touches, give 709.4 TB, a 1.41× overstatement. We use the exact ceiling for capacity results and report the hourly value only as a resolution warning.

4.3

Retained State Sits Idle Across Heavy-tailed Gaps

A session’s KV state remains potentially reusable between model calls, but retaining it consumes scarce GPU memory while the session is idle. Within-session idle gaps, measured between consecutive model calls of a session and pooling intra- and inter-turn gaps (Section 2.2), have similar medians across the three traces that report per-call tokens: 8.38 s for Research, 9.87 s for Assist, and 7.87 s for Chat. Their tails are substantially longer, with P99 gaps of approximately 20, 50, and 93 minutes, respectively (Figure 8). The longest observed Research gap is 7.9 days, indicating that some sessions are parked and later resumed. In the two traces with a usable turn counter, intra-turn gaps dominate the short-gap body of the distribution, while inter-turn gaps account for most of the total idle time: in Research, inter-turn gaps constitute 7.0% of gaps but 92.6% of total idle time (median 125.0 s versus 7.5 s for intra-turn gaps), and in Chat, whose maximum gap is bounded by its 24-hour window, they constitute 27.4% of gaps and 97.3% of idle time (146.5 s versus 4.6 s). This range makes any single retention timeout a compromise: a short timeout discards KV that returning sessions would have reused, forcing reconstruction, while a long timeout holds state for sessions that stay inactive for hours or days. Section 5.1 quantifies this trade-off by varying the idle interval after which a session’s KV state is reclaimed.

4.4

Reuse Is Associated with Staying on One Deployment

The preceding analysis treats resident KV as logically available whenever a session returns. In a distributed fleet, however, reusable KV may remain local to the serving deployment that 13

constructed it. Using the call-level region and deployment fields available for Research, Assist, and Chat, we examine how deployment continuity is associated with reuse. 4.4.1

Long Research sessions span multiple deployments.

We summarize placement continuity using single-region (SR) and single-deployment (SD) session subsets. Both are defined per model: every model a session invokes stays within one region (SR) or on one deployment (SD), although different models may use different regions or deployments. Placement stability differs substantially across workloads. Only 45.0% of Research sessions are SD and 57.6% are SR, compared with 80.2% and 83.3% for single-model Assist, and 79.1% and 80.0% for Chat. A Research session touches a mean of 4.48 deployments across 3.04 regions, reflecting its longer, multi-model orchestration path. The SR and SD subsets are structurally different from the complete workloads. For example, the median Research depth falls from 13 calls overall to 5 in SR and 3 in SD. Their aggregate reuse and capacity values therefore cannot be interpreted as the causal effect of applying a placement policy. We use these subsets only to describe placement stability; the per-transition analysis below provides the cleaner comparison of reuse immediately following a deployment change. 4.4.2

Reuse is strongly associated with deployment continuity.

For each model call, we condition per-call reuse on its transition from the preceding call in the session. We distinguish a continuation on the preceding call’s deployment, a model switch, a same-model move to another deployment within the region, a same-model cross-region move, and a session’s first call. Figure 9 reports the resulting mean reuse. For Research, same-deployment continuations achieve mean reuse of 88.9%. When the same model is served by another deployment within the region, mean reuse falls to 29.0%; for a cross-region move, it falls to 23.5%. These transitions account for only 2.0% and 0.9% of calls, but their reuse is less than one-third that of same-deployment continuations; same-deployment reuse is stable across the eight observed days. The single-model Assist trace removes the model-switch confound. Its mean reuse is 64.8% for same-deployment continuations, 27.7% after a cross-deployment move, and 7.5% after a cross-region move. The consistent decline in both Research and Assist provides evidence that deployment continuity is strongly associated with preserving reusable KV state. Chat does not show the same pattern. Model switches account for 64.0% of its calls, while same-model deployment and region moves together account for less than 2.5%. These rare moves show higher mean reuse than same-deployment continuations, but the categories can differ in session depth and composition, so we treat Chat as selection-confounded for this comparison and base the deployment-continuity finding on Research and the single-model Assist control. Together, these measurements show why holding all KV state is costly: under the reference representation a tail session can exceed one GPU’s memory, live sessions add up to a fleet-wide capacity problem, much of the state idles for long intervals, and its reuse is associated with the deployment continuity that routing provides. The next section evaluates what residency policies can recover under these constraints.

14

⋆ Klive

Chat

Fleet ceiling ⋆ Klive (TB)

50 25 Same Model CrossCrossdeployment switch deployment region

400

75

evict >1h idle

50 200 0

Session's first call

25

1m 5m 15m 1h

6h

idle cap θ

Figure 9: KV reuse is associated with deployment continuity in the agentic traces. For Research, mean reuse falls from 88.9% on same-deployment continuations to 29.0% when the same model moves to another deployment. The single-model Assist control shows a similar decline from 64.8% to 27.7%. Chat is selectionconfounded: its same-model deployment moves are rare, and the categories can differ in session composition.

5

duty cycle

hold-all-live

75

0

reconstruction

1d

∞

0

Reconstruction rate, duty cycle (%)

Assist

n/a

per-call reuse (%)

Research 100

Figure 10: An idle cap substantially reduces fleet KV demand. For Research, reclaiming state after one idle hour reduces the ⋆ from 501.8 to 140.8 TB exact fleet ceiling Klive while increasing reconstruction from 4.07% to 4.53%. The dotted line (right axis) is the duty cycle, engine time as a share of resident session time. The penalty rises more rapidly when the cap enters the short-gap body of the distribution.

What Can Be Done? Residency Policies Under Constrained Memory

Because holding every session’s KV state is costly, we replay residency policies on the observed call streams. We first sweep an idle cap over the complete Research trace, then compare destructive eviction with multi-resolution retention (MRR), which keeps displaced KV as a recoverable int4 copy, under fixed budgets with one KV object per session, and finally measure the miss rate that remains with unlimited memory when KV objects are deployment-local. A trace-driven residency replay processes an ordered call stream C under a pooled memory budget B, shared by all sessions in the stream, and KV residency policy π: it handles calls in timestamp order and keeps the KV states that π retains within B. A call whose KV state is absent when it arrives is a miss and requires reconstruction: its whole context must be recomputed by prefill. With mπ,B misses, the replayed reconstruction (miss) rate is Mπ (B) = mπ,B |C| , Eq. (7); MLRU , MFNU , and MMRR denote it for the policies below. The replay treats a session’s current KV state as one variable-sized object, updates its size at each call, and applies π whenever resident KV exceeds B. Because each replay starts with an empty cache, its first access to each KV object in C misses, independent of the policy and budget. This first-touch floor, the number of distinct KV objects divided by |C|, is the compulsory-miss rate of classical cache analysis, and every policy reaches it once B is at least the stream’s no-eviction peak.

5.1

Reclaiming Idle State: Nominally Live State Is Not Always Residencyworthy

The hold-all-live ceiling is deliberately pessimistic: a session whose consecutive calls are days apart holds its peak KV state throughout the gap. We therefore ask which live sessions are residency-worthy, likely to resume soon enough that keeping their KV resident pays off, and 15

parameterize liveness by an idle cap θ: once a session has been idle for θ, its KV state is evicted from HBM and its next model call requires reconstruction. Each session is still sized at its peak lifetime working set, and θ = ∞ recovers the hold-all-live model. Figure 10 shows that most of the hold-all-live ceiling can be reclaimed at little reconstruction cost. A one-hour cap lowers the Research ceiling 3.6×, from 501.8 to 140.8 TB, while the replayed reconstruction rate over the complete trace rises only from 4.07% at θ = ∞, where only first touches count, to 4.53%. The long idle tail thus inflates the ceiling, yet few calls follow such gaps. The trade-off remains favorable at a 15-minute cap, where the ceiling falls to 97.2 TB and reconstruction reaches 5.2%. As the cap falls further below the approximately 20-minute P99 idle gap (Section 4.3), reconstruction rises quickly, to 6.7% at five minutes and 14.1% at one minute. The source of this penalty shifts with the cap: inter-turn gaps (Section 4.3) are 98.0– 99.8% of the gaps reclaimed at caps of 15 minutes or longer, whereas at one minute intra-turn gaps contribute 5.23 of the 10.04 added percentage points. Fleet provisioning should therefore distinguish nominally live sessions from those likely to resume soon, rather than treating each session’s first-to-last-call span as continuously resident.

5.2

Capacity Limits Destructive Eviction

The idle-cap analysis charges each resident session its peak lifetime working set; we next replay current-context KV residency under a fixed memory budget B. The replay processes the 176,007 model calls that 9,486 sessions issue in the busiest Research hour, so its first-touch floor is 5.4%, and sizes each session by its current context at each call rather than its lifetime peak. Without eviction, the current-context peak in this hour ⋆ = 47.0 TB, a different quantity from the whole-trace, lifetime-peak ceiling K ⋆ . We set is Khr live ⋆ = 4.7 TB, a deliberately constrained operating point at which policy differences B = 0.1 Khr are visible, and sweep the full budget range in Figure 11. Here B is a pooled budget: the replay treats the KV memory of all replicas serving the stream as one tier and selects victims across the whole stream. Physical replicas, each a tensor-parallel group of GPUs, enforce residency on their own memory, so at the same total capacity we expect per-replica caches to reconstruct more often than the pooled replay reports. Because of its lower volume, Assist is replayed over ⋆ its complete retained stream, whose no-eviction current-context peak we denote Kwin (8.1 TB). We compare two destructive policies, which discard an evicted session’s KV. LRU evicts the least recently accessed session [50, 52]. The clairvoyant furthest-next-use (FNU) policy applies Belady’s rule [5]: it knows the next access to each resident KV object and evicts the object whose next access lies furthest in the future. Belady’s rule minimizes misses when all objects have the same size; because session KV objects vary in size and grow with context, FNU is here a strong clairvoyant heuristic rather than an optimum [6]. At B = 4.7 TB, LRU reconstructs context for 62.9% of Research model calls and FNU for 31.1%. Future knowledge thus approximately halves reconstruction, yet nearly a third of calls still rebuild their context.

5.3

Retaining a Lower-precision Copy: Multi-resolution Retention

In the replay, MRR replaces eviction with demotion: when resident KV exceeds B, it retains the least recently used session as an int4 copy, a quarter of its fp16 size, which stays in HBM and is charged to the same budget. A later call to a demoted session is modeled as promoting the copy back to fp16 instead of reconstructing its context, building on KV quantization and 16

75 50 25 0

2

5

10 20

50 100

⋆ budget B (% of Khr )

(a) Research, ⋆ Khr = 47.0 TB.

busiest

hour:

⋆ B = 0.1Kwin

100

LRU FNU MRR (int4) first-touch floor

75 50 25 0

2

5

10 20

50 100

⋆ budget B (% of Kwin )

⋆ (b) Assist, full window: Kwin = 8.1 TB.

reconstruction rate Mπ(B) (%)

reconstruction rate Mπ(B) (%)

reconstruction rate Mπ(B) (%)

⋆ B = 0.1Khr

100

⋆ B = 0.1Khr

100 75 50 25 0

2

5 10 20 50 100 ⋆ budget B (% of Khr )

⋆ (c) Chat, busiest hour: Khr = 1,876.7 TB.

Figure 11: Capacity constrains destructive eviction, while multi-resolution retention ⋆ , Research call-level reconstruction rates M (B) reduces reconstruction. At B = 0.1Khr π are 62.9% for LRU, 31.1% for FNU, and 13.6% for int4 MRR. Chat also exhibits substantial policy separation, whereas Assist does not. Research and Chat use their busiest-hour replay streams; Assist uses its complete retained stream because of its lower volume. Budgets are ⋆ in (a) pooled and expressed as fractions of each stream’s no-eviction current-context peak: Khr ⋆ and (c), Kwin in (b). MRR keeps int4 copies within the same pooled memory budget. The dashed line marks the first-touch floor (Section 5.2). MRR reaches this floor when displaced KV instead moves to a host-memory tier four times the pooled memory budget (Section 5.3). The legend in (b) applies to all panels. compression [32, 30]; VeriCache likewise keeps a compressed KV copy close to the GPU [57]. MRR chooses which session to demote by recency alone, without future knowledge; only when every other session is already int4 and the budget is still exceeded does it discard int4 copies, least recently used first, and a session whose copy was discarded is reconstructed on its next call. The replay does not measure how int4 KV changes responses, but published evaluations of open Llama models report modest costs. Adding 4-bit KV quantization to an already weight- and activation-quantized model raises WikiText-2 perplexity by 0.12 for Atom (W4A4 Llama-7B) [63] and 0.14 for QoQ (W4A8 Llama-2-7B) [27]. Group-quantized W4A8KV4 Llama-2 models from 7B to 70B lose at most 1.03 points of FP16 average zero-shot accuracy; the 70B model has the reference KV geometry used in our byte estimates. W4A8KV4 Llama-3.1-8B loses 0.14 points of its BF16 LongBench average [27]. These evaluations use single-request benchmarks rather than agentic sessions. With the same pooled HBM budget, MRR reduces the replayed reconstruction rate to 13.6%, 2.3× below FNU. Because MRR demotes in LRU’s victim order, the drop from LRU’s 62.9% isolates the effect of keeping a smaller, lower-precision copy instead of destroying the state, at a potentially modest quality cost. This does not contradict Belady’s result, which applies to policies that keep or discard whole objects: a quarter-size copy lets the same budget hold up to four times as much session state, and MRR’s rate at B nearly equals LRU’s at 4B (13.56% versus 13.35%). The advantage is therefore one of capacity, not foresight; at the tightest budget ⋆ , FNU is lower (67.60% versus 72.92%; Figure 11a). Demoting displaced evaluated, 0.02 Khr KV to a separate host-memory (CPU DRAM) tier, rather than keeping the int4 copy in HBM, lowers the rate further: a host tier equal to the pooled HBM budget gives 9.46%, and one twice as large gives 5.57%, close to the 5.4% first-touch floor (dashed line in Figure 11) that a

17

tier four times the budget reaches. Doing so moves 1,026.1 TB of int4 KV between host and GPU memory in the hour, 285.0 GB/s on average. The replay charges no promotion latency, dequantization, or transfer cost and treats a promoted int4 copy as equivalent to an fp16 hit. Counting per-group quantization metadata enlarges each int4 copy beyond a quarter of fp16 and raises the in-HBM MRR rate to between 14.99% and 19.25%. These results therefore motivate retaining recoverable KV representations across memory tiers, subject to the quality, latency, and bandwidth costs the replay omits.

5.4

Policy Separation Differs Across Workloads

The capacity result is not uniform across workloads. Assist has comparable reconstruction rates of 12.6%, 11.9%, and 11.1% for LRU, FNU, and MRR, respectively, all close to its 10.8% first-touch floor, which leaves better policies little to recover at the 10% pooled budget. Chat, despite smaller per-session working sets, separates the three policies at 40.7%, 28.4%, and 22.5%, above its 16.4% first-touch floor; its 9,749 TB hold-all-live ceiling over 24 h comes from concurrency, not session size. These workloads expose two routes to fleet GPU-memory pressure: deep agentic sessions create large per-session working sets, while Chat creates a large aggregate footprint through concurrency. Capacity planning must therefore account for both the distribution of session working sets and the number of concurrently live sessions; neither session size nor workload category alone determines whether KV residency becomes HBM capacity-bound.

5.5

Routing: Deployment-local KV Creates an Independent Floor

The replay so far models one KV object per session. To represent deployment-local state, we repeat the busiest-hour Research replay with KV objects keyed by both session and deployment. Over the same 176,007 calls, this increases the number of modeled cache objects from 9,486 to 35,588, because a session routed among deployments may establish separate state at each one. With unlimited residency capacity, reconstruction occurs only on the first access to each modeled cache object. Session-only keying gives the 5.4% first-touch floor of Section 5.2, while session-and-deployment keying raises it to 20.2%. Deployment-local state thus imposes a routing floor that neither more memory nor better eviction within a deployment can remove; lowering it requires deployment affinity, KV replication, or migration. This replay assumes that KV remains local to the deployment identifier reported by the telemetry and is not automatically shared across deployments. Together, the capacity and placement results identify two distinct sources of reconstruction. Limited GPU memory displaces reusable KV even under clairvoyant FNU eviction, while deployment changes make state unavailable under deployment-local caching. Recoverable retention, such as MRR, mitigates the first; affinity-aware routing, replication, or migration mitigates the second.

5.6

Lessons learned

Relative to prior serving work, our production measurements offer four lessons. First, requestlevel KV management [20, 50] does not consider reuse across calls within a session. In Research, 89.3% of calls are internal and 80.3% of context tokens hit the prefix cache, motivating session-

18

level residency decisions. Second, retaining KV throughout a session’s lifetime can overestimate demand: a one-hour idle cap reduces the Research ceiling from 501.8 to 140.8 TB, while reconstruction increases from 4.07% to 4.53%. Third, eviction policy has limited impact under tight memory: clairvoyant FNU reconstructs 31.1% of calls, while retaining an int4 copy within the same budget reduces this to 13.6%. Finally, locality-aware routing [29, 38, 45] is relevant in production, since mean reuse falls from 88.9% to 29.0% after a same-model deployment change.

5.7

Open Problems and Limitations

Our measurements expose two complementary opportunities for agentic LLM serving: preserving reusable KV under limited GPU memory and coordinating its placement with application control flow. We frame these as research directions rather than evaluated system designs. Reversible, multi-resolution KV residency. MRR isolates one effect, keeping displaced KV recoverable instead of destroying it; it does not show that int4 retention is the best mechanism. Residency mechanisms leave the context unchanged and decide only whether, where, and at what precision its KV is kept: discard and recompute, exact offload to host or remote storage with a fetch on resumption [41, 29, 1], or quantized retention as in MRR [32, 17, 30]. Context reduction, such as compaction, rolling summaries, or agent memories that keep a pointer and retrieve content on demand [19, 37, 9], changes the context itself: it can change the response, and because prefix caching reuses only an unchanged prefix, each edit invalidates the cached KV after it. Context reduction and KV residency can be combined: reduction shrinks Ws , residency retains the remaining state, and our replay already sizes sessions after the application’s own reductions (Section 3.2). Exact offload avoids MRR’s precision loss but moves four times the bytes of an int4 copy; it beats reconstruction only if fetch bandwidth exceeds βRprefill , at least about 1 GB/s for fp16, and its latency adds to the resumed call unless the fetch is issued during the preceding idle gap. Hybrids such as VeriCache decode from a compressed copy and verify against a full-precision copy fetched from a lower tier [57]. The open problem is to decide, for each session, whether its state should be retained exactly, quantized, offloaded, summarized, or discarded and reconstructed, given the workload’s idle gaps, reuse and quality tolerance, while meeting latency and output-quality objectives. Application-aware placement and lifecycle control. Model-class predictability, excursionconditioned reuse (Section 3.4), and deployment continuity (Sections 4.4 and 5.5) motivate coordinating application control flow with KV placement and retention; a context reduction (Section 3.2) likewise lowers the session’s current KV requirement and provides a natural point for releasing or demoting superseded state. Current serving interfaces reveal little about an agent’s expected next model, tool interval, context reduction, or likelihood of resumption. Prior systems expose parts of this structure through semantic variables, program-level scheduling, cache-aware routing, and application hints [26, 33, 16, 38, 2]. A broader interface could allow the orchestrator to communicate model role, expected continuation, and session-lifecycle events explicitly, while the serving layer returns memory pressure and placement constraints. Designing this interface without compromising isolation, load balance, or failure tolerance remains an open problem. Such mechanisms also require portable measurement. In our traces, three workloads report provider-cached tokens per call, while Code uses session-cumulative cache_read

19

accounting. A provider-neutral definition of cache objects, KV reuse, deployment continuity, and reconstruction would enable residency policies to be compared consistently across serving stacks [36, 50, 52]. Scope and limitations. Our results describe four Microsoft M365 workloads observed over specific windows between late June and early August 2026 and should not be interpreted as universal properties of agentic or chat serving. Cached tokens reported by the LLM endpoint measure the reuse realized by the deployed infrastructure, whereas our byte-valued working sets scale linearly with the assumed per-token KV size β. The deployment identifier, the finest reported placement label, may not expose worker-level placement or cache sharing. Finally, several results depend on analysis choices: the model-class boundary (Section 3.4.1), the observation window, the liveness definition (Section 5.1), the cache-object key (Section 5.5), and the pooling of the replay budget across replicas (Section 5.2). We describe each analysis choice alongside the corresponding result. Studying the open problems above requires workloads with the structure measured here; because the production traces cannot be released, the next section addresses this gap.

6

AgenticTraceSim: Reproducible Agentic Traces

The production traces contain sensitive serving telemetry and cannot be released without risking client and operational confidentiality. We therefore plan to release AgenticTraceSim, a parametric trace generator that captures the principal distributions and dependencies observed in Research without exposing production events or user content, together with a residency replay (LRU, FNU, and an unbounded-tier MRR variant whose int4 tier never fills) that accepts synthetic or user-supplied call streams. AgenticTraceSim models six workload components: time-varying session arrivals, session depth (a lognormal mixture), context growth, model-class transitions, per-call reuse, and idle gaps. The components are coupled to preserve the relationships important to residency, including depth with peak context and duration, and Spine reuse with intervening model-class excursions. The generator exposes workload knobs for cache effectiveness, the number of model classes, and the session arrival rate, and the planned release also includes SR and SD population profiles, refitted to those subsets, for sensitivity analysis; these do not model deployment transitions and are not a placement-policy simulator. We validate both the generated distributions and their principal dependencies. For depth, idle gaps, and per-call reuse, the Kolmogorov–Smirnov distance from the measured Research distributions is at most 0.10. Context is less closely matched at the per-call level because it is calibrated to reproduce the residency-relevant per-session peak: generated P50/P90/P99 working sets are 21.6/55.9/82.5 GiB, versus 25.7/59.2/81.6 GiB measured. The generator also reproduces the measured depth–peak-context and depth–duration rank correlations within 0.05. Synthetic traces also preserve the downstream residency conclusions. Under the same re⋆ , so LRU rates play, with budgets as fractions of each stream’s own no-eviction peak (not Khr are not comparable to Section 5.2’s 62.9%), matched real and synthetic workloads retain the reconstruction-rate ordering LRU ≥ FNU ≥ unbounded-tier MRR at every evaluated budget (Figure 12). This unbounded-tier variant of multi-resolution retention demotes displaced KV to a separate int4 tier sized to the no-eviction peak, so the tier never fills and its rate equals 20

reconstruction rate Mπ(B) (%)

LRU FNU unbounded-tier MRR measured synthetic

80 60 40 20 0

1

2

5

10

20

50

budget B (% of no-eviction peak)

100

Figure 12: AgenticTraceSim preserves downstream residency-policy behavior. Matched measured and synthetic workloads retain the ordering LRU ≥ FNU ≥ unboundedtier MRR at every evaluated budget. This MRR variant uses a separate int4 tier that never fills, unlike the capacity-shared variant in Figure 11. The measured and generated LRUto-multi-resolution improvements are 2.45× and 2.50×, respectively, with overlapping bootstrap intervals.

the first-touch floor; it is not the capacity-shared variant of Section 5.3. The LRU-to-multiresolution improvement is 2.45× on the matched measured Research sample and 2.50× on the synthetic trace, with overlapping bootstrap intervals. A marginals-only generator fails to reproduce this improvement, showing that the modeled dependencies are necessary for policy evaluation. The generator and replay produce the synthetic side of every comparison in this section, including the LRU ≥ FNU ≥ unbounded-tier MRR ordering. The generator reproduces shapes, ratios, and orderings, not absolute fleet magnitudes, which depend on production arrival volume, observation window, and placement topology. The measured side of these comparisons, the capacity-shared MRR replay of Section 5.3, and the deployment-local routing and placement results (the generator emits no deployment identifiers) require the production traces; the production values in Sections 4 and 5 therefore remain measured results rather than generator outputs. The AgenticTraceSim trace generator and additional appendices offering supplementary material on the trace analysis will be released in the near future.

7

Related Work

Agentic workload characterization. TraceLab characterizes Claude Code and Codex sessions, the GitHub Copilot study measures production coding-agent traffic at scale, and the ReAct study instruments ReAct-style agents on five benchmarks [66, 28, 60]; Section 2.1 and Table 1 compare them with this work. KVCache in the Wild characterizes production KV-cache reuse in single- and multi-turn requests at a large cloud provider [52]; we focus on multi-call agentic sessions and their deployment placement. State retention, migration and caching bounds. CachedAttention and Pensieve retain multi-turn KV state across memory tiers [12, 59], while HCache reconstructs KV state from stored intermediate activations [13]. Llumnix supports cross-instance request and KV-cache migration [46]. These systems provide candidate mechanisms for the capacity and locality requirements characterized here; we do not compare their implementation-level latency or transfer costs. Moreover, because FNU is not optimal for variable-size objects, we treat it only as a clairvoyant reference policy; flow-based offline bounds provide a more principled optimality benchmark but are outside our present replay [6]. 21

LLM serving and KV management. PagedAttention and vLLM established paged KV management for high-throughput inference [20]; Sarathi-Serve co-schedules prefill and decode with chunked prefills [3], whereas Splitwise, DistServe, and Mooncake run the two phases on separate resources and transfer KV between them [40, 65, 41]. Prefix-caching systems improve reuse via structured caching, cache-aware scheduling, and distributed KV transport [64, 15, 45, 30, 29]. These systems provide KV-management mechanisms; we measure the session- and fleet-level demand they must serve in production agentic workloads. Agent-aware serving and orchestration. Parrot, Teola, ALTO, Autellix, and Pie optimize serving for whole LLM applications rather than individual requests [26, 48, 43, 33, 16]. More recent systems use agent semantics for KV prefetching, pruning, and cache management, and for admission control, program-aware scheduling, and GPU allocation [38, 23, 7, 8, 18, 53]. InferCept and Continuum decide whether to keep, discard, or swap a request’s KV while it waits on a tool call [1, 22], a per-request counterpart of the idle cap in Section 5.1. Model-routing systems choose among models, reasoning modes, or quantized model variants to balance quality, cost, and latency [35, 51, 10, 47]. Our measurements inform these directions by quantifying model-class predictability, idle-gap structure, deployment-conditioned reuse, and the separate reconstruction costs of limited memory and routing. Context reduction and external memory. Prompt and context compression reduce the state that must be materialized into KV [55, 39, 34, 19], while agent-memory systems externalize information beyond the active context window [37, 9, 42, 56, 25]. These approaches determine what enters the model context; we characterize how the resulting KV state is reused, retained, and placed across a production serving fleet; Section 5.7 discusses how the two compose. KV compression and tiering. When KV capacity is insufficient, prior work evicts selected tokens [62, 31, 54, 24, 14], quantizes retained state [32, 17, 61], or moves state across memory tiers [44, 21, 30, 57]. These techniques define representations and storage tiers for displaced KV; our trace-driven replay asks a complementary workload-level question: how much reconstruction remains under destructive eviction, and how much of it a retained lower-precision copy removes.

8

Conclusion

We characterized the KV working set of four production Microsoft M365 workloads spanning three agentic applications and enterprise chat. Agentic sessions repeatedly re-enter LLMs through internal model-tool loops, accumulate context, and reuse much of it across idle intervals; in Research, a P99 session’s estimated KV exceeds one H100’s HBM, and a predictable Spine class holds most call-context KV bytes. Retaining all of this state is costly, and the resulting memory pressure is not determined by per-session size alone: deep agentic sessions create large individual working sets, while enterprise chat creates substantial fleet demand through concurrency. Reclaiming state after one idle hour lowers the Research hold-all-live ceiling by 3.6× while raising the replayed reconstruction rate from 4.07% to 4.53%. Beyond idle reclamation, the busiest-hour residency replay identifies two distinct sources of reconstruction. Under a pooled

22

memory budget equal to 10% of the busiest-hour current-context peak, even clairvoyant destructive eviction by Belady’s furthest-next-use rule (FNU) reconstructs context for 31.1% of Research calls, while int4 MRR, which keeps evicted KV as a recoverable copy, reduces this rate to 13.6%. Separately, even with unlimited memory, modeling KV as deployment-local raises the first-touch floor from 5.4% to 20.2%. Efficient agentic serving therefore requires both memory mechanisms that preserve recoverable KV under capacity pressure and placement mechanisms that maintain or move reusable state across deployments. Because the production traces contain sensitive enterprise telemetry, we plan to release AgenticTraceSim and a residency replay for synthetic or user-supplied call streams, for reproducible counterfactual studies. The generator preserves the workload dependencies needed to reproduce the reconstruction-rate ordering of LRU, FNU, and unbounded-tier MRR without exposing production events. Together, the measurements and the planned artifacts support the study of fleet-scale KV residency and routing policies that account for session working-set size, expected reuse, idle duration and deployment placement.

23

References [1] Reyna Abhyankar, Zijian He, Vikranth Srivatsa, Hao Zhang, and Yiying Zhang. InferCept: Efficient intercept support for augmented large language model inference. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 81–95. PMLR, 2024. [2] Saurabh Agarwal, Marco Laju, Jayanth Srinivasa, Myungjin Lee, and Aditya Akella. Software-defined agentic serving. arXiv preprint arXiv:2601.03197, 2026. [3] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve. In Proc. USENIX OSDI, 2024. [4] Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multihead checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895–4901, Singapore, 2023. Association for Computational Linguistics. [5] L. A. Belady. A study of replacement algorithms for a virtual-storage computer. IBM Systems Journal, 5(2):78–101, 1966. [6] Daniel S. Berger, Nathan Beckmann, and Mor Harchol-Balter. Practical bounds on optimal caching with variable object sizes. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 2(2):1–38, June 2018. [7] Anish Biswas, Kanishk Goel, Srivarshinee S, Jayashree Mohan, Alind Khare, Anjaly Parayil, Ramachandran Ramjee, and Chetan Bansal. Sutradhara: An intelligent orchestrator-engine co-design for tool-based agentic inference. arXiv preprint arXiv:2601.12967, 2026. [8] Qiaoling Chen, Zhisheng Ye, Tian Tang, Peng Sun, Boyu Tian, Guoteng Wang, Shenggui Li, Yonggang Wen, Zhenhua Han, and Tianwei Zhang. CONCUR: high-throughput agentic batch inference of llm via congestion-based concurrency control. arXiv preprint arXiv:2601.22705, 2026. [9] Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. Mem0: Building production-ready AI agents with scalable long-term memory. In ECAI 2025: 28th European Conference on Artificial Intelligence, volume 413 of Frontiers in Artificial Intelligence and Applications, pages 2993–3000. IOS Press, 2025. [10] Yinwei Dai, Zhuofu Chen, Anand Iyer, and Ravi Netravali. Aragog: Just-in-time model routing for scalable serving of agentic workflows. arXiv preprint arXiv:2511.20975, 2025. [11] Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Erkang Zhu, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, Peter Chang, Ricky Loynd, Robert West, Victor Dibia, Ahmed Awadallah, Ece Kamar, Rafah Hosn, and Saleema Amershi. Magentic-One: A generalist multi-agent system for solving complex tasks. arXiv preprint arXiv:2411.04468, 2024.

24

[12] Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. Cost-efficient large language model serving for multi-turn conversations with cachedattention. In Proceedings of the 2024 USENIX Annual Technical Conference (USENIX ATC 2024), pages 111–126. USENIX Association, 2024. [13] Shiwei Gao, Youmin Chen, and Jiwu Shu. Fast state restoration in LLM serving with HCache. In Proceedings of the Twentieth European Conference on Computer Systems (EuroSys 2025), pages 128–143, New York, NY, USA, 2025. Association for Computing Machinery. [14] Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive KV cache compression for LLMs. In International Conference on Learning Representations, 2024. [15] In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. Prompt cache: Modular attention reuse for low-latency inference. In Proceedings of Machine Learning and Systems, volume 6, 2024. [16] In Gim, Zhiyao Ma, Seung-seob Lee, and Lin Zhong. Pie: A programmable serving system for emerging LLM applications. In Proceedings of the 31st ACM Symposium on Operating Systems Principles (SOSP), pages 415–430. Association for Computing Machinery, 2025. [17] Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. KVQuant: Towards 10 million context length LLM inference with KV cache quantization. In Proc. NeurIPS, 2024. [18] Hao Kang, Ziyang Li, Weili Xu, Xinyu Yang, Yinfang Chen, Junxiong Wang, Beidi Chen, Tushar Krishna, Chenfeng Xu, and Simran Arora. ThunderAgent: A simple, fast and program-aware agentic inference system. arXiv preprint arXiv:2602.13692, 2026. [19] Minki Kang, Wei-Ning Chen, Dongge Han, Huseyin A. Inan, Lukas Wutschitz, Yanzhi Chen, Robert Sim, and Saravan Rajmohan. ACON: Optimizing context compression for long-horizon LLM agents. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pages 55321–55375. PMLR, 2026. [20] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP), pages 611–626. Association for Computing Machinery, 2023. [21] Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. InfiniGen: Efficient generative inference of large language models with dynamic KV cache management. In Proc. USENIX OSDI, 2024. arXiv:2406.19707. [22] Hanchen Li, Qiuyang Mang, Runyuan He, Qizheng Zhang, Huanzhi Mao, Xiaokun Chen, Hangrui Zhou, Alvin Cheung, Joseph E. Gonzalez, and Ion Stoica. Continuum: Efficient 25

and robust multi-turn LLM agent scheduling with KV cache time-to-live. In ICLR 2026 Workshop on Lifelong Agents: Learning, Aligning, Evolving, 2026. [23] Junjie Li, Jiong Lou, and Jie Li. IntentKV: Cross-turn intent-aware KV cache pruning for agent inference. arXiv preprint arXiv:2606.09916, 2026. [24] Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. SnapKV: LLM knows what you are looking for before generation. In Proc. NeurIPS, 2024. arXiv:2404.14469. [25] Zhiyu Li, Shichao Song, Hanyu Wang, Simin Niu, Ding Chen, Jiawei Yang, Chenyang Xi, Huayi Lai, Jihao Zhao, Yezhaohui Wang, Junpeng Ren, Zehao Lin, Jiahao Huo, Tianyi Chen, Kai Chen, Kehang Li, Zhiqiang Yin, Qingchen Yu, Bo Tang, Hongkang Yang, ZhiQin John Xu, and Feiyu Xiong. MemOS: An operating system for memory-augmented generation (MAG) in large language models. arXiv preprint arXiv:2505.22101, 2025. [26] Chaofan Lin, Zhenhua Han, Chengruidong Zhang, Yuqing Yang, Fan Yang, Chen Chen, and Lili Qiu. Parrot: Efficient serving of LLM-based applications with semantic variable. In Proc. USENIX OSDI, 2024. [27] Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. QServe: W4A8KV4 quantization and system co-design for efficient LLM serving. In Proceedings of Machine Learning and Systems (MLSys), 2025. [28] Banruo Liu, Haoran Qiu, Íñigo Goiri, Rodrigo Fonseca, Ricardo Bianchini, and Esha Choukse. Agentic coding in the wild: Characterizing GitHub Copilot traces at production scale. arXiv preprint arXiv:2608.00101, 2026. [29] Yuhan Liu, Yihua Cheng, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaoting Feng, Yuyang Huang, Samuel Shen, Rui Zhang, Kuntai Du, and Junchen Jiang. LMCache: An efficient KV cache layer for enterprise-scale LLM inference. arXiv preprint arXiv:2510.09665, 2025. [30] Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, and Junchen Jiang. CacheGen: KV cache compression and streaming for fast large language model serving. In Proceedings of the ACM SIGCOMM 2024 Conference. Association for Computing Machinery, 2024. [31] Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression at test time. In Proc. NeurIPS, 2023. arXiv:2305.17118. [32] Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. KIVI: A tuning-free asymmetric 2bit quantization for KV cache. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 32332–32344. PMLR, 2024.

26

[33] Michael Luo, Xiaoxiang Shi, Colin Cai, Tianjun Zhang, Justin Wong, Yichuan Wang, Chi Wang, Yanping Huang, Zhifeng Chen, Joseph E. Gonzalez, and Ion Stoica. Autellix: An efficient serving engine for LLM agents as general programs. arXiv preprint arXiv:2502.13965, 2025. [34] Jesse Mu, Xiang Lisa Li, and Noah Goodman. Learning to compress prompts with gist tokens. In Proc. NeurIPS, 2023. [35] Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. RouteLLM: Learning to route LLMs from preference data. In Proc. ICLR, 2025. [36] OpenAI. Prompt caching. https://developers.openai.com/api/docs/guides/ prompt-caching, 2025. OpenAI API documentation, accessed October 2026. [37] Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. MemGPT: Towards LLMs as operating systems. arXiv preprint arXiv:2310.08560, 2023. [38] Zaifeng Pan, Ajjkumar Patel, Yipeng Shen, Zhengding Hu, Yue Guan, Wan-Lu Li, Lianhui Qin, Yida Wang, and Yufei Ding. KVFlow: Efficient prefix caching for accelerating LLMbased multi-agent workflows. In Proc. NeurIPS, 2025. [39] Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Ruhle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. LLMLingua-2: Data distillation for efficient and faithful task-agnostic prompt compression. In Findings of the Association for Computational Linguistics: ACL 2024, pages 963–981, Bangkok, Thailand, 2024. Association for Computational Linguistics. [40] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Inigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative LLM inference using phase splitting. In Proceedings of the 51st Annual International Symposium on Computer Architecture, pages 118–132. IEEE, 2024. [41] Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: Trading more storage for less computation — a KVCache-centric architecture for serving LLM chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST 25), pages 155–170, Santa Clara, CA, February 2025. USENIX Association. [42] Preston Rasmussen, Pavlo Paliychuk, Travis Beauvais, Jack Ryan, and Daniel Chalef. Zep: A temporal knowledge graph architecture for agent memory. arXiv preprint arXiv:2501.13956, 2025. [43] Keshav Santhanam, Deepti Raghavan, Muhammad Shahir Rahman, Thejas Venkatesh, Neha Kunjal, Pratiksha Thaker, Philip Levis, and Matei Zaharia. ALTO: An efficient network orchestrator for compound AI systems. In Proceedings of the 4th Workshop on Machine Learning and Systems (EuroMLSys), pages 117–125. Association for Computing Machinery, 2024. 27

[44] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Re, Ion Stoica, and Ce Zhang. FlexGen: High-throughput generative inference of large language models with a single GPU. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 31094–31116. PMLR, 2023. [45] Vikranth Srivatsa, Zijian He, Reyna Abhyankar, Dongming Li, and Yiying Zhang. Preble: Efficient distributed prompt scheduling for LLM serving. In Proc. ICLR, 2025. arXiv:2407.00023. [46] Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. Llumnix: Dynamic scheduling for large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 173–191, Santa Clara, CA, July 2024. USENIX Association. [47] Zuhair Ahmed Khan Taha, Mohammed Mudassir Uddin, and Shahnawaz Alam. AgentCompress: Task-aware compression for affordable large language model agents. arXiv preprint arXiv:2601.05191, 2026. [48] Xin Tan, Yimin Jiang, Yitao Yang, and Hong Xu. Teola: Towards end-to-end optimization of LLM-based applications. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, pages 1302–1316. Association for Computing Machinery, 2025. [49] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. [50] vLLM Project. Automatic prefix caching. https://docs.vllm.ai/en/stable/ design/prefix_caching/, 2026. vLLM documentation, accessed October 2026. [51] Chen Wang, Xunzhuo Liu, Yuhan Liu, Yue Zhu, Xiangxi Mo, Junchen Jiang, and Huamin Chen. When to reason: Semantic router for vLLM. In NeurIPS ML for Systems Workshop, 2025. [52] Jiahao Wang, Jinbo Han, Xingda Wei, Sijie Shen, Dingyan Zhang, Chenguang Fang, Rong Chen, Wenyuan Yu, and Haibo Chen. KVCache cache in the wild: Characterizing and

28

optimizing KVCache cache at a large cloud provider. In 2025 USENIX Annual Technical Conference (USENIX ATC 25), 2025. [53] Otto White, Marcel Wagenländer, Britannio Jarrett, Xijin Zhou, Yanda Tao, Pedro Silvestre, Guo Li, Huanzhou Zhu, Lluís Vilanova, and Peter Pietzuch. Scepsy: Serving agentic workflows using aggregate LLM pipelines. arXiv preprint arXiv:2604.15186, 2026. [54] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In Proc. ICLR, 2024. [55] Fangyuan Xu, Weijia Shi, and Eunsol Choi. RECOMP: Improving retrieval-augmented LMs with compression and selective augmentation. In Proc. ICLR, 2024. arXiv:2310.04408. [56] Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A-Mem: Agentic memory for LLM agents. In Advances in Neural Information Processing Systems, volume 38, 2025. [57] Jiayi Yao, Samuel Shen, Kuntai Du, Shaoting Feng, Dongjoo Seo, Rui Zhang, Yuyang Huang, Yuhan Liu, Shan Lu, and Junchen Jiang. VeriCache: Turning lossy KV cache into lossless LLM inference. arXiv preprint arXiv:2605.17613, 2026. [58] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023. [59] Lingfan Yu, Jinkun Lin, and Jinyang Li. Stateful large language model serving with pensieve. In Proceedings of the Twentieth European Conference on Computer Systems (EuroSys 2025), pages 144–158, New York, NY, USA, 2025. Association for Computing Machinery. [60] Yichao Yuan, Ankita Nayak, Souvik Kundu, and Nishil Talati. Agentic AI workload characteristics. In 2026 IEEE International Symposium on Workload Characterization (IISWC), 2026. To appear. [61] Yuxuan Yue, Zhihang Yuan, Haojie Duanmu, Sifan Zhou, Jianlong Wu, and Liqiang Nie. WKVQuant: Quantizing weight and key/value cache for large language models gains more. arXiv preprint arXiv:2402.12065, 2024. [62] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2O: Heavy-hitter oracle for efficient generative inference of large language models. In Proc. NeurIPS, 2023. [63] Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. Atom: Low-bit quantization for efficient and accurate LLM serving. In Proceedings of Machine Learning and Systems (MLSys), 2024. [64] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. SGLang: Efficient execution of structured language model programs. In Proc. NeurIPS, 2024. 29

[65] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving. In Proc. USENIX OSDI, 2024. [66] Kan Zhu, Mathew Jacob, Chenxi Ma, Yi Pan, Stephanie Wang, Arvind Krishnamurthy, and Baris Kasikci. TraceLab: Characterizing coding agent workloads for LLM serving. arXiv preprint arXiv:2606.30560, 2026.

30

Related documents

• The KV Cache Working Set: Online Capacity Planning for LLM Inference Systems#1064290
• PDEU-Bench: Benchmarking the Personalized Planning Lifecycle of Tool-Calling LLM Agents#1129540
• Beyond Agent Architecture: Execution Assumptions and Reproducibility in LLM-Based Trading Systems#1061727
• Not Every Call Needs a Frontier Model: Per-Call-Site Evaluation of Small Language Models in a Deployed Agentic Home-Automation System#1233060
• MicroIntent: Intent-Based Placement Strategy for Microservice Application in the Compute Continuum Using LLMs#667986