O BSERVATION , N OT P REDICTION : C ONVERSATION -L EVEL D ISAGGREGATED S CHEDULING FOR AGENTIC S ERVING
arXiv:2606.01839v1 [cs.DC] 1 Jun 2026
Jianru Ding 1 Ryien Hosseini * 1 Pouya Mahdi Gholami * 1 Mingyuan Xiang * 1 Henry Hoffmann 1
A BSTRACT LLM-based agents resolve a user task through many turns of dependent inference and tool calls, producing a workload whose total cost is unknown when the task arrives. Existing multi-turn systems keep the turn as the scheduling unit and decide, turn by turn, whether to disaggregate prefill from decode. That decision rests on the turn’s decode length, tool behavior, and KV growth, quantities that are not observable when the scheduler must act, forcing the system to predict them. We show this dependence on prediction is imposed by the scheduling unit, not the workload. Raising the scheduling unit from the turn to the conversation converts turn-level irregularity into a stable, two-phase structure: 1) a compute-bound turn-1 prefill followed by 2) a long, memory-bound tail. Thus, with the conversation as the scheduling unit, placement reduces to reading the first-turn input length and per-decoder KV occupancy, both directly observable. We instantiate this principle in ConServe, which routes the first-turn prefill to a high-throughput prefiller, transfers the KV cache exactly once, and pins the conversation to a single decoder for its entire tail, with no learned model of decode-side cost. Against a per-turn prediction baseline, ConServe reduces p95 time-to-first-effective-token (the latency of a conversation’s first user-visible output) by 51.08% and improves energy efficiency by 7.51% while preserving last-turn TBT and SLOs; mapping the two phases onto heterogeneous GPU tiers adds a further 22.75% in energy efficiency. 1
I NTRODUCTION
ing. For single-shot inference this approach is harmless, as the unit the system schedules coincides with the unit the user values. Agentic workloads break this coincidence. What the user values is the conversation’s final, externally meaningful output, yet most intermediate turns emit only tool calls that are never read by the user. The result is a scheduling–value mismatch: the system optimizes perturn latency while the user waits on a conversation-level response. Consequently, existing serving systems that optimize each turn in isolation optimize the wrong objective. To name the value the user actually cares about, we introduce time-to-first-effective-token (TTFET): the time until the first token of genuine, user-visible progress, as opposed to the internal tokens that drive tool calls. TTFET is thus a property of a conversation, not of any turn.
Traditional large language model (LLM) serving is built around a simple contract: a request arrives, the model prefills its prompt and decodes a response, and the request departs (Vaswani et al., 2017). Agentic workloads break this contract. A user task is no longer resolved by a single model invocation but by a sequence of dependent inference calls interleaved with tool use, environmental feedback, and accumulating state (Yao et al., 2023). The served object is therefore not a request but a stateful, multi-turn program which moreover exhibits a characteristic temporal asymmetry: its first turn encodes a long task description (e.g., instructions, repository state, retrieved context) in a single compute-bound prefill, while later turns typically append only short tool outputs and observations, dominated by memory-bound decoding over a key-value (KV) cache that only grows. A conversation is therefore, typically, structured as one heavy prefill followed by a long, light, memory-bound tail.
Existing multi-turn serving systems do recognize that later turns differ from the first. Building on prefill–decode disaggregation, which separates the compute-bound prefill from latency-sensitive decoding onto distinct workers (Zhong et al., 2024), these systems add append-prefill routing, KVcache retention, and tool-aware scheduling to avoid recomputing shared state across turns (Li et al., 2025; Abhyankar et al., 2024; Luo et al., 2025). Yet these designs share a single design principle: the decision unit remains the turn. For each turn, such a system decides whether to disaggregate it (i.e., whether to pay a remote prefill and a KV transfer or
This shift is not merely quantitative. Current generic LLM serving systems typically take the request (equivalently, a single prefill–decode turn) as the atomic unit of schedul*
Equal contribution 1 University of Chicago. Correspondence to: Jianru Ding <[email protected]>. Preprint
1
process it locally) and reaches that decision by profiling or predicting the turn’s cost (He et al., 2026; Li et al., 2026).
straction at the conversation level: it routes the first-turn prefill as the compute-bound phase, transfers the resulting KV cache exactly once, and then runs the first-turn decode and every subsequent append-prefill/decode turn as a single memory-bound phase. Under ConServe, a conversation is therefore pinned for its entire memory-bound tail, avoiding repeated KV movement across turns. This paradigm shows that placement should be reactive rather than predictive. New conversations are assigned using observable quantities: input-token counts for the initial prefill and decoder KV-cache utilization for the memory-bound tail. ConServe overprovisions decode replicas to absorb variation in conversation length; when a prefiller is saturated, or when decoders approach saturation under bursts of long conversations, the scheduler routes the next conversation elsewhere rather than migrating one already placed.
In this work, we first argue that this per-turn scheduling decision is structurally brittle. The cost that drives the decision cannot be known in advance: a turn’s prefill is fixed by its input length and can be read directly, but other key behavior (e.g., how many tokens it will decode, whether it will call a tool, and how much it will enlarge the KV cache) is unobservable at the moment the scheduler must act. Yet such decode-side quantities are what determine whether disaggregating the turn pays off, so per-turn placement must predict them. Any predictor, however welltuned, will eventually misjudge a turn and route it incorrectly (we characterize this prefill/decode predictability gap empirically in Section 3). We argue this prediction is not a requirement of the workload but of the scheduling unit: partitioning a conversation into independently scheduled turns discards precisely the structure that would have made placement deterministic. Thus, in prior work, the unit, not the predictor, is the source of the fragility.
We evaluate ConServe on representative agentic workloads (Jimenez et al., 2024), using a Qwen3-0.6B model backbone and a NVIDIA A40 GPU cluster, against collocated, fully disaggregated, and per-turn prediction-based baselines. Compared to the per-turn baseline (He et al., 2026), ConServe reduces p95 TTFET by 51.08% and improves energy efficiency by 7.51% while maintaining similar lastturn time-between-tokens (TBT) and preserving servicelevel objectives (SLOs). It does so with a deliberately minimal policy: one disaggregation per conversation and no learned cost model. On heterogeneous hardware, ConServe preserves these latency results while improving energy efficiency by a further 22.75% over its homogeneous configuration. These gains show that the relevant abstraction is the conversation, not the turn. We therefore present conversation-level scheduling as a principled paradigm for agentic serving, and ConServe as a minimal instantiation of this principle. We make the following contributions:
Key Insight. We schedule the conversation, not the turn. At turn granularity, agentic serving is irregular: decode lengths, tool calls, append-prefill sizes, and KV growth are difficult to infer when placement decisions are made. At conversation granularity, this irregularity collapses into a stable two-phase structure: an initial compute-bound prefill that materializes task state, followed by a memorybound trajectory consisting of the first decode and all subsequent append-prefill/decode turns. Later turns may contain append-prefill work, but their input is small relative to the accumulated KV cache, so execution is dominated by KV movement and attention over existing state. This restores the original prefill–decode abstraction but at conversation-level granularity: one compute-bound phase, one memory-bound phase, and a KV transfer at the boundary.
• Conversation-level scheduling. We identify a scheduling–value mismatch in agentic serving: systems optimize per-turn latency, while users wait for conversation-level outcomes. We argue that the conversation is therefore the correct unit of scheduling, and introduce time-to-first-effective-token (TTFET) and last-turn time-between-tokens (TBT) as metrics that capture this objective. Raising the unit to the conversation replaces turn-level prediction with directly observable state, eliminating the source of brittleness in prior multiturn schedulers.
Two consequences follow. First, scheduling need not predict turn-level outcomes such as future decode lengths or tool behavior; it can instead rely on the same kind of observable phase structure that classical serving systems already expose. Second, heterogeneous serving follows by construction: under heterogeneous cluster environments, the same phase split maps compute-intensive initial prefills to high-throughput GPUs and the long memory-bound tail to devices with sufficient memory bandwidth and capacity. ConServe. We instantiate this principle in ConServe1 , a conversation-level disaggregation scheduler for agentic serving. ConServe mirrors the original prefill–decode ab1
So named as to reflect two commitments: ConServe serves stateful, multi-turn agentic conversations and conserves energy by spending it only on the phase that requires it.
2
• ConServe. We introduce a disaggregation scheduler that routes first-turn prefills to high-power GPUs, transfers KV state exactly once, and executes a conversation’s entire tail locally on a separate (possibly low-power) decoder. Committing to one placement per conversation from observable signals allows ConServe to work without a learned cost model.
• Empirical demonstration of structural brittleness. We show that per-turn scheduling is brittle in a structural sense: under a per-turn prediction baseline, SLO violations and energy efficiency degrade linearly in prediction error rate, while ConServe’s conversation-level placement holds constant by construction.
sumes generation with the tool result appended to the context (Yao et al., 2023; Schick et al., 2023; Shinn et al., 2023; Qin et al., 2024). We refer to the full sequence of turns required to complete a single user request as a conversation. A conversation begins with a detailed, initial prompt that establishes the task and environment context, followed by a variable number of turns, each consisting of an LLM generation step and a subsequent tool invocation whose result is fed back as additional input (Liu et al., 2024b). To improve context efficiency, modern agentic systems frequently restrict tool-call outputs to task-relevant information, avoiding the inclusion of complete files, logs, or other large artifacts in the prompt (Anthropic, 2026).
• Heterogeneity and energy as corollaries. Because placement reads only observable states, tier-aware allocation and conversation-level energy optimization follow without additional machinery, and both are largely unaddressed by prior multi-turn systems.
2
BACKGROUND
2.1
LLM Serving Techniques
Unfortunately, neither collocated nor fully disaggregated prefill-decode architectures are well matched to this workload structure. In a collocated deployment, chunked prefill (Agrawal et al., 2024) splits the long prefill into smaller chunks interleaved with decode steps, but a turn1 prompt spanning thousands of tokens still occupies the GPU for many chunk iterations, delaying collocated decodes throughout the chunking window. Full prefill-decode disaggregation eliminates this interference by routing prefill and decode to dedicated GPUs (Zhong et al., 2024; Patel et al., 2024). In a multi-turn setting, however, this design routes every turn through the prefill node, disregarding a key structural property of agentic workloads: turn 2+ appends are short incremental prefills that could run directly on the decoder holding the cached prefix, with minimal contention (Figure 5). Full disaggregation forfeits this optimization opportunity in turn 2+ requests.
Transformer-based language models (Vaswani et al., 2017) generate text autoregressively in two stages (Radford et al., 2018; 2019). Prefill processes the input prompt in a single forward pass and produces a key-value (KV) cache for every input token. Decode then generates each subsequent token by reading the full KV cache and appending one new entry. Prefill is consequently compute-bound; decode is memory-bound, gated by KV cache bandwidth at every step (Pope et al., 2023; Patel et al., 2024). Efficient management of this growing KV cache is critical for serving throughput (Gao et al., 2024). A few techniques have been proposed and adopted in practice to address these characteristics and improve serving quality for user-facing LLM inference requests. Prefix caching (Zheng et al., 2024) reuses the KV cache across requests sharing a common input prefix, eliminating redundant prefill computation. Dynamic batching (Yu et al., 2022) batches forward passes from different requests at each iteration so finished requests can leave and new requests can join continuously, raising GPU utilization. Its downside is that a long prefill batched into ongoing decode iterations stalls the decoders for the prefill’s duration; chunked prefill (Agrawal et al., 2024) mitigates this by splitting long prefills into smaller chunks that interleave with decode, bounding the per-step stall. Prefill-decode (PD) disaggregation (Zhong et al., 2024; Patel et al., 2024) eliminates the problem entirely by assigning prefill and decode to separate replicas, exploiting the compute-bound vs memory-bound asymmetry to raise aggregate GPU utilization at the cost of transferring KV state between replicas. 2.2
Two concurrent systems route incremental prefills in multiturn disaggregated serving, and both retain the turn as the decision unit. AMPD (He et al., 2026) chooses, for each Turn 2+ prefill, between local execution and remote execution on a prefill node using real-time queue states and an offline cost model; remote execution reads the history KV from the decoder and writes the new KV back, a bidirectional transfer that departs from the one-way producerconsumer contract of standard disaggregation. PPD (Li et al., 2026) makes the same per-turn choice through an offline lookup table indexed by context length, input-output ratio, and load. Keeping the turn as the unit has two consequences. First, each decision conditions on quantities unavailable when it is made: AMPD’s cost model omits the collocated-decode interference that dominates local execution cost, and PPD’s table is indexed by a request’s output length, unknown at arrival. Second, both optimize per-turn TTFT and TBT, and neither measures TTFET or last-turn TBT, the conversation-level metrics that determine userperceived performance (§1). Both consequences trace to scheduling the turn rather than the conversation. The case for per-turn routing is also narrowest where agentic workloads sit: PPD’s dynamic routing is motivated by Turn 2+
Multi-turn Agentic Conversations
Agentic LLM applications (Chase, 2022; Richards, 2023) follow a multi-turn ReAct execution loop, in which the model reasons about a user request prompt, emits a structured tool call (e.g., an API invocation, code execution, or database query), suspends until the tool returns, and re-
3
requests with heavy prefills, but agentic appends are uniformly short (§3), the regime in which local execution already dominates and the per-turn decision collapses to a fixed local policy. ConServe adopts that policy at conversation granularity, conditioning only on observable state and optimizing the metrics users experience. 2.3
sands of tokens, whereas prompts appended in later turns are far smaller, on the order of hundreds. Output token counts, in contrast, are unpredictable and exhibit large variance. To build the quantitative foundation to understand the impact of serving techniques on agentic conversations, we profile agentic workloads across computational phases, inputs tokens, context lengths, and GPU power budgets and analyze the impact of each variable on agentic serving.
Serving with Heterogeneous GPUs
Several recent systems serve LLMs on heterogeneous GPU clusters. Some formulate GPU allocation as constrained optimization over cost or throughput, assigning GPU types to model replicas or partitions (Griggs et al., 2024; Mei et al., 2025; Jiang et al., 2024). Others route the prefill and decode phases onto distinct GPU tiers, matching the compute-bound prefill and memory-bound decode to hardware suited to each (Patel et al., 2024; Jiang et al., 2025b;a; Shi et al., 2024). Although these approaches differ in optimization formulation and partitioning strategy, they share a common insight: LLM inference places non-uniform demands on compute, memory capacity, and memory bandwidth, making it profitable to map workload to GPU capabilities rather than deploying a single homogeneous GPU type. All of this work, however, targets single-turn inference. At conversation granularity, agentic workloads exhibit a stable two-phase structure, a compute-bound firstturn prefill followed by a memory-bound trajectory, which remains unexploited.
3
Profiling Setup. Experiments are conducted on NVIDIA A40 GPUs running Qwen3-0.6B in bf16 precision. This model size is chosen to leave sufficient GPU memory headroom to accommodate the large and growing KV caches of multi-turn agentic conversations, making the experimental setup representative of realistic agentic serving conditions. For heterogenous GPU experiments, we power cap GPUs at 2/3 of their Thermal Dynamic Power (TDP) to represent lower-power GPUs that has less compute capacity but similar memory hardware. While results are model- and hardware-specific, the general qualitative observations hold broadly across model sizes and GPU generations as the underlying computational properties are architectural rather than configuration-specific. 3.1
We analyze the compute-bound prefill phase under multiturn agentic conversations. We characterize each turn individually and use Time-to-first-token (TTFT) – latency from request arrival to the first generated token – for comparison. Our results show a clear picture: (1) first turn prefill latency is predictably expensive, but (2) caching and reusing the KV cache could substantially improve prefill latency at later turns, and (3) KV cache transfer between model replicas is relatively inexpensive in agentic scenarios. These insights motivate the design of ConServe in §4.
C HARACTERIZATION OF AGENTIC W ORKLOADS Appended input
Output
100,000 10,000 1,000 100 10
prefill prefill ( 1024 tokens): 1.47 ns·L² + 10.5 µs·L + 13.8 ms (R 2=1.000) prefill w/ prefix-cache hit prefill w/ prefix-cache hit (median): 27 ms
6000 1
2
3
4
5
6
Turn index
7
8
9
TTFT (ms)
Tokens (log scale)
Prefill Phase
10
Figure 1. Input/Output token distribution of the first 10 turns from agentic traces. For the appended input, turn-1 is the input prompt, turn-2+ is the tool call response.
While LLM and agentic workloads share similar computational phases, agentic workloads behave differently due to their underlying characteristics, as shown empirically in Figure 1. The figure depicts the input and output token distributions of agentic traces generated from SWEbench bm25 13K with swe-agent, using Qwen3-Coder30B-A3B-Instruct as the trace-generation model. Agentic conversations begin with an initial input of tens of thou-
4000 2000 0
102
103
Input tokens
104
Figure 2. Characterization of TTFT under large inputs with and without caching. Uncached agentic prompts (≥ 104 ) incur quadratic latency overheads but remain predictable as shown in the label (R2 = 1.0).
4
Initial prefills are expensive but predictable.. We pro-
4.8 ms
60 40 20 102
104 103 Input tokens
10
35 30 25 20 15 10 5 0 5
Figure 3. KV-transfer overhead is constant at short input lengths (< 1024) and scales linearly at long inputs (≥ 1024). KV-transfer overhead between GPUs is significant when input is short but is dominated by Prefill latency in agentic workloads where inputs are tens of thousands of tokens long.
der high batch sizes or long contexts lengths where memory bandwidth is saturated. Given the long context length of agentic tasks, these workloads are memory-bound during the decode stage. Prefix-caching changes what impacts collocation overheads. We conduct a series of experiments to understand the impact of collocating prefill and decode stages on a per iteration basis. In the first experiment, we begin a number of decode tasks and then introduce a prefill request after 58 – 92 decode iterations and measure the collocated iterations latency. Figure 5.a depicts these results across a range of initial decode tasks (batch size) and prefill size requests. In our second experiment, we maintain a consistent batch size and introduce the prefill request after a range of iterations. Figure 5.b depicts these results across a range of prefill size requests. Both figures depict results with and without prefix-cache hits. As shown, prefix-caching significantly improves results by approximately one order of magnitude in both experiments. However, when the decoding context gets larger to where memory cost exceeds to prefill compute cost, the iteration latency is largely affected by the decoding context length. At 262,144 active kv-cache tokens, the iteration latency is determined by the decoding instead of the collocated prefill. Hence, any prediction of such collocated iteration is invalid if the active kv-cache usage is not taken into account.
KV cache transfer between GPUs is marginal in agentic workloads. Prefill-decode disaggregation – i.e., when prefill and decode phases occur on separate model replicas – incurs additional KV cache transfer between the prefill and decode GPU. Figure 3 depicts the latency overhead of KVtransfer and its relative impact on TTFT across a range of input tokens. While KV-transfer contributes significantly to TTFT at low input tokens (> 20%), it scales linearly with respect to input tokens and its impact is thus overshadowed by quadratic prefill latency at long inputs, precisely the operating domain of agentic workloads. Moreover, if no disaggregation happens for the rest of the turns given huge speedup by prefix caching, the KV-transfer overhead becomes negligible. ConServe relies on this insight when scheduling the first turn prefill stage for agentic requests. Decode Phase
Output tokens are generated autoregressively across many iterations throughout the decode stage. We conclude that (1) agentic workloads are memory bound, (2) prefixcaching reduces collocation overheads, and (3) decoding latency is unpredictable. Our conclusions inform the the design of ConServe in §4. Agentic workloads are memory-bound. We profile the average TBT across a range of batch sizes and KV lengths. As shown in Figure 4, mean TBT is relatively stable in low memory configurations but increases substantially un-
80
0
Prefix caching reduces TTFT to near-constant in multiturn agentic workloads. Figure 2 includes the latency of prefill stage with prefix-caching across a range of input tokens. As shown, prefix caching reduces TTFT to near-constant, reducing latency by two orders of magnitude under long inputs. Since relatively small prompts are appended to the whole history context after the first turn, prefix caching prior prompts and only calculating new KV cache entries for the appended tokens would substantially speed up the prefill stage at later turns.
3.2
KV-transfer latency ( 1024 tokens): const KV-transfer latency ( 1024 tokens): linear 0.98 µs/tok + 5.3 ms (R 2=0.981) KV-transfer latency / TTFT
100
KV-transfer share of TTFT (%)
KV-transfer latency (ms)
file prefill latency across a range of input lengths; results are shown in Figure 2. TTFT is nearly constant for short prompts, but grows quadratically once attention becomes the dominant bottleneck. Prior work often models prefill latency as linear (Databricks, 2023), which is accurate when linear projection layers dominate the quadratic attention term. First turn agentic prompts, however, can reach tens of thousands of tokens, where attention is no longer negligible and a quadratic fit is substantially more accurate. In both regimes, TTFT is determined primarily by input length, allowing prefill latency to be estimated accurately from the input-token count alone. ConServe uses this observation to route incoming conversations away from saturated prefill workers before they incur queueing delay.
5
Decoding latency is highly variable. Our experimental results indicate that decoding latency is highly variable, and unlike the prefill stage, scheduling decisions should not rely on estimating decoding latency. We record the TBT for 64 different, batched prompts across 32 runs and report the iteration-level results in Figure 6. While the mean TBT generally increases as the context length and number of iterations increase, individual TBT remains unpredictable.
Prefill tokens 4096 16384 32768 Prefix-cache miss hit a. Varying decoder batch size b. Varying decoder context length
B·L = 64k
128 17.8 18.8 24.1 36.5
35
64 16.1 15.9 16.5 21.6 34.1
8 14.5 15.1 15.4 14.9 14.2 14.6 17.9 29.9 4 15.5 14.7 14.9 14.8 15.0 15.9 14.8 17.6 29.5
Iteration latency (ms)
16 14.3 14.6 14.8 15.5 14.6 18.7 30.4
30
Mean TBT (ms)
Batch size B
32 15.1 14.9 15.8 15.6 20.7 33.9
25
20
2 15.9 15.4 15.4 14.9 15.0 14.4 14.6 15.8 17.7 1 13.8 12.5 13.1 13.0 14.2 13.6 13.9 14.4 13.7 128
256
512 1024 2048 4096 8192 16384 32768 Per-request context length L (tokens)
20
21
22
23
Batch size
24
25
210
211
212
213
214
215
Individual context length (tokens)
Figure 5. Iteration latency of collocated prefill and decode stages. (a) Latency of a prefill request arriving after 58 – 92 decode iterations across a range decoder batch sizes. (b) Latency of a prefill request arriving after a range of decoding iterations with a fixed batch size (8). Prefix-caching significantly improves collocation overhead.
P5 P95 TBT Spread
300 250 200 150 100
TBT (ms)
Moreover, as shown in Figure 5.b, iteration latency under collocation and prefix-caching depends on the prefill input size and the context length of running decode tasks. Finally, the total number of output tokens is input-dependent and cannot be known apriori (Figure 1. Overall, while decoding latency is correlated with memory bandwidth utilization, we find that accurately estimating the per iteration or end-to-end decoding latency is a challenging task that depends on many complex or unknown factors. Hence, ConServe’s design opts to not rely on such estimates for scheduling purposes.
50 30 25 20 15 10
800
1000
P99 TBT
Mean TBT
1200 1400 1600 1800 Decode Iteration Counter
2000
Figure 6. High variance in TBT throughout a long decode
Heterogeneous Serving
Power capping has a marginal effect on the decode stage of agentic workloads. We repeat the experiments in Figure 4 on power-capped GPUs and show the results in Figure 8. As depicted, decoding is marginally impacted when memory is saturated, i.e., the operating regime where agentic operations occur. This insight allows ConServe to delegate decoding to less powerful GPUs when possible.
We vary the GPU power caps to simulate the impact of heterogeneous agentic serving on the prefill and decode phase. We find that prefill and decode are impacted significantly and marginally, respectively. These findings further motivate ConServe’s design choices in §4. Power capping impacts the prefill stage heavily but has a marginal effect on prefix caching. We repeat the experiments in Figure 2 on power-capped GPUs and show the difference in performance in Figure 7. Since prefill is compute-bound at high input token ranges, power-capping significantly increases TTFT. On the other hand, prefill with prefix-caching remains relatively stable irrespective of input size. This insight harmonizes well with ConServe’s heterogeneous mapping: turn 1 prefill stages are scheduled on the best available GPUs, whereas later prefill stages with prefix-cache hits can occur on less powerful decode GPUs with marginal performance penalties.
102
101
15
Figure 4. Heat map of mean time between tokens (TBT) across a range of batch sizes and context lengths. The dashed white line highlights the boundary between unsaturated (lower left) and saturated (upper right) memory bandwidth regions. Given the long context length of agentic tasks, agentic workloads lay in the saturated memory bandwidth domain.
3.3
103
4
6
C ON S ERVE
ConServe instantiates conversation-level scheduling with a minimal placement policy. As illustrated in Figure 9, the system runs on a prefill-decode disaggregated architecture: one model replica is dedicated to processing input prompts (the prefiller node), and the remaining replicas are dedicated to generating output tokens (the decoder replicas). A conversation is bound to one decoder replica at arrival, and the binding persists for the conversation’s lifetime. The first-turn prefill is routed to the prefiller where the long ini-
40 TTFT change (%)
Online Scheduling
prefill prefill w/ prefix-cache hit
Prefiller
Prompt for
Compute-Bound
Turn-1 Prefill
Prefill Latency Modeling
Profilied Offline
30
KV Cache for
Turn-1 Decode Prompt
20
ConServe Scheduler Decoder Pool
Memory-Bound Tail
10
Decoder 1
0
Prefiller Context
Monitoring
102
103
Input tokens
Decoder
Decoder 2
KV-Cache Utilization
...
Monitoring
104
Decoder N
Conversation KV Occupancy
pinned
Prefill Completion
Figure 7. TTFT difference between uncapped GPUs and GPUS capped at 2/3 TDP. Power capping impacts the prefill stage heavily but has a marginal effect on prefix caching.
Compute Bound
Memory Bound
Data Path
Monitoring Signal
Figure 9. ConServe System Architecture. B·L = 64k 128 +3.4% -2.9% +2.3% +2.2%
10
can therefore decide whether to admit a new conversation, where to bind it, and when to scale out by reading current system state rather than forecasting decode behavior.
64 +1.2% +0.1% +2.7% +1.9% +2.0%
16 +12.5% +9.4% +6.5% -5.7% +3.9% +1.8% +1.4% 8 +12.7% +5.8% -3.0% -0.0% +3.9% +10.1% +1.5% +1.6% 4 -6.9% -1.2% +2.3% +9.0% -0.6% -10.0% +5.3% +1.5% +1.0%
5 Mean TBT (%)
Batch size B
32 -2.9% -0.8% -5.4% -1.7% +2.0% +2.7%
0
4.1
Provisioning N is bounded by two constraints on the decoder side. The first is throughput: aggregate decoder throughput must process the tokens consumed and generated across each conversation’s lifetime. The second is memory: each decoder pins its bound conversations and holds their KV cache in GPU memory, limiting concurrent conversations per replica to B slots, where B is determined by GPU memory capacity and per-conversation peak KV footprint.
5
2 +5.3% +6.0% -2.0% -0.0% +1.8% +2.3% -1.7% -9.4% +2.0% 1 -1.1% +6.5% -2.3% +1.9% +0.8% +3.9% -0.8% -4.1% -1.8% 128
256
10
512 1024 2048 4096 8192 16384 32768 Per-request context length L (tokens)
Figure 8. Heat map of mean TBT difference uncapped GPUs and GPUS capped at 2/3 TDP across a range of batch sizes and context lengths. The dashed black line highlights the boundary between unsaturated (lower left) and saturated (upper right) memory bandwidth regions. Power capping has a marginal effect in the saturated memory bandwidth domain.
Let R denote the arrival rate, Td the per-decoder token throughput, Ld the mean per-conversation token volume handled by decoders (turn-1 decode plus all turn 2+ prefill and decode), and W the mean wall-clock lifetime including time spent on external tool calls. N decoders must satisfy both constraints simultaneously:
tial prompt is processed; the resulting KV cache is transferred once to the bound decoder. The first-turn decode and every subsequent turn’s incremental prefill and decoding then execute on that decoder with full KV cache reuse across turns. The system makes one routing decision per conversation and predicts no decode-side quantity at any point. A ConServe deployment consists of one prefill node and N decoder replicas, with N chosen such that the prefill node saturates before the decoder pool under the workload’s input-to-output token ratio. This deliberate overprovisioning of decoders places the system’s throughput ceiling on the prefill side, where the relevant signal (input token rate) is observable at admission time and the cost-pertoken relationship is deterministic (§3.1). The scheduler
Instance Configuration
7
N · Td ≥ R · Ld
(throughput)
(1)
N ·B ≥R·W
(memory)
(2)
The prefill node saturates at arrival rate R∗ = Tp /Lin , where Tp is the prefill input throughput and Lin the mean first-turn input length. ConServe provisions N as an integer more than satisfying both inequalities at R = R∗ . This places the throughput bottleneck on the prefill node, where input token rate maps deterministically to expected utilization via the offline-profiled latency curve. The specific value of N used in our evaluation is given in §5.1.
4.2
Reactive Scheduling
Collocated 100
AMPD
ConServe
E2E
1
1.5 0.5
1
1 100
p95
10 5
SLO violation (%)
1
A new conversation enters the system at its first turn. The first-turn prefill is routed unconditionally to the prefill node. Once the prefill completes and the KV cache is transferred, the scheduler binds the conversation to the decoder replica with the lowest current KV cache occupancy. The binding is persistent: every subsequent turn for this conversation executes on the same decoder, with no re-evaluation of where it should go.
100 25
0
0.5
1
1.5 0.5
Request rate (conv / s)
1.5
Figure 10. Normalized agentic performance results over multiple request arrival rates (lower is better). AMPD has 10% wrong prediction rate.
Because the two signals are direct measurements of system state, the same observability extends to capacity management. Imminent prefill saturation is detectable from the input token rate against the profiled curve, and decoder saturation surfaces as aggregate KV cache pressure across the pool. Provisioning additional capacity in either tier follows from the same observations the scheduler already makes. 4.3
SLO (5×)
Last-turn TBT
10 5
gmean
The scheduler operates on two signals, both directly observable. The first, the prefill latency curve, is profiled offline as a deterministic function of input token count (§3.1); given an incoming conversation’s first-turn prompt length, the scheduler reads off the prefill node’s expected utilization in constant time. The second is per-decoder active KV cache occupancy. Because every turn reuses the conversation’s accumulated history, every byte of KV cache held for a live conversation is part of its active working set; the scheduler decrements occupancy on conversation termination so the signal always reflects only currently active state rather than allocated-but-idle memory. Both signals are properties of state the system already maintains, not forecasts of what the workload will do next.
Full Disagg
TTFET
Collocated
Full Disagg
102
104 103 TTFT (ms)
AMPD
ConServe
1.0
CDF
0.8 0.6 0.4 0.2 0.0
Heterogeneous GPU Mapping
ConServe segments a conversation into a compute-bound first-turn prefill and a memory-bound tail, mirroring the phase structure that prefill-decode disaggregation exploits. This segmentation maps cleanly onto heterogeneous GPU tiers. Memory-bound workloads scale less aggressively with newer hardware than compute-bound workloads do, so decoder replicas, which run the memory-bound tail, can execute on older lower-power GPUs without proportional throughput loss, while the prefill node takes the newest high-power GPU available.
102 TBT (ms)
Figure 11. Overall TTFT and TBT distribution at RPS 1.634.
5
E VALUATION
Our evaluation addresses the following questions: • Q1: Does ConServe match or beat all baselines on latency?
This mapping requires no changes to the scheduling logic. The same two observable signals drive placement on either uniform or heterogeneous hardware, and the conversationlevel commitment ensures KV state is transferred between tiers at most once per conversation. Energy efficiency emerges as a consequence: the system spends highpower compute only on the turn that requires it, while the memory-bound tail runs on hardware that consumes substantially less power.
• Q2: Does ConServe maintain SLO at saturation? • Q3: Does ConServe avoid the routing error sensitivities of per-turn approaches? • Q4: Does the heterogeneous variant translate to energy efficiency wins? 8
5.1
Experiment Setup
with our offline profiling results at best effort. Because in all our experiments, executing the turn 2+ prefill locally on the decoders is more effecient than migrating it to the prefiller, all reported AMPD results except Figure 12 induce a 10% wrong prediction rate, i.e., the scheduler migrates the turn 2+ prefill to the prefiller for 10% of all turns.
Workload and hardware We evaluate on agentic traces generated from SWE-bench bm25 13K with sweagent, using Qwen3-Coder-30B-A3B-Instruct as the tracegeneration model. Traces are replayed on a 4-GPU NVIDIA A40 machine with Qwen3-0.6B as the served model. The smaller served model leaves sufficient memory headroom for the long KV caches characteristic of multiturn agentic conversations. To simulate the serving performance on heterogeneous GPUs, we power cap 3 GPUs to 2/3 of their TDP (300W) to represent prior-generation GPUs with less compute capacity but similar memory hardware.
Metrics. We report three latency metrics: TTFET, lastturn TBT, and end-to-end (E2E) latency, motivated in §2.2. Together they capture the three user-facing performance dimensions that matter for multi-turn agentic conversations. For comparison with prior work, we additionally report conventional per-turn TTFT and TBT distributions, though these metrics conflate intermediate tool-call turns with the final user-facing reply and therefore do not reflect end-user experience.
System configuration. Each served model replica occupies one GPU. The prefiller sustains roughly 25k input tokens per second, while each decoder produces 1k output tokens per second and accommodates approximately 300k tokens of KV cache. We allocate one prefiller and three decoders across the four-GPU machine. As a sanity check following the analysis in §4.1, an average of 15k input tokens and 1k output tokens per conversation would require at least 1.67 decoders per prefiller. The 3 decoder configuration is a guarantee that the prefiller saturates before the decoders.
5.2
Q1: Agentic Conversation Performance
Agentic performance. Figure 10 reports normalized geometric mean and p95 latencies for TTFET, last-turn TBT, and E2E across the five systems at request rates from 0.5 to 1.5 conversations per second under Poisson arrivals. We additionally report results at 1.634 conv/s, which is not a Poisson rate but a synthesized arrival pattern that holds the prefiller exactly at its saturation throughput, isolating behavior at the system’s capacity limit. For AMPD we fixed a wrong prediction rate of 10%, with further discussion in section 5.4.
Implementation and Baselines. We implement ConServe on with vLLM (Kwon et al., 2023) as the LLM serving engine and LMCache (Cheng et al., 2025) as the Prefill-Decode disaggregation manager. We configure the prefiller model replica with a vLLM standard 8192 chunked token size per batch to prevent aggressive queueing due to long context. We compare against three baselines.
• Full Disaggregation routes every turn through a dedicated prefill node.
ConServe outperforms AMPD by reducing up to 19.17% and +51.08% on geometric mean and p95 TTFET. At low to moderate loads (0.5 to 1.25 conv/s), ConServe, AMPD, and Collocated remain within 2× of the baseline. At saturation (1.5 to 1.634 conv/s), ConServe holds steady while Collocated and AMPD degrade sharply at p95, reflecting their respective structural failure modes: Collocated suffers from prefill-decode contention growing with load, and AMPD accumulates routing-decision errors that compound into prefiller queue pressure (§5.4). Full Disaggregation is uncompetitive at every operating point, with gmean TTFET and E2E exceeding 10× the baseline. Routing every turn through the prefill node forfeits the cross-turn KV cache reuse that dominates efficiency in multi-turn workloads, and additionally pays a per-turn KV transfer cost.
• We also implement AMPD (He et al., 2026) at our best effort, which performs per-turn predictionbased disaggregation for multi-turn workloads. As bi-directional KV cache transfer (between two model replicas) and management is infeasible to implement in a reasonable amount of time, we simulate the latency of KV cache transfer and stall requests for that period. We further post-process the experiment trace
Last-turn TBT inverts the ordering. Full Disaggregation achieves the lowest TBT across all loads, since its decoder replicas process only decode tokens with no prefill interference. At saturation, ConServe’s gmean last-turn TBT (2.49× baseline) is 91.85% higher than Full Disaggregation’s (1.35×). This penalty is the price of preserving cross-turn KV cache reuse, and is dwarfed by the prefill latency that Full Disaggregation incurs on every turn:
• Collocated uses all four GPUs as mixed-batch replicas, and runs each conversation on a single replica with prefill and decode batched together. Each model replica is configured with a 2944 chunked token size. This size is derived from our offline profiling, such that the iteration latency is within 5x baseline SLO, as practiced in prior work (Stojkovic et al., 2025).
9
its gmean TTFET reaches 18.78× baseline against ConServe’s 2.53×, translating to 4.7× worse E2E. AMPD produces essentially identical TBT to ConServe despite disaggregating roughly 10% of turn 2+ prefills, showing that partial disaggregation at this rate gains no measurable TBT benefit while still paying the TTFET cost of those routing decisions (§5.4). Collocated has the worst p95 TBT at saturation (7.52× baseline, 72% worse than ConServe), reflecting the prefill-decode contention from batching long initial prefills with ongoing decoder workloads.
Collocated violates both TTFET and last-turn TBT SLOs at 1.634 conv/s, since prefill-decode contention degrades both phases simultaneously. AMPD violates only the TTFET SLO, because its per-turn routing errors accumulate on the prefiller but leave the decoder workload unchanged (§5.4). Full Disaggregation violates the TTFET and E2E SLOs at nearly 100% across all loads, with last-turn TBT remaining under the SLO due to its interference-free decoder design. The structural reason for ConServe’s SLO robustness is that the prefiller’s load is bounded by design. Only turn-1 prefills route to the prefill node, and the rate at which they arrive is gated by incoming conversation arrival. Since turn1 prefill latency is a deterministic function of input token count (§3.1), the system operates exactly at the prefiller’s saturation capacity without exceeding it. Neither baseline has this property: Collocated absorbs all load on a single batched replica pool, and AMPD’s per-turn routing injects unpredictable additional prefiller load that breaks the operating bound.
A deeper look at LLM metrics. Figure 11 shows the perturn TTFT and TBT CDFs across all turns and conversations. Full Disaggregation pays both the full prefill cost and the KV transfer on every turn, producing the rightmost TTFT distribution; its prefill-free decoders produce the leftmost TBT distribution. The TTFT CDF resolves into three regimes for the remaining systems. Below the 0.7 percentile, ConServe, AMPD, and Collocated overlap: these are turn 2+ prefills, which complete quickly regardless of where they execute. Between 0.7 and 0.8, ConServe pulls ahead significantly. This regime contains turn-1 prefills, which in Collocated batch with ongoing turn 2+ prefills on the same replicas, and in AMPD wait behind the queue formed by wrongly disaggregated turn 2+ prefills on the prefill node. Above 0.8, the ordering inverts: Collocated becomes best because its four-GPU pool prevents compute saturation, while AMPD becomes worst as its accumulated prefill queue stretches into the tail.
5.4
AMPD’s per-turn predictions are susceptible to three sources of error: the cost model does not account for decoder KV cache utilization, which directly affects decode throughput (§3.2); when incremental prefills batch with heavy ongoing decode, the resulting latency has high variance that offline profiling cannot capture; and most consequentially, the cost model does not account for prefiller queueing pressure, so each erroneously disaggregated turn 2+ prefill adds to the prefiller’s queue and delays all subsequent prefill workload. ConServe sidesteps all three by scheduling on incoming input prompt size alone, which is fully observable and makes the prefiller’s saturation point deterministic, enabling predictable autoscaling.
The TBT CDF completes the picture. ConServe and AMPD produce nearly identical distributions across the full range; AMPD’s 10% per-turn disaggregation does not meaningfully change which prefills reach the decoders. Below the 0.7 percentile, Collocated outperforms both because it dedicates four GPUs to decoding while ConServe and AMPD dedicate only three, giving Collocated onethird more aggregate memory bandwidth. Above the 0.7 percentile, this advantage inverts as long turn-1 prefills batch with ongoing decode on Collocated’s replicas and inflate per-step latency, while ConServe and AMPD insulate the decoders from this contention. 5.3
Figure 12 shows that AMPD is highly sensitive to even small cost-estimation errors. At a 0% wrong-prediction rate, AMPD routes every turn 2+ prefill locally and reduces to ConServe, so the curves begin at ConServe’s operating point. At just 5% wrong predictions, SLO violations reach roughly 7.8% for TTFET and 6.3% for E2E. As the wrong-prediction rate climbs to 50%, both gmean and p95 latencies grow linearly and SLO violations exceed 50%. The linearity follows directly from the queueing mechanism: each wrong decision adds one unit of unanticipated load to a system operating at saturation, so the resulting queue delay scales with the cumulative excess load. Our evaluation runs a finite trace subset, after which the prefiller queue drains. In a sustained production workload, the queueing pressure continues to accumulate, and the SLO violations observed here form a lower bound on AMPD’s deployment-scale impact. Last-turn TBT remains flat throughout, since routing errors affect interme-
Q2: Serving Throughput and Quality
Figure 10 (third row) reports SLO violation rates. We set the SLO threshold at 5× the baseline latency for each metric, where the baseline is measured by executing a single request without batching or interference. This 5× threshold follows standard practice in prior LLM serving work. ConServe sustains zero SLO violations across all three metrics at every request rate tested, including the 1.634 conv/s saturation point. The baselines fail at saturation in metricspecific patterns that reveal their underlying failure modes.
Q3: Pitfalls of Per-Turn Routing
10
TTFET
5 4
E2E
300W / 300W
SLO (5× baseline)
a. Normalized gmean latencies
100 50 25
ConServe
6
Last-turn TBT
10 7 6 5 4 3 2 1
3 2
12 10
b. Normalized p95 latencies ConServe
14
100 50 25
8
10 7 6 5 4 3 2 1
6 4
c. SLO violation ConServe
60% 50% 40% 30% 20% 10% 0%
300W / 200W
TTFET p95
E2E p95
SLO (5×)
Last-turn TBT p95
Tokens per Joule
80 70 60 50 40 30
cated isagg Collo Full D
AMPD ConServe
20
catedull Disagg F
Collo
AMPD ConServe
Figure 13. P95 agentic performance and energy efficiency on heterogeneous GPUs.
d. Energy efficiency ConServe
tokens / J
64
rather than computing. The effect compounds with error rate: AMPD’s tokens-per-joule declines monotonically toward 58 tokens/J at a 50% wrong-prediction rate, while ConServe holds constant by construction.
62 60 58
0%
10%
20%
30%
Wrong-prediction rate
40%
50%
5.5
ConServe segments each conversation into a computebound first-turn prefill and a memory-bound tail, mirroring the prefill-decode abstraction at conversation granularity. If this segmentation is faithful, the memory-bound tail should tolerate a reduced power budget on the decoder GPUs: its throughput is gated by memory bandwidth, not by the compute capacity a power cap removes. Figure 13 tests this, comparing a homogeneous configuration (300W prefill, 300W decoders) against a heterogeneous one (300W prefill, 200W decoders).
Figure 12. Comparison between ConServe and AMPD over different wrong prediction rate.
diate turns but never reach the user-visible final turn. ConServe has no equivalent degradation curve: it makes no perturn routing decision and therefore generates no prediction error. The same routing errors degrade energy efficiency (Figure 12.d). A wrongly migrated turn 2+ prefill carries substantial reused KV cache; mixing this memory-heavy work with the compute-bound turn-1 prefills on the prefiller lowers the prefiller’s utilization. The misrouted conversations also incur longer latency, and these delays accumulate into a longer wall-clock time to serve the full workload. At the 1.634 conv/s saturation point, with 10% wrong predictions, AMPD takes 15% longer than ConServe to drain the workload while delivering 7.5% lower tokens-per-joule. Because the workload processes a fixed token volume, the lower efficiency reflects higher total energy; that the 15% time penalty exceeds this energy increase indicates the GPUs spend the extra time under-utilized, idling on queue
Q4: Serving with Heterogeneous GPUs
ConServe realizes two energy wins. Its scheduling alone is energy-efficient in the homogeneous configuration, since confining compute-bound work to one prefill node and running the memory-bound tail without per-turn disaggregation avoids the wasted transfers and idle cycles that lower the baselines’ tokens-per-joule. Capping the decoders to 200W then adds a further gain: tokens-per-joule rises by 22.75% while p95 TTFET, last-turn TBT, and E2E remain essentially unchanged. The memory-bound tail absorbs the power cap, so the heterogeneous mapping does not trade away ConServe’s scheduling advantage but compounds it.
11
The baselines do not share this property. Collocated batches prefill and decode on the same GPUs, so reducing
Agentic and Application-Aware Serving. A growing body of work optimizes serving for the structure of agentic applications. Parrot recovers prompt sharing and request dependencies through application-level dataflow (Lin et al., 2024). A second line targets the tool-call period that separates an agent’s turns: InferCept retains, discards, or swaps the paused KV cache while an external call runs (Abhyankar et al., 2024), Continuum retains KV state across pauses under a time-to-live policy (Li et al., 2025), and Sutradhara co-designs orchestrator and engine to overlap tool execution with subsequent prefill and decode (Biswas et al., 2026). KVFlow evicts and prefetches cache entries according to an agent execution graph (Pan et al., 2025), and Autellix raises the scheduling unit to the agentic program to order calls against head-of-line blocking (Luo et al., 2025). Several of these optimize conversation-level rather than per-turn objectives: Continuum targets job completion time, and Sutradhara optimizes final-answer latency, the same quantity ConServe captures as TTFET. All of them operate within a conversation whose physical placement is fixed, managing its cache, overlapping its tool calls, and ordering its requests on an instance already chosen. None makes the placement decision itself: where the first-turn prefill runs, when its KV state transfers, and which decoder holds the conversation for its lifetime. ConServe commits this decision once from observable state and is orthogonal to this line of work, which caches, overlaps, and schedules the calls within a conversation ConServe has already placed.
decoder power to 200W starves the compute-bound prefill: its p95 TTFET worsens by 11.15% under the heterogeneous configuration. Full Disaggregation cannot meet the SLO in either configuration, leaving its energy numbers moot. AMPD gains the same power-cap benefit as ConServe, since it also runs decode on the capped GPUs, but it carries the per-turn routing penalties characterized in §5.5.
6
R ELATED W ORK
LLM Request Scheduling. LLM request scheduling has been studied along several axes. Iteration-level scheduling batches and preempts at the granularity of individual decode steps to raise utilization (Yu et al., 2022; Sun et al., 2024), and recent schedulers route or reorder requests using predicted output lengths or learned rankings (Fu et al., 2024; Jain et al., 2025). A parallel line schedules across the prefill-decode split: disaggregation systems decide where each phase executes and how the two scale independently (Zhong et al., 2024; Patel et al., 2024), while intra-GPU and unified designs revisit where the split is drawn (Shi et al., 2025; Gao et al., 2025; Wang et al., 2025). Elastic systems extend this to autoscaling prefill and decode capacity in response to load (Zhang et al., 2025; Lai et al., 2025). Production frameworks fold all of these together: NVIDIA Dynamo unifies KV-cache-aware routing, conditional prefill-decode disaggregation, SLO-driven autoscaling, and tiered KV cache management in one serving system (NVIDIA, 2025). Whether these mechanisms are studied in isolation or integrated this way, the unit of scheduling remains the request, or its constituent phases and iterations. ConServe raises the unit to the conversation: it commits one placement per conversation and never revisits it, which is what lets scheduling condition on observable state rather than predicted per-request cost. KV Cache Management. A separate line of work targets how key-value state is stored and reused rather than where the computation that produces it runs. Hierarchical and distributed cache layers move KV state across memory tiers and across the cluster to extend reuse beyond a single instance (Gao et al., 2024; Qin et al., 2025; Hu et al., 2024; Cheng et al., 2025), and compression and offloading shrink or relocate the cache to fit larger working sets (Li et al., 2024; Liu et al., 2024a). These mechanisms operate at a different layer than ConServe, which schedules computation and assumes only standard local prefix caching beneath it. The two compose: richer cache management widens what can be reused, while ConServe’s conversation-level placement keeps the reusable prefix on the instance that will read it, so after the first turn it never crosses the network again.
7
C ONCLUSION
We introduce ConServe, a conversation-level disaggregation scheduler for agentic serving. ConServe raises the scheduling unit from the turn to the conversation, which collapses turn-level irregularity into a stable two-phase structure: a compute-bound first-turn prefill followed by a memory-bound tail. ConServe routes the first-turn prefill to a high-power prefiller provisioned as the system bottleneck, transfers the KV cache exactly once, and pins each conversation to a decoder replica for its lifetime, conditioning placement only on observable turn-1 input length and per-decoder KV occupancy rather than predicted per-turn cost. Experiments on agentic workloads show that ConServe reduces p95 TTFET by 51.08% over per-turn routing while sustaining zero SLO violations at saturation, and its phase split maps onto heterogeneous GPU tiers for a further 22.75% gain in energy efficiency.
R EFERENCES Reyna Abhyankar, Zijian He, Vikranth Srivatsa, Hao Zhang, and Yiying Zhang. 2024. InferCept: Efficient Intercept Support for Augmented Large Language 12
Model Inference. arXiv:2402.01869 [cs.LG] https: //arxiv.org/abs/2402.01869
Tyler Griggs, Xiaoxuan Liu, Jiaxiang Yu, Doyoung Kim, Wei-Lin Chiang, Alvin Cheung, and Ion Stoica. 2024. M\’elange: Cost efficient large language model serving by exploiting gpu heterogeneity. arXiv preprint arXiv:2404.14527 (2024).
Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2024, Santa Clara, CA, USA, July 10-12, 2024, Ada Gavrilovska and Douglas B. Terry (Eds.). USENIX Association, 117–134.
Wenhao He, Youhe Jiang, Penghao Zhao, Quanqing Xu, Eiko Yoneki, Bin Cui, and Fangcheng Fu. 2026. Efficient multi-round llm inference over disaggregated serving. arXiv preprint arXiv:2602.14516 (2026).
Cunchen Hu, Heyang Huang, Junhao Hu, Jiang Xu, Xusheng Chen, Tao Xie, Chenxi Wang, Sa Wang, YunAnthropic. 2026. Models, Usage, and gang Bao, Ninghui Sun, and Yizhou Shan. 2024. MemLimits in Claude Code. https:// Serve: Context Caching for Disaggregated LLM Serving support.claude.com/en/articles/ with Elastic Memory Pool. arXiv:2406.17565 [cs.DC] 14552983-models-usage-limits-in-claude-code. https://arxiv.org/abs/2406.17565 Accessed: 2026-05-30. Kunal Jain, Anjaly Parayil, Ankur Mallick, Esha Choukse, Anish Biswas, Kanishk Goel, Srivarshinee S, Jayashree Xiaoting Qin, Jue Zhang, Íñigo Goiri, Rujia Wang, Mohan, Alind Khare, Anjaly Parayil, Ramachandran Chetan Bansal, Victor Rühle, Anoop Kulkarni, Steve Ramjee, and Chetan Bansal. 2026. Sutradhara: An InKofsky, and Saravan Rajmohan. 2025. Intellitelligent Orchestrator-Engine Co-design for Tool-based gent Router for LLM Workloads: Improving PerAgentic Inference. arXiv:2601.12967 [cs.DC] https: formance Through Workload-Aware Load Balancing. //arxiv.org/abs/2601.12967 arXiv:2408.13510 [cs.DC] https://arxiv.org/ abs/2408.13510 Harrison Chase. 2022. LangChain. https://github. com/langchain-ai/langchain Youhe Jiang, Fangcheng Fu, Xiaozhe Yao, Taiyi Wang, Bin Yihua Cheng, Yuhan Liu, Jiayi Yao, Yuwei An, Xiaokun Cui, Ana Klimovic, and Eiko Yoneki. 2025a. ThunderChen, Shaoting Feng, Yuyang Huang, Samuel Shen, Serve: High-performance and Cost-efficient LLM ServKuntai Du, and Junchen Jiang. 2025. LMCache: An ing in Cloud Environments. In Proceedings of the Eighth Efficient KV Cache Layer for Enterprise-Scale LLM InConference on Machine Learning and Systems, MLSys ference. arXiv preprint arXiv:2510.09665 (2025). 2025, Santa Clara, CA, USA, May 12-15, 2025, Matei Zaharia, Gauri Joshi, and Yingyan (Celine) Lin (Eds.). Databricks. 2023. LLM Inference PerOpenReview.net/mlsys.org. formance Engineering: Best Practices. https://www.databricks.com/blog/ Youhe Jiang, Ran Yan, Xiaozhe Yao, Yang Zhou, Beidi llm-inference-performance-engineering-best-practices. Chen, and Binhang Yuan. 2024. HexGen: Generative Inference of Large Language Model over Heterogeneous Environment. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 (Proceedings of Machine Learning Research), Ruslan Salakhutdinov, Zico Kolter, Katherine A. Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (Eds.). PMLR / OpenReview.net, 21946–21961.
Yichao Fu, Siqi Zhu, Runlong Su, Aurick Qiao, Ion Stoica, and Hao Zhang. 2024. Efficient LLM Scheduling by Learning to Rank. arXiv:2408.15792 [cs.LG] https://arxiv.org/abs/2408.15792 Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. 2024. Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention. arXiv:2403.19708 [cs.CL] https: //arxiv.org/abs/2403.19708 Lei Gao, Chaoyi Jiang, Hossein Entezari Zarch, Daniel Wong, and Murali Annavaram. 2025. DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing. arXiv preprint arXiv:2511.04791 (2025).
Youhe Jiang, Ran Yan, and Binhang Yuan. 2025b. HexGen2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.
13
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve
Real-World GitHub Issues? arXiv:2310.06770 [cs.CL] https://arxiv.org/abs/2310.06770
Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, and Junchen Jiang. 2024a. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving. In Proceedings of the ACM SIGCOMM 2024 Conference, ACM SIGCOMM 2024, Sydney, NSW, Australia, August 4-8, 2024. ACM, 38–56.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023, Jason Flinn, Margo I. Seltzer, Peter Druschel, Antoine Kaufmann, and Jonathan Mace (Eds.). ACM, 611–626.
Michael Luo, Xiaoxiang Shi, Colin Cai, Tianjun Zhang, Justin Wong, Yichuan Wang, Chi Wang, Yanping Huang, Zhifeng Chen, Joseph E. Gonzalez, and Ion Stoica. 2025. Autellix: An Efficient Serving Engine for LLM Agents as General Programs. arXiv:2502.13965 [cs.LG] https://arxiv.org/abs/2502.13965
Ruiqi Lai, Hongrui Liu, Chengzhi Lu, Zonghao Liu, Siyu Cao, Siyang Shao, Yixin Zhang, Luo Mai, and Dmitrii Ustiugov. 2025. TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity. arXiv preprint arXiv:2512.03416 (2025).
Yixuan Mei, Yonghao Zhuang, Xupeng Miao, Juncheng Yang, Zhihao Jia, and Rashmi Vinayak. 2025. Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, ASPLOS 2025, Rotterdam, The Netherlands, 30 March 2025 - 3 April 2025, Lieven Eeckhout, Georgios Smaragdakis, Kaitai Liang, Adrian Sampson, Martha A. Kim, and Christopher J. Rossbach (Eds.). ACM, 586–602.
Hanchen Li, Runyuan He, Qiuyang Mang, Qizheng Zhang, Huanzhi Mao, Xiaokun Chen, Hangrui Zhou, Alvin Cheung, Joseph Gonzalez, and Ion Stoica. 2025. Continuum: Efficient and robust multi-turn llm agent scheduling with kv cache time-to-live. arXiv preprint arXiv:2511.02230 (2025). Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. SnapKV: LLM Knows What You are Looking for Before Generation. arXiv:2404.14469 [cs.CL] https://arxiv.org/ abs/2404.14469
NVIDIA. 2025. NVIDIA Dynamo: A Low-Latency Distributed Inference Framework for Scaling Reasoning AI Models. https://developer.nvidia.com/ dynamo.
Zongze Li, Jingyu Liu, Zhen Xu, Yineng Zhang, Tahseen Rabbani, and Ce Zhang. 2026. Not All Prefills Are Equal: PPD Disaggregation for Multi-turn LLM Serving. arXiv:2603.13358 [cs.NI] https://arxiv. org/abs/2603.13358
Zaifeng Pan, Ajjkumar Patel, Zhengding Hu, Yipeng Shen, Yue Guan, Wan-Lu Li, Lianhui Qin, Yida Wang, and Yufei Ding. 2025. KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows. arXiv:2507.07400 [cs.DC] https://arxiv.org/ abs/2507.07400
Chaofan Lin, Zhenhua Han, Chengruidong Zhang, Yuqing Yang, Fan Yang, Chen Chen, and Lili Qiu. 2024. Parrot: Efficient Serving of LLM-based Applications with Semantic Variable. In 18th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2024, Santa Clara, CA, USA, July 10-12, 2024, Ada Gavrilovska and Douglas B. Terry (Eds.). USENIX Association, 929–945. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. 2024b. AgentBench: Evaluating LLMs as Agents. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. In 51st ACM/IEEE Annual International Symposium on Computer Architecture, ISCA 2024, Buenos Aires, Argentina, June 29 - July 3, 2024. IEEE, 118–132.
14
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2023. Efficiently Scaling Transformer Inference. In Proceedings of the Sixth Conference on Machine Learning and Systems, MLSys 2023, Miami, FL, USA, June 4-8, 2023, Dawn Song, Michael Carbin, and Tianqi Chen (Eds.). mlsys.org.
Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading More Storage for Less Computation - A KVCache-centric Architecture for Serving LLM Chatbot. In 23rd USENIX Conference on File and Storage Technologies, FAST 2025, Santa Clara, CA, February 25-27, 2025, Haryadi S. Gunawi and Vasily Tarasov (Eds.). USENIX Association, 155–170.
Xiaoxiang Shi, Colin Cai, Junjia Du, and Zhihao Jia. 2025. Nexus: Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Serving. arXiv preprint arXiv:2507.06608 (2025). Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.).
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
Jovan Stojkovic, Chaojie Zhang, Íñigo Goiri, Josep Torrellas, and Esha Choukse. 2025. DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency. In 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 1348–1362. doi:10.1109/ hpca61900.2025.00102
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language understanding by generative pre-training. (2018). https: //cdn.openai.com/research-covers/ language-unsupervised/language_ understanding_paper.pdf
Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. 2024. Llumnix: Dynamic Scheduling for Large Language Model Serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, 173–191.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. (2019). https://cdn.openai.com/ better-language-models/language_ models_are_unsupervised_multitask_ learners.pdf
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (Eds.). 5998–6008.
Toran Bruce Richards. 2023. AutoGPT. https: //github.com/Significant-Gravitas/ AutoGPT
Chao Wang, Pengfei Zuo, Zhangyu Chen, Yunkai Liang, Zhou Yu, and Ming-Chang Yang. 2025. Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving. arXiv:2508.01989 [cs.DC] https://arxiv.org/ abs/2508.01989
Timo Schick, Jane Dwivedi-Yu, Roberto Dessı̀, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.). Tianyao Shi, Yanran Wu, Sihang Liu, and Yi Ding. 2024. GreenLLM: Disaggregating Large Language Model Serving on Heterogeneous GPUs for Lower Carbon Emissions. arXiv:2412.20322 [cs.AR] https: //arxiv.org/abs/2412.20322
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
15
Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Genera-
tive Models. In 16th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2022, Carlsbad, CA, USA, July 11-13, 2022, Marcos K. Aguilera and Hakim Weatherspoon (Eds.). USENIX Association, 521–538. Dingyan Zhang, Haotian Wang, Yang Liu, Xingda Wei, Yizhou Shan, Rong Chen, and Haibo Chen. 2025. BlitzScale: Fast and Live Large Model Autoscaling with O(1) Host Caching. In 19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25). USENIX Association, 275–293. Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark W. Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Structured Language Model Programs. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (Eds.). Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. In 18th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2024, Santa Clara, CA, USA, July 10-12, 2024, Ada Gavrilovska and Douglas B. Terry (Eds.). USENIX Association, 193–210.
16