AGENT S ERVE S IM: A Hardware-aware Simulator for Multi-Turn LLM Agent Serving
arXiv:2606.09613v1 [cs.CL] 8 Jun 2026
Rakibul Hasan Rajib Mengxin Zheng Qian Lou University of Central Florida {ra963432, mengxin.zheng, qian.lou}@ucf.edu
Abstract Multi-turn LLM agents interleave model calls with external tool invocations, shifting serving from stateless request processing to stateful program execution. Serving these workloads requires scheduling, KV-cache management, and routing policies that use program-level context, including turn dependencies, tool-induced gaps, and reusable KV state. Evaluating such policies directly on real systems is costly, since each design point may require dedicated accelerator time across arrival rates, model scales, serving-instance counts, and memory hierarchies. Simulation offers a scalable alternative, but existing LLM serving simulators target stateless request-level workloads and therefore omit the core dynamics of agent serving: multi-turn program execution, cross-turn cache locality, and KV-cache residency during tool gaps. We present AGENT S ERVE S IM, a hardware-aware simulator for multi-turn LLM agent serving. AGENT S ERVE S IM evaluates serving policies at program granularity through composable modules: a Program Orchestrator preserves program identity and turn order, a Tool Simulator materializes tool-induced gaps, a Session-Aware Router maintains program-to-instance affinity for cache-aware dispatch, and a KV Residency Model tracks policy-defined KV placement across HBM, host DRAM/CXL, and eviction. Across real serving deployments and hardware configurations, AGENT S ERVE S IM reproduces real-system behavior within 6% error across key performance metrics while running entirely on commodity CPUs. These results show that AGENT S ERVE S IM enables controlled, repeatable exploration of agent-serving policies without requiring exhaustive deployment on costly accelerators.
1
Introduction
Modern LLM deployments are rapidly shifting from isolated, single-turn chat queries to multi-turn agentic systems. Autonomous coding assistants (e.g., Cursor, Claude Code, Codex), function-calling pipelines, web-search agents, and long-horizon planners execute not as single prompt-completion pairs, but as stateful programs that interleave many LLM calls with external tool invocations [Yao et al., 2023, Jimenez et al., 2024, Patil et al., 2025]. These tool invocations vary widely in duration, from millisecond-scale operations such as cat to minute-scale test runs such as pytest. As a result, the user-perceived metric is the program’s job-completion time (JCT) rather than the latency of any individual LLM call, breaking several load-bearing assumptions of today’s request-oriented LLM serving stack. This shift has already led to agent-serving policies that reason across turns: InferCept [Abhyankar et al., 2024] preserves, swaps, or discards KV state across tool-induced interruptions; Continuum [Li et al., 2026] uses time-to-live policies to manage KV retention during tool gaps; and Autellix [Luo et al., 2025] prioritizes LLM calls using program-level execution history rather than request arrival order. Evaluating how scheduling, routing, and KV-cache management policies interact in agent serving is costly on real deployments, since each configuration requires substantial engineering effort and Preprint.
accelerator infrastructure. Simulation is therefore a natural tool for design-space exploration, but existing LLM serving simulators are designed for stateless workloads and do not capture the dynamics of agent execution. Vidur [Agrawal et al., 2024] and APEX [Lin et al., 2025] treat each request as an independent prefill-decode pair with fixed input and output lengths. LLMServingSim [Cho et al., 2024] and its 2.0 extension [Cho et al., 2026] provide fine-grained hardware modeling, but their workload model still assumes independent requests. As a result, these simulators lack key abstractions required for agent workloads, including program identity, tool-call gaps, and cross-turn KV reuse. Table 1 summarizes this gap. Our goal is to build an agent-serving simulator that is faithful enough to study these policy interactions, yet flexible enough to explore configurations that are costly or unavailable on real systems. Table 1: Comparison of LLM serving simulators on agent-relevant capabilities. Simulator Vidur [Agrawal et al., 2024] APEX [Lin et al., 2024] LLMServingSim [Cho et al., 2024] LLMServingSim 2.0 [Cho et al., 2026] AGENT S ERVE S IM (ours)
Multi-turn Tool gap Program-aware Session-aware Cross-turn KV programs modeling scheduling routing cache mgmt. ✗ ✗ ✗ ✗ ✓
✗ ✗ ✗ ✗ ✓
✗ ✗ ✗ ✗ ✓
✗ ✗ ✗ ✗ ✓
✗ ✗ ✗ partial ✓
Building an agent-serving simulator is challenging because agent serving is stateful program execution rather than independent request processing. Program identity must persist as unrelated requests interleave with an agent’s turns under continuous batching, allowing program-level objectives such as attained service to depend on execution history accumulated across the whole program. Tool-gap duration determines whether retaining KV state saves prefill work or wastes scarce memory, so these gaps must be represented explicitly. Routing must account for cross-turn cache locality, since cross-instance dispatch can duplicate or recompute a program’s shared prefix. Finally, KV-cache management must track whether cached blocks remain in HBM, move to host DRAM/CXL, or are evicted during tool gaps, since this choice affects both memory pressure and future prefill cost. We present AGENT S ERVE S IM, a hardware-aware simulator for multi-turn LLM agent serving. Rather than treating turns as independent requests, AGENT S ERVE S IM represents each agent as a program record with a persistent identifier, turn index, and outstanding-tool state, allowing the Program Orchestrator to release turn t+1 only after turn t and its tool gap complete in simulated time. The Tool Simulator injects gaps from replayed traces or sampled distributions, exposing the idle interval over which KV-retention policies act. For multi-instance deployments, the Session-Aware Router uses a program-to-instance affinity table to route later turns to the same instance when capacity permits, otherwise charging migration or re-prefill cost. The KV Residency Model assigns policy-defined lifetimes to cached KV blocks using a RadixAttention-style prefix-cache structure and tracks whether they remain in HBM, move to host DRAM/CXL, or are evicted during tool gaps. Concretely, our contributions are as follows: • A program-centric simulation architecture for agent serving (§3). We introduce a simulation architecture that treats the agent program, rather than the individual LLM call, as the unit of execution. This abstraction preserves turn dependencies, tool-induced idle intervals, routing locality, and cross-turn KV residency, enabling scheduling, routing, and KV-management policies to be studied within a single hardware-aware framework. • Validation against real serving deployments (§5). We validate AGENT S ERVE S IM against real system runs using end-to-end program latency, and aggregate throughput. Across workloads, model scales, hardware platforms, and arrival rates, the simulator runs on commodity CPUs while matching real deployments to within 6% on the main performance metrics. • Hardware-aware design-space analysis (§6).We use AGENT S ERVE S IM to systematically study how prefix caching, scheduling, routing, and KV-cache management policies interact under agent workloads. Our analysis reveals the conditions under which agent-aware policies provide benefits, and highlights the impact of factors such as prefix reuse, session locality, and tool latency on performance across hardware configurations. 2
2
Background and Motivation
2.1
From Stateless Inference to Multi-Turn Agent Serving
Stateless LLM serving treats each prompt-completion pair as an independent request. Systems such as vLLM [Kwon et al., 2023] and SGLang [Zheng et al., 2024] optimize this regime with continuous batching, paged attention, and request-level prefix caching. Scheduling is request-keyed, KV state is typically tied to a single generation, and the main latency metrics are per-request TTFT and TPOT. Agentic serving changes the unit of execution from a request to a multi-turn program. Each program issues many correlated LLM calls separated by external tool invocations, so user-perceived latency is program job-completion time (JCT), not the latency of an individual call. KV state also becomes program-scoped: later turns often extend the same conversation with tool outputs, making cross-turn prefix reuse systematic rather than incidental. Realizing this benefit requires KV entries to survive tool gaps ranging from milliseconds to minutes. These properties make scheduling, KV management, and routing program-aware. Continuous batching interleaves turns from many programs and can delay short programs behind long ones. KV retention during tool gaps becomes a deadline-bound decision based on expected tool duration, memory pressure, and reload or recomputation cost. In multi-instance deployments, later turns should preferably return to the same serving engine to preserve prefix-cache locality. Recent systems such as Autellix [Luo et al., 2025], Continuum [Li et al., 2026], and InferCept [Abhyankar et al., 2024] address parts of this space, but the combined policy design space is broader than what any single deployment can easily characterize.
2.2
Limitations of Existing Serving Simulators
Simulation is widely used to study serving-system trade-offs because it avoids running every design point on real hardware. Vidur [Agrawal et al., 2024] and APEX [Lin et al., 2025] model individual prefill-decode requests and predict per-request latency under batching policies. LLMServingSim [Cho et al., 2024] and its extension [Cho et al., 2026] add fine-grained hardware modeling, including heterogeneous accelerators, multi-tier memory, MoE expert offloading, and prefill-decode disaggregation. However, these simulators target stateless workloads in which each request is independent. They lack program identity, so program-level objectives such as attained service cannot be expressed; they do not inject tool-induced inter-turn gaps, so tool-duration-dependent KV retention cannot be evaluated; and cross-instance routing reduces to stateless load balancing rather than cache-locality preservation. Table 1 summarizes this gap. As a result, recent agent-serving systems must be evaluated on real deployments, which is feasible for a few platforms but does not scale to systematic sweeps over policies, hardware, memory hierarchies, and arrival rates.
2.3
The Need for an Agent-Serving Simulator
The agent-serving techniques above appear local to individual layers, such as scheduling, caching, KV management, and routing, but their effects are tightly coupled. A program-level scheduler that ignores KV state can co-locate related calls yet still incur re-prefill cost; a TTL policy that ignores scheduler state can evict KV blocks needed imminently; and an affinity router that ignores memory pressure can overload individual instances. The best trade-off depends on hardware capacity and bandwidth, arrival rate, and workload-specific tool latencies. Empirically exploring these interactions is expensive. Each policy combination requires an implementation, each hardware platform requires a deployment, and each design point can consume hours of accelerator time. Some configurations, such as larger clusters or hypothetical memory tiers, may not be available at all. What is missing is a simulation framework that jointly models these dimensions, reproduces real-system behavior under relevant policies, and runs cheaply enough to make policy and configuration sweeps practical. 3
Figure 1: AGENT S ERVE S IM architecture. The Program Orchestrator advances each program turn by turn, routing New Turn events through the Session-Aware Router to a Model Serving Group. There, the scheduler queues turns, the KV Residency Model manages KV state across memory tiers, and the System Simulator executes the resulting operator graphs. After Turn Complete, the Tool Simulator materializes the next inter-turn gap.
3
AGENT S ERVE S IM
AGENT S ERVE S IM targets multi-turn LLM agent serving, where performance depends on programlevel dependencies, tool-induced gaps, routing locality, and cross-turn KV residency. Stateless request-level simulators cannot capture this regime because they do not preserve program identity across turns, model cache-aware routing across instances, or treat KV state as time- and tier-dependent. AGENT S ERVE S IM therefore makes the agent program the unit of execution, exposes scheduling, routing, and KV management as composable policy interfaces, and layers agent-aware semantics above operator-graph execution, hardware modeling, and continuous batching. The rest of this section describes the Program Orchestrator (§3.1), Tool Simulator (§3.2), Session-Aware Router (§3.3), Program-Aware Batch Scheduler (§3.4), and KV Residency Model (§3.5). 3.1
Program Orchestrator
A multi-turn agent program is a sequence of dependent LLM calls, but a stateless simulator admits each call independently and loses the dependency structure that program-level metrics need. The Program Orchestrator restores it, and must (i) track turn order and outstanding-tool state per program, (ii) propagate program identity through every downstream event so router, scheduler, and residency model act at program granularity, and (iii) hold turn t+1 in simulated time until turn t’s tool gap elapses; without (iii) all turns of one program collapse into independent LLM calls. We implement it as a per-program state machine with four events (New Turn, Turn Complete, Invoke Tool, Tool Complete). Each program record carries a turn index, an outstanding-tool flag, and the program identity. The barrier suppresses New Turn for p while p’s flag is set and clears it on Tool Complete. Per-program JCT then reduces to the simulated interval from p’s first New Turn to its final Turn Complete, and attained service, accumulated tokens, and queue position become consumable by the scheduler and residency model. 4
3.2
Tool Simulator
Agent turns are separated by external tool invocations whose durations range from milliseconds (e.g., cat) to minutes (e.g., pytest) in our workload. The Tool Simulator materializes these tool-waiting intervals so that latency-sensitive policies, especially KV retention, can be evaluated faithfully. It must support two needs: replaying captured tool schedules for real-vs.-sim validation, and generating hypothetical tool durations for counterfactual sweeps. The Tool Simulator provides two modes. In replay mode, it returns the captured duration for each tool call, keeping real and simulated runs on the same inter-turn schedule. In generative mode, it samples durations from per-tool distributions conditioned on tool name, such as grep, python, or git. The module carries no GPU state; during tool gaps, KV state is managed by the KV Residency Model (§3.5). Together, the two modes support both faithful trace reproduction and controlled exploration of alternative tool-latency regimes. 3.3
Session-Aware Router
When a deployment scales to multiple instances, stateless routing rules (round-robin, random, leastloaded) scatter successive turns of one program across engines and force a re-prefill or KV migration on every turn, destroying the cross-turn locality that single-instance deployments enjoy for free. The Session-Aware Router preserves that locality, and must (i) keep successive turns of one program on the same engine by default, (ii) handle home-engine over-capacity without silently sacrificing locality, and (iii) surface the cost of any locality-breaking fallback to the scheduler so policies can weigh it. A program-to-MSG affinity table routes every turn after the first to the program’s home Model Serving Group. On home-MSG over-capacity the router falls back to wait, migrate (paying the simulated KV-transfer cost via the System Simulator), or re-prefill elsewhere; the chosen cost surfaces as a scheduler signal. Multi-instance routing then becomes a cache-locality question rather than load balancing with a JCT-neutral answer. 3.4
Program-Aware Batch Scheduler
A flat request-keyed queue cannot express program-level priorities (program-FCFS, attained service) or per-turn KV-disposition decisions (preserve, swap, discard). The Program-Aware Batch Scheduler closes this gap, and must (i) key the queue on program identity, (ii) expose program-level state (attained service, history) to a pluggable policy hook, and (iii) preserve existing admission and continuous-batching logic so competing policies differ only in ordering and KV disposition. The scheduler reads the program identity stamped by the Orchestrator and consults a policy hook returning an ordering and a per-turn KV-disposition decision. Request-level FCFS, program-level FCFS, attained-service prioritization, and interruption-aware variants share the same admission and continuous-batching path; competing policies thus differ only in their ordering and KV-disposition logic, not in the underlying engine. 3.5
KV Residency Model
A program’s KV state must survive each tool gap for the next turn to reuse it. The retain, swap, or evict decision depends on gap duration and recompute or reload cost, which a capacity-only single-tier manager cannot express. The KV Residency Model addresses this by tagging per-program KV nodes with policy-defined deadlines, handling memory pressure explicitly, spanning HBM, host DRAM, and CXL, and exposing swap-to-host as a separate policy primitive. A deadline-aware residency controller sits above a RadixAttention-style prefix cache shared across within-program turns and cross-program prefixes. On tool-gap entry, the Orchestrator assigns deadlines to the program’s KV nodes. Eviction respects these deadlines unless memory pressure forces an override, in which case AGENT S ERVE S IM records a TTL-miss event. Policies may set deadlines adaptively from tool-duration and prefill or reload costs, or use fixed thresholds for ablations. Interruption-aware policies can separately invoke swap-to-host when the predicted gap exceeds the swap-and-restore cost. Thus, time-bounded retention, multi-tier placement, and policy-controlled offload share a single memory substrate. 5
4
Experimental Setup
In this section, we define the experimental setup used throughout our evaluation. The setup specifies how workloads, hardware configurations, and serving policies are expressed, how each agent turn traverses the architectural layers of §3, and how simulated outputs are compared with real-system measurements. We describe the simulator interface and runtime workflow in §4.1, and the shared validation testbed in §4.2. 4.1
Simulator I/O and Runtime Workflow
Inputs. AGENT S ERVE S IM takes (1) an Agentic Workload that describes each program as an ordered sequence of LLM calls and the wall-clock duration of every interleaved tool invocation; (2) a Cluster Configuration specifying node count, device pool composition, memory hierarchy, and interconnect parameters; (3) per-device Hardware Profiles produced by an operator-level profiler; and (4) a Tool Configuration that selects between replay mode (use captured tool durations verbatim) and generative mode (sample from a fitted per-tool distribution). Outputs. AGENT S ERVE S IM reports two streams. Runtime Statistics aggregate engine-level metrics (throughput, memory usage, per-request TTFT/TPOT) over simulated time. Program Statistics aggregate per-program metrics (end-to-end JCT, KV-cache hit), which are the primary target of validation (§5). Runtime workflow. Simulation begins with a one-time initialization that constructs one Model Serving Group (MSG) per LLM, binds devices to the configured pool, and installs the chosen policies. The runtime loop proceeds through the Program Orchestrator (§3.1), Tool Simulator (§3.2), SessionAware Router (§3.3), and Program-Aware Batch Scheduler (§3.4). Within an MSG, a per-operator performance model generates the turn’s operator graph; an operation mapper and scheduler from LLMServingSim 2.0 [Cho et al., 2026] sequence operators across the MSG’s heterogeneous device pool; and a modified ASTRA-sim/Chakra back-end [Won et al., 2023] executes the graph, accounting for inter-device communication, NIC synchronization, and multi-tier memory access. Once the turn completes, the KV Residency Model (§3.5) decides whether the program’s KV stays in HBM, swaps to DRAM or CXL, or is discarded before the next turn. 4.2
Validation Configuration
Hardware platforms. We validate AGENT S ERVE S IM on three NVIDIA GPU platforms spanning consumer, data-center, and next-generation accelerators: RTX 3090 (24 GB GDDR6X), H100-SXM (80 GB HBM3), and B200 (180 GB HBM3e). Each run uses a single Slurm-allocated node with 64–128 GB host DRAM and 8 CPU cores. We use vLLM [Kwon et al., 2023] as the real serving system. Simulator configuration. AGENT S ERVE S IM is parameterized with platform-specific device specifications matching the real systems (Appendix A, Table 3). Operator-level latency profiles are collected for each (device, model) pair using a vLLM-based layerwise profiler adapted from [Kim et al., 2025]. Host-DRAM and CXL bandwidths are treated as static cluster-configuration parameters. Workload. We use the SWE-Bench Verified trace (Appendix B). Each cell submits 50 programs under Poisson arrivals at JPS ∈ {0.02, 0.04, 0.06, 0.08, 0.1}. For each turn, we replay the captured prompt, generated-token count, and tool duration on both real and simulated sides, so JCT differences reflect serving-system behavior rather than agent-side nondeterminism. Policies and baselines. We evaluate four policies implemented in both AGENT S ERVE S IM and the real serving stack: (i) vLLM-FCFS [Kwon et al., 2023], request-level FCFS with end-of-turn KV eviction; (ii) Autellix [Luo et al., 2025], program-level attained-service scheduling; (iii) InferCept [Abhyankar et al., 2024], interruption-aware KV management that chooses whether to discard, retain, or swap a program’s KV state at each tool gap; and (iv) Continuum [Li et al., 2026], programlevel FCFS with cost-model-guided KV retention. We do not compare directly against existing simulators because they lack the program identity, explicit tool gaps, and cross-turn KV-residency state required for this evaluation, as summarized in Table 1. 6
5
Results
5.1
Job Completion Time (JCT) validation
Per-program job completion time (JCT) captures the user-perceived latency of an agent program and is the primary end-to-end outcome of multi-turn serving. Accurately reproducing JCT is therefore a prerequisite for establishing the simulator’s validity, and we use it to evaluate AGENT S ERVE S IM’s end-to-end fidelity.
real sim (e = +3.26%)
0.025 0.050 0.075 0.100
mean JCT (s)
150
90 80 70
Arrival rate (programs/s) RTX 3090 / Llama-3.1-8B / TP=2
InferCept
80
100
200
100
Autellix
real sim (e = 4.90%)
70 60 50
0.025 0.050 0.075 0.100 Arrival rate (programs/s) H100 / Llama-3.1-8B / TP=1
Continuum mean JCT (s)
vLLM-FCFS mean JCT (s)
mean JCT (s)
250
real sim (e = +3.98%)
0.025 0.050 0.075 0.100 Arrival rate (programs/s) B200 / Llama-3.1-8B / TP=1
150 140 130 120 110 100
real sim (e = 2.78%)
0.025 0.050 0.075 0.100 Arrival rate (programs/s) B200 / Llama-3.1-70B / TP=2
Figure 2: Mean per-program JCT vs JPS, real (solid, circles) vs AGENT S ERVE S IM (dashed, crosses), one representative policy per (HW, model) configuration. The remaining (HW × policy) cells are in Appendix E (Figure 9). Figure 2 overlays real and simulated mean JCT vs. JPS for every (configuration, policy) cell. Real (solid, circles) and AGENT S ERVE S IM (dashed, crosses) curves closely overlap across all panels. Table 2 reports the per-cell mean relative error, which stays below 5% on every (configuration, policy) pair of the 80-cell sweep. The error signs cluster by configuration: AGENT S ERVE S IM mildly over-predicts on B200 / Llama-3.1-8B (within +2.6% to +4.2%) and mildly under-predicts on B200 / Llama-3.1-70B and H100 / Llama-3.1-8B (within −1.9% to −4.8%), indicating per-platform calibration bias rather than per-policy bias. Table 2: Mean relative error of AGENT S ERVE S IM’s per-program JCT prediction per (configuration, policy) cell. Positive values indicate over-prediction. Configuration RTX 3090 / Llama-3.1-8B B200 / Llama-3.1-8B B200 / Llama-3.1-70B H100 / Llama-3.1-8B
5.2
vLLM-FCFS
Continuum
Autellix
InferCept
+3.26% +3.53% −2.05% −4.53%
+2.12% +2.78% −2.60% −4.55%
+3.97% +2.64% −2.67% −4.71%
+4.70% +4.15% −1.89% −4.79%
Throughput validation
Instantaneous throughput is the engine-level signal produced by AGENT S ERVE S IM’s operator and memory models. It captures runtime dynamics such as queue buildup, service stability under load, and drain after arrivals stop. Validating it shows that AGENT S ERVE S IM matches dynamic execution behavior, not only final program-level outcomes. This matters because scheduling and KV-management policies can depend on temporal effects that aggregate metrics hide, such as recovery after long tool calls. We measure throughput as generated tokens per unit of real or simulated time over uniform windows. Figure 3 overlays real and simulated throughput on a representative cell per configuration; per-cell MAPE and signed aggregated error are annotated in-panel. The two curves track each other through admission, steady state, and drain on every platform. Aggregated throughput error stays below 2% across all 80 (configuration, policy, JPS) cells. The full per-cell breakdown is in Appendix I (Table 8)
6
Hardware-aware Design Space Exploration
The validation in §5 shows that AGENT S ERVE S IM reproduces real-system JCT and throughput, enabling design-space questions that would be costly to answer using real deployments. Using the 7
Gen throughput (tokens / s)
300 250 200 150 100 50 0
H100 / Llama-3.1-8B / TP=1 vLLM-FCFS, JPS=0.04
B200 / Llama-3.1-8B / TP=1 Autellix, JPS=0.06
mean err = +1.92%
RTX3090 / Llama-3.1-8B / TP=2 InferCept, JPS=0.06
mean err = -0.02%
400
300
300
200
100 0
250 500 750 1000 Time since first arrival (s)
400
B200 / Llama-3.1-70B / TP=2 Continuum, JPS=0.08 mean err = +0.71%
300
200
0
mean err = -0.93%
0
200 400 600 800 Time since first arrival (s)
200
100
100
0
0
0
200 400 600 800 1000 Time since first arrival (s)
real sim
0
200 400 600 800 Time since first arrival (s)
Figure 3: Instantaneous throughput (generated tokens per second) vs time, real (solid) vs AGENT S ERVE S IM (dashed). One representative cell per configuration is shown; the curves track each other through admission, steady state, and drain on each platform. Mean throughput error is annotated in-panel. SWE-Bench Verified workload, we study five questions: whether multi-instance routing preserves single-instance prefix-cache hit rates (§6.1); when prefix reuse is high enough to affect policy choice (§6.2); how tool latency shifts the choice between static-threshold and adaptive KV retention (§6.3); how much agent-aware speedup comes from prefix caching alone; and when CXL bandwidth matters for KV tiering below HBM. We defer the prefix-caching ablation and CXL sweep to Appendices G and H; per-cell breakdowns appear in Appendices H–L. 6.1
Session affinity is critical when cross-turn cache reuse is high
In a multi-instance deployment, routing deter100 Round-robin Least-loaded 178 98 mines whether later turns of a program reuse Session-aware 96 176 the same prefix cache or trigger cross-instance 94 174 92 misses. We compare three policies on a two- 172 90 88 instance H100 / Llama-3.1-8B / TP=1 clus- 170 86 ter with vLLM-FCFS scheduling: round-robin, 0.04 0.05 0.06 0.07 0.08 0.09 0.10 0.04 0.06 0.08 0.1 Arrival rate (programs/s) Arrival rate (programs/s) least-loaded, and session-aware routing, which (a) (b) pins each program to the instance that served its first turn. Session-aware routing reaches Figure 4: Cross-instance routing on a two-instance a 96.26% hit rate at every JPS, matching the H100 / Llama-3.1-8B / TP=1 cluster with vLLMsingle-instance ceiling. Round-robin and least- FCFS scheduling. (a) p95 JCT vs arrival rate. loaded fall to 92.6–93.0%, losing 3.3–3.6 per- (b) NPU prefix-cache hit rate. Round-robin and centage points because the same prefix is ma- least-loaded lose 3.3–3.6 pp hit rate relative to terialized on multiple instances. p95 JCT fol- session-aware routing because consecutive turns lows the same trend: session-aware routing is may land on different engines. 1.2–5.1 s faster at every arrival rate. Thus, multiinstance routing is not merely a load-balancing problem; it is a cache-locality problem. The default policy should preserve session affinity, especially for longer programs or larger per-program KV states, where each miss incurs a full re-prefill. Session-aware
p95 JCT (s)
Which workloads benefit from agent-aware scheduling depends on how much prefix is reused across turns. We sweep the trace’s prefixreuse rate η, the fraction of consecutive turns whose prompt shares a prefix with the previous turn, across η ∈ {0.1, 0.3, 0.5, 0.7, 0.9} while holding per-turn token counts fixed, and compare two agent-aware policies that §F identified as representative: Autellix (program-level attained service) and InferCept (interruptionaware kv retention), on B200 / Llama-3.170B / TP=1 at JPS= 0.06. p95 JCT decreases monotonically with η for both policies (Fig-
Autellix InferCept
3500 3000 2500 2000 1500 1000 500
0.2
0.4 0.6 Prefix reuse rate
(a)
0.8
NPU prefix-cache hit (%)
Policy choice matters only at high prefix-reuse rate
p95 JCT (s)
6.2
Least-loaded
NPU prefix-cache hit (%)
Round-robin
80
Autellix InferCept ideal hit =
60 40 20 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Prefix reuse rate
(b)
Figure 5: Prefix-reuse sweep on B200 / Llama3.1-70B / TP=1 at JPS= 0.06 under Autellix and InferCept. (a) p95 JCT vs η. (b) NPU prefix-cache hit rate vs η; dashed line is hit = η.
8
ure 5), but the gap between them grows from 1.8× at η = 0.1 (2,601.1 vs 3,783.1 s) to 3.3× at η = 0.9 (538.3 vs 1,751.4 s). The hit-rate panel explains the gap directly: InferCept tracks the ideal hit = η line within 4 pp across the sweep, while Autellix lags by 17 to 23 pp at high η. The takeaway is that policy choice is most consequential at η ≥ 0.5: below η = 0.3 recompute dominates absolute JCT and the gap shrinks below the calibration error in §5.1, while above η = 0.5 picking the wrong policy can cost a multiplicative factor on JCT. Mean-JCT companion plot and per-cell numbers are in Appendix K. 6.3
As tool latency grows, agent-aware policies pull ahead
At each tool gap, the engine must decide 100 1200 vLLM-FCFS InferCept 95 whether to preserve a program’s KV for the Continuum 1000 90 next turn or release it. We compare three reten85 800 80 tion rules: vLLM-FCFS as the baseline, static75 600 70 TTL Continuum with a fixed 2 s pinning thresh400 65 old, and InferCept with a per-request preserve0.25 0.5 1 2 4 0.25 0.5 1 2 4 Tool-latency scale k (× baseline) Tool-latency scale k (× baseline) or-discard decision based on the cheaper cost. (a) (b) We scale each recorded tool_ms by k ∈ {0.25, 0.5, 1, 2, 4} while holding arrivals and Figure 6: Tool-latency sweep on B200 / Llamaprompts fixed on B200 / Llama-3.1-70B / TP=1 3.1-70B / TP=1 at JPS= 0.06. (a) p95 JCT vs at JPS= 0.06. tool-latency scale k. (b) Cache hit rate vs k. Below For k ≤ 0.5, the three policies are nearly iden- k = 0.5 the policies overlap; from k = 1 onward, tical (Figure 6): most gaps remain below the static-TTL Continuum and InferCept improve both 2 s threshold, KV stays pinned, and mean JCT tail JCT and cache locality over vLLM-FCFS. differs by less than 0.2 s. From k = 1 onward, longer gaps make the retention decision consequential. At k = 1, static-TTL Continuum reduces mean JCT by 26.7% relative to FCFS (190.9 vs. 260.5 s) and improves cache hit rate by 8.5 percentage points; the advantage persists at k = 2 and k = 4. InferCept also improves tail latency and locality over FCFS for all k ≥ 1, but does not surpass static-TTL Continuum on this workload. The reason is that SWE-Bench tool times are sharply bimodal, with short shell calls and a sparse set of longer network calls, so the fixed 2 s threshold falls between the modes and already makes the right preserve-or-discard decision. As tool latency grows, agent-aware policies pull ahead, and for bimodal traces such as SWE-Bench, a static-threshold retention rule is often sufficient to capture most of the gain.
7
InferCept
Continuum
Cache hit rate (%)
p95 JCT (s)
vLLM-FCFS
Limitations and Future Work
AGENT S ERVE S IM inherits the analytical-network and profile-based-operator assumptions of existing LLM serving simulators[Cho et al., 2026, Lin et al., 2024]. Tool gaps are replayed from real traces, so AGENT S ERVE S IM does not model agent-side non-determinism (a different run of the same agent may make different tool calls); for predictive use cases we provide the generative tool-distribution mode, but its predictive validity is not the focus of this paper. Finally, our validation covers two model sizes (8B, 70B) and three hardware platforms; extension to TPUs and to MoE agents (e.g., Mixtral) is left to future work.
8
Conclusion
We presented AGENT S ERVE S IM, a hardware-aware simulator for multi-turn LLM agent serving. AGENT S ERVE S IM carries program identity and tool latencies through simulation, exposes programaware scheduling and session-aware routing as policy hooks, and models cross-turn KV residency across HBM, host DRAM, and CXL. Validated against real deployments on heterogeneous hardware, AGENT S ERVE S IM reproduces key performance metrics within 6% error across all tested arrival rates. Using AGENT S ERVE S IM, we study how prefix caching, routing, prefix reuse, KV-tier bandwidth, and tool-time distributions shape agent-serving performance, including configurations beyond available clusters. These sweeps run entirely on commodity CPUs and require no GPU-hours per design point. 9
References Reyna Abhyankar, Zijian He, Vikranth Srivatsa, Hao Zhang, and Yiying Zhang. Infercept: Efficient intercept support for augmented large language model inference. In Forty-first International Conference on Machine Learning, Vienna, Austria, July 2024. Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Vidur: A large-scale simulation framework for llm inference. In MLSys, 2024. Jaehong Cho, Minsu Kim, Hyunmin Choi, Guseul Heo, and Jongse Park. Llmservingsim: A hw/sw co-simulation infrastructure for llm inference serving at scale. In IEEE IISWC, 2024. Jaehong Cho, Hyunmin Choi, Guseul Heo, and Jongse Park. Llmservingsim 2.0: A unified simulator for heterogeneous and disaggregated llm serving infrastructure. arXiv preprint arXiv:2602.23036, 2026. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? In ICLR, 2024. Wonung Kim, Yubin Lee, Yoonsung Kim, Jinwoo Hwang, Seongryong Oh, Jiyong Jung, Aziz Huseynov, Woong Gyu Park, Chang Hyun Park, Divya Mahajan, and Jongse Park. Pimba: A Processing-in-Memory Acceleration for Post-Transformer Large Language Model Serving. In Proceedings of the 58th IEEE/ACM International Symposium on Microarchitecture, page 292–307, 2025. doi: 10.1145/3725843.3756121. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In SOSP, 2023. Hanchen Li, Qiuyang Mang, Runyuan He, Qizheng Zhang, Huanzhi Mao, Xiaokun Chen, Hangrui Zhou, Alvin Cheung, Joseph Gonzalez, and Ion Stoica. Continuum: Efficient and robust multi-turn llm agent scheduling with kv cache time-to-live. arXiv preprint arXiv:2511.02230, 2026. Yi-Chien Lin, Woosuk Kwon, Ronald Pineda, and Fanny Nina Paravecino. Apex: An extensible and dynamism-aware simulator for automated parallel execution in llm serving. arXiv preprint arXiv:2411.17651, 2024. Yi-Chien Lin, Woosuk Kwon, Ronald Pineda, and Fanny Nina Paravecino. Apex: An extensible and dynamism-aware simulator for automated parallel execution in llm serving, 2025. URL https://arxiv.org/abs/2411.17651. Michael Luo, Xiaoxiang Shi, Colin Cai, Tianjun Zhang, Justin Wong, Yichuan Wang, Chi Wang, Yanping Huang, Zhifeng Chen, Joseph E. Gonzalez, and Ion Stoica. Autellix: An efficient serving engine for llm agents as general programs. arXiv preprint arXiv:2502.13965, 2025. Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Bfcl v4: Berkeley function calling leaderboard. https://gorilla.cs.berkeley.edu/leaderboard.html, 2025. Princeton NLP. mini-swe-agent: A minimal agent for swe-bench. https://github.com/SWEagent/mini-swe-agent, 2025. William Won, Taekyung Heo, Saeed Rashidi, Srinivas Sridharan, Sudarshan Srinivasan, and Tushar Krishna. ASTRA-sim2.0: Modeling hierarchical networks and disaggregated systems for largemodel training at scale. In IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), 2023. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In ICLR, 2023. Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, et al. Sglang: Efficient execution of structured language model programs. In NeurIPS, 2024.
10
A
Hardware configurations
Table 3 lists the device specifications used to parameterise AGENT S ERVE S IM for each (hardware, model, parallelism) cell in §4. All cells except RTX3090 run on a single node with one Slurm allocation per cell; the host-DRAM column is the Slurm –mem allocation rather than the full host capacity. Table 3: Device specifications used to parameterise AGENT S ERVE S IM for each (hardware, model, parallelism) cell. HW
Model
TP
Device mem (GB)
Mem bw (GB/s)
Interconnect (GB/s)
Host mem (GB)
GPUs
RTX 3090 H100 B200 B200
Llama-3.1-8B Llama-3.1-8B Llama-3.1-8B Llama-3.1-70B
2 1 1 2
24 80 180 180
936 3,350 8,000 8,000
64 900 1,800 1,800
64 64 64 128
2 1 1 2
B
Agentic Workload Characterization
We collect multi-turn agent traces by running mini-swe-agent [Princeton NLP, 2025] against an OpenRouter endpoint serving Qwen3.6-plus on SWE-Bench Verified, capturing every LLM call (with its prompt and generated tokens) and every tool invocation (with its wall-clock duration) along the trajectory. Trace statistics. Table 4 reports per-program turn counts, total token counts, and per-call tool duration over the captured workload. The trace exhibits a pronounced long tail: programs average ∼42 turns and accumulate hundreds of thousands of tokens of context across their trajectory. The per-call tool-duration distribution is heavy-tailed: a 4,125 ms standard deviation against a 1,161 ms mean means a handful of slow calls (e.g., pip install, in-repo binary executions) dominate aggregate tool time. Table 4: Trace statistics for the captured SWE-Bench Verified workload (µ ± σ). Tool time / call is the wall-clock duration of an individual tool invocation; Tokens / prog is the sum of input and output tokens over the entire program trajectory. Dataset SWE-Bench Verified (Qwen3.6)
# Programs
# Turns / prog
Tool time / call (ms)
Tokens / prog
50
41.9 ± 14.9
1161 ± 4125
356031 ± 184632
Tool-duration CDFs. Figure 7 plots the empirical CDF of tool durations for the five most frequent tools in the SWE-Bench capture, after canonicalising shell commands of the form cd /testbed && <cmd> that mini-swe-agent uses inside the SWE-Bench container. Three regimes are visible: sed is the tightest distribution, finishing under 300 ms even at p99 because it streams over already-loaded files; grep and cat share a similar p50 around 160–180 ms but stretch into a multi-second tail (p99 ≈ 3 s) when the agent scans or dumps large source trees; and python (one-shot execution of small scripts and in-repo entry-points) sits an order of magnitude higher at p50 (600 ms), while git reaches 9 s at p99. The p99(git) / p50(sed) spread is ∼ 60×, with even longer tails for the rarer pip install and full-suite pytest invocations that we omit from the top-5. This spread is exactly the dynamic range that motivates adaptive KV retention rather than a fixed retain-or-evict policy. Prefix-reuse rate η. We define η at turn t as the fraction of its input tokens that were already present at turn t − 1, and report the per-program median across turns. A high η implies that retaining KV across turns is a near-pure win; a low η means the prefix mostly turns over. SWE-Bench has η ≈ 0.95 in the median program, validating the cross-turn-reuse premise behind retention-based policies such as Continuum. 11
1.0
Empirical CDF
0.8 0.6 0.4 grep (n=443) sed (n=436) python (n=391) cat (n=390) git (n=154)
0.2 0.0 1 10
102
103 Tool-call duration (ms, log scale)
104
105
Figure 7: Empirical CDF of tool durations (log x) for the five most common tools in the SWE-Bench capture. Sample counts per tool are shown in the legend. The ∼ 60× spread between fast text utilities and project-local executables drives the design of an adaptive KV-retention policy.
C
TTFT and TPOT validation
Time-to-first-token (TTFT) and time-per-output-token (TPOT) are standard per-request LLM serving metrics. TTFT captures queueing and prefill latency before the first generated token, while TPOT captures steady-state decode latency between output tokens. Validating both ensures that AGENT S ERVE S IM reproduces the phase-level timing behavior of serving, rather than matching end-to-end latency while hiding errors between prefill and decode.
TPOT (ms)
TTFT (ms)
Figure 8 compares real and simulated TTFT and 30 12 TPOT on H100 / Llama-3.1-8B / TP=1 at JPS= 0.02. 25 10 TPOT is decode-bound and barely varies with pol20 8 icy: real values cluster in 11.69–11.78 ms while 15 6 AGENT S ERVE S IM predicts 12.01 ms uniformly, a 10 4 ∼2.5% over-prediction matching the H100 calibra5 2 tion drift in §5.1 and §5.2. TTFT is more variable 0 0 vLLM-FCFS Autellix InferCept Continuum vLLM-FCFS Autellix InferCept Continuum because it includes queueing: real values span 23.61– vLLM-FCFS (real) Autellix (real) InferCept (real) Continuum (real) vLLM-FCFS (sim) Autellix (sim) InferCept (sim) Continuum (sim) 26.14 ms while AGENT S ERVE S IM predicts a tighter 24.96–25.04 ms band, within ±6% of the real perFigure 8: Mean TTFT (left) and TPOT (right) policy mean. The simulator captures the steady-state on H100 / Llama-3.1-8B / TP=1 at JPS = 0.02, level of both metrics, though it flattens some policyreal (solid) vs AGENT S ERVE S IM (hatched), induced TTFT spread at this low-load cell. across the four scheduling policies.
D Cost Comparison of Real-System and Simulator-Based Exploration The validation grid exercises 80 cells across RTX 3090, H100-SXM, and B200 GPUs, with each real-system cell occupying a dedicated accelerator for the full run. Even a single low-load cell at JPS= 0.02 requires about 2,200 seconds for 50 programs, so the full grid costs tens of GPU-hours across platforms that may not be widely available. In contrast, AGENT S ERVE S IM produces the same output metrics on a commodity CPU machine, requires no GPU occupancy, and allows cells to run in parallel. After calibration (Table 2), the advantage is larger for design-space exploration: CXL bandwidth sweeps, larger multi-instance topologies, and prefix-reuse counterfactuals are simulatoronly in our study, avoiding hardware configurations that would otherwise be costly. 12
Table 5: Resource cost of real-system runs versus AGENT S ERVE S IM. Hardware required GPU occupancy Cell-level parallelism
E
Real-system runs
AGENT S ERVE S IM
RTX 3090, H100-SXM, B200 Dedicated per cell Limited by available GPUs
Commodity CPU machine None Parallel across CPU cores
Per-cell JCT validation panels
Figure 9 reports the per-cell mean JCT overlay for the (hardware, model, policy) cells omitted from Figure 2 in §5.1.
F
Policy-level ranking preservation
Speedup ranking preservation is the consistency property that makes a simulator useful for policy comparison. A simulator may reproduce absolute JCT and throughput, yet still be of limited value if it fails to preserve the relative ordering of policies at load points where they differ. We therefore validate that AGENT S ERVE S IM agrees with the real system on which scheduling and KV-management policy yields lower JCT at every tested load. For each non-baseline policy P ∈ {Continuum, Autellix, InferCept}, we compute speedup over vLLM-FCFS in both the real system and AGENT S ERVE S IM, defined as the ratio of FCFS mean program JCT to that of P . Comparing real and simulated speedups at each JPS tests whether the simulator preserves policy ordering under matched load. Figure 10 compares real and simulated speedups on H100 / Llama-3.1-8B / TP=1 across five arrival rates. In every case, the simulator preserves the sign of the speedup (i.e., whether P outperforms FCFS), and the magnitude error remains below 0.1%. Since the policies cluster within ±3% of FCFS on these workloads, the task is not to distinguish large effects; we revisit this limitation in §7.
G
Prefix caching is the dominant first-order knob
Recent agent-aware policies report large JCT improvements over request-level baselines, but each bundles a scheduling change with prefix caching across turns, leaving the relative contribution of the two unresolved. We disentangle them by toggling enable_prefix_caching on and off while holding the scheduling policy (Autellix) fixed, on B200 / Llama-3.1-8B / TP=1 across the four arrival rates from §5.1. With caching on, p95 JCT stays flat with load, growing only 3% between JPS= 0.04 and JPS= 0.1 (125.5 to 129.2 s). With caching off, the same sweep nearly doubles p95 JCT to 274.4 s at JPS= 0.1, a 2.13× tail-latency penalty. The mean-JCT speedup from caching grows from 1.24× at JPS= 0.04 to 1.71× at JPS= 0.1, smaller than the corresponding p95 ratio at every load, indicating the cache benefit lands disproportionately on the tail. The takeaway is that prefix caching, not scheduling policy, is responsible for most of the JCT improvement that agent-aware policies show on this workload at moderate-to-high load; scheduling decisions act on the residual after caching has saturated. NPU prefix-cache hit rate with caching on is 96.26% at every arrival rate. The mean-JCT view shows the same flat-vs-steep pattern as p95 with smaller absolute magnitude: with caching on, mean JCT increases by less than 1.3% between JPS= 0.04 and JPS= 0.1 (72.5 to 73.4 s); with caching off, it grows from 89.5 to 125.2 s, a 1.71× inflation at the highest arrival rate. The mean speedup is monotone in load and smaller than the p95 speedup at every JPS we tested (Table 6, last column), consistent with the observation that cache misses concentrate in the tail of the program-JCT distribution.
H
KV memory tiering: when does CXL bandwidth matter?
We compare five tier configurations under matched load on B200 / Llama-3.1-70B / TP=1 with the Continuum scheduling policy: HBM-only (eviction triggers full re-prefill), HBM + CPU DRAM 13
Autellix
mean JCT (s)
250
InferCept
Continuum
200 150 real sim (e = +3.97%)
100 0.02
0.04
0.06
0.08
real sim (e = +4.70%)
0.10 0.02
0.04
0.06
0.08
real sim (e = +2.12%)
0.10 0.02
0.04
0.06
0.08
0.10
Arrival rate (programs/s) RTX 3090 / Llama-3.1-8B / TP=2
Arrival rate (programs/s) RTX 3090 / Llama-3.1-8B / TP=2
Arrival rate (programs/s) RTX 3090 / Llama-3.1-8B / TP=2
vLLM-FCFS
InferCept
Continuum
mean JCT (s)
100 90 80 70
real sim (e = 4.72%)
0.02
0.04
0.08
0.10 0.02
0.04
0.08
0.10 0.02
real sim (e = 4.74%)
0.04
0.06
0.08
Arrival rate (programs/s) H100 / Llama-3.1-8B / TP=1
Autellix
0.10
Continuum
70 60 50
real sim (e = +3.36%)
0.02
0.04
0.06
0.08
Arrival rate (programs/s) B200 / Llama-3.1-8B / TP=1
0.10 0.02
real sim (e = +2.47%)
0.04
150 140 130 120 110 100
0.06
0.04
0.06
0.10 0.02
real sim (e = +2.61%)
0.04
0.08
Arrival rate (programs/s) B200 / Llama-3.1-70B / TP=2
0.10 0.02
0.06
0.06
0.10
InferCept
real sim (e = 2.84%)
0.04
0.08
Arrival rate (programs/s) B200 / Llama-3.1-8B / TP=1
Autellix
real sim (e = 2.22%)
0.02
0.08
Arrival rate (programs/s) B200 / Llama-3.1-8B / TP=1
vLLM-FCFS mean JCT (s)
0.06
Arrival rate (programs/s) H100 / Llama-3.1-8B / TP=1
vLLM-FCFS
80 mean JCT (s)
0.06
Arrival rate (programs/s) H100 / Llama-3.1-8B / TP=1
real sim (e = 4.98%)
0.08
Arrival rate (programs/s) B200 / Llama-3.1-70B / TP=2
0.10 0.02
real sim (e = 2.07%)
0.04
0.06
0.08
Arrival rate (programs/s) B200 / Llama-3.1-70B / TP=2
0.10
Figure 9: Mean per-program JCT vs JPS for the (HW, model, policy) cells not shown in Figure 2. Real (solid, circles) vs AGENT S ERVE S IM (dashed, crosses); rows are configurations (RTX 3090/8B, H100/8B, B200/8B, B200/70B) and columns are the three policies omitted from the headline figure for that row. The y-axis is shared within each row. SWE-Bench, 50 programs, Poisson arrivals (seed=42).
14
1.08 Speedup over vLLM-FCFS
1.06 1.04 1.02 1.00 0.98 0.96 0.94 0.92
0.02
0.04
0.06 Arrival rate (programs/s)
0.08
Autellix (real) Autellix (sim)
InferCept (real) InferCept (sim)
Continuum (real) Continuum (sim)
0.1
Figure 10: Speedup of each scheduling policy over vLLM-FCFS at five JPS values, real (solid bars) vs AGENT S ERVE S IM (hatched bars), on H100 / Llama-3.1-8B / TP=1. 275
Mean-JCT speedup
250 p95 JCT (s)
2.0
caching on caching off
225 200 175 150 125
0.04
0.05
0.06 0.07 0.08 0.09 Arrival rate (programs/s)
1.5
1.37x
1.71x
1.0 0.5 0.0
0.10
1.24x
1.53x
0.04
0.06 0.08 Arrival rate (programs/s)
(a)
0.1
(b)
Figure 11: Prefix-caching ablation on B200 / Llama-3.1-8B / TP=1 under Autellix. (a) p95 JCT vs arrival rate with caching on (solid) vs off (dashed); caching off inflates p95 by 2.13× at JPS= 0.1. (b) per-JPS mean-JCT speedup from caching. 275
caching on caching off
110 100 90
225 200 175 150
80 70
caching on caching off
250 p95 JCT (s)
Mean JCT (s)
120
125
0.04 0.05 0.06 0.07 0.08 0.09 0.10 Arrival rate (programs/s)
0.04 0.05 0.06 0.07 0.08 0.09 0.10 Arrival rate (programs/s)
Figure 12: Mean (left) and p95 (right, repeated from Figure 11(a)) per-program JCT vs arrival rate with prefix caching on (solid, filled circles) vs off (dashed, crosses) on B200 / Llama-3.1-8B / TP=1, Autellix. Table 6: Per-cell JCT and NPU prefix-cache hit rate for the caching ablation. Speedup is the ratio of mean JCT with caching off to mean JCT with caching on at the same arrival rate; the larger p95 inflation at JPS= 0.1 (2.13×) compared to the mean inflation (1.71×) is the magnitude of the tail effect noted in §G. Caching ON
Caching OFF
JPS
mean (s)
p95 (s)
hit %
mean (s)
p95 (s)
hit %
speedup
0.04 0.06 0.08 0.1
72.5 72.8 73.1 73.4
125.5 127.3 128.5 129.2
96.3 96.3 96.3 96.3
89.5 99.7 112.2 125.2
150.1 173.4 211.5 274.4
— — — —
1.24× 1.37× 1.53× 1.71×
15
offload (host-bound, ∼ 64 GB/s effective), and HBM + CXL with effective bandwidths of 32, 64, and 128 GB/s. The full twenty-five-cell grid (five tiers × five arrival rates from JPS= 0.04 to 0.12) is in Table 7. HBM only HBM+CPU HBM+CXL_32 HBM+CXL_64 HBM+CXL_128
350
HBM only HBM+CPU HBM+CXL_32 HBM+CXL_64 HBM+CXL_128
700 p95 JCT (s)
Mean JCT (s)
400
300 250 200
600 500 400
0.04
0.06 0.08 0.10 Arrival rate (programs/s)
0.12
0.04
0.06 0.08 0.10 Arrival rate (programs/s)
0.12
Figure 13: Mean (left) and p95 (right) per-program JCT vs arrival rate across five KV-memory tiers on B200 / Llama-3.1-70B / TP=1 under Continuum. HBM-only inflates 2.45× across the JPS sweep (170 → 418 s); HBM + CPU recovers 28% of that loss at JPS= 0.12, and HBM + CXL_128 recovers 30% at JPS= 0.10. HBM-only collapses under load. At JPS= 0.04 all five tiers collapse to within 0.3 s of each other (170.3–170.8 s) because the working set fits in HBM (NPU hit 96.24%, T2 hit ≤ 0.11%). HBM-only then degrades super-linearly as load grows: mean JCT climbs 190.9 → 238.9 → 354.2 → 417.8 s across JPS= 0.06, 0.08, 0.10, 0.12, a 2.45× inflation end-to-end and an absolute 247 s gap from the JPS= 0.04 baseline. NPU hit rate falls in lockstep (96.24% → 94.45 → 92.59 → 90.96 → 90.20), so each arrival increment translates into more re-prefills on the eviction path.
100
dotted line at JPS color: HBM+CPU baseline
300 280 260 240 220 200 180
Prefix-cache hit (%)
Mean JCT (s)
Tier-2 lift grows monotonically with load. Adding a CPU-DRAM second tier flattens the curve substantially. HBM + CPU is 6.3% / 16.0% / 22.9% / 28.0% faster than HBM-only at JPS= 0.06 / 0.08 / 0.10 / 0.12 (Figure 13). The absolute gain at JPS= 0.12 is 117 s of mean JCT (HBM-only 417.8 s versus HBM + CPU 300.7 s). HBM + CXL configurations match or exceed the CPU lift, with the largest separation at JPS= 0.10: HBM + CXL_128 reduces mean JCT to 246.3 s, a 30.5% improvement over HBM-only and 9.8% over HBM + CPU.
jps=0.04 jps=0.06 jps=0.08 jps=0.1 jps=0.12
95 90 85 80
32
64 CXL effective bandwidth (GB/s)
128
solid = NPU hit, hatched = T2
0.04
0.06
Figure 14: Mean JCT vs CXL effective bandwidth at five arrival rates. Coloured dotted lines mark the HBM + CPU baseline at each JPS. The CXL bandwidth gradient is largest at JPS= 0.10 (9.8% of CPU recovered), then narrows at JPS= 0.12
0.08
0.1
Arrival rate (programs/s)
HBM only HBM+CPU
HBM+CXL_32 HBM+CXL_64
0.12
HBM+CXL_128
Figure 15: Prefix-cache hit decomposition into NPU-resident (solid) and tier-2-resident (hatched) components per (tier, JPS) cell. HBMonly bars carry no hatching by construction. T2 hit rate climbs from ≤ 0.1% at JPS= 0.04 to 6.0–6.8% at JPS= 0.12
The CXL bandwidth gradient peaks at JPS= 0.10. The cross-tier comparison at fixed JPS (Figure 14) shows a clean monotone gain from CPU to CXL_128 at JPS= 0.08 (188.5 vs 200.7 s, 16
Table 7: Per-cell KV-tiering numbers on B200 / Llama-3.1-70B / TP=1 under Continuum. Five tiers (HBM-only, HBM + CPU, HBM + CXL at 32, 64, 128 GB/s) × five arrival rates (JPS= 0.04, 0.06, 0.08, 0.10, 0.12). Tier
JPS
mean JCT (s)
p95 JCT (s)
NPU hit %
T2 hit %
HBM only
0.04 0.06 0.08 0.1 0.12
170.5 190.9 238.9 354.2 417.8
392.7 436.4 471.2 633.0 777.9
96.24 94.45 92.59 90.96 90.20
— — — — —
HBM+CPU
0.04 0.06 0.08 0.1 0.12
170.4 178.8 200.7 273.0 300.7
392.7 408.3 409.8 479.9 516.9
96.24 94.60 92.79 90.44 90.10
0.02 1.65 3.43 5.73 6.08
HBM+CXL_32
0.04 0.06 0.08 0.1 0.12
170.8 179.6 196.5 268.2 306.0
393.8 411.6 422.1 496.2 530.5
96.24 94.81 92.71 90.75 90.39
0.11 1.54 3.64 5.60 5.96
HBM+CXL_64
0.04 0.06 0.08 0.1 0.12
170.5 177.4 197.4 272.8 303.7
392.9 409.4 413.6 476.1 529.5
96.24 94.63 92.78 91.13 89.92
0.11 1.72 3.57 5.22 6.43
HBM+CXL_128
0.04 0.06 0.08 0.1 0.12
170.3 176.8 188.5 246.3 293.7
392.5 408.7 413.5 442.7 492.4
96.24 94.64 92.47 90.75 89.57
0.11 1.71 3.88 5.61 6.78
5.7%) and JPS= 0.10 (246.3 vs 273.0 s, 9.8%). At JPS= 0.12 the CXL_128 advantage on mean compresses to 2.3% (293.7 vs 300.7 s), but p95 still favours CXL_128 by 4.7% (492.4 vs 516.9 s). The likely interpretation: at saturating load the eviction queue itself becomes the bottleneck, so faster transfer no longer shifts the average – but the tail keeps benefiting because CXL_128 retains more state in the second tier (T2 hit 6.78%, vs 6.08% on CPU). The CXL-bandwidth gradient is therefore a real lever from JPS= 0.08 through 0.10 and shifts from a mean lever to a tail-only lever at 0.12.
Hit-rate decomposition explains the JCT trends. The stacked bars in Figure 15 show NPU hit declining smoothly with load (96.24% → 94.4–94.8% → 92.5–92.8% → 90.4–91.1% → 89.6– 90.4%) while T2 hit climbs from ≤ 0.1% at JPS= 0.04 to 6.0–6.8% at JPS= 0.12. At the highest load the second tier is on the path for roughly one program-second in fifteen, large enough to drive the 117 s mean-JCT gap reported above.
Implications. A CPU-DRAM second tier captures most of the available benefit at moderate load and is essentially free at low load. CXL bandwidth is a secondary lever whose payoff grows with arrival rate up to a saturation point, peaking at 9.8% over CPU at JPS= 0.10 before the eviction queue dominates. Both findings are now well above the ±4% calibration drift reported in Table 2, so the CXL-vs-CPU gap should read as a load-conditional design choice rather than as suggestive evidence: above the saturation load CXL pays back, at saturation it pays back only on tail latency, and well below saturation it does not pay back. We expect the curves to shift further with workloads that hold larger per-program KV state, where each spilled token compounds into larger re-prefill or transfer time. 17
Table 8: Per-(configuration, policy, JPS) aggregated throughput error (%). Each entry is the signed percent error of mean generated throughput over the run. Aggregated throughput error (%)
I
Policy
JPS
B200 / 8B TP=1
B200 / 70B TP=2
H100 / 8B TP=1
RTX 3090 / 8B TP=2
vLLM-FCFS vLLM-FCFS vLLM-FCFS vLLM-FCFS vLLM-FCFS
0.02 0.04 0.06 0.08 0.10
−1.11 +0.05 +0.04 +0.56 +1.22
−0.99 −0.24 +0.00 +0.09 +0.45
−0.92 −0.93 −0.93 −0.95 −0.94
−0.92 −0.93 −0.93 −0.93 −0.92
Continuum Continuum Continuum Continuum Continuum
0.02 0.04 0.06 0.08 0.10
−0.27 −0.02 +0.13 +0.76 +0.88
−0.43 −0.17 +0.10 +0.71 +0.54
−0.92 −0.93 −0.93 −0.95 −0.94
−0.93 −0.92 −0.93 −0.93 −0.93
Autellix Autellix Autellix Autellix Autellix
0.02 0.04 0.06 0.08 0.10
−0.38 −0.18 −0.02 +0.21 +0.54
−0.40 −0.11 −0.03 +0.14 +0.52
−0.92 −0.93 −0.93 −0.95 −0.94
−0.92 −0.92 −0.93 −0.92 −0.92
InferCept InferCept InferCept InferCept InferCept
0.02 0.04 0.06 0.08 0.10
−0.29 −0.07 −0.00 +0.42 +0.71
−0.31 −0.09 +0.03 +0.13 +0.49
−0.92 −0.93 −0.93 −0.95 −0.94
−0.93 −0.92 −0.93 −0.93 −0.93
Per-(config, policy, JPS) throughput accuracy
Table 8 provides the full per-cell breakdown of the throughput validation summarised by Figure 3 and the headline numbers in §5.2. Each entry is the signed percent error of mean generated throughput over the entire run, where positive values indicate over-prediction by AGENT S ERVE S IM and negative values indicate under-prediction.
J
Multi-instance routing: mean JCT and per-cell numbers
91.5 91.0 90.5 90.0 89.5 89.0 88.5 88.0
Round-robin Least-loaded Session-aware
178 p95 JCT (s)
Mean JCT (s)
This appendix complements the p95 + hit-rate plot in §6.1 with the matching mean-JCT panel (Figure 16) and the per-cell numerical breakdown (Table 9). The mean-JCT view shows the same session-aware advantage as p95 with smaller magnitude: across the four arrival rates the spread between routes is at most 3.5% (mean) versus 1.2–5.1 s on p95. As with the caching ablation in Appendix G, the hit-rate benefit lands disproportionately on the tail because each cross-instance prefix miss inflates a single program’s JCT by a re-prefill cost rather than amortising across all programs. Round-robin Least-loaded Session-aware
176 174 172 170
0.04 0.05 0.06 0.07 0.08 0.09 0.10 Arrival rate (programs/s)
0.04 0.05 0.06 0.07 0.08 0.09 0.10 Arrival rate (programs/s)
Figure 16: Mean (left) and p95 (right, repeated from Figure 4) per-program JCT vs arrival rate on the two-instance H100 / Llama-3.1-8B / TP=1 cluster, vLLM-FCFS scheduling.
18
Table 9: Per-(routing policy, JPS) JCT and NPU hit rate on the two-instance H100 / Llama-3.18B / TP=1 cluster. Routing policy
JPS
mean JCT (s)
p95 JCT (s)
NPU hit %
Round-robin
0.04 0.06 0.08 0.1
88.0 88.9 89.1 89.1
170.6 173.6 175.5 179.0
92.65 92.71 92.68 92.67
Least-loaded
0.04 0.06 0.08 0.1
89.1 89.5 89.7 91.5
170.8 173.8 175.4 177.5
92.98 92.86 92.83 92.89
Session-aware
0.04 0.06 0.08 0.1
88.7 89.2 89.5 89.8
169.2 170.6 172.8 174.0
96.26 96.26 96.26 96.26
Autellix InferCept
Autellix InferCept
3500 3000
1500
p95 JCT (s)
Mean JCT (s)
2000
1000
2500 2000 1500 1000
500 0.2
0.4 0.6 Prefix reuse rate
500
0.8
0.2
0.4 0.6 Prefix reuse rate
0.8
Figure 17: Mean (left) and p95 (right, repeated from Figure 5(a)) per-program JCT vs prefix-reuse rate η on B200 / Llama-3.1-70B / TP=1 at JPS= 0.06, under Autellix and InferCept.
K
Prefix-reuse sweep: mean JCT and per-cell numbers
This appendix complements the p95 + hit-rate plot in §6.2 with the matching mean-JCT panel (Figure 17) and the per-cell numerical breakdown (Table 10). Both come from the same B200 / Llama3.1-70B / TP=1 sweep at JPS= 0.06 under Autellix and InferCept. The mean-JCT view shows the same monotone-in-η ordering as p95 with smaller magnitudes. The InferCept-vs-Autellix mean-JCT ratio grows from 1.6× at η = 0.1 (1,412.9 vs 2,264.8 s) to 2.4× at η = 0.9 (288.3 vs 681.5 s), narrower than the 3.3× p95 gap at the same η. As in the caching ablation, the cache benefit lands disproportionately on the tail.
Table 10: Per-(policy, η) JCT and NPU hit rate for the prefix-reuse sweep in §6.2. Policy
η
mean JCT (s)
p95 JCT (s)
NPU hit %
Autellix
0.1 0.3 0.5 0.7 0.9
2264.8 2134.8 1897.1 1449.1 681.5
3783.1 3596.4 3322.7 2928.6 1751.4
12.25 19.18 27.56 39.01 68.35
InferCept
0.1 0.3 0.5 0.7 0.9
1412.9 1208.0 990.6 762.1 288.3
2601.1 2264.5 1912.8 1516.2 538.3
17.61 31.82 45.45 59.77 86.03
19
L
Tool-latency sweep: per-cell numbers
Table 11 reports the per-(policy, tool-latency scale k) JCT and cache hit rate that back the line and bar panels in Figure 6. The k = 1 column corresponds to the natural SWE-Bench tool-time distribution; other columns scale every per-turn tool_ms by k. Table 11: Per-(policy, tool-latency scale k) JCT and cache hit rate on B200 / Llama-3.1-70B / TP=1 at JPS= 0.06. k
mean JCT (s)
p95 JCT (s)
cache hit %
vLLM-FCFS
0.25 0.5 1 2 4
139.1 150.9 260.5 298.8 592.4
363.2 378.4 641.8 678.0 1201.3
96.26 96.23 86.00 86.79 67.10
InferCept
0.25 0.5 1 2 4
139.2 151.1 223.8 300.1 608.2
363.2 378.5 456.5 534.4 1116.5
96.26 96.23 93.01 89.77 74.33
Continuum
0.25 0.5 1 2 4
139.2 151.2 190.9 282.7 468.3
363.2 378.7 436.4 505.6 836.0
96.26 96.22 94.45 91.99 83.24
Policy
20