Coda: Exploiting Admission Flexibility for Coding-Agent Serving

C ODA: Exploiting Admission Flexibility for Coding-Agent Serving Youhe Jiang University of Cambridge

arXiv:2610.03088v1 [cs.DC] 2 Oct 2026

Ehsan K. Ardestani Meta

Fangcheng Fu SJTU

Binhang Yuan HKUST

Zhan Shu Meta

Adnan Aziz Meta

Abstract

Yi Xu Meta

Time Session 1

Coding agents powered by large language models (LLMs) repeatedly alternate between model inference and tool calls, creating long-lived sessions with reusable key-value (KV) states and asynchronous request resumptions. Logical readiness, however, does not ensure efficient admission in a shared serving system. Through direct trace analysis and trace-driven replay, we identify two mismatches: reusable KV states reside across storage tiers and incur unequal computational costs for state preparation, while requests with heterogeneous context lengths can decode inefficiently together. Our central insight is that exploiting admission flexibility can improve serving performance while preserving request progress. We present C ODA, a coding-agent serving system that realizes this insight through a readiness-informed admission layer incorporating two mechanisms. Tiered-Aging state admission exploits bounded flexibility in admission order to improve admission efficiency and preserve request progress. Compatibility-Aware execution admission exploits flexibility in attention grouping within each model iteration to reduce mixed-context interference and improve shared decoding efficiency. For multi-worker configurations, C ODA introduces a separate routing layer that considers KV-state residency, worker load, and request-to-worker context-length compatibility to guide efficient request placement. We evaluate C ODA across different models and workloads against state-of-theart coding-agent serving systems and vLLM. Across the single-worker and multi-worker experiments, C ODA improves output-token and SLO-compliant throughput by 20.3% and 70.5% on average, with peak gains of 29.3% and 140.2%.

1

Krishna Malladi Meta

Fix a bug

Session 2

6K

Set up an environment

18K

Search code Apply fix 28K 12K

Implement a feature

Session 3

10K

4K

Read files 8K

Check deps Model inference

22K

Unit test 44K

52K

Apply patch

Integration test 14K 16K

Install packages

Smoke test

Tool execution

Tool return

Figure 1: Illustrative multi-session coding-agent workflows. Independent sessions alternate between model inference and tool execution. Tool calls return asynchronously, and their outputs extend subsequent model inputs, causing session contexts to grow at different rates.

these workflows through integrated model inference and tool execution. As Figure 1 illustrates, concurrent sessions repeatedly pause for tools, resume asynchronously, and accumulate context across model rounds [21, 24]. A request becomes logically ready when user input arrives or a tool return enables its session to continue. Logical readiness determines whether a request may proceed, but it does not determine whether that request should be prepared next or how efficiently it will share GPU execution with other ready requests. Admitting a request commits GPU resources and changes how other requests share those resources. Two characteristics of codingagent workloads make this distinction particularly important. First, readiness does not imply equal computational cost for preparation. Long-lived coding-agent sessions accumulate reusable interaction histories across successive model and tool invocations. Preserving the KV cache avoids repeated prefix computation, but the aggregate KV state of concurrent sessions can exceed available GPU high-bandwidth memory (HBM) capacity, motivating storage across HBM, CPU DRAM, and lower storage tiers [16, 18]. During tool-induced idle periods, memory pressure may evict a session’s KV state from HBM before its next request becomes ready [12, 21, 24].

Introduction

LLM coding agents extend software automation from code generation to multi-step development workflows, including repository-level repair, code editing, environment configuration, and test execution [3, 23, 26, 30]. Systems such as OpenAI Codex [17], Claude Code [1], and Cursor [5] support 1

Flexibility in execution grouping. Admission can account for context-length compatibility when organizing the participating requests for attention execution. Compatibility-aware grouping can improve shared decoding efficiency when its computational savings exceed the computational cost of additional execution groups. Existing systems improve coding-agent serving through cache management, session placement, and resource coordination. CacheWise combines prefix-aware scheduling based on additional KV-block demand with predictive KV-cache eviction informed by tool calls [21]. SMetric uses session reuse and worker load to guide request routing [22], while MORI manages GPU and CPU residency through relative program idleness and tier admission [24]. AlignedServe instead forms batches from waiting requests with similar KV-cache lengths [2]. Existing systems therefore address individual aspects of residency, placement, or context-aware batching, but do not jointly provide bounded preparation-aware ordering and within-iteration attention grouping for asynchronously resuming coding-agent sessions. We present C ODA, a coding-agent serving system that decouples logical readiness from physical GPU execution through a readiness-informed admission layer. Tiered-Aging state admission realizes bounded admission-order flexibility by combining the computational cost of KV-state preparation, waiting age, and a progress guard. Compatibility-Aware execution admission realizes execution-grouping flexibility by applying context-compatible attention groups only when conservative predicted computational savings exceed their additional computational overhead. For multi-worker configurations, a separate routing layer considers context-length compatibility only within KV-affinity and load safeguards. Our contributions are summarized as follows: Contribution 1: We characterize coding-agent workloads through direct trace analysis and replay (§3). We examine how changing KV-state residency creates unequal computational costs for state preparation, and how uneven context growth and tool returns create heterogeneous decode contexts. These findings show why logical readiness alone is insufficient and motivate flexible admission order and execution grouping. Contribution 2: We design and implement C ODA with two admission mechanisms (§4): Tiered-Aging state admission (§4.2) and Compatibility-Aware execution admission (§4.3). A complementary routing layer supports multi-worker configurations (§5). We integrate these mechanisms as an orchestration layer connected to the inference backend (§6). Contribution 3: We evaluate C ODA across models and workloads in single-worker and multi-worker configurations (§7). Compared with state-of-the-art coding-agent serving systems and a vLLM baseline, C ODA improves single-worker outputtoken throughput by 17.5% on average and up to 23.9%, and SLO-compliant throughput by 96.1% on average and up to 140.2%. In the multi-worker experiments, output-token

Consequently, equally ready requests can require substantially different preparation: one may reuse HBM-resident state, whereas another requires restoration, storage access, or prefix recomputation. In our trace-driven replay (§3), 24.45% of resumed requests are not HBM-ready but account for 42.45% of total startup computational cost. Admission based on arrival order or reusable-prefix availability alone can therefore allow computationally expensive state preparation to delay several computationally inexpensive session continuations. Second, readiness does not imply efficient shared execution. Long-lived coding-agent sessions accumulate context at different rates, while asynchronous tool returns bring sessions at different stages back into execution [21, 30]. Consequently, short-context requests frequently execute alongside requests with substantially larger KV states. Although each request is ready to proceed, heterogeneous context lengths can reduce attention-kernel efficiency during shared decoding [2]. In our trace-driven replay (§3), 82.84% of multi-request decode iterations have a largest-to-smallest context-length ratio of at least 4×. For pure-decode iterations, at fixed request count and total context-token volume, the observed context composition incurs a 25.95% higher median computational cost per iteration than a near-uniform composition. Admission based only on individual readiness can therefore create a heterogeneous active population; treating that population as a single attention group leaves context-length compatibility unaddressed. Exploiting these opportunities is challenging. Prioritizing requests with low computational costs of state preparation can reduce blocking, but may repeatedly bypass sessions whose KV states reside in lower tiers. Conversely, strict ready-time order prevents reordering-based bypass but sacrifices this efficiency opportunity. Similarly, organizing heterogeneous contexts into separate attention groups can improve attention efficiency, but additional launches and smaller groups may outweigh the benefit; this tradeoff must be assessed online as the active population changes. In multi-worker configurations, context-compatible placement can also conflict with KV affinity and load balance. Logical readiness should determine eligibility, but not admission order or execution grouping. Our approach exploits bounded flexibility between readiness and execution to reorder state preparation and reorganize attention groups. In this paper, admission is twofold: state admission determines which logically ready request is prepared next, while execution admission determines how the current participating requests are organized for execution. These two decisions expose flexibility in admission order and execution grouping: Flexibility in admission order. Admission can account for differences in the computational costs of KV-state preparation when selecting which logically ready request is prepared next. Bounded reordering can improve admission efficiency while guarding against indefinite bypass. 2

Increasing storage capacity

80 GiB

KV access GPU HBM Direct KV access

384 GiB

CPU DRAM Host-resident KV state

2 TiB

SSD Persistently stored KV state

to reuse cached state: HBM-resident KV can be accessed directly, CPU-resident KV requires transfer to the GPU, and SSD-resident KV additionally requires storage I/O [6, 15]. If reusable state is unavailable, the corresponding prefix must be recomputed. Restoring cached state also does not eliminate prefill computation for newly appended input tokens. Consequently, retaining a reusable prefix avoids redundant computation, but its current storage location still affects the computational cost of resuming a request. Systems for coding-agent serving. Recent systems exploit session structure, tool behavior, and KV reuse to improve coding-agent serving. CacheWise combines prefix-aware scheduling with predictive KVCache eviction: it prioritizes requests requiring fewer additional KV blocks and uses tool information to predict relative future reuse for eviction decisions [21]. SMetric introduces balanced session-centric scheduling, using actual local-prefix hits to preserve session locality and load-aware guards to prevent overloaded workers from retaining requests solely for reuse [22]. MORI derives relative program idleness from inference and tool-call timings, coordinating KV-state residency and program admission across GPU and CPU tiers [24]. ThunderAgent represents multi-turn workflows as LLM programs and coordinates program execution through memory-pressure-aware pause and restore decisions, together with tool-environment management [12]. For interactive LLM serving, Bidaw prioritizes SSD KV loading according to KV size and waiting time to reduce storage-induced request blocking [7]. CacheWise does not combine tier-specific preparation time with a waiting bound. SMetric does not consider context-length compatibility, and MORI, ThunderAgent, and Bidaw do not optimize within-iteration attention grouping. None of these systems jointly addresses admission order and execution organization. AlignedServe groups requests with similar prefix lengths by controlling which requests enter the next batch, potentially delaying requests and requiring starvation handling; C ODA instead keeps the participating request set unchanged and reorganizes only its attention computation [2]. These systems establish the importance of locality, residency, and programlevel resource control in coding-agent and interactive LLM serving. C ODA instead places admission flexibility at the center of its design, which is based on careful coding-agent workload characterizations and jointly considers admission order and execution grouping within a unified admission layer. Motivation. In coding-agent serving, ready requests differ in both the work required to prepare their KV states and their context-length compatibility for shared decoding. These differences make admission consequential for shared serving performance. A returning session may require substantial KV restoration and prefill work, while another can proceed with little preparation. Meanwhile, asynchronous resumptions bring requests with different context lengths into shared decoding, where their execution efficiency depends on how they are organized together. Logical readiness therefore

GPU decoding Reads HBM-resident KV CPU-to-GPU restoration 31.5 GB/s over PCIe SSD-to-CPU loading Up to 6.9 GB/s

Figure 2: Illustrative example of multi-tier KV storage. Lower tiers provide larger storage capacity. HBM-resident KV is directly accessible for decoding, whereas DRAM-resident KV requires GPU restoration and SSD-resident KV first requires loading into DRAM.

throughput improves by 23.1% on average and up to 29.3%, while SLO-compliant throughput improves by 45.0% on average and up to 59.0%. Component ablations and runtime analysis further identify the contributions of Tiered-Aging state admission, Compatibility-Aware execution admission, and routing to serving performance.

2

Background and Motivation

Coding-agent serving background. Coding-agent workflows interleave model inference with development tools for repository inspection, code modification, command execution, and testing [23, 26]. A session comprises multiple dependent model requests, with tool outputs incorporated into the input of subsequent requests [21, 27]. Each model request involves prefill computation for input tokens and iterative decoding for output generation. Retaining the corresponding key-value (KV) state allows subsequent requests to reuse the session prefix rather than repeatedly compute it [6, 15]. Unlike isolated inference requests, coding-agent sessions retain growing interaction histories and repeatedly pause for tool execution, producing long-lived KV states and asynchronous resumptions [4, 14, 30]. Concurrent sessions compete for GPU memory, motivating KV storage across GPU HBM, CPU DRAM, and lower storage tiers [15, 18]. Meanwhile, serving engines combine requests into batches, so requests from different session stages may decode together [13,28]. Admission therefore affects more than an individual request: admission order determines which request’s state-preparation work occurs next and shapes which contexts execute concurrently. Multi-tier KV storage. GPU serving systems extend KVcache storage across GPU HBM, CPU DRAM, and SSDs to balance access efficiency and capacity across the hierarchy [6, 18, 19]. GPU HBM provides high-bandwidth access to KV state during model execution, but its limited capacity must also accommodate model weights and runtime buffers [6, 13]. CPU DRAM provides additional capacity for reusable KV state, while SSDs offer a larger-capacity storage tier [15]. As illustrated in Figure 2, these tiers differ in the work required 3

3

● p50 ◆ p95 0.32 / 3.21

HBM DRAM

0.62 / 3.82

SSD Recompute

1.76 / 12.63

100

101

8.80 / 45.47

102

Startup computational cost (s, log scale)

100

Startup-work share (%)

leaves two important choices: which ready request should have its KV state prepared next, and how the participating requests should be organized for execution. Understanding whether these choices offer meaningful performance opportunities requires careful characterization of the underlying coding-agent workload behavior. The following section characterizes KV-state residency and startup computational costs (Findings 1-3), within-worker context composition and decode efficiency (Findings 4-7), and cross-worker context composition (Finding 8). These findings motivate our stateadmission, execution-admission, and routing mechanisms, respectively.

HBM

DRAM

SSD/Recomp.

0

40

80

75 50 25 0

20

60

100

Trace-replay time (min)

Figure 3: Left: p50 (circle) and p95 (diamond) startup computational costs, including KV-state movement and prefill, for resumed requests grouped by their required KV-state preparation path. Right: each stacked bar shows the share of total startup work attributable to HBM-ready, DRAM-restoration, and SSD-loading/recomputation paths for each successive 10-minute trace-replay interval.

Coding-Agent Workload and System Characterization

Trace and replay methodology. We characterize the complete TraceLab corpus [30], which contains 357,161 model rounds, 432,510 tool calls, and 4,281 physical sessions. We obtain request- and session-level properties directly from the trace. For runtime state not recorded by TraceLab, including physical KV-tier placement, concurrent request populations, decode iterations, and worker assignments, we use policyneutral trace-driven replays that preserve request tokens, session order, and observed tool-call intervals. Specifically, we use a common hierarchy replay with a 30-minute warm-up followed by a two-hour measurement interval. The evaluation configuration consists of four A100-80GB workers, each with 33.91 GiB of usable HBM KV-cache capacity, 384 GiB of shared DRAM, and a 2-TiB SSD tier. Reusable KV states are distributed across storage tiers. At the end of the measurement interval, HBM holds 21 resident KV-state copies (129.5 GiB), DRAM holds 57 (383.4 GiB), and SSD holds 316 (2,047.3 GiB). These copy counts overlap because the hierarchy maintains lower-tier shadows. During the interval, 39,271 requests resume existing sessions after user input or tool-call returns. Among them, 9,601 (24.45%) are not HBM-ready: 6,769 (17.24%) require CPU-to-GPU restoration from DRAM, 2,763 (7.04%) require SSD-to-CPU loading followed by GPU restoration, and 69 (0.18%) require prefix recomputation. The SSD remains near capacity and performs 1,858 evictions. Lower-tier restoration generates 64,517.2 GiB of CPU-to-HBM traffic, including 15,705.3 GiB that first crosses the SSD-to-CPU link. Finding 1. Reusable coding-agent KV state spans the storage hierarchy. A reusable prefix may reside below HBM and require substantial state movement before execution.

identities are replaced. Thus, KV-state residency is a transient property rather than a fixed session attribute. Finding 2. Reusable KV-state residency changes continuously as asynchronous tool-call returns reshape storage-tier residency. Request startup computational costs vary across KV-state preparation paths. For each request in the trace replay, startup computational cost captures the KV-state movement and prefill work required before decoding can begin. The p50/p95 startup computational costs are 0.320/3.214 s for HBM-ready requests, 0.623/3.820 s for requests requiring DRAM restoration, 1.760/12.633 s for requests requiring SSD loading, and 8.797/45.466 s for requests requiring prefix recomputation. Although only 7.21% of resumed requests require SSD loading or recomputation, these paths contribute 23.03% of total startup computational cost. DRAM restoration is required by 17.24% of resumed requests and contributes another 19.42%. Across successive 10-minute tracereplay windows, requests collectively incur 3,238-4,266 s of startup computational cost. The HBM-ready, DRAMrestoration, and SSD-loading/recomputation paths contribute 51.22-71.40%, 11.79-24.78%, and 16.81-27.78% of this computational cost, respectively. Figure 3 summarizes the distribution and temporal composition of request startup computational cost. Finding 3. Coding-agent request startup computational costs vary substantially across KV-state preparation paths, and each path’s contribution to the system’s startup computational cost changes over time.

KV-state residency changes continuously. Asynchronous tool-call returns repeatedly reactivate different coding-agent sessions, changing which reusable KV states reside in each storage tier. Between adjacent 10-minute boundaries, 80.00100.00% of HBM-resident session identities, 79.59-91.67% of DRAM-resident identities, and 34.95-53.24% of SSD-resident

Session context growth and tool-call durations vary widely. Across 3,801 multi-round sessions, the ratio between the maximum and first-round context lengths is 4.45× at p50, 13.96× at p90, and 37.77× at p99; 54.64% of sessions grow by at least 4×. Tool-call duration also varies widely. Among 419,735 completed tool calls, duration is 0.209 s at p50, 8.866 s at p90, 4

Session percentile

75

n=419,735

50

Top 1%: 69.8% Top 5%: 88.7% Top 10%: 94.9% Top 25%: 98.6% Top 50%: 99.7%

25 0

0

25

50

75

60

82.84% ≥ 4× 52.8% 30.02% ≥ 8×

40 21.1%

16.9%

20 0

100

n=102,809

9.0%

0.24%

<2

2-4

4-8

8-16 ≥ 16

Within-iteration context ratio (max/min)

Tool calls ranked by duration (%)

cost increase (%)

Tool-call duration concentration

Decode computationa

100

Iteration share (%)

Cumulative tool time (%)

Context growth (max/first, ×)

Session context growth 256 n=3,801 128 54.64% of sessions grow ≥ 4× p99: 37.77× 64 32 p90: 13.96× 16 8 p50: 4.45× 4 2 1 0 25 50 75 100

Median

80 60

p95

n=76,689

box: p25-p75; whisker: p10-p90

40 20 0

Median: +25.95% Output rate: -22.58%

<2

2-4

4-8

8-16 ≥ 16

Within-iteration context ratio (max/min)

Figure 5: Left: share of multi-request decode iterations grouped by their within-iteration context-length ratio, defined as the maximum divided by the minimum materialized context length. Right: decode computational cost increase relative to a near-uniform context composition with the same request count and total context tokens, grouped by the observed context-length ratio. Boxes span p25 to p75, center lines mark the median, whiskers span p10 to p90, and diamonds mark p95.

Figure 4: Left: maximum-to-first context-growth ratio across session percentiles for 3,801 multi-round sessions. Right: cumulative share of total tool-call elapsed time contributed by 419,735 completed positive-duration tool calls ranked from longest to shortest. Dashed diagonal: uniform contribution.

30.013 s at p95, and 180.002 s at p99. Figure 4 summarizes these session-growth and tool-return distributions. Finding 4. Uneven context growth and widely varying toolcall durations cause coding-agent sessions to reach different context lengths and resume execution asynchronously.

containing the same number of requests and exactly the same total context tokens. As shown in the right panel of Figure 5, the observed composition has a 25.95% higher median computational cost per iteration, and 95.84% of the iterations incur at least 10% higher computational cost. Across the same output-token work, the output rate decreases by 22.58%, from 320.791 to 248.368 tokens/s. Finding 7. Decode efficiency depends on within-batch context composition even when request count and total contexttoken volume remain unchanged.

Context lengths vary widely within decode iterations. We examine each context vector immediately after the standard continuous batching planner forms a decode iteration. Across 102,809 multi-request iterations, the largest materialized context is at least 2× the smallest in 99.76% of iterations, at least 4× in 82.84%, and at least 8× in 30.02%. The largestto-smallest context-length ratio is 6.32× at p50 and 19.69× at p95, while the context-length CV is 0.58 at p50 and 0.93 at p95. The left panel of Figure 5 summarizes this withiniteration heterogeneity.

Decode-context composition differs across workers. We replay multi-worker serving with SMetric’s KV-reuse- and load-aware routing [22]. At each routing decision, we record SMetric’s selected worker and the other workers that satisfy the same KV-affinity and load constraints, and measure the context-length distance on each worker. Among 12,160 routing decisions, 3,135, or 25.78%, retain at least two candidate workers after these constraints. In 2,444 of these decisions, or 77.96%, the most context-compatible candidate differs from SMetric’s selected worker. Selecting that candidate reduces the mean context-length distance by 34.25%, showing frequent opportunities to reduce mixed-context interference identified in Finding 7.

Finding 5. Standard continuous batching frequently places coding-agent requests with widely different materialized context lengths in the same decode iteration. High prefix reuse makes coding-agent execution decodedominant. Tool-follow-up requests account for 87.89% of requests in the trace and achieve 97.55% token-weighted prefix reuse. Consequently, returning sessions largely reuse their accumulated prefixes and prefill only newly appended tokens. During the stable trace replay, 752,143 of 754,670 model iterations, or 99.67%, contain only decode work. These puredecode iterations account for 79.34% of aggregate modelexecution time, compared with 13.69% for mixed prefilldecode iterations and 6.97% for prefill-only iterations.

Finding 8. Even after preserving KV reuse and balancing load, workers can have different decode-context compositions, making context-length compatibility a meaningful routing signal.

Finding 6. High prefix reuse makes coding-agent execution decode-dominant, with pure-decode iterations consuming 79.34% of model-execution time.

4

Within-batch context heterogeneity reduces decode efficiency. We analyze 76,689 pure-decode iterations from the trace-replay. For each iteration, we compare its observed context-length composition with a near-uniform composition

C ODA introduces an admission layer between logical request readiness and physical GPU execution. The layer maintains three complementary readiness views. (i) Logical readiness 5

System Design: The C ODA Admission Layer

indicates whether a request is eligible to proceed. (ii) State readiness characterizes the work required to make a request’s KV state available for GPU execution. (iii) Execution readiness characterizes whether a request selected by state admission can be efficiently integrated into the current model iteration. C ODA bridges the gaps between these readiness views through controlled admission.

State Readiness

Logical Readiness User Req Ready Reqs Tool Return

4.1

Admission Model

HBM

DRAM

SSD

Recom.

KV State GPU Active Incre. Prefix

Execution Readiness Complete Batch Model Execution

Current Context

Compat. Cohorts

Figure 6: Illustration of the three readiness stages in a request’s lifecycle. Logical readiness indicates that a request is available after a user request arrives or a tool call returns. State readiness characterizes the work required to make its KV state available in GPU HBM, including tier restoration, recomputation, and incremental-input prefill. Execution readiness indicates whether the request can execute efficiently with the currently active requests: compatible contexts execute as one complete batch, while heterogeneous contexts are organized into compatible cohorts (detailed in §4.3).

We define three readiness views applied in sequence along the path from request availability to GPU execution: • Logical readiness. Logical readiness characterizes whether a request is eligible to proceed. A request becomes logically ready when (i) a user submits a new request or (ii) a tool call returns and the agent session can continue. • State readiness. State readiness characterizes the work required to make a request’s KV state available in GPU HBM. The three cases follow the storage hierarchy from highest to lowest readiness: (i) the KV state is resident in HBM and immediately usable; (ii) reusable KV state resides in CPU DRAM and requires restoration; or (iii) reusable KV state exists only on SSD and must be loaded, or no reusable copy exists and the prefix must be recomputed.

(ii) Execution admission determines how the participating requests are organized for attention execution according to their execution readiness. The following subsections describe Tiered-Aging state admission (§4.2) and Compatibility-Aware execution admission (§4.3).

• Execution readiness. Execution readiness characterizes the context-length compatibility between a state-selected request and the current participating requests. It distinguishes two conditions: (i) the request is compatible with the active context composition and can execute efficiently with the existing population; or (ii) the request introduces substantial context heterogeneity, requiring the resulting population to be reorganized for efficient model execution.

4.2

Tiered-Aging State Admission

Admission challenge. State admission selects which logically ready request has its KV state prepared next. Long-lived coding-agent sessions accumulate large reusable prefixes that may move across storage tiers during tool-induced idle periods, while asynchronous tool returns continuously reshape the ready population. Consequently, newly ready requests incur substantially different computational costs for state preparation (Findings 1-4). C ODA uses Tiered-Aging state admission to exploit these differences while preserving progress. Tier-aware computational cost of state preparation. As described in §4.1, the current placement of a request’s KV state determines the work required before GPU execution. A coding-agent continuation contains a persistent session prefix and an incremental suffix introduced by a user request or tool return. The reusable prefix may be distributed across storage tiers, while the incremental suffix requires prefill computation. At decision time t, C ODA estimates the computational cost of state preparation for each request i:

Together, these views define the admission path: the admission layer uses logical readiness to form the candidate set, state readiness to select which request is prepared next, and execution readiness to organize the participating requests for efficient attention execution. Figure 6 illustrates this threestage admission path. Dynamic readiness. Readiness evolves rapidly in codingagent serving. User requests and asynchronous tool returns continuously change the logically ready population; longlived session KV states grow and move across storage tiers; and resumed sessions repeatedly reshape the active context composition (Findings 1-5). Consequently, state and execution readiness may change after a request first becomes logically ready. C ODA reevaluates these views at their corresponding decision opportunities rather than fixing them at request arrival. Admission decisions. To bridge these dynamically changing readiness views, the admission layer makes two decisions: (i) State admission determines which logically ready request has its KV state prepared next according to its state readiness.

prefill

Cstate (t) = Trestore (t) + Tload (t) + Ti i i i

(t),

(1)

where the three terms capture CPU-to-GPU restoration, SSDto-HBM loading, and prefill computation for incremental suffix tokens or unavailable prefix tokens, respectively. HBMresident portions contribute no computational cost for restora6

T0

tion or loading, while different portions of the same KV state may contribute to different terms. Let F (t) denote the logically ready requests that satisfy the current HBM-capacity constraint; all other requests are temporarily infeasible. If F (t) = ∅, no state-admission decision is made. State computational cost preference. Using the tier-aware computational cost, C ODA prefers the feasible request requiring the least work to make its KV state HBM-ready: iSCF (t) = arg min Cstate (t). i i∈F (t)

A-HBM B-DRAM C-SSD D-HBM T0:

Cstate (t) i , 1 + wi (t)/τ

Computational Age cost

C"

DRAM

T3

E-HBM

F-DRAM

Computational Age cost

C"

HBM DRAM

SSD

SSD

Admit D: HBM Hit T2: Computational Age cost HBM

(2)

T2

T1: A D

HBM

C"

DRAM SSD Admit B: DRAM to HBM

This preference applies the shortest-job-first principle, which prioritizes shorter jobs to reduce aggregate waiting [20]. Here, “job size” refers to state-preparation work rather than the request’s total execution time. For coding-agent workloads, prioritizing shorter preparation paths prevents a returning session with a large lower-tier KV state from blocking multiple HBMready session continuations. Age-adjusted preference. Selecting iSCF (t) improves immediate efficiency, but asynchronous tool returns may continuously introduce computationally inexpensive HBM-ready continuations and repeatedly bypass sessions whose growing ready KV states reside in lower tiers. Let ri denote the most recent time at which a user-request arrival or tool completion made request i logically ready. Its ready-wait age is ready wi (t) = t − ri . Time spent executing the preceding tool is excluded because the request is not yet logically ready. C ODA uses this age to adjust the state computational cost: b i (t) = C

T1

Admit A: HBM Hit

T3: Computational Age cost HBM

C"

DRAM SSD Admit C: SSD to DRAM to HBM

Figure 7: Illustrative example of Tiered-Aging state admission. A-HBM means that Request A arrives with its KV state resident in HBM. At successive admission opportunities, requests with HBM-resident KV state and less additional prefill work are initially preferred because they incur lower computational costs for state preparation, while aging progressively increases the admission preference of requests that remain ready. C ODA selects the request with the lowest age-adjusted state computational cost. Once the bound is reached, the oldest feasible request takes precedence. The guard overrides the efficiency preference but never violates the HBM feasibility constraint. Figure 7 illustrates how the computational cost of state preparation and waiting age affect admission order. Dynamic reevaluation. At every state-admission opportunity, C ODA recomputes F (t), Cstate (t), and wi (t) while retaining i ready each request’s ready timestamp ri . User-request arrivals and tool completions immediately introduce new candidates, while tier movement during tool-induced idle periods changes their state computational costs. Tier-aware computational cost therefore captures where a persistent session’s KV state resides when the session resumes, while age captures how long the continuation has waited after becoming logically ready. State admission outcome. After selecting i∗ (t), C ODA performs the state-preparation work indicated by the request’s current state readiness, combining HBM reuse, CPU restoration, SSD loading, and prefix recomputation as needed. Once the required KV state becomes available in HBM, the request proceeds to execution admission. State admission therefore determines which request is prepared next, while execution admission determines how the participating requests execute.

(3)

where τ > 0 controls how quickly waiting weakens the state computational cost preference.1 Requests with lower computational costs of state preparation are therefore preferred initially, while accumulated age progressively increases the competitiveness of requests with more computationally expensive state preparation. Progress-bounded selection. Age adjustment progressively relaxes the state computational cost preference, while a final progress guard strictly bounds how long feasible requests can be repeatedly bypassed. Let o(t) = arg maxi∈F (t) wi (t) denote the oldest feasible request. C ODA makes the Tiered-Aging decision as follows:  o(t), wo(t) (t) ≥ Wmax , i∗ (t) = (4) b i (t), otherwise, arg min C i∈F (t)

4.3 Compatibility-Aware Execution Admission

where Wmax > 0 bounds how long the state-efficiency preference may override ready-time order.1 Below this bound,

Admission challenge. Execution admission determines how the requests participating in a model iteration are organized for attention execution. The contexts of long-lived coding-agent sessions grow at different rates, while the sessions resume asynchronously after tool calls. Consequently,

1 The aging timescale τ can be calibrated offline on representative

workloads to balance SLO-compliant throughput and admission waiting, while Wmax can be configured according to the evaluation configuration’s admission-waiting tolerance. Both remain fixed across evaluation workloads; the settings are reported in Appendix A.

7

10K

requests with substantially different materialized context lengths frequently execute together, reducing decode efficiency (Findings 4-7). C ODA therefore uses CompatibilityAware execution admission to select an efficient execution plan based on the current context composition. Context-length view. At an eligible pure-decode iteration occurring at time t, let D (t) denote the participating requests, n(t) = |D (t)|, and Li (t) the materialized context length of request i. Let πj , j = 1, . . . , n(t), be the request ranked j-th by context length, so that

89K

66K

14K

43K

30K

30K

Order by current attention context: compute tokens + one scheduled token

Selected Cut k* = 5

10K

14K

25K

30K

30K

Shorter-context cohort

43K

66K

89K

Longer-context cohort

Per model layer: full-row QKV projection + one K/V scatter to the paged cache K/V for all requests is written to the paged KV cache once; both launches then read it.

Attention launch 1: shorter cohort Q rows [0: k*) → output rows [0: k*)

(5)

o2

Attention launch 2: longer cohort Q rows [k*: n) → output rows [k*: n)

writes output [k*: n)

writes output [0: k*)

o1

Lπ1 (t) ≤ Lπ2 (t) ≤ · · · ≤ Lπn(t) (t).

25K

o3

o4

o5

o6

o7

o8

In the same layer: full-row output projection + residual/norm + dense/MoE

These context lengths are obtained directly from the current execution state rather than from predictions of generation length or future tool-return time. This allows C ODA to observe the context heterogeneity created by coding-agent sessions at different stages of context growth. Compatibility-aware cohort construction. To evaluate whether cohorting is beneficial, each candidate cut k partitions the context-ordered requests into a shorter-context cohort and a longer-context cohort:

Figure 8: Illustrative example of Compatibility-Aware execution admission. C ODA reorders a heterogeneous puredecode batch by context length and partitions it into shorterand longer-context cohorts. Each model layer computes QKV and updates the shared paged KV cache once, after which the two cohorts execute through separate attention launches and write to disjoint regions of the same output buffer. All requests remain in the iteration, and the remaining layer computation proceeds over the complete reordered batch.

Dkshort (t) = {π1 , . . . , πk }, Dklong (t) = {πk+1 , . . . , πn(t) }. (6) To avoid inefficiently small cohorts, each cohort must contain at least m(t) = max(Mmin , ⌈ρ n(t)⌉) requests. Here, Mmin ∈ N+ is the minimum efficient cohort size, and 0 < ρ ≤ 1/2 prevents highly imbalanced splits.2 Accordingly, K (t) contains the integer cuts satisfying m(t) ≤ k ≤ n(t) − m(t). If n(t) < 2m(t), this set is empty, and C ODA retains completepopulation execution. Compatibility-aware preference. When K (t) ̸= ∅, C ODA compares each feasible cut’s estimated computational cost of two-cohort attention with that of complete-population attention and selects the cut with the largest predicted relative gain: b full (t) − C b cohort C (t) k k∗ (t) = arg max . (7) full k∈K (t) b (t) C

executing attention in two context-compatible launches. Both cohorts execute within the same model iteration and together cover exactly the requests in D (t). Thus, execution admission changes the physical organization of attention without duplicating complete model execution or changing the participating request set. Figure 8 illustrates this cohort-based organization of attention execution. Dynamic reevaluation. At every eligible pure-decode iteration, C ODA reconstructs D (t) and reevaluates its materialized context composition. Session progress, request completion, and asynchronous tool returns may therefore change the preferred organization from one iteration to the next. C ODA falls back to complete-population execution when the population is insufficiently heterogeneous, the execution shape is unsupported, or the predicted gain is insufficient.

full

(t) denotes the estimated attention-execution time b cohort of the complete population, while C (t) denotes the comk bined attention-execution time of the two cohorts, including their additional computational overhead. The two-cohort organization is applied only when its conservative predicted relative gain is at least Γ = 0.05; otherwise, C ODA retains complete-population execution.2 Compatibility-aware execution. Under the selected cohort organization, the model performs a single shared dense/MoE traversal and preserves normal KV-update semantics while b Here, C

5

System Design: The C ODA Routing Layer

For coding-agent serving with multiple workers, C ODA introduces a routing layer that determines the worker assignment for each arriving request. The layer considers three complementary placement signals. (i) KV reuse captures the reusable session prefix available at each worker. (ii) Worker load captures the pressure created by active requests. (iii) Context-length compatibility captures how closely the arriving context matches each worker’s active decode population. C ODA applies these signals hierarchically: it preserves useful KV affinity when load-safe and uses context-length compatibility to select among the remaining load-eligible workers

2 The minimum cohort size M min , minimum cohort fraction ρ, gain threshold Γ, and computational cost model for attention execution are calibrated offline once for each model-hardware configuration and remain fixed across workloads, as detailed in Appendix A.

8

Before: reuse- + load-aware GPU 0

GPU 1

GPU 0

10K 12K

8K

48K

T0

After: + context compatibility-aware

48K

w 40K w 50K w 45K w 48K

10K 12K 8K

GPU 1

Tool starts New chunk

Pressure-triggered Reclaim

Tool returns Reuse

Async publish

Release HBM copy

Restore

CPU Shadow

Remain avaliable

Figure 10: Illustrative example of CPU-shadow offloading. A completed KV-state chunk is asynchronously published from GPU HBM to the CPU shadow. Under HBM pressure, the GPU copy can be reclaimed without an eviction-time transfer, while the CPU copy remains available for subsequent restoration and reuse.

(Finding 8). Reuse- and load-aware placement. C ODA adopts SMetric’s reuse- and load-aware policy [22] as its parent routing policy. A session’s first request initially selects a least-loaded worker. For a session continuation, the router prefers the worker with the largest actual local-prefix hit and preserves this affinity while the worker remains within the configured overload bound. This policy exploits the locality of long-lived coding-agent sessions while limiting load imbalance caused by asynchronous tool-return bursts. Compatibility-aware placement. When KV affinity does not determine the destination, C ODA considers workers at the minimum load or one active request above that minimum, and selects among them according to context-length compatibility. For an arriving request i at routing time t, its compatibility with worker w is: ℓ 1 log2 arr . ∑ |Cw (t)| ℓ∈C (t) Li (t)

T2

GPU HBM

w 40K w 50K w 45K w 48K

Figure 9: Illustrative example comparing routing decisions. When KV affinity is inconclusive, reuse- and load-aware routing selects the minimum-load GPU, while C ODA selects a load-safe GPU whose active contexts better match the arriving 48K request.

di,w (t) =

T1

for KV-state preparation and model execution.

6

Implementation

Runtime integration. We implement C ODA as an orchestration layer with approximately 2.9 KLoC of core policy and planning code, connected to the inference backend through an adapter. The adapter adds hooks to request dispatch, waitingqueue selection, and model execution. At each hook, C ODA reads the current runtime state and returns a routing, stateadmission, or execution-admission decision. The backend continues to handle memory allocation, KV-state movement, preemption, model execution, and sampling. Routing and admission paths. The dispatcher maintains asynchronously refreshed worker snapshots and uses a consistent snapshot version for each arriving request. The resulting routing decision is attached to the request before normal dispatch. At each state-admission opportunity, C ODA examines a bounded frontier of at most 64 candidate entries and queries the KV manager for reusable-prefix matches, KV-block demand, and usable HBM capacity. Candidate inspection is read-only, and the selected entry is promoted through one remove-and-prepend operation before native scheduling continues. For an accepted execution plan, C ODA reorders the persistent batch through state-safe row swaps and constructs cohort-specific views of the required attention metadata. Each model layer performs one complete-population KV update, invokes the existing attention operator once for each contextcompatible cohort, and writes each cohort’s results into the corresponding region of the original output buffer. Dense or MoE computation and sampling remain unchanged. CPU-shadow offloading and HBM reclaim. The KV connector asynchronously publishes completed KV-state chunks to a shared CPU shadow in DRAM and restores matching KVstate chunks on demand. Once publication completes, the corresponding HBM copy can be reclaimed without an evictiontime GPU-to-CPU transfer. Figure 10 illustrates CPU-shadow publication, HBM reclaim, and on-demand restoration. When

(8)

w

Here, Cw (t) is the multiset of materialized context lengths of worker w’s active decoders, |Cw (t)| is the number of those decoders, ℓ is one decoder’s context length, and Larr i (t) is the arriving request’s reusable prefix plus incremental input. The resulting di,w (t) is a dimensionless routing context-length distance; a smaller value indicates a more compatible worker. C ODA selects the eligible worker with the smallest distance, forming more homogeneous decode populations while preserving the parent policy’s reuse and load protections. Figure 9 illustrates how context-length compatibility can change the routing decision when KV affinity is inconclusive. Dynamic reevaluation. C ODA refreshes local KV reuse, worker load, and active context composition at every routing opportunity. These measurements reflect only the current system state and do not rely on predicted output lengths or future tool-return times. When context telemetry is unavailable or no load-eligible worker has an active decode population, C ODA retains the reuse- and load-aware destination. Routing changes only the arriving request’s assignment and does not migrate requests that are already GPU-active. Once assigned, the request enters the selected worker’s admission layer (§4) 9

89.6

93.9 74.0

94.9

49%↓

157

16%↓

0.921 124

27%↓

0.758 114 36%↓

1.46

0.672 189

102 80.5

54.1

58.8

34.2

37.1

103

57%↓

33%↓ 42%↓

57%↓

1.99

1.2

123

2.56

5%↓

2.53

80%↓

1.64

231 3.07

44%↓

2.1

57%↓

501

1.06 156

Q3 7%↓

40%↓

636

642

366 10%↓

2.08

vLLM 771

47%↓

1.46

44%↓

205 2.53

5%↓

222 2.24

6%↓

325 1.87

12%↓

445

18%↓

565

570

SMetric

Q2

20%↓

48%↓

4%↓

684

352

56.6

• RQ1: How does C ODA compare with existing systems in throughput, SLO-compliant throughput, and latency across workloads and model sizes?

Figure 11: Throughput comparison for Qwen3-30B on Q1-Q3 under setup 2: output, request, and SLO TPS.

• RQ2: How do routing, state admission, and execution admission contribute to C ODA’s performance, and what runtime behavior explains their benefits?

7.1

482

38%↓

We design the evaluation3 to answer two research questions:

612

618

CacheWise

Q1 9%↓

Output TPS (tokens/s) Request TPS (requests/s)

Evaluation

741

1.85

SLO TPS (tokens/s)

7

Coda

70%↓

decode execution requires additional HBM blocks, the backend automatically invokes HBM reclaim. Following CacheWise’s cache-reclaim integration [21], C ODA determines the HBM reclaim victims using a tool-call-aware policy. Within a bounded victim window, it prioritizes valid, uniquely owned KV states predicted to remain inactive longer based on toolcall identity and elapsed tool-call time. Incomplete, shared, or unverified KV states retain their native LRU order.

latencies for end-to-end (E2E) latency, time between tokens (TBT), and time to first token (TTFT), as well as prefix reuse rate and queue length.

Experimental Setup 7.2

Hardware Environment. We consider four configurations: (i) Setup 1: One NVIDIA A100-80GB GPU for admissiononly ablation. (ii) Setup 2: 4 NVIDIA A100-80G GPUs, NVLink at 300 GB/s intra-node. (iii) Setup 3: 16 NVIDIA A100-80G GPUs across two nodes (8 GPUs/node), NVLink at 300 GB/s intra-node, InfiniBand at 50 GB/s inter-node. Baseline systems. We compare C ODA with three baseline systems. (i) CacheWise [21] improves KV-cache reuse through prefix-aware scheduling and predictive eviction guided by tool-call metadata. (ii) SMetric [22] balances load and KVcache reuse by routing each session’s first request based on load and subsequent requests based on cache locality. (iii) vLLM [13] serves as an unoptimized baseline using roundrobin routing and FIFO request admission. Workload traces. We derive workload traces by subsampling TraceLab [30] at three request arrival rates, denoted Q1, Q2, and Q3, corresponding to 3, 5, and 7 requests/s. The subsampling procedure preserves the key long-tailed properties of the original traces, including the distribution of tool-call intervals. For single-GPU admission ablation, we use M1, M2, and M3, which contain approximately one quarter of the sessions in Q1, Q2, and Q3, respectively, retaining complete sessions with unchanged token counts and tool-call intervals. Models and metrics. We use Qwen3-Coder-30B-A3B-FP8 and Qwen3-235B-A22B-FP8 as the served models; both are popular open-source models [25]. Each Qwen3-30B replica is served on a single GPU, whereas each Qwen3-235B replica is served on eight GPUs. Following prior work [21, 22], we evaluate overall throughput, SLO-compliant throughput, tail

End-to-end Evaluation (RQ1)

We first examine whether C ODA improves both serving throughput and latency. The subsequent ablations separate the contributions of routing, state admission, and execution admission, while the decision-level measurements examine how these mechanisms affect runtime behavior. 30B end-to-end throughput. C ODA achieves the highest output-token, completed-request, and SLO-compliant throughput on all three workloads (Figure 11). Relative to the strongest competing baseline for each metric and trace, the gains are 4.2-12.1%, 4.9-10.0%, and 15.9-44.3%, respectively. The larger gains in SLO-compliant throughput show that the improvement extends beyond raw token generation to output that meets the latency targets. For example, on Q3, C ODA produces 642.1 output tokens/s and 103.2 SLO-compliant output tokens/s, compared with CacheWise’s 598.3 and 71.5. Relative to vLLM, the output-token and SLO-compliant throughput gains reach 56.9% and 70.3%. The advantage over CacheWise and SMetric further shows that the gains are not limited to replacing the unoptimized baseline. We isolate the contributions of routing and admission in Figure 14. 30B request latency (TTFT and E2E). Figure 12 reports TTFT and E2E latency, capturing request startup and completion, respectively. Relative to SMetric, C ODA reduces p50 and p95 TTFT by 22.0-53.5% and 8.0-67.6%, respectively. It also reduces p50 E2E latency by 12.1-33.0%. Its p95 E2E latency is comparable on Q1 and 23.2-34.2% lower on Q2 and Q3. Relative to CacheWise, E2E latency is comparable or lower, with p50 reductions of 8.6-9.8% on Q2 and Q3 and a p95 reduction of 5.9% on Q1. Compared with vLLM, C ODA reduces p95 TTFT and E2E latency by 66.0-68.6% and

3 All research reported in this paper, including access to and use of models,

datasets, and code, was conducted by the university-affiliated authors.

10

CacheWise

0

0

33%↓

10%↓

1%↑ 28%↑ 33%↑

34%↑

36%↓

80%↓

21%↓ 17%↓ 24%↑

2%↑ 11%↑

60.9

21%↑ 27%↑

22.0 13.4

3%↑

17%↑ 20%↑

23%↑

16%↓

18%↓

7%↓

8%↓

8%↓

12%↓ 12%↑

5%↑ 7%↑

61.9

12%↑ 18%↑

38%↓ 4%↑

8%↑

12%↑ 12%↑

22.1 14.4

4%↓

3%↓

4%↓

30%↓

17%↓

50.0

7%↑ 6%↑

18.8 11.3

E2E Latency SLO TPS (seconds) (tokens/s)

42%↑

34%↑

31%↑ 33%↑

2%↓ 28%↑ 33%↑

Q3

35.7 17.8

14%↑

7%↓ 5%↑ 4%↑

53.5

50.9

5%↑ 1%↑ 8%↑

17.2

6%↑ 8%↑

34.4

18.0 11.6

3%↑ 11%↑

16%↑ 26%↑

24.2

51.6

3%↑

90.2 71.3

48.4

196

54.4

86.4 68.7

3%↑

143

105 96.7

21%↑

137

64.0

187

166

251

18%↓

248

6%↑

301

227

72.5

771 642 636 501 366 231 124 103 102 80.5 58.8 37.1 111 83.2 55.5 27.7 0

CacheWise and 3.1-7.2% higher than SMetric. Its mean and p50 E2E latency are 6.1-15.7% and 8.2-25.9% lower than CacheWise’s, respectively, while its p95 latency is comparable or up to 20.6% lower. Against SMetric, the mean E2E latency remains comparable, and the p50 and p95 latencies are comparable or lower by up to 11.2% and 14.3%, respectively. Overall, these comparisons demonstrate C ODA’s throughput and E2E latency advantages for the 235B model.

Q3

9%↑ 22%↑

240

Q2

17.7 10.9

238

684 570 565 445 325 205 114 94.9 93.9 74.0 54.1 34.2 96.8 72.6 48.4 24.2 0

P95

Q2

Figure 14: Routing and admission ablation for Qwen3-30B on Q1-Q3 under setup 2: output, SLO TPS, and E2E latency.

P95 3%↓

288

P50

SMetric

P50 12%↓

291

6%↓

288

Q1 16%↓

349

22.1 12.7

E2E Latency (seconds)

Output TPS (tokens/s)

Mean

741 618 612 482 352 222 189 157 156 123 89.6 56.6 65.8 49.3 32.9 16.4 0

Q1

Output TPS (tokens/s)

69%↑

68%↑

93.3 13%↑

59.8

31.7

60.9

Figure 12: Latency comparison for Qwen3-30B on Q1-Q3 under setup 2: TTFT, observed TBT, and E2E latency. Coda

Coda w/o Admission Opt Coda w/o Both Opt

Mean

7.65 5.08 19.4 4%↓ 3%↑ 0%↓ 11%↑ 3%↓ 15%↓ 54%↑ 9%↓ 54%↑ 26%↓ 15%↑ 53%↑ 17%↑ 43%↑ 6%↓

Q3

4%↑ 9%↑

74.0 55.5 37.0 18.5 0 126 94.3 62.9 31.4 0 127 95.1 63.4 31.7 0

22.0 13.4

36%↑

29%↑ 29%↑

61.9

2%↑

23%↑

46.4 29.0

8.58 6.38 71.3 23.6 15%↑ 12%↑ 11%↑ 1%↓ 17%↑ 23%↑ 18%↓ 39%↑ 10%↓ 34%↑ 21%↓ 43%↑ 16%↑ 55%↑ 11%↑ 37%↑ 3%↑ 66%↑

Q2

8%↑ 10%↑

83.2 62.4 41.6 20.8 0 103 77.1 51.4 25.7 0 117 87.7 58.5 29.2 0

22.1 14.4

6%↑

4%↑ 24%↑ 20%↑

12%↑ 12%↑

7%↑ 3%↑

50.0

43%↑

33.4 25.6

6.5 4.15 56.1 20.3 16%↑ 0%↑ 11%↑ 7%↑ 19%↑ 9%↓ 1%↑ 10%↑ 1%↑ 22%↑ 3%↓ 8%↑ 10%↑ 54%↑ 1%↓ 32%↑ 0%↑

67%↑

Q1

18.8 11.3

E2E Observed TBT TTFT (seconds) (milliseconds) (seconds)

73.2 54.9 36.6 18.3 0 82.8 62.1 41.4 20.7 0 105 79.1 52.7 26.4 0

Coda Coda w/o Routing Opt

vLLM

P95

9%↑

SMetric P50

9%↑ 4%↑

CacheWise Mean

17%↑ 20%↑

Coda

0

7.3

Ablation and Mechanism Analysis (RQ2)

Component ablation. Figure 14 isolates the contributions of routing and admission by disabling routing optimization, admission optimization, or both. Compared with the routingonly variant, the full system gains 3.7-20.8% in output-token throughput and 12.1-17.3% in SLO-compliant throughput by enabling admission. Compared with the admission-only variant, enabling routing yields gains of 2.7-10.1% and 8.2-35.7% in these two metrics, respectively. With both optimizations enabled, C ODA outperforms the variant with neither optimization by 4.2-33.2% in output-token throughput and 15.8-80.3% in SLO-compliant throughput. Its p50 E2E latency is also 12.1-33.0% lower than this variant’s, while p95 is comparable on Q1 and 23.2-34.2% lower on Q2 and Q3. These gains support the complementary roles of routing in shaping each worker’s request and KV-state composition and admission in controlling preparation order and attention grouping. Thus, the combined design achieves the highest output-token and SLO-compliant throughput across Q1-Q3 while maintaining comparable or lower E2E latency than every ablated variant. Single-GPU admission ablation. We examine C ODA’s admission performance in greater detail using single-GPU experiments (Figure 15). Compared with CacheWise, C ODA achieves 3.1-14.3% higher output-token throughput and 8.8104.7% higher SLO-compliant throughput, with both gains

Figure 13: Throughput and E2E latency comparison for Qwen3-235B on Q1-Q3 under setup 3: output TPS and mean, p50, and p95 E2E latency.

36.5-43.1%, respectively. Taken together, these results show that C ODA improves request responsiveness over SMetric and vLLM while maintaining comparable or lower completion latency than CacheWise. 30B token-delivery latency (TBT). C ODA achieves comparable or lower TBT than CacheWise, including p95 reductions of 10.9-18.7%. Relative to vLLM, p50 TBT is comparable on Q1 and 11.1-17.0% lower on Q2 and Q3; p95 TBT is comparable on Q1 and Q2 but 6.1% higher on Q3. Compared with SMetric, p50 and p95 TBT are comparable on Q1 but higher on Q2 and Q3. SMetric’s lower TBT on these two traces is consistent with smaller effective decode batches: the decodeconcurrency in Figure 16 is 13.0% and 22.6% higher for C ODA. In summary, C ODA combines higher throughput with comparable or lower TBT than CacheWise, while SMetric’s lower TBT on Q2 and Q3 accompanies lower throughput. 235B results. Figure 13 presents the attainable performance of C ODA on the larger model in a larger cluster. C ODA achieves 11.7-18.3% higher output-token throughput than 11

CacheWise

P95

35%↓

14%↓

Trace Q1 Q2 Q3 Total/mean

191%↓

105%↓ 10%↓

6%↑ 37%↑ 56%↑

Trace M1 M2 M3 Total/mean

M2

M3

Output tok/s 130.792 130.933 134.800 132.587 143.473 148.732 146.622 163.902 167.652

C ODA Dist. 1.059 1.048 1.324 1.155

Reduction 24.9% 29.8% 23.7% 26.1%

Useful tok/s 1535.248 1542.755 1564.598 1784.415 1785.320 1819.360 2040.227 2085.903 2189.557

Changes 16 37 77 130

CacheWise TBT 69.30 ms 81.79 ms 84.00 ms 78.36 ms

C ODA TBT 65.98 ms 69.47 ms 71.74 ms 69.06 ms

Reduction 4.8% 15.1% 14.6% 11.9%

and 36.6%, respectively. Its output-token and SLO-compliant throughput are comparable to CacheWise’s on M1, with larger gains on M2 and M3. Adding CA while keeping TA unchanged yields further gains of 2.9%, 2.9%, and 15.6% in these three mean throughput metrics, respectively. The additional SLO-compliant throughput gain ranges from 10.0% to 25.6% across traces. The combined policy achieves the highest values for all three metrics on every trace, demonstrating the complementary benefits of state and execution admission. Routing-decision changes. Table 2 examines how contextlength compatibility affects worker selection relative to SMetric’s reuse- and load-aware routing. During the five-minute measurement windows, C ODA selects a different worker in 57, 80, and 83 decisions on Q1, Q2, and Q3, respectively. For these changed decisions, its selected workers reduce the mean context-length distance score by 24.9%, 29.8%, and 23.7% relative to SMetric’s choices. Across all 220 changed decisions, the pooled score decreases from 1.563 to 1.155, a 26.1% reduction. These results show that C ODA’s compatibility signal changes placement toward closer context-length matches, complementing the throughput benefits established by the routing ablation in Figure 14. Admission-choice changes. Table 3 compares C ODA’s admission choices and mean observed TBT with CacheWise’s. Within the five-minute decision windows, C ODA makes 16, 37, and 77 different admission choices on M1, M2, and M3, respectively. Over the 10-minute measurement windows, its mean TBT is comparable to CacheWise’s on M1 and 15.1% and 14.6% lower on M2 and M3. Averaging the three tracelevel means equally gives 69.06 ms for C ODA versus 78.36 ms for CacheWise, an 11.9% reduction. These TBT measurements describe token delivery over each 10-minute window, not only requests with changed admission choices. Together with Table 1, this comparison shows that C ODA changes admission behavior while achieving higher throughput and lower average token-delivery latency. KV reuse. Figure 16 shows the contributions of GPU-resident

Table 1: Cumulative ablation of single-GPU admission policies on M1-M3. TA applies Tiered-Aging state admission to the CacheWise baseline; CA further adds CompatibilityAware execution admission. Policy CacheWise +TA +CA CacheWise +TA +CA CacheWise +TA +CA

SMetric Dist. 1.411 1.493 1.735 1.563

Table 3: Admission-choice changes in C ODA relative to CacheWise, with mean observed TBT for both systems.

Figure 15: Performance comparison on M1-M3 under setup 1: output, SLO TPS, and mean, p50, and p95 E2E latency.

Workload M1

Changes 57 80 83 220

18.6

87.0

142%↓

M3

168

5%↓ 1%↓

11%↑

20%↑ 39%↑ 54%↑

79%↓

16.8

201 166 131 95.6 60.4 22.3 18.3 14.3 10.3 6.26 111 83.3 55.5 27.8 0

23.2 12.4

32%↓

M2 12%↓

149

72.6

178 147 116 84.8 53.5 20.2 16.7 13.1 9.59 6.06 108 81.2 54.1 27.1 0

P50

21.8 11.4

13%↓

3%↓

6%↑

20%↑ 27%↑ 45%↑

69.5

104%↓

13.9

9%↓

135

5%↓ 5%↑

162 133 105 76.8 48.5 16.7 13.8 10.9 7.93 5.01 105 78.4 52.3 26.1 0

23.1 11.4

E2E Latency (seconds)

SLO TPS (tokens/s)

Output TPS (tokens/s)

Mean

M1

Table 2: Changes in C ODA’s routing decisions relative to SMetric. Distances are averaged over changed decisions.

SMetric/vLLM

5%↑ 8%↑

Coda

SLO tok/s 12.788 12.370 13.917 9.378 13.402 16.830 9.070 16.883 18.567

increasing from M1 to M3. Against the shared SMetric/vLLM FIFO baseline, its output-token and SLO-compliant throughput improve by 12.5-35.2% and 104.4-190.7%, respectively, while mean, p50, and p95 E2E latency decrease by 27.339.0%, 44.9-55.6%, and 6.0-20.2%. Compared with CacheWise, C ODA maintains comparable E2E latency on average across M1-M3; the equal-weight means of p50 and p95 latency are 3.8% and 2.4% lower, respectively. These results show that C ODA’s admission layer increases output-token and SLO-compliant throughput while maintaining comparable average E2E latency to CacheWise. Admission-policy ablation. Table 1 examines two successive admission refinements to the CacheWise baseline: TieredAging state admission (TA; §4.2), followed by CompatibilityAware execution admission (CA; §4.3). In addition to output-token and SLO-compliant throughput, we report useful throughput, which counts each session’s initial prefix once, appended input, and in-window output for completed requests, divided by the window duration. Compared with CacheWise, TA increases the equal-weight mean output-token, useful, and SLO-compliant throughput across M1-M3 by 6.9%, 1.0%, 12

Coda

CacheWise

Decode Concurrency Cache Reuse (requests) (%)

GPU hit 108 80.8

88

Q1

84

83

53.9

100 62

75.1

SMetric KV-store hit

82

Q2

78

73

50.1

95.2 55

71.4

25.0

23.8

0 28.5

0 36.9

0 50.9

23

23

23

20

27.7

30

26

27

26

78

38.2

14.3

18.5

25.5

7.13

9.23

12.7

0

0

0

38

[1] Anthropic. Claude Code Overview. https://code. claude.com/docs/en/overview. Accessed: September 7, 2026.

Q3

72

66

47.6

26.9

21.4

References

vLLM

42

31

52

[2] Fengyao Bai, Hongbin Zhang, Zhitao Chen, Jiangsu Du, Zhiguang Chen, and Yutong Lu. Alignedserve: Orchestrating prefix-aware batching to build a highthroughput and computing-efficient llm serving system. Proceedings of the ACM on Management of Data, 4(3 (SIGMOD):1–25, 2026.

32

[3] Islem Bouzenia and Michael Pradel. You name it, i run it: An llm agent to execute tests of arbitrary projects. Proceedings of the ACM on Software Engineering, 2(ISSTA):1054–1076, 2025.

Figure 16: GPU and KV-store hit rates and the mean decodeconcurrency on Q1-Q3 under setup 2.

and KV-store hits to total reuse. C ODA achieves total reuse rates of 88.3%, 82.1%, and 78.0% on Q1-Q3, respectively, exceeding CacheWise by 4.3-5.6 percentage points, SMetric by 5.6-11.6 points, and vLLM by 25.7-27.0 points. The source breakdown also distinguishes total reuse from GPU residency: on Q1, CacheWise has a larger GPU-hit share, but C ODA achieves higher total reuse through additional KV-store hits. C ODA thus achieves higher total reuse across all three traces by drawing on both GPU-resident KV state and the KV store. Decoding concurrency. The lower panel of Figure 16 demonstrates cluster-wide decoding concurrency. C ODA yields 23.3, 30.2, and 37.5 concurrent requests on Q1-Q3, respectively. Its concurrency is comparable to SMetric’s on Q1 and 13.0% and 22.6% higher on Q2 and Q3, respectively. It is also 16.718.1% higher than vLLM’s. On Q3, however, CacheWise has a larger concurrency but lower output-token and SLOcompliant throughput than C ODA. This comparison reinforces the importance of combining KV reuse with efficient execution rather than maximizing decoding concurrency alone.

8

[4] Chaokun Chang, Yukun Zhou, Kaihua Fu, Dakai An, Tianyu Feng, Hanfeng Lu, Sheng Yao, Pu Guo, Yinghao Yu, Yizhou Shan, et al. From llm inference to agentic workloads: Characterization and implications for serving systems. arXiv preprint arXiv:2608.15127, 2026. [5] Cursor. Cursor Agent. https://cursor.com/docs/ agent/overview. Accessed: September 7, 2026. [6] Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. {Cost-Efficient} large language model serving for multi-turn conversations with {CachedAttention}. In 2024 USENIX annual technical conference (USENIX ATC 24), pages 111–126, 2024. [7] Shipeng Hu, Guangyan Zhang, Yuqi Zhou, Yaya Wei, Ziyan Zhong, and Jike Chen. Bidaw: Enhancing {KeyValue} caching for interactive {LLM} serving via bidirectional {Computation–Storage} awareness. In 24th USENIX Conference on File and Storage Technologies (FAST 26), pages 101–116, 2026.

Conclusion

[8] Youhe Jiang, Fangcheng Fu, Xiaozhe Yao, Guoliang He, Xupeng Miao, Ana Klimovic, Bin Cui, Binhang Yuan, and Eiko Yoneki. Demystifying cost-efficiency in llm serving over heterogeneous gpus. arXiv preprint arXiv:2502.00722, 2025.

We presented C ODA, a coding-agent serving system that exploits admission flexibility to improve serving efficiency while preserving request progress. Through direct trace analysis and replay, we showed that ready requests can incur substantially different KV-state computational costs for state preparation and execute inefficiently together because of heterogeneous context lengths. Guided by these findings, C ODA introduces Tiered-Aging state admission, Compatibility-Aware execution admission, and a separate routing layer for multi-worker configurations. Experiments across models and workloads demonstrate gains in outputtoken and SLO-compliant throughput, while component ablations identify the contributions of admission and routing. These results highlight the benefit of treating admission as a first-class optimization target.

[9] Youhe Jiang, Fangcheng Fu, Xiaozhe Yao, Taiyi Wang, Bin Cui, Ana Klimovic, and Eiko Yoneki. Thunderserve: High-performance and cost-efficient llm serving in cloud environments. Proceedings of Machine Learning and Systems, 7, 2025. [10] Youhe Jiang, Ran Yan, Xiaozhe Yao, Yang Zhou, Beidi Chen, and Binhang Yuan. Hexgen: Generative inference of large language model over heterogeneous environment. arXiv preprint arXiv:2311.11514, 2023. 13

[11] Youhe Jiang, Ran Yan, and Binhang Yuan. Hexgen-2: Disaggregated generative inference of llms in heterogeneous environment. In International Conference on Learning Representations, volume 2025, pages 3985– 4002, 2025.

[22] Jiahao Wang, Kaizhan Lin, Kaixi Zhang, Jinbo Han, Xingda Wei, Sijie Shen, Chenguang Fang, Wenyuan Yu, Rong Chen, and Haibo Chen. Smetric: Rethink llm scheduling for serving agents with balanced sessioncentric scheduling. arXiv preprint arXiv:2607.08565, 2026.

[12] Hao Kang, Ziyang Li, Weili Xu, Xinyu Yang, Yinfang Chen, Junxiong Wang, Beidi Chen, Tushar Krishna, Chenfeng Xu, and Simran Arora. Thunderagent: A simple, fast and program-aware agentic inference system. arXiv preprint arXiv:2602.13692, 2026.

[23] Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. In International Conference on Learning Representations, volume 2025, pages 65882–65919, 2025.

[13] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pages 611–626, 2023.

[24] Tian Xia, Hanchen Li, Zhifei Li, Xiaokun Chen, Hao Kang, Yifan Qiao, Yi Xu, and Ion Stoica. Idleness is relative: Exploiting tool-call idle windows for offloading in agentic systems with mori. arXiv preprint arXiv:2606.00866, 2026.

[14] Banruo Liu, Haoran Qiu, Íñigo Goiri, Rodrigo Fonseca, Ricardo Bianchini, and Esha Choukse. Agentic coding in the wild: Characterizing github copilot traces at production scale. arXiv preprint arXiv:2608.00101, 2026.

[25] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.

[15] Yuhan Liu, Yihua Cheng, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaoting Feng, Yuyang Huang, Samuel Shen, Rui Zhang, Kuntai Du, et al. Lmcache: An efficient kv cache layer for enterprise-scale llm inference. arXiv preprint arXiv:2510.09665, 2025.

[26] John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528–50652, 2024.

GDS Backend. https: [16] LMCache Project. //docs.lmcache.ai/kv_cache/storage_ backends/gds.html. Accessed September 2026. [17] OpenAI. Codex CLI. https://learn.chatgpt.com/ docs/codex/cli. Accessed: September 7, 2026.

[27] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.

[18] Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Heyi Tang, Feng Ren, Teng Ma, Shangming Cai, Yineng Zhang, Mingxing Zhang, et al. Mooncake: A kvcachecentric disaggregated architecture for llm serving. ACM Transactions on Storage, 22(4):1–38, 2026.

[28] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for {Transformer-Based} generative models. In 16th USENIX symposium on operating systems design and implementation (OSDI 22), pages 521– 538, 2022.

[19] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning, pages 31094–31116. PMLR, 2023. [20] Wayne E Smith. Various optimizers for single-stage production. Naval Research Logistics Quarterly, 3(12):59–66, 1956.

[29] Li Zhang, Youhe Jiang, Guoliang He, Xin Chen, Han Lv, Qian Yao, Ningsheng Ma, Fangcheng Fu, and Kai Chen. Lmdeploy accelerates mixed-precision llm inference with turbomind. arXiv preprint arXiv:2508.15601, 2025.

[21] Shubham Tiwari, Tapan Chugh, Nash Rickert, Simon Peter, Ratul Mahajan, and Haiying Shen. Cachewise: Understanding workloads and optimizing kvcache management for efficiently serving llm coding agents. arXiv preprint arXiv:2606.16824, 2026.

[30] Kan Zhu, Mathew Jacob, Chenxi Ma, Yi Pan, Stephanie Wang, Arvind Krishnamurthy, and Baris Kasikci. Tracelab: Characterizing coding agent workloads for llm serving. arXiv preprint arXiv:2606.30560, 2026. 14

A

Profiling Details

per-cohort total and maximum context lengths. Conservative gain estimate. When a paired profile is available, it directly estimates the relative gain of a full-versuscohort execution pair and applies a held-out residual bound. Otherwise, C ODA derives the conservative gain from the b full (t) and the sum of the upper lower duration bound of C duration bounds of the two cohorts plus the profiled launch b cohort allowance in C (t). Thus, uncertainty makes cohort execuk tion harder, rather than easier, to select. Missing, unsupported, or out-of-domain profiles retain complete-population execution. Cohort parameters. The implemented controller considers cohort construction only with at least four participating requests and a complete-population maximum-to-minimum context-length ratio of at least 1.5. It uses Mmin = 2, ρ = 0.35, an additional-launch allowance of 0.3 ms, and Γ = 0.05. Consequently, each cohort contains at least two requests and at least 35% of the participating population, and cohort execution must provide at least 5% conservative predicted gain after accounting for its additional launch. Gain ties favor the more balanced split and then the smaller cut index. These gates are validated with the configuration profile and remain fixed across workloads. Workload independence. Profiling is performed offline, and C ODA performs no online fitting or per-trace retuning. Runtime decisions use only current KV-state residency, missingprefix work, ready-wait age, and materialized context lengths; they do not use future output lengths or future tool-return times. Workload dynamics therefore change the inputs to the admission decisions but not the calibrated computational cost functions or their parameters.

Profile scope. C ODA profiles each model–hardware–servingengine configuration once and binds the resulting artifacts to that configuration. All estimated times are represented in milliseconds. Changing the model, GPU, KV representation, transfer topology, or serving engine requires reprofiling; within one configuration, profiles and policy parameters remain fixed across workloads. Computational cost calibration for state preparation. We profile the operations that contribute to Cstate (t) separately. i CPU-to-GPU restoration is measured over multiple transfer sizes and represented by a piecewise-linear H2D service-time curve. At runtime, Trestore (t) combines the profiled time for i the request’s CPU-resident KV bytes with the outstanding completion time in its transfer domain. Prefill is profiled as a function of the number of tokens that lack reusable KV prefill state; Ti (t) is obtained from the unavailable-prefix and incremental-suffix token counts. The last measured segment is used for positive-slope extrapolation beyond either profile’s largest point. HBM-resident portions incur no computational cost for restoration, and the evaluated CPU-shadow configuration incurs no eviction-time D2H transfer. The current prototype and reported experiments do not configure an SSD tier, so Tload (t) = 0; an SSD-enabled configuration requires a sepi arately measured loading curve. Requests whose required KV blocks exceed current HBM capacity remain outside F (t). Tiered-Aging parameters. The evaluated configuration uses τ = 5,000 ms and Wmax = 20,000 ms. After a request waits for τ, an otherwise unchanged state computational cost is divided by two; Wmax sets the threshold for oldest-request priority, subject to HBM feasibility. Equal age-adjusted computational costs are resolved by raw state computational cost, additional KV blocks, and ready-time order. These parameters are configuration-level controls, not values learned from request arrivals or tool-call durations, and remain fixed across all evaluated traces. For another configuration, Wmax can be configured from the admission-waiting tolerance, while τ can be calibrated through an offline sweep on representative workloads to balance SLO-compliant throughput and admission waiting. Computational cost calibration for execution. We profile pure-decode attention using exact vectors of materialized context lengths on the target model–hardware–serving-engine configuration. Each measurement compares the completepopulation path with candidate two-cohort paths while excluding the shared dense/MoE traversal and common KV update. We use two warmup executions, five measured repetitions, and two passes per shape. Whole context shapes, rather than individual repetitions, are partitioned across fitting, calibration, and validation sets. The duration model uses batch size, log total context length, maximum context length, and context-length coefficient of variation. The paired gain model additionally uses the context-length ratio, cut fraction, and 15

Related documents

• Coda: Exploiting Admission Flexibility for Coding-Agent Serving#1187766
• MTAC-IFBench: Benchmarking Instruction-Following in Multi-Turn Agentic Coding#1084892
• ParaAgent: Reinforcing Parallel Acting in Open-World Tool Environments#1118330
• Terminal-Bench-LILT: Multilingual Agentic Coding Benchmark Grounded in Language, Region, and Culture#792891
• Terminal-Bench-LILT: Multilingual Agentic Coding Benchmark Grounded in Language, Region, and Culture#788887