TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving

Preprint

TAO: T OOL -A WARE J OINT S CHEDULING FOR LLMBASED AGENTIC S ERVING Shuhuai Li Shanghai Jiao Tong University

arXiv:2610.37230v1 [cs.DC] 27 Oct 2026

Dongdong Ge Shanghai Jiao Tong University

Shengdong Ye Shanghai Jiao Tong University Yinyu Ye Stanford University

Jianghao Lin∗ Shanghai Jiao Tong University

A BSTRACT Agentic services alternate between model inference on clustered GPUs and external tools on distributed CPUs. In this scenario, existing systems (1) overlook tool-return readiness in service evaluation, wasting limited resources, and (2) use heuristic strategies to decide admission and placement on GPUs, resulting in suboptimal scheduling decisions. To bridge this gap, we present TAO, a tool-aware joint scheduling framework for LLM-based agentic serving. Its value interface combines near-term tool readiness, node-specific cache credit with time-aware decay, and token footprint; a dynamic program jointly selects and places waiting sessions under capacity constraints. Across the 12 reported model–workload–concurrency settings, TAO reduces latency by up to 26.8%, reduces mean time to first token (TTFT) by up to 36.3%, and increases decode throughput by up to 29.5% relative to the best evaluated state-of-the-art baseline for each metric.

1

I NTRODUCTION

Recent advances in language models have expanded their use beyond basic chatbots to complex agents, making CPU-based tool execution a substantial source of latency alongside GPU-based inference (Qiao et al., 2026; Huang et al., 2026). Specifically, an agent operates through an agent session that alternates between LLM inference and external tools, such as compilers and retrievers, to complete multi-step tasks (Anthropic, 2026a; OpenAI, 2026). When tools run outside the inference service, a session’s demand for GPU computation becomes interruptible. We call this serving scenario Clustered-GPUs with Distributed-CPUs (CGDC), illustrated in Figure 1(Left): clustered GPUs, i.e., nodes, provide shared inference capacity, while external tool calls introduce delays between LLM requests1 . Hence, efficient agentic serving must account for both inference demand and tool timing to sustain throughput and session responsiveness. We find that existing serving systems generally follow an evaluate-then-decide scheduling pattern (Guy et al., 2025; SGLang Team, n.d.; vLLM Production Stack Team, n.d.). They evaluate node features at the session level and then decide where and when to serve each session. This separates the problem into two related challenges: (1) how to value a session’s next model turn, and (2) how to select and place sessions under shared capacity constraints. For session evaluation, existing systems in CGDC scenarios generally overlook tool-call latency when evaluating established workload features. Cache-aware systems reuse KV prefixes and balance inference load (Zheng et al., 2024; Srivatsa et al., 2025; Qin et al., 2025; NVIDIA, 2025); workflow-aware systems anticipate future calls (Wadlom et al., 2026; Pan et al., 2026); and latencyaware schedulers prioritize sessions using latency or remaining-work estimates (Choi et al., 2025; Chen et al., 2025). Tool timing provides another useful signal, yet timing is utilized only as a surrogate for KV reuse or token count: Li et al. (2025) predicts tool duration to choose KV-cache retention times, while Kang et al. (2026) discounts the token footprint of sessions executing tools as time elapses. Such features commonly enter fixed scoring rules or manually chosen thresholds. ∗ 1

Corresponding author. E-mail: linjianghao@sjtu.edu.cn. One agent session consists of multiple LLM inference requests (i.e., steps) after tool calls.

1

Preprint

Evaluate then Decide

Clustered GPUs

Distributed CPUs CGDC

GPU Node 1

Evaluation Signal

Scheduling Decision M erging Heuristic

Submitted Session in Waiting Pool

w/ Searching Service

KV Cache Affinity

Developer Requests

Scheduler

GPU Node 2 w/

Evaluate + Decide

CPU Resource SLO Awareness

Maintainer AI -powered Clients

w/

Tool Calls

OR

Total Token Count

Short but Low Cache

OR

Urgent but Tool Running

Single Feature Priority Session Still Waiting in Pool

...... I nference Latency

Eval.

Rule-Based Selection

GPU Node 3 Online Service

M erged with Weights

Tool-Return Readiness

Overlooked

Tool-Return Ready & High Cache Reuse

Figure 1: Left: In the CGDC serving scenario, external tool calls create intervals between LLM requests and opportunities to reallocate inference capacity. Right: Two scheduling challenges motivate TAO, including (1) overlooking tool-return readiness and using heuristic merging for evaluation, and (2) making myopic and suboptimal decisions about session admission and placement. For decision making, existing systems usually apply heuristic scheduling strategies to each session, leading to myopic and suboptimal placements. In llm-d and SGLang (Guy et al., 2025; SGLang Team, n.d.), per-session scoring and routing rules assign inference opportunities individually. Yet sessions compete for the same node capacity. In particular, assigning one session can prevent a more valuable combination of other sessions from being served. Furthermore, the heuristic selection and placement rules can also lead to suboptimal choices (Peng et al., 2026; Yuan et al., 2026). As shown in Figure 1(Right), they fail to formulate scheduling jointly across multiple sessions. To address these challenges, we propose TAO, a Tool-Aware jOint scheduling framework for LLMbased agentic serving. To incorporate tool latency into session evaluation, we propose tool-return readiness and explicitly integrate it with established features such as KV-cache reuse. Following Kang et al. (2026), we interrupt sessions during tool calls and put them into a waiting pool. Toolreturn readiness estimates which sessions will need inference within the next scheduling window. For each session–node pair, we form a value triplet from readiness, time-decayed KV-cache reuse, and token footprint. Given these evaluation triplets, we design a joint scheduling algorithm in the decision making stage that considers all sessions in the waiting pool. Across deployments in different scenarios, joint optimization considers how the admission of each session affects the placements available to others. The optimizer maximizes the additive evaluation triplets using dynamic programming. Their coordinates evolve with cache age and tool progress, allowing distinct signals to guide decisions over time without a weighted sum. Separating the optimization procedure from session evaluation makes the framework extensible. We summarize our contributions as follows: • We identify the Clustered GPUs with Distributed CPUs (CGDC) serving scenario in the era of LLM-based agents, which shifts the key bottleneck of latency to tool calls. • We propose TAO, a tool-aware joint scheduling framework for LLM-based agentic serving. It improves the evaluate-then-decide paradigm through session evaluation with explicit tool-return readiness and decision making with joint modeling and optimization. • TAO can be generalized and extended to different workloads such as multi-agent orchestration and Looped Agents. Moreover, it is also compatible with non-agent chatbot scenarios. • Across the 12 reported model–workload–concurrency settings, TAO reduces latency by up to 26.8%, reduces mean time to first token (TTFT) by up to 36.3%, and increases decode throughput by up to 29.5% relative to the state-of-the-art baseline for each metric.

2

BACKGROUND AND P ROBLEM D ESCRIPTION

2.1

AGENT S ESSION AND DISTRIBUTED RESOURCES

An agent session is a persistent interaction that alternates between model requests and external tool calls. During a model request, it uses a serving node; during tool execution, it awaits a result outside the inference service. A scheduler records each session’s state and controls session admission, while the serving engine manages physical KV-cache residency and eviction. 2

Preprint

A later model request can reuse the KV prefix produced by earlier turns and avoid prefill computation (Kwon et al., 2023). Because an agent session carries its instructions, history, and tool observations across turns, reuse depends on both the agent session history and the node selected for its next request. SGLang and Preble exploit related prefix structure in model programs and distributed scheduling (Zheng et al., 2024; Srivatsa et al., 2025). Our CGDC separates clustered GPU inference from distributed tool execution. Client or remote tools introduce variable delays between model requests, while their CPU and RAM are managed outside the GPU admission budget (Anthropic, 2026a; OpenAI, 2026; Chang et al., 2026). Tool latency therefore determines when an agent session may need GPU resources for inference again. 2.2

SESSION INTERRUPTION AND SESSION RESTART

During tool execution, a session can release its GPU resources safely, although its cached prefix may subsequently be evicted. In contrast, preempting an active inference session requires recomputation. Repeated recomputation and reconstruction cause KV-cache thrashing, wasting capacity on state that must be rebuilt. This phenomenon motivates cache-aware engine and scheduling work (Kwon et al., 2023; Zheng et al., 2024; Srivatsa et al., 2025; NVIDIA, 2025), as well as policies that weigh retaining state against serving other work (Li et al., 2025; Abhyankar et al., 2024). Therefore, at a tool boundary, the scheduler can apply session interruption. Specifically, a scheduler can interrupt the agent session’s model admission and release its node resources while the external tool continues. When inference service is needed again, session restarting restores admission on an eligible node and resumes the existing interaction. Session interruption and session restarting describe session state transitions. They are decoupled from the underlying inference engine. We find that the decisions made for these two operations can be based on the same session evaluation. Thus, we optimize toward a unified goal for maximizing the evaluation. Session interruption can be decided by ordering sessions according to their values. Interrupted sessions join new arrivals in a waiting pool. We focus on unified session evaluation and restarting decisions from this pool.

3

M ETHOD

3.1

F RAMEWORK OVERVIEW

We present TAO, which combines agentic session evaluation with decision making, i.e. dynamic programming for joint selection and placement. The detailed formulation appears in Section 3.2. Figure 2 illustrates the pipeline of our method. First, session evaluation maps agent session and node observations to assignment values (Section 3.2), including KV cache with time-aware decay (Section 3.3) and tool-return readiness (Section 3.4). Second, an exact dynamic program jointly selects waiting agent sessions and assigns them to nodes under residual capacity constraints (Sections 3.2 and 3.5). Separating session evaluation from decision making lets the evaluation interface evolve while preserving the scheduling procedure. 3.2

F ORMULATION AND AGENTIC SESSION EVALUATION

At each restart scheduling decision, let the waiting pool P contain the agent sessions awaiting inference admission, including interrupted sessions and new arrivals. Let B be the nodes in the frozen snapshot. Each agent session i has admission cost wi , and each node b has residual capacity Cb . Let E ⊆ P × B denote eligible assignments, and let xi,b ∈ {0, 1} indicate whether session i is admitted to node b. The feasible set is ) ( X X X = x: xi,b ≤ 1, wi xi,b ≤ Cb , xi,b = 0 if (i, b) ∈ /E . (1) b∈B

i∈P

TAO assigns each agent session–node pair the lexicographic value vi,b = (ki,b , ri , ti ), 3

(2)

Preprint

Clustered GPUs

Distributed CPUs

External Tools Separate CPU / RAM budget

Session Evaluation

Cache Observations

Vi,b = (ki,b, ri, ti)

Snapshot

2

Triplet Evaluation Token Count: ti

Waiting Sessions

Session Prefix (i)

Waiting: ri = 0

s1

s1

Ready: ri = 1

vi,b

s2 Node Cache (b)

s3 Tools running

Tools Return Ready for Inference

Admission & Placement

DP Matrix

Online Validation

Select & Place

GPU b1

𝑇 = ∑𝑤𝑖 𝑥𝑖,𝑏

s2

𝑇 ≤ 𝐶𝑏 ?

s3

…

s4

Free Capacity

Decision Making Asynchronous Calculation

Tool-Return Readiness: ri 1.2

Reusable Prefix with Affinity

KV-Cache in Use

Cb1 Cb3 Engine-managed KV Allocation, Reuse & Eviction

KV-Cache Credit: ki,b Time Decay

New KV-Cache Demand Evitable KV-Cache

GPU b3

Inference Nodes b ∈ B

Tool Observations

1

…

GPU b1

GPU b2

…

Figure 2: The TAO evaluate-then-decide framework in CGDC. External tools on distributed CPUs provide historical tool-call durations, while the serving engines on clustered GPUs provide cache observations. Then, the session evaluation component constructs triplets (decayed KV-cache credit, binary readiness, token count). The decision making component jointly selects and places waiting agent sessions. The executor validates the plan against live state before applying it. The resulting assignments schedule sessions to eligible nodes. where ki,b is decayed cache credit, ri is the binary readiness signal, and ti is token count. We jointly select and place sessions to maximize their aggregate value: XX x⋆ ∈ arg lexmax xi,b vi,b . (3) x∈X

i∈P b∈B

The evaluation triplet’s order remains fixed while its values evolve over time. Our design rationale is detailed as follows: • KV-Cache credit takes precedence, but its value decays quickly after interruption, because prefix reuse directly avoids repeated prefill, motivating cache-aware scheduling (NVIDIA, 2025). The age-based discount in Section 3.3 favors recent cache evidence, which is often most useful when a tool returns quickly. • As unrefreshed credit decays and the first component is equalized, tool-return readiness can distinguish agent sessions nearing a return from those still far from their next model turn (Section 3.4). • Token footprint then favors larger accumulated contexts among plans tied on cache credit and readiness. The optimizer accounts for the greater capacity cost of such contexts when choosing the complete set of assignments. 3.3

T IME - AWARE CACHE VALUE FOR C LUSTERED GPU S

Cached / prompt tokens (%)

TAO estimates reusable KV state from observations associated with each target node. It uses 100 Descriptive fit an observed cached-token count when available, Median (95% CI) or estimates the count from observed token vol80 ume and cache rate. A node without matching evidence receives zero credit. Figure 3 reports 60 observed cache-token retention in DeepSeek40 V4 Flash traces from two nodes with 4×H100 per node; the decline in binned medians mo20 tivates discounting older observations. Points show binned medians with bootstrap 95% inter0 vals; the curve is a descriptive exponential fit 101 102 103 over the populated bins through 1,800 s. Older Pause age before next request (s) observations receive less credit: 0 ki,b (t) = ki,b 2−τi,b /h , Figure 3: Observed cache-token retention in (4) DeepSeek-V4 Flash traces from two nodes with τi,b = max(0, t − tobs ). i,b 4×H100 per node. 4

Preprint

0 Here ki,b is the observed cache credit and h is its decay half-life. We use h = 10 seconds and round the resulting credit for valuation. The discount makes the admission value responsive to observation age; the serving engine continues to control physical allocation, reuse, and eviction. The half-life is a scheduler parameter, distinct from the fitted trace time constant. Dynamo similarly adjusts overlap value under excess prefill load, and ThunderAgent discounts token demand over time (NVIDIA, 2025; Kang et al., 2026).

3.4

T OOL - RETURN READINESS IN A DISTRIBUTED CPU

In our CGDC deployment, external executors manage tool CPU and RAM, while TAO controls GPU admission. Tool-return readiness connects these resource domains by identifying sessions likely to request model service within a short horizon. Tool timing reflects external execution conditions; the GPU admission budget accounts for the returning session’s inference footprint. Tool interfaces can expose progress or return deadlines: Claude Code’s Monitor forwards background-work events, while OpenCode reports termination after a Bash timeout (Anthropic, 2026b; OpenCode Contributors, 2026). The predictor combines these hints, parsed sleep commands, and historical tool durations, using direct timing evidence first and falling back to the tool-specific history, global history, or a five-second b f gives default (Li et al., 2025). For a call with elapsed time ei (t), the historical estimate D b f − ei (t)}. dbi (t) = max{0, D

(5)

The implementation adds a finite-sample margin to the historical mean; Appendix A specifies this estimator in detail. Binary readiness. Let oi (t) indicate observed tool completion, ai (t) indicate an established tool execution candidate with an eligible prediction source, and gi (t) indicate valid prediction timestamps refreshed within 30 seconds. Write zi (t) for the earlier available predicted completion or exposed return deadline, treating an unavailable deadline as +∞. With horizon H and overdue-probe indicator bi (t), we define  oi (t) = 1, 1, ri (t) = ai (t)bi (t), (6) oi (t) = 0, zi (t) ≤ t,  ai (t)gi (t)1[zi (t) ≤ t + H], otherwise. TAO uses H = 1 second; the scheduler’s session interruption valuation uses its configured scheduling interval. Overdue predictions receive temporary probe opportunities instead of remaining ready indefinitely. Observed completion takes precedence over prediction checks. This signal prioritizes near-term returns among plans tied on higher-precedence values. 3.5

J OINT SCHEDULING WITH DYNAMIC PROGRAMMING

The optimizer is a capacity-state dynamic program for the multiple-knapsack and generalizedassignment structure (Kellerer et al., 2004; Shmoys & Tardos, 1993). Let ∆ be the capacity quantum, w̄i = ⌈wi /∆⌉, and C̄b = ⌊Cb /∆⌋. For candidate j, let Dj (s) be the best objective attainable from the first j candidates at the used capacity s, and let uj,b embed the value and count for assigning j to node b:   lex Dj (s) = lexmax Dj−1 (s), max [Dj−1 (s − w̄j eb ) + uj,b ] . (7) b eligible

Only nonnegative states within C̄ are allowed; D0 (0) = 0 and other initial states are unreachable. Skipping or assigning each candidate exhausts the feasible choices, so induction gives exactness for the frozen, discretized problem; rounding can exclude plans feasible before discretization. Node-specific cache observations are decayed and folded into assignment values during O(nm) preprocessing; cache-equivalent nodes can be grouped into g ≤ m aggregate capacity dimensions without moving physical KV blocks. The planner runs in an isolated asynchronous worker (detailed in Appendix B) and revalidates before application; recorded preparation plus completed-solve work is below 0.12% of runtime in the four-replica sweep (Appendix I). 5

Preprint

Table 1: Main results for DeepSeek-V4 Flash on 2 nodes with 4×H100 per node and Qwen3-32B on 4 nodes with 2×H100 per node. DeepSeek uses Terminal-Bench to study more demanding tasks at moderate concurrency; Qwen3-32B uses SWE-verified at higher concurrency to exercise the capacity of its smaller model replicas. Bold and underline mark the best and second-best distinct values in each concurrency-level/metric column. DeepSeek-V4 Flash, 2 nodes with 4×H100 per node 16 workers Benchmark

SWE-Lite

Terminal-Bench

32 workers

64 workers

Latency ↓

TTFT ↓

Decode ↑

Latency ↓

TTFT ↓

Decode ↑

Latency ↓

TTFT ↓

Decode ↑

Dynamo vLLM Scheduler w/ RR vLLM Scheduler w/ CA ThunderAgent TAO (Ours) Rel. Imprv. (%)

121.993 122.814 116.011 123.564 113.264 +2.4

0.146 0.177 0.183 0.379 0.135 +7.4

874.085 888.489 889.590 796.215 934.967 +5.1

152.297 158.841 150.431 215.727 142.332 +5.4

0.176 0.217 0.226 2.708 0.153 +13.3

1393.267 1145.560 1256.064 961.791 1477.463 +6.0

269.801 290.686 253.228 407.180 185.410 +26.8

0.351 0.508 0.410 7.447 0.224 +36.3

1548.472 1299.754 1561.307 969.018 2021.523 +29.5

Dynamo vLLM Scheduler w/ RR vLLM Scheduler w/ CA ThunderAgent TAO (Ours) Rel. Imprv. (%)

536.600 342.464 304.802 361.061 298.224 +2.2

1.371 0.306 0.319 0.336 0.239 +21.7

494.866 820.754 723.889 876.138 798.624 -8.8

579.650 375.182 394.677 380.108 333.587 +11.1

1.252 0.499 0.331 1.402 0.264 +20.3

651.151 1071.063 1146.434 1053.042 1161.438 +1.3

688.531 467.410 393.065 391.580 376.647 +3.8

1.315 0.582 0.387 1.010 0.324 +16.2

760.736 1597.497 1654.108 1376.539 1551.387 -6.2

Method

Qwen3-32B, 4 nodes with 2×H100 per node 96 workers Benchmark

SWE-Lite

SWE-verified

128 workers

160 workers

Latency ↓

TTFT ↓

Decode ↑

Latency ↓

TTFT ↓

Decode ↑

Latency ↓

TTFT ↓

Decode ↑

Dynamo vLLM Scheduler w/ RR vLLM Scheduler w/ CA ThunderAgent TAO (Ours) Rel. Imprv. (%)

821.558 775.303 811.604 716.272 634.694 +11.4

2.098 5.109 5.339 2.051 1.827 +10.9

2074.598 1400.577 1614.671 2104.896 1996.793 -5.1

857.418 975.382 881.068 839.274 829.361 +1.2

7.924 11.179 11.707 8.705 7.907 +0.2

1203.708 1359.789 1456.411 2036.280 2051.807 +0.8

1203.712 1088.036 1179.340 1028.478 1030.973 -0.2

17.208 16.645 18.304 16.209 14.554 +10.2

1239.372 1433.349 1285.413 1712.363 1827.561 +6.7

Dynamo vLLM Scheduler w/ RR vLLM Scheduler w/ CA ThunderAgent TAO (Ours) Rel. Imprv. (%)

736.823 777.584 816.201 699.801 642.017 +8.3

1.985 5.082 5.731 2.396 1.392 +29.9

1862.845 1931.333 1522.641 2184.933 2354.709 +7.8

1041.911 933.347 1016.513 911.097 933.678 -2.5

10.441 12.586 14.263 10.500 9.092 +12.9

1820.919 1804.505 1670.856 2045.875 2128.599 +4.0

1286.167 1244.525 1264.406 1202.215 1094.318 +9.0

22.469 24.221 23.199 21.608 17.619 +18.5

1667.794 1091.002 1619.768 1993.620 2087.191 +4.7

Method

4

E XPERIMENTS

4.1

E XPERIMENT S ETUP

Models and workloads. We evaluate DeepSeek-V4 Flash, Qwen3-32B, and GLM-4.7 (DeepSeek-AI et al., 2026; Yang et al., 2025; Team et al., 2025) on the SWE-bench Lite workload (SWE-Lite) using the mini-SWE-agent harness (MiniSWE) (mini-SWE-agent Contributors, 2026; Jimenez et al., 2024). For DeepSeek, we also use Claude Code on Terminal-Bench 2.0 (Merrill et al., 2026) for challenging tasks. For Qwen, we test with SWE-verified to exercise the serving capacity of its smaller model replicas. We further evaluate GLM-4.7 on SWE-Lite to test single-node performance. Appendix F reports an InferenceX trace-replay evaluation. Deployment. DeepSeek runs on 2 nodes with 4×H100 per node; Qwen runs on 4 nodes with 2×H100 per node. Each node hosts one tensor-parallel model replica. GLM-4.7 runs on one node with 8×H100. Appendix E.1 reports a lower-concurrency DeepSeek check on the same single-node topology. Baselines. vLLM Scheduler w/ RR and vLLM Scheduler w/ CA denote the vLLM schedulers with round-robin and cache-aware policies, respectively (vLLM Production Stack Team, n.d.), which align with the policy in SGLang (Zheng et al., 2024). These baselines cover load spreading and prefix-locality scheduling. Dynamo is a KV-aware scheduler (NVIDIA, 2025). ThunderAgent 6

Preprint

Table 2: High-concurrency extensibility showcase for a larger model: GLM-4.7 on SWE-Lite and SWE-verified through MiniSWE, one node with 8×H100 (TP8), and 300 tasks per run. GLM-4.7, one node with 8×H100 (TP8) 96 workers

128 workers

160 workers

Method

Latency ↓ TTFT ↓

Decode ↑

Latency ↓

TTFT ↓

Decode ↑

Latency ↓

TTFT ↓

Decode ↑

Dynamo vLLM Scheduler w/ RR vLLM Scheduler w/ CA ThunderAgent TAO (Ours) Rel. Imprv. (%)

2870.313 2687.766 2606.523 999.624 887.663 +11.2

21.677 20.927 19.553 7.014 6.777 +3.4

290.532 310.800 318.687 773.250 869.628 +12.5

3779.196 3633.795 3435.779 1360.382 1261.906 +7.2

41.865 39.502 37.665 14.243 14.103 +1.0

287.844 294.582 307.983 731.052 813.653 +11.3

4786.720 4390.961 4314.458 1747.028 1476.190 +15.5

59.548 53.588 56.326 23.345 20.066 +14.1

262.215 272.460 297.280 668.940 805.939 +20.5

Dynamo vLLM Scheduler w/ RR SWE-verified vLLM Scheduler w/ CA ThunderAgent TAO (Ours) Rel. Imprv. (%)

3164.578 2938.216 2928.613 1146.121 951.720 +17.0

26.761 23.967 24.505 12.590 7.752 +38.4

286.215 292.731 300.339 691.506 856.764 +23.9

4545.301 4136.027 3935.660 1533.286 1243.886 +18.9

52.588 47.930 46.963 18.516 15.281 +17.5

256.606 269.838 284.906 688.310 858.140 +24.7

8217.092 6149.466 5989.491 1846.499 1548.253 +16.2

86.008 71.452 71.239 26.045 21.943 +15.7

163.572 214.622 220.933 683.551 839.149 +22.8

Benchmark

SWE-Lite

uses token-count scheduling and best-fit-decreasing placement (Kang et al., 2026). TAO combines predictive cache-aware session evaluation with joint dynamic-programming placement. Metrics. We report mean task end-to-end latency, mean time to first token (TTFT), and decode throughput in tokens per second (TPS). Task latency includes both model inference and tool execution. Component analyses also measure prefix-cache hit rate, tool-end-to-first-token delay, and pause wait, which includes the tool call latency of an interrupted session. 4.2

M AIN R ESULTS

Table 1 compares five baselines across different levels of concurrency, models, and topologies. Taken together, these results make TAO the strongest overall choice: it combines consistently low latency and TTFT with high decode throughput more effectively than any single baseline, consistent with our focus on responsiveness after tool calls and improved KV reuse. DeepSeek-V4 Flash. We evaluate on challenging benchmarks with moderate concurrency. On SWE-Lite, , relative to the best competing method in each column, latency falls by 2.4–26.8%, TTFT by 7.4–36.3%, and decode throughput rises by 5.1–29.5%. On Terminal-Bench, TAO also leads on latency and TTFT at all three concurrency levels. Qwen3-32B. We use higher concurrency to exercise the serving capacity of its smaller model replicas. On SWE-Lite, TAO achieves the lowest TTFT at all three worker counts and the highest decode throughput at 96, 128 and 160 workers. On SWE-verified, TAO leads on TTFT and decode throughput throughout, with reductions of 12.9–29.9% in TTFT and gains of 4.0–7.8% in decode throughput. It also reduces latency by 8.3% and 9.0% at 96 and 160 workers; at 128 workers, latency is 2.5% higher than ThunderAgent. GLM-4.7. As a larger-model setting, GLM-4.7 provides a high-concurrency extensibility showcase on one node with 8×H100. Table 2 reports the separate worker-grouped results: on SWE-Lite at 160 workers, improvements reach 15.5% in latency, 14.05% in TTFT, and 20.48% in decode throughput. On all 500 SWE-verified tasks, TAO reduces mean task latency by 16.2–18.9% and client TTFT by 15.7–38.4%, while increasing decode throughput by 22.8–24.7% relative to ThunderAgent, the SOTA baseline on each metric. Source of Improvement. Figure 4 illustrates the serving dynamics at 160 workers. During the middle of the serving window, TAO processes ongoing requests more efficiently and steadily admits additional sessions under limited capacity, avoiding sharp bursts in concurrency. Appendix D provides the corresponding traces at other workers. 7

Preprint

TAO (60.0 min)

vLLM w/ RR (176.8 min)

ThunderAgent (70.9 min)

vLLM w/ CA (164.3 min)

Dynamo (189.4 min)

Running requests (Requests)

160

120

80

40

0

0

600

1,200

1,800

2,400

3,000

3,600

4,200

4,800

5,400

6,000

6,600

7,200

7,800

8,400

9,000

9,600 10,200 10,800 11,400

Time since each run's first client request (seconds)

Figure 4: Running-request traces for GLM-4.7 on SWE-Lite at 160 workers (TP8). Each run starts at its first client request. TAO completes the workload sooner while sustaining fewer concurrent running requests for much of the run. Table 3: Workload compatibility on mixed workloads with chatbot scenarios. Deployed on DeepSeekV4 Flash with 2 nodes 4×H100. Latency and TTFT are in seconds, and decode throughput is in token/s. 32 workers Latency ↓

Method

TTFT ↓

64 workers Decode ↑

Latency ↓

128 workers

TTFT ↓ Decode ↑

Latency ↓

TTFT ↓ Decode ↑

Terminal-Bench + SWE-Lite (389 tasks) Dynamo vLLM Scheduler w/ RR vLLM Scheduler w/ CA ThunderAgent TAO (Ours) Rel. Imprv. (%)

378.542 419.926 373.900 361.778 317.476∗ +12.2

0.524 897.840 1.088 832.544 0.707 853.498 1.149 908.751 0.553∗ 1019.827∗ -5.5 +12.2

575.024 866.508 519.631 535.400 398.911 +23.2

0.786 3.511 0.911 4.200 0.609 +22.5

949.396 681.442 879.750 1053.911 1144.432 +8.6

1065.101 1328.289 1002.259 679.212 585.804 +13.8

1.331 4.823 1.568 7.235 0.840 +36.9

723.171 635.044 751.131 1337.560 1396.591 +4.4

574.021 468.963 988.401 1264.527 1409.613 +11.5

1417.212 1232.168 1214.511 523.676 563.114 -7.0

3.166 10.884 3.938 16.870 19.123 -504.0

482.381 496.417 501.852 1140.862 1013.157 -11.2

AutomationBench (800 tasks) Dynamo vLLM Scheduler w/ RR vLLM Scheduler w/ CA ThunderAgent TAO (Ours) Rel. Imprv. (%)

147.378 293.286 129.716 107.485 101.064 +6.0

0.330 1.849 0.409 0.444 0.246 +25.6

1174.370 555.398 1283.423 1487.097 1604.639 +7.9

593.958 691.139 312.150 237.577 197.023 +17.1

1.449 3.616 1.046 5.368 0.539 +48.5

Compatibility. TAO supports heterogeneous agent workloads and composes with engine-level KV-cache management. Table 3 evaluates a mixture of Terminal-Bench and SWE-Lite alongside AutomationBench at 32, 64, and 128 workers, reporting the available results. Appendix E.2 demonstrates integration with TokenCake (Bian et al., 2025) and discusses the relationship to program-aware scheduling in Agentix (Luo et al., 2026) and workflow-aware cache management. Appendix E.1 provides the complementary low-concurrency DeepSeek evaluation on a single TP8 replica. 4.3

I MPROVEMENT B REAKDOWN

As shown in Section 4.2, the worker sweep exposes a latency–throughput tradeoff: the lowest latency and TTFT occur at the smallest worker count, while peak decode throughput occurs at 64 workers for DeepSeek SWE-Lite and Terminal-Bench, 128 workers for Qwen SWE-Lite, and 96 workers for SWE-verified. We use DeepSeek at 32 workers as a balanced operating point and examine how cache-aware valuation, tool readiness, and joint placement improve reuse, tool-return responsiveness, and admission decisions. Cache-aware valuation. In a fixed-workload replay of 218 tasks with DP placement held fixed for the ThunderAgent control and cache-aware variants, cache-token ordering raises prefix-cache hit rate 8

Preprint

Readiness Delta

Disabled Enabled

0.6 1.00

0.4

0.95

0.2 0.0 0.0

0.2

0.90 0.30

0.35

0.4

0.6

0.40

1.0

10 0.8 8 6 +4.03%

4

+2.13%

2

0.6 0.4 0.2

+0.15 pp

0

0.5 s

0.0

0.8

Tool end to first token (s)

Prediction Accuracy

+10.88%

CDF

CDF

0.8

Gain (%; cache hit in pp)

Earlier Return 1.0

de

in

co

De

s/m

t es

qu Re

e d ch t En Ca Hi oolT

0.0

0.2

0.4

0.6

0.8

Absolute error (s)

Figure 6: Readiness effectiveness in the matched 32-worker SWE-Lite traces. From left to right, the panels show the tool-end-to-first-token CDF, the aggregate readiness delta, and absolute error of the structured-history duration predictions. Tool-end measures the time from tool completion to the first output token. The left inset magnifies 0.3–0.4 s. The dashed line in the right panel marks 0.5 s, half of the 1 s scheduling period.

by 0.19 percentage points and reduces mean pause wait by 4.5% relative to ThunderAgent. Adding cache-credit decay raises hit rate by 0.31 percentage points and decode throughput by 1.68% relative to ThunderAgent. Figure 5 summarizes these changes; bars are percentage points for cache hit and percentages for the other metrics, with positive values denoting improvement.

Gain vs. ThunderAgent (pp or %)

Tool readiness. Enabling readiness in the paired SWE-Lite runs increases decode throughput by 4.03% and reduces tool-end-to-first-token delay by 10.88%. Mean pause wait falls from 4 1.067 to 0.972 s. Figure 6 shows the returndelay distributions and prediction error. Across 3 10,967 completed tools, the final duration pre2 diction has a mean absolute error of 0.410 s, 1 with 82.9% of errors within 0.5 s. The returndelay CDFs show that readiness shifts model 0 responses earlier after tool completion. The middle panel summarizes the serving gains, and the cache hit physical TTFT pause decode (pp) mean prefill right panel shows the prediction accuracy that Cache-token Cache-token + decay informs readiness valuation. The return CDFs contain 10,481 and 10,673 observations, and the prediction panel contains 10,967 predictions. Figure 5: Cache-aware valuation in the 32-worker replay, relative to the ThunderAgent control with Joint Scheduling. Joint optimization im- DP placement. Reductions in prefill, TTFT, and proves the lexicographic scheduling objective in pause wait are shown as positive improvements. six of the 23 replay snapshots with multiple candidates (26.1%), matching the short-first heuristic in the remaining 17. Across all 35 reconstructable snapshots, including single-candidate cases, DP improves six (17.1%) and matches the heuristic in 29 (82.9%). Detailed cache-credit and admission examples appear in Appendix H. 4.4

C OMPONENT A BLATIONS

Table 4 compares the complete TAO configuration with variants of its value function, pause ordering, and resumption planner. The complete configuration achieves the highest decode throughput and cache-hit rate, together with the lowest mean TTFT, post-tool delay, and pause wait. Removing cache credit lowers decode throughput from 1417.97 to 1398.35 tok/s; removing readiness lowers it to 1363.06 tok/s. Using the pause ordering from ThunderAgent yields 1355.20 tok/s, while replacing DP resumption with best-fit-decreasing (BFD) yields 1233.72 tok/s. Both valuation and decision making contribute to the observed performance. 9

Preprint

Table 4: Component variants on SWE-Lite with DeepSeek-V4 Flash, 2 nodes with 4×H100 per node, and 32 workers. Each row uses the indicated component variant of TAO. Decode is in tok/s and hit is prefix-cache reuse (%). Mean and P90 timing values are reported separately in seconds. The best results are in bold and the second-best results are underlined. Configuration

TAO (Ours) w/o Cache Value w/o Readiness w/o Cache Value & Readiness w/o Session Evaluation w/o Decision Making

5

Decode ↑ Hit (%) ↑

1417.97 1398.35 1363.06 1373.51 1355.20 1233.72

96.31 96.24 96.16 96.24 96.20 96.05

TTFT ↓

Tool-end→first ↓

Pause ↓

Mean

P90 Mean

P90 Mean

P90

0.154 0.157 0.161 0.182 0.162 0.195

0.241 0.246 0.253 0.278 0.257 0.300

0.299 0.309 0.358 0.317 0.350 0.354

1.051 1.047 1.079 1.059 1.061 1.119

0.203 0.209 0.227 0.213 0.226 0.235

0.972 1.028 1.067 1.025 1.025 1.081

R ELATED W ORK

PagedAttention and RadixAttention improve engine-level KV management, while Preble, Mooncake, and Dynamo exploit distributed cache locality (Kwon et al., 2023; Zheng et al., 2024; Srivatsa et al., 2025; Qin et al., 2025; NVIDIA, 2025). SMetric combines session affinity with load balancing (Wang et al., 2026a), and production schedulers support extensible scoring (Guy et al., 2025). Autellix prioritizes programs using service history, while ThunderAgent introduces the lifecycle and waiting pool on which TAO builds (Luo et al., 2025; Kang et al., 2026). InferCept and Continuum manage KV retention across tool calls, with Continuum using historical tool-duration distributions (Abhyankar et al., 2024; Li et al., 2025). TAO brings cache opportunity and tool readiness into a joint schedulerlevel admission and placement objective, complementing engine-level KV management. Parrot and Helium exploit workflow dependencies, while KVFlow and CacheScout guide residency or prefetching using execution structure (Lin et al., 2024; Wadlom et al., 2026; Pan et al., 2025; Zhang et al., 2026). TOPAS jointly considers prefix residency and admission through bounded search and greedy packing (Ni et al., 2026). Conveyor, Sutradhara, and PASTE reduce exposed tool latency through partial execution, overlap, or speculation (Xu et al., 2024; Biswas et al., 2026; Sui et al., 2026). TAO converts tool timing into near-term model-service opportunities that inform joint admission and placement. AgentSysBench and MARS motivate accounting for tool delays and deployment boundaries (Chang et al., 2026; Wang et al., 2026b). TAO couples an agentic value interface with joint placement and asynchronous revalidation, using the insights from classical multiple-knapsack and generalized-assignment machinery (Kellerer et al., 2004; Shmoys & Tardos, 1993). It becomes popular to formulate some optimization problems in LLM and agentic serving and solve online, from edge scenarios (Li et al., 2026) to large-scale infrastructure (Bu et al., 2026; Chen et al., 2026).

6

C ONCLUSION

Tool execution makes agentic inference demand intermittent, while accumulated context makes the choice of node consequential. TAO brings these two properties into a common scheduling objective through cache credit, tool-return readiness, and token footprint. Joint dynamic-programming placement maximizes this objective under snapshot capacity constraints, and asynchronous revalidation connects the plan to live execution. Across the evaluated DeepSeek-V4 Flash and Qwen3-32B settings, this design delivers improvements of up to 26.8% in latency, 36.3% in mean TTFT, and 29.5% in decode throughput. The resulting architecture gives agentic serving an extensible way to translate workload knowledge into coordinated admission and placement. AI USE STATEMENT We used generative AI tools to assist with manuscript editing and wording review. We did not use generative AI to run experiments, fabricate measurements, or decide which experimental artifacts are valid. We reviewed all AI-assisted text and figure suggestions against the source code, experiment records, cited literature, and the compiled manuscript. The authors take responsibility for the final content, claims, and artifacts in this submission. 10

Preprint

R EFERENCES Reyna Abhyankar, Zijian He, Vikranth Srivatsa, Hao Zhang, and Yiying Zhang. InferCept: Efficient intercept support for augmented large language model inference. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 81–95. PMLR, 2024. URL https://proceedings.mlr.press/v235/ abhyankar24a.html. Anthropic. How tool use works. https://platform.claude.com/docs/en/ agents-and-tools/tool-use/how-tool-use-works, 2026a. Official documentation; accessed September 11, 2026. Anthropic. Claude Code tools reference: Monitor tool. https://code.claude.com/docs/ en/tools-reference, 2026b. Accessed September 8, 2026. Richard Bellman. Dynamic Programming. Princeton University Press, 1957. Zhuohang Bian, Feiyang Wu, Teng Ma, and Youwei Zhuo. Tokencake: A KV-cache-centric serving framework for LLM-based multi-agent applications. arXiv preprint arXiv:2510.18586, 2025. URL https://arxiv.org/abs/2510.18586v2. Anish Biswas, Kanishk Goel, Srivarshinee S, Jayashree Mohan, Alind Khare, Anjaly Parayil, Ramachandran Ramjee, and Chetan Bansal. Sutradhara: An intelligent orchestrator-engine co-design for tool-based agentic inference. arXiv preprint arXiv:2601.12967, 2026. URL https://arxiv.org/abs/2601.12967v3. Tianci Bu, Yuan Lyu, Zixi Chen, Chendong Song, Hong Liang, Tsepten Gurung, Yuwei Fan, Yinyu Ye, and Zijie Zhou. Tackling the data-parallel load balancing bottleneck in llm serving: Practical online routing at scale. arXiv preprint arXiv:2605.06113, 2026. Chaokun Chang, Yukun Zhou, Kaihua Fu, Dakai An, Tianyu Feng, Hanfeng Lu, Sheng Yao, Pu Guo, Yinghao Yu, Yizhou Shan, Bo Li, Binhang Yuan, and Wei Wang. From llm inference to agentic workloads: Characterization and implications for serving systems. arXiv preprint arXiv:2608.15127, 2026. URL https://arxiv.org/abs/2608.15127v1. Chandra Chekuri and Sanjeev Khanna. A polynomial time approximation scheme for the multiple knapsack problem. SIAM Journal on Computing, 35(3):713–728, 2005. doi: 10.1137/ S0097539700382820. Jinyuan Chen, Jiuchen Shi, Quan Chen, and Minyi Guo. Kairos: Low-latency multi-agent serving with shared LLMs and excessive loads in the public cloud. arXiv preprint arXiv:2508.06948, 2025. URL https://arxiv.org/abs/2508.06948. Zixi Chen, Yinyu Ye, and Zijie Zhou. Online linear programming for multi-objective routing in llm serving. arXiv preprint arXiv:2607.03948, 2026. Seungbeom Choi, Jeonghoe Goo, Eunjoo Jeon, Mingyu Yang, and Minsung Jang. ELIS: Efficient LLM iterative scheduling system with response length predictor. arXiv preprint arXiv:2505.09142, 2025. URL https://arxiv.org/abs/2505.09142. DeepSeek-AI, Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chengyu Hou, Chenhao Xu, Chenze Shao, Chong Ruan, Conner Sun, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Donghao Li, Dongjie Ji, Erhang Li, Fang Wei, Fangyun Lin, Fangzhou Yuan, Feiyu Xia, Fucong Dai, Guangbo Hao, Guanting Chen, Guoai Cao, Guolai Meng, Guowei Li, Han Yu, Han Zhang, Hanwei Xu, Hao Li, Haofen Liang, Haoling Zhang, Haoming Luo, Haoran Wei, Haotian Yuan, Haowei Zhang, Haowen Luo, Haoyu Chen, Haozhe Ji, Hengqing Zhang, Honghui Ding, Hongxuan Tang, Huanqi Cao, Huazuo Gao, Hui Qu, Hui Zeng, J Yang, JQ Zhu, Jia Luo, Jia Song, Jia Yu, Jialiang Huang, Jialu Cai, Jian Liang, Jiangting Zhou, Jiasheng Ye, Jiashi Li, Jiaxin Xu, Jiewen Hu, Jieyu Yang, Jin Chen, Jin Yan, Jingchang Chen, Jingli Zhou, Jingting Xiang, Jingyang Yuan, Jingyuan Cheng, Jingzi Zhou, Jinhua Zhu, Jiping Yu, Joseph Sun, Jun Ran, Junguang Jiang, Junjie Qiu, Junlong Li, Junmin Zheng, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Kexing Zhou, Kezhao Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Wang, Leyi 11

Preprint

Xia, Li Zhang, Liang Zhao, Lihua Guo, Lingxiao Luo, Linwang Ma, Linyan Zhu, Litong Wang, Liyu Cai, Liyue Zhang, Longhao Chen, MS Di, MY Xu, Max Mei, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Mingxu Zhou, Minmin Han, Ning Wang, Panpan Huang, Panpan Wang, Peixin Cong, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qingyang Li, Qinyu Chen, Qiushi Du, Qiwei Jiang, Rui Tian, Ruifan Xu, Ruijie Lu, Ruiling Xu, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runqian Chen, Runqiu Yin, Runxin Xu, Ruomeng Shen, Ruoyu Zhang, Ruyi Chen, SH Liu, Shanghao Lu, Shangmian Sun, Shangyan Zhou, Shanhuang Chen, Shaofei Cai, Shaoheng Nie, Shaoqing Wu, Shaoyuan Chen, Shengding Hu, Shengyu Liu, Shiqiang Hu, Shirong Ma, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, Shuying Yu, Songyang Zhou, Tao Ni, Tao Yun, Tian Jin, Tian Pei, Tian Ye, Tianle Lin, Tianran Ji, Tianyi Cui, Tianyuan Yue, Tingting Yu, Tun Wang, W Zhang, WL Xiao, Wangding Zeng, Wei An, Weilin Zhao, Wen Liu, Wenfeng Liang, Wenjie Pang, Wenjing Luo, Wenjing Yao, Wenjun Gao, Wenkai Yang, Wenlve Huang, Wenqing Hou, Wentao Zhang, Wenting Ma, Xi Gao, Xiang He, Xiangwen Wang, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaokang Zhang, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingchen Liu, Xingkai Yu, Xingyou Li, Xinyu Yang, Xinyu Zhang, Xu Chen, Xuanyu Wang, Xuecheng Su, Xueyin Chen, Xuheng Lin, Xuwei Fu, YC Yan, YQ Wang, YW Ma, Yanfeng Luo, Yang Zhang, Yanhong Xu, Yanru Ma, Yanwen Huang, Yao Li, Yao Li, Yao Xu, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Qian, Yi Shao, Yi Yu, Yichao Zhang, Yifan Ding, Yifan Shi, Yijia Wu, Yiliang Xiong, Yiling Ma, Ying He, Ying Tang, Ying Zhou, Yingjia Luo, Yinmin Zhong, Yishi Piao, Yisong Wang, Yixiang Zhang, Yixiao Chen, Yixuan Tan, Yixuan Wei, Yiyang Ma, Yiyuan Liu, Yonglun Yang, Yongqiang Guo, Yongtong Wu, Yu Wu, YuKun Li, Yuan Cheng, Yuan Ou, Yuanfan Xu, Yuanhao Li, Yuduan Wang, Yuehan Yang, Yuer Xu, Yuhan Wu, Yuhao Meng, Yuheng Zou, Yukun Zha, Yunfan Xiong, Yupeng Chen, Yuping Lin, Yuqian Cao, Yuqian Wang, Yushun Zhang, Yuting Yan, Yutong Lin, Yuxian Gu, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuxuan Zhou, Yuyang Zhou, Yuzhen Huang, ZF Wu, Zehao Wang, Zehua Zhao, Zehui Ren, Zekai Zhang, Zhangli Sha, Zhe Fu, Zhe Ju, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zheren Gao, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhixian Huang, Zhixuan Chen, Zhiyu Wu, Zhizhou Ren, Zhongyu Wu, Zhuoshu Li, Zhuping Zhang, Zian Xu, Zihao Wang, Zihua Qu, Zihui Gu, Zijia Zhu, Zilin Li, Zipeng Zhang, Ziwei Xie, Ziyi Gao, Ziyi Wan, Zizheng Pan, and Zongqing Yao. Deepseek-v4: Towards highly efficient million-token context intelligence, 2026. URL https://arxiv.org/abs/2606.19348. Bochao Feng, Jianjiang Li, Haojie Wang, Lin Qiao, Yinghui Li, Yukun Yan, and Jidong Zhai. Decoupling readiness from release for tail-aware scheduling of agentic LLM workflows. arXiv preprint arXiv:2609.10964, 2026. URL https://arxiv.org/abs/2609.10964v1. Version 1, September 10, 2026. In Gim, Zhiyao Ma, Seung-seob Lee, and Lin Zhong. Pie: A programmable serving system for emerging LLM applications. In Proceedings of the ACM SIGOPS Symposium on Operating Systems Principles, 2025. doi: 10.1145/3731569.3764814. URL https://arxiv.org/abs/ 2510.24051v1. Dongxin Guo, Jikun Wu, and Siu Ming Yiu. SAGA: Workflow-atomic scheduling for AI agent inference on GPU clusters. arXiv preprint arXiv:2605.00528, 2026. URL https://arxiv. org/abs/2605.00528v2. Version 2, June 19, 2026. Nili Guy, Vita Bortnikov, Etai Lev Ran, Robert Shaw, and Clayton Coleman. Intelligent inference scheduling with llm-d. https://llm-d.ai/blog/ intelligent-inference-scheduling-with-llm-d, 2025. Published September 3, 2025. Jialiang Huang, Hongxuan Tang, Jingchang Chen, Yuxuan Liu, Yixiao Chen, Yuan Cheng, Yi Tao, Jingli Zhou, Yupeng Chen, Haoyu Chen, Jiarui Wang, Shengkai Lin, Chuqi Zhang, Bryan Lee Teng, Lian Guo, Zhe Fu, Wenjun Gao, Yisong Wang, Liang Zhao, Zehao Wang, Ziwei Xie, Yongqiang Guo, Peixin Cong, Ziyi Gao, Shuiping Yu, Hanwei Xu, Zuofan Wu, Zhizhou Ren, Yuyang Zhou, Bowei Zhang, Zhihuan Huang, Qihao Zhu, Lei Wang, Tianle Lin, Han Yu, Jiewen Hu, Dejian Yang, Shuo Yang, Shanghao Lu, Shaoyuan Chen, Junjie Qiu, Zhangli Sha, Yinmin Zhong, Yongtong Wu, Shiyu Wang, Wei Liu, Bingzheng Xu, Longhao Chen, Qiushi Du, Yuzhen Huang, Shirong Ma, Yaohui Wang, Mingshu Chen, Tongrui Xiong, Y. C. Yan, Haowen Luo, Haofen Liang, Xiaokang 12

Preprint

Zhang, Weihao Zeng, Runxin Xu, Peiyi Wang, Jinhua Zhu, Ruoyu Zhang, Wenkai Yang, Qi Tang, Jiping Yu, Tian Ye, Ruizhe Pan, Honghui Ding, Xiaodong Liu, Lingxiao Luo, Zhihong Shao, Yuhan Wu, Jibai Lu, Wen Liu, Haoling Zhang, Jingcheng Hu, Yaoyang Ye, Chaofan Lin, Zhaochen Zhang, Jianan Tong, Hengxu Wu, Zhihao Li, Yicheng Wang, Luyao Wang, Yuzhuo Bai, Lingyue Fu, Ruifan Xu, Y. Z. Wang, Zonglin Li, Mingqi Wei, Haiyang Shen, Chengyuan Zhang, Chao Jin, Zili Zhang, R. H. Yang, Xinbo Xu, Jian Zhou, Ruidong Zhu, Yuzhe Guo, Zelun Pan, Shaoheng Nie, Erhang Li, Shuhan Lin, Zheng Liu, Anshuo Chen, Zilong Lyu, Sinuo Cao, Rui Yu, Chuhao Wang, Junyi Guo, Junxiao Song, Kaifeng Chen, Menghao Ye, Junxian Li, Di Wu, Haiyang Ma, Yilun Wang, Haoran Yang, Yizai Cai, Shichun Liu, Yiping Wang, Junbo Sun, Shicheng Xu, Xiao Bi, Ying He, Yichao Zhang, Mingxing Zhang, Liyue Zhang, Panpan Huang, and Wenfeng Liang. Deepseek elastic compute (dsec): A sandbox infrastructure for effective agentic training at scale, 2026. URL https://arxiv.org/abs/2609.22978. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. SWE-bench: Can language models resolve real-world Github issues? In International Conference on Learning Representations, 2024. URL https://openreview.net/forum? id=VTF8yNQM66. Hao Kang, Ziyang Li, Weili Xu, Xinyu Yang, Yinfang Chen, Junxiong Wang, Beidi Chen, Tushar Krishna, Chenfeng Xu, and Simran Arora. ThunderAgent: A simple, fast and program-aware agentic inference system. arXiv preprint arXiv:2602.13692, 2026. URL https://arxiv. org/abs/2602.13692v3. Version 3, June 30, 2026. Hans Kellerer, Ulrich Pferschy, and David Pisinger. Knapsack Problems. Springer, 2004. doi: 10.1007/978-3-540-24777-7. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS Symposium on Operating Systems Principles, 2023. URL https://arxiv.org/abs/2309.06180v1. Hanchen Li, Runyuan He, Qiuyang Mang, Qizheng Zhang, Huanzhi Mao, Xiaokun Chen, Hangrui Zhou, Huanchen Zhang, Alvin Cheung, Joseph Gonzalez, and Ion Stoica. Continuum: Efficient and robust multi-turn LLM agent scheduling with KV cache time-to-live. arXiv preprint arXiv:2511.02230, 2025. URL https://arxiv.org/abs/2511.02230v7. Version 7, September 8, 2026. Shuhuai Li, Jianghao Lin, Dongdong Ge, and Yinyu Ye. Moe-spac: Efficient moe inference based on speculative activation utility in heterogeneous edge scenarios. arXiv preprint arXiv:2603.09983, 2026. Xiaocheng Li, Chunlin Sun, and Yinyu Ye. Simple and fast algorithm for binary integer and online linear programming. arXiv preprint arXiv:2003.02513, 2020. URL https://arxiv.org/ abs/2003.02513v3. Chaofan Lin, Zhenhua Han, Chengruidong Zhang, Yuqing Yang, Fan Yang, Chen Chen, and Lili Qiu. Parrot: Efficient serving of LLM-based applications with semantic variable. In 18th USENIX Symposium on Operating Systems Design and Implementation, pp. 929–945. USENIX Association, 2024. URL https://www.usenix.org/conference/osdi24/ presentation/lin-chaofan. Michael Luo, Xiaoxiang Shi, Colin Cai, Tianjun Zhang, Justin Wong, Yichuan Wang, Chi Wang, Yanping Huang, Zhifeng Chen, Joseph E. Gonzalez, and Ion Stoica. Autellix: An efficient serving engine for LLM agents as general programs. arXiv preprint arXiv:2502.13965, 2025. URL https://arxiv.org/abs/2502.13965v1. Michael Luo, Xiaoxiang Shi, Colin Cai, Tianjun Zhang, Justin Wong, Yichuan Wang, Chi Wang, Yanping Huang, Zhifeng Chen, Joseph E. Gonzalez, and Ion Stoica. Agentix: An efficient serving engine for LLM agents as general programs. In 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26), pp. 2443–2459. USENIX Association, 2026. URL https://www.usenix.org/conference/nsdi26/presentation/luo. 13

Preprint

Andreas Maurer and Massimiliano Pontil. Empirical bernstein bounds and sample variance penalization. arXiv preprint arXiv:0907.3740, 2009. Mike Merrill, Alexander Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Shin, Thomas Walshe, E Kelly Buchanan, et al. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces. In International Conference on Learning Representations, volume 2026, pp. 40903–40986, 2026. mini-SWE-agent Contributors. mini-SWE-agent. https://mini-swe-agent.com/ latest/, 2026. Official documentation; accessed September 8, 2026. Hongqiu Ni, Han Tian, Chi Zhang, Guopeng Li, and Haisheng Tan. TOPAS: Workflow-aware prefix-state scheduling for multi-agent LLM serving. arXiv preprint arXiv:2608.25523, 2026. URL https://arxiv.org/abs/2608.25523v1. Version 1, August 26, 2026. NVIDIA. Nvidia dynamo: A distributed inference framework. https://docs.nvidia.com/ dynamo/, 2025. Technical documentation; scheduler code audited in the literature ledger at commit 1e3478dfe09bdd305d7c58f412fda2dfe5bec7ed. OpenAI. Computer use. https://developers.openai.com/api/docs/guides/ tools-computer-use, 2026. Official documentation; accessed September 11, 2026. OpenCode Contributors. Improve bash timeout retry hint. https://github.com/ anomalyco/opencode/pull/22390, 2026. Pull request 22390, merged April 14, 2026; timeout-termination metadata. Zaifeng Pan, Ajjkumar Dahyalal Patel, Yipeng Shen, Zhengding Hu, Yue Guan, WanLu Li, Lianhui Qin, Yida Wang, and Yufei Ding. KVFlow: Efficient prefix caching for accelerating LLM-based multi-agent workflows. In Advances in Neural Information Processing Systems, volume 38, pp. 126246–126265, 2025. doi: 10.52202/ 085713-4208. URL https://papers.neurips.cc/paper_files/paper/2025/ hash/b7971d31a7d5eb0f1eed2f8f6f368195-Abstract-Conference.html. Zaifeng Pan, Qianxu Wang, Zhengding Hu, Chang Chen, Yue Guan, Yanbo Zhou, Steven Swanson, and Yufei Ding. Smoothagent: Efficient long-horizon llm-based agent serving with lookahead context engineering. arXiv preprint arXiv:2607.00151, 2026. You Peng, Youhe Jiang, Wenshuang Li, Xu Xu, Ke Zhou, Jiawei Jiang, Chen Wang, and Binhang Yuan. Hexagent: Efficient agentic llm serving via workflow-and heterogeneity-aware scheduling. arXiv preprint arXiv:2605.16637, 2026. David Pisinger. An exact algorithm for large multiple knapsack problems. European Journal of Operational Research, 114(3):528–541, 1999. doi: 10.1016/S0377-2217(98)00120-9. Zongming Qiao, Ziyue Hua, Zirui Ou, Zihao Yue, Zihan Jiang, Zhuo Huang, Zhiyang Chen, Zhixian Zheng, Zhipeng Xu, Zhengrui Ma, et al. Mimo-v2. 6: Scaling reinforcement learning towards self-improvement. arXiv preprint arXiv:2610.11959, 2026. Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: Trading more storage for less computation—a KVCachecentric architecture for serving LLM chatbot. In 23rd USENIX Conference on File and Storage Technologies, pp. 155–170. USENIX Association, 2025. URL https://www.usenix.org/ conference/fast25/presentation/qin. SemiAnalysis. AgentX MVP: Multi-turn coding-agent trace replay. https://github. com/SemiAnalysisAI/agentx-harness/blob/master/docs/tutorials/ agentx-mvp.md, 2026a. Official documentation; accessed September 8, 2026. SemiAnalysis. InferenceX: Open source continuous inference benchmark research platform. https: //github.com/SemiAnalysisAI/InferenceX, 2026b. Official repository; accessed September 8, 2026. 14

Preprint

SGLang Team. SGLang Model Gateway. https://docs.sglang.io/docs/advanced_ features/sgl_model_gateway, n.d. Technical documentation; accessed September 7, 2026. David B. Shmoys and Eva Tardos. An approximation algorithm for the generalized assignment problem. Mathematical Programming, 62:461–474, 1993. doi: 10.1007/BF01585178. Vikranth Srivatsa, Zijian He, Reyna Abhyankar, Dongming Li, and Yiying Zhang. Preble: Efficient distributed prompt scheduling for LLM serving. In International Conference on Learning Representations, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/ hash/5bc342f48de8264779952fac378f96dc-Abstract-Conference.html. Yifan Sui, Han Zhao, Rui Ma, Zhiyuan He, Hao Wang, Jianxun Li, Kaiqiang Xu, Kai Chen, and Yuqing Yang. Parallelizing tool execution and LLM generation for low-latency agent serving. arXiv preprint arXiv:2603.18897, 2026. URL https://arxiv.org/abs/2603.18897v3. GLM-4.5 Team, Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, Kedong Wang, Lucen Zhong, Mingdao Liu, Rui Lu, Shulin Cao, Xiaohan Zhang, Xuancheng Huang, Yao Wei, Yean Cheng, Yifan An, Yilin Niu, Yuanhao Wen, Yushi Bai, Zhengxiao Du, Zihan Wang, Zilin Zhu, Bohan Zhang, Bosi Wen, Bowen Wu, Bowen Xu, Can Huang, Casey Zhao, Changpeng Cai, Chao Yu, Chen Li, Chendi Ge, Chenghua Huang, Chenhui Zhang, Chenxi Xu, Chenzheng Zhu, Chuang Li, Congfeng Yin, Daoyan Lin, Dayong Yang, Dazhi Jiang, Ding Ai, Erle Zhu, Fei Wang, Gengzheng Pan, Guo Wang, Hailong Sun, Haitao Li, Haiyang Li, Haiyi Hu, Hanyu Zhang, Hao Peng, Hao Tai, Haoke Zhang, Haoran Wang, Haoyu Yang, He Liu, He Zhao, Hongwei Liu, Hongxi Yan, Huan Liu, Huilong Chen, Ji Li, Jiajing Zhao, Jiamin Ren, Jian Jiao, Jiani Zhao, Jianyang Yan, Jiaqi Wang, Jiayi Gui, Jiayue Zhao, Jie Liu, Jijie Li, Jing Li, Jing Lu, Jingsen Wang, Jingwei Yuan, Jingxuan Li, Jingzhao Du, Jinhua Du, Jinxin Liu, Junkai Zhi, Junli Gao, Ke Wang, Lekang Yang, Liang Xu, Lin Fan, Lindong Wu, Lintao Ding, Lu Wang, Man Zhang, Minghao Li, Minghuan Xu, Mingming Zhao, Mingshu Zhai, Pengfan Du, Qian Dong, Shangde Lei, Shangqing Tu, Shangtong Yang, Shaoyou Lu, Shijie Li, Shuang Li, Shuang-Li, Shuxun Yang, Sibo Yi, Tianshu Yu, Wei Tian, Weihan Wang, Wenbo Yu, Weng Lam Tam, Wenjie Liang, Wentao Liu, Xiao Wang, Xiaohan Jia, Xiaotao Gu, Xiaoying Ling, Xin Wang, Xing Fan, Xingru Pan, Xinyuan Zhang, Xinze Zhang, Xiuqing Fu, Xunkai Zhang, Yabo Xu, Yandong Wu, Yida Lu, Yidong Wang, Yilin Zhou, Yiming Pan, Ying Zhang, Yingli Wang, Yingru Li, Yinpei Su, Yipeng Geng, Yitong Zhu, Yongkun Yang, Yuhang Li, Yuhao Wu, Yujiang Li, Yunan Liu, Yunqing Wang, Yuntao Li, Yuxuan Zhang, Zezhen Liu, Zhen Yang, Zhengda Zhou, Zhongpei Qiao, Zhuoer Feng, Zhuorui Liu, Zichen Zhang, Zijun Yao, Zijun Yao, Zikang Wang, Ziqiang Liu, Ziwei Chai, Zixuan Li, Zuodong Zhao, Wenguang Chen, Jidong Zhai, Bin Xu, Minlie Huang, Hongning Wang, Juanzi Li, Yuxiao Dong, and Jie Tang. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models, 2025. URL https://arxiv.org/abs/2508.06471. vLLM Production Stack Team. vLLM Production Stack. https://docs.vllm.ai/ projects/production-stack/en/latest/, n.d. Technical documentation; accessed September 7, 2026. Noppanat Wadlom, Junyi Shen, and Yao Lu. Efficient llm serving for agentic workflows: A data systems perspective. arXiv preprint arXiv:2603.16104, 2026. URL https://arxiv.org/ abs/2603.16104. Jiahao Wang, Kaizhan Lin, Kaixi Zhang, Jinbo Han, Xingda Wei, Sijie Shen, Chenguang Fang, Wenyuan Yu, Rong Chen, and Haibo Chen. SMetric: Rethink LLM scheduling for serving agents with balanced session-centric scheduling. arXiv preprint arXiv:2607.08565, 2026a. URL https://arxiv.org/abs/2607.08565v1. Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. In International Conference on Learning Representations, volume 2025, pp. 65882–65919, 2025. Yifei Wang, Hancheng Ye, Yechen Xu, Cong Guo, Chiyue Wei, Qinsi Wang, Dongting Li, Tingjun Chen, Hai “Helen” Li, Danyang Zhuo, and Yiran Chen. MARS: Efficient, adaptive co-scheduling 15

Preprint

for heterogeneous agentic systems. arXiv preprint arXiv:2604.26963, 2026b. URL https: //arxiv.org/abs/2604.26963v2. Version 2, June 15, 2026. Yechen Xu, Xinhao Kong, Tingjun Chen, and Danyang Zhuo. Conveyor: Efficient tool-aware LLM serving with tool partial execution. arXiv preprint arXiv:2406.00059, 2024. URL https: //arxiv.org/abs/2406.00059. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388. Yichao Yuan, Mosharaf Chowdhury, and Nishil Talati. Kairos: Stateful, context-aware power-efficient agentic inference serving. arXiv preprint arXiv:2604.16682, 2026. Rui Zhang, Chaeeun Kim, Shaoting Feng, Kuntai Du, Yuhan Liu, Yi Zhong, Cheng-Wei Ching, Junchen Jiang, and Liting Hu. Learning agent execution for KV-cache management in agentic serving. arXiv preprint arXiv:2608.14624, 2026. URL https://arxiv.org/abs/2608. 14624v1. Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. SGLang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems, volume 37, pp. 62557–62583, 2024. doi: 10.52202/079017-2000. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/ 724be4472168f31ba1c9ac630f15dec8-Abstract-Conference.html.

16

Preprint

A

H ISTORICAL T OOL -D URATION E STIMATOR

The history-based predictor in Section 3.4 uses a lightweight summary of completed calls. A finitesample margin augments the historical mean, allowing the estimate to account for observed variability and the amount of available history. For calls without direct timing evidence, we maintain statistics of completed durations keyed by the structured tool name f . Let nf , d¯f , s2f , and Rf denote the sample count, mean, sample variance, and observed duration range. We use the following conservative upper estimate as a duration heuristic (Maurer & Pontil, 2009): ¯ nf = 1,  df , s b 2 Df = (8) 2sf log(3/δ) 3Rf log(3/δ)  + , nf ≥ 2, d¯f + nf nf with δ = 0.2. We use global completed-call statistics when tool-specific history is empty. Subtracting b f gives the remaining-duration estimate. The finite-sample margin is a scheduling elapsed time from D heuristic; it does not provide calibrated coverage for an individual call. Continuum also exploits tool-duration history to choose a KV-retention TTL (Li et al., 2025). Our estimate converts this historical timing evidence into a signal for scheduler admission. If both tool-specific and global histories are empty, the implementation uses a five-second duration default. The remaining-time estimate subtracts elapsed time and clips at zero, as in equation (5). Exposed timing hints take precedence as described in the main text.

B

E XTENDED M ETHOD D ETAILS

B.1

J OINT SELECTION AND PLACEMENT ALGORITHM

Algorithm 1: TAO joint selection and placement Input: frozen candidates P , nodes B, capacities C, eligibility E, quantum ∆ Output: a revalidated set of agent session–node assignments Compute wi and vi,b = (ki,b , ri , ti ) through session evaluation w̄i ← ⌈wi /∆⌉; C̄b ← ⌊Cb /∆⌋ Embed each value to obtain ui,b Initialize D0 [0] ← 0; other states are unreachable for candidate j = 1, . . . , |P | do Dj ← Dj−1 , recording skip predecessors for each reachable state s in Dj−1 do for each eligible node b with sb + w̄j ≤ C̄b do s′ ← s + w̄j eb z ← Dj−1 [s] + uj,b if s′ is new or z improves Dj [s′ ] then Store z and predecessor (s, j, b) If the budget is exceeded, obtain a BFD plan and skip to revalidation Otherwise choose the best final state and backtrack to recover the plan Revalidation: Discard the plan if a selected agent session’s live signature or eligibility changed return the longest plan prefix feasible under current admission capacities

At every scheduling time, wi is updated with the memory demand of the total token count minus the shared token capacity for session i, and Cb is updated with the remaining capacity of node b from the metric observation. Algorithm 1 summarizes TAO, from session evaluation to joint scheduling and live-state revalidation. B.2

A SYNCHRONOUS EXECUTION

The scheduler solves a frozen snapshot in an isolated worker. Before applying the plan, it checks that selected agent sessions remain eligible and match their recorded state signatures, then applies 17

Preprint

the longest prefix permitted by current capacities. Changed selected-agent session signatures cause the plan to be discarded. Revalidation preserves feasibility under current admission constraints. We quantify the effect of stale values, discarded plans, and partial application through a loss bound relative to the fresh-snapshot optimum, and measure these execution outcomes alongside solve time and plan age. Conditional loss bound. Let s and t be planning and application times, xs the exact snapshot plan, and yt the revalidated plan (empty if discarded). Write Xs∆ and Xt∆ for the quantized feasible sets. For analysis, represent the lexicographic objective by one fixed scalar score Uτ that preserves its order on the finite plans and values under comparison; the implementation still compares vectors. Define ϵs,t = max |Ut (x) − Us (x)|, x∈Xs∆

As,t = max Ut (x) − x∈Xt∆

max

x∈Xs∆ ∩Xt∆

Ut (x),

Ts,t = [Ut (xs ) − Ut (yt )]+ .

(9)

These terms capture value drift, opportunities introduced by feasibility changes, and value lost during execution. The empty plan ensures a nonempty intersection. Then max Ut (x) − Ut (yt ) ≤ As,t + 2ϵs,t + Ts,t .

x∈Xt∆

(10)

The proof in Appendix C inserts the best common feasible plan and applies snapshot optimality plus two value-drift terms. If every plan has at most K assignments and each scalar assignment value changes by at most ηs,t , then ϵs,t ≤ Kηs,t . With an additional drift bound ηs,t ≤ L(t − s), the value-drift term becomes 2KL(t − s); readiness jumps require separate allowances. An unquantized comparator adds the quantization gap, and a fallback adds its snapshot suboptimality gap. The bound connects planner freshness to retained scheduling value: short plan age limits value drift under the stated assumptions, while revalidation losses appear explicitly in Ts,t . Its comparator is the fresh placement optimum on the same realized state; Appendix C details this scope.

C

A SYNCHRONOUS S NAPSHOT L OSS : P ROOF AND S COPE

Loss relative to a fresh snapshot. We compare the applied plan with the optimal placement at application time on the same observed system state. Let s be the planning time and t the application time. Write Xs∆ and Xt∆ for the discretized feasible sets, embedding both in the union of candidate– node assignments. The empty plan belongs to both sets. Let xs be the exact snapshot solution and yt ∈ Xt the applied plan, feasible under the current admission contract; a discarded plan has yt = 0. To express loss numerically, let Uτ (x) be a fixed positive weighted sum of the objective coordinates in equation (3), with weights that preserve their lexicographic order on the finite collection of plans and objective values considered at s and t. Such weights exist for finite sets, with a scale determined by the coordinate ranges and gaps. The scalar score serves the analysis; the implementation compares vectors directly. In particular, xs maximizes Us over Xs∆ . Extend the additive objective to all assignments in the union, including plans that have become infeasible. Define the value variation, loss from changes in feasibility, and execution loss as ϵs,t = max |Ut (x) − Us (x)|, x∈Xs∆

As,t = max Ut (x) − x∈Xt∆

max

x∈Xs∆ ∩Xt∆

Ut (x),

Ts,t = [Ut (xs ) − Ut (yt )]+ .

(11)

Here [z]+ = max(z, 0). New candidates, new eligibility, and additional capacity can contribute to As,t ; truncation or discard contributes to Ts,t . Changes in feature values contribute to ϵs,t . Conditional bound.

For every exact snapshot solve and feasible applied plan, max Ut (x) − Ut (yt ) ≤ As,t + 2ϵs,t + Ts,t .

x∈Xt∆

18

(12)

Preprint

To prove this, choose z maximizing Ut over the intersection of the two feasible sets. Snapshot optimality gives Us (z) ≤ Us (xs ), and hence Ut (z) − Ut (yt ) ≤ [Ut (z) − Us (z)] + [Us (xs ) − Ut (xs )] + [Ut (xs ) − Ut (yt )] ≤ 2ϵs,t + Ts,t . Adding As,t proves equation (12). If the comparator uses the original, undiscretized capacities, add the nonnegative quantization gap Qt = maxx∈Xt Ut (x) − maxx∈Xt∆ Ut (x) to its right-hand side. The bound isolates conditions under which asynchronous planning retains value. If every plan contains at most K assignments and each scalar assignment value changes by at most ηs,t , then ϵs,t ≤ Kηs,t . A further assumption ηs,t ≤ L(t − s) yields a variation term 2KL(t − s); binary readiness require explicit jump allowances and do not satisfy this assumption automatically. Summing equation (12) over scheduling ticks bounds cumulative loss against fresh-snapshot optima on the realized states, using one common scalarization. It does not bound end-to-end latency or regret against a globally optimal online policy, whose earlier decisions can produce different future states. A uniform numerical bound requires corresponding bounds on state variation, feasibility changes, and discarded value. A fallback solve additionally contributes its snapshot suboptimality gap.

D

A DDITIONAL GLM-4.7 S ERVING T RACES

Figures 7 and 8 extend Figure 4 to 96 and 128 workers on the 300-task SWE-Lite workload with GLM-4.7 TP8. Each trace starts at its run’s first client request and shows the observed number of running backend requests. In both settings, TAO reaches the end of the observed workload window earlier than the displayed baselines. The curves describe natural-live runs with differing generated trajectories. TAO (53.9 min)

vLLM w/ RR (151.5 min)

ThunderAgent (60.1 min)

vLLM w/ CA (148.1 min)

Dynamo (165.9 min)

Running requests (Requests)

100

75

50

25

0

0

600

1,200

1,800

2,400

3,000

3,600

4,200

4,800

5,400

6,000

6,600

7,200

7,800

8,400

9,000

9,600

Time since each run's first client request (seconds)

Figure 7: GLM-4.7 running-request traces on SWE-Lite at 96 workers (TP8), aligned to each run’s first client request.

TAO (59.2 min) ThunderAgent (66.2 min)

vLLM w/ RR (164.5 min) vLLM w/ CA (154.6 min)

Dynamo (173.5 min)

Running requests (Requests)

120

90

60

30

0

0

600

1,200

1,800

2,400

3,000

3,600

4,200

4,800

5,400

6,000

6,600

7,200

7,800

8,400

9,000

9,600

10,200

Time since each run's first client request (seconds)

Figure 8: GLM-4.7 running-request traces on SWE-Lite at 128 workers (TP8), aligned to each run’s first client request. 19

Preprint

Table 5: Low-concurrency fallback check for DeepSeek-V4 Flash on SWE-Lite, one TP8 node (8×H100), with a 4–64-worker sweep. Each run serves 300 tasks. Latency is the mean task E2E time, TTFT is the mean client time to first token, and decode is aggregate throughput. Boldface and underlining indicate the best and second-best distinct values. Workers

Method

E2E latency (s)

TTFT (s)

Decode (tok/s)

4 4 4 4 4

vLLM Scheduler w/ RR vLLM Scheduler w/ CA Dynamo ThunderAgent TAO (Ours)

87.911 88.390 92.549 90.099 89.479

0.126 0.125 0.102 0.120 0.117

317.855 318.357 307.382 312.368 315.629

8 8 8 8 8

vLLM Scheduler w/ RR vLLM Scheduler w/ CA Dynamo ThunderAgent TAO (Ours)

102.868 101.698 103.813 100.610 102.913

0.139 0.142 0.112 0.130 0.132

528.074 555.818 565.893 527.068 522.838

16 16 16 16 16

vLLM Scheduler w/ RR vLLM Scheduler w/ CA Dynamo ThunderAgent TAO (Ours)

121.158 126.907 126.670 147.709 126.702

0.160 0.176 0.130 0.789 0.153

831.120 861.089 884.575 722.445 861.198

32 32 32 32 32

vLLM Scheduler w/ RR vLLM Scheduler w/ CA Dynamo ThunderAgent TAO (Ours)

160.981 166.657 172.228 272.845 171.389

0.211 0.218 0.171 3.919 0.195

1085.654 1232.259 1147.924 766.776 1240.768

64 64 64 64 64

vLLM Scheduler w/ RR vLLM Scheduler w/ CA Dynamo ThunderAgent TAO (Ours)

225.967 225.513 229.670 475.033 226.268

0.278 0.279 0.231 9.332 0.423

1788.345 1658.347 1704.425 794.991 1667.524

E

C OMPATIBILITY

E.1

FALLBACK C OMPATIBILITY

Table 5 evaluates five schedulers on a single TP8 node across 4–64 workers. Each run serves the 300task SWE-Lite workload through MiniSWE. This experiment checks that TAO remains competitive when concurrency is low and scheduling has less opportunity to improve performance. At 4–8 workers, its mean task latency stays within 2.3% of the best baseline, it retains over 92% of the best baseline’s decode throughput, and its mean TTFT is only 15–20 ms above the best baseline. Thus, TAO incurs little task-latency degradation at low concurrency while maintaining competitive throughput. Together with its leading performance in the main evaluation (Section 4.2), this fallback behavior supports choosing TAO as the default scheduler across the evaluated operating regimes. E.2

KV-C ACHE M ANAGEMENT C OMPATIBILITY

TokenCake uses workflow dependencies and agent priorities to guide its Space Scheduler, and toolstall events and timing prediction to guide offload and upload in its Time Scheduler (Bian et al., 2025). Agentix prioritizes and preempts LLM calls using program-level service history to reduce request- and program-level blocking (Luo et al., 2026). It operates at the request-scheduling layer, so integrating it with TAO would require coordinating the two admission policies. KVFlow represents a workflow as an Agent Step Graph and uses steps-to-execution to prioritize cache eviction and prefetching (Pan et al., 2025). CacheScout learns execution transitions online to guide cache management without requiring a predefined graph (Zhang et al., 2026). These mechanisms provide information about future cache use. In our formulation, such information can inform the program–node values supplied to session evaluation, while engine-local cache policies can operate beneath TAO’s admission and placement decisions. This is an architectural extension for KVFlow and CacheScout, while the implemented composition evaluated here uses TokenCake. With TokenCake enabled, the combined system completes validated 300-task SWE-Lite runs on DeepSeek-V4 Flash TP4×2 at 16 and 32 workers. Relative to the same engine behind consistenthash routing, the combination increases decode throughput from 785.159 to 886.772 tok/s and 20

Preprint

Table 6: Frontend composition over TokenCake on DeepSeek-V4 Flash TP4×2, SWE-Lite (300 tasks), with 100 GiB total CPU KV capacity. Consistent-hash rows are retained validated runs; cache-aware and TAO rows are from the later campaign. Bold and underline rank the TokenCake configurations within each worker group. Light-blue rows reproduce the original TAO results from Table 1 for reference and are excluded from ranking and relative-improvement calculations. Latency ↓ (s)

TTFT ↓ (s)

Decode ↑ (tok/s)

16

TokenCake Cache-aware w/ TokenCake TAO w/ TokenCake Rel. Imprv. (%) TAO (Ours)

126.891 137.907 122.940 +3.1 113.264

0.178 0.268 0.177 +0.7 0.135

785.159 760.166 886.772 +12.9 934.967

32

TokenCake Cache-aware w/ TokenCake TAO w/ TokenCake Rel. Imprv. (%) TAO (Ours)

162.905 284.667 159.581 +2.0 142.332

0.220 0.715 0.223 -1.5 0.153

1292.584 778.611 1347.373 +4.2 1477.463

64

TokenCake Cache-aware w/ TokenCake TAO w/ TokenCake Rel. Imprv. (%) TAO (Ours)

509.018 695.986 274.892 +46.0 185.410

1.590 3.022 0.593 +62.7 0.224

790.491 613.004 1414.547 +78.9 2021.523

Workers

Frontend

Table 7: InferenceX replay on 2 nodes with 4×H100 per node. Latency is reported in seconds per request (s/req). TTFT is recovered client time (s), and decode is node-window decode TPS. These replay measurements use a fixed trace corpus and are reported separately from natural-live task results. Method

16 workers

32 workers

64 workers

Latency (s/req) TTFT (s) Decode (tok/s) Latency (s/req) TTFT (s) Decode (tok/s) Latency (s/req) TTFT (s) Decode (tok/s) Dynamo vLLM Scheduler w/ RR vLLM Scheduler w/ CA TAO (Ours) Rel. Imprv. (%)

5.545 7.121 5.916 5.382 +2.9

10.844 10.861 7.421 4.756 +35.9

133.158 107.478 115.947 138.701 +4.2

3.234 5.069 3.103 3.234 -4.2

26.542 23.058 9.006 10.302 -14.4

204.257 127.991 211.262 215.075 +1.8

5.154 4.589 3.825 3.532 +7.7

163.674 45.852 25.145 27.190 -8.1

71.085 87.050 109.413 120.469 +10.1

from 1292.584 to 1347.373 tok/s, respectively (12.9% and 4.2%). These runs establish operational compatibility. Table 6 compares three frontends over the TokenCake engine. At 16 and 32 workers, TAO lowers mean task latency relative to consistent hashing by 3.1% and 2.0%, while increasing decode throughput by 12.9% and 4.2%. At 32 workers, client TTFT increases slightly from 0.220 to 0.223 s. Both combined runs record balanced request routing across the two replicas (5816/5393 and 5328/5683 requests) and tool-prediction updates (22085 and 21776). Neither records a program pause, resume, or migration, and their final offload series are absent. The evidence therefore establishes lifecycle and serving compatibility under these loads; attribution to DP selection or overlapping offload would require runs that activate those mechanisms. Separate transfer qualifications verify actual GPU–CPU movement.

F

I NFERENCE X R EPLAY

TAO also achieves the highest decode throughput at every tested concurrency level on InferenceX, a fixed trace corpus served by DeepSeek on 2 nodes with 4×H100 per node. The replay uses 393 AgentX traces with a 1M-token context limit and 1,800-second profiling windows (SemiAnalysis, 2026a;b); workers count the concurrent session trees. Table 7 reports client-visible TTFT and decode throughput, together with seconds per request computed as 60/R, where R is native request throughput in requests per minute. This last metric is the reciprocal of aggregate request throughput within the profiling window. Decode throughput improves on the best competing method by 4.2%, 1.8%, and 10.1% at 16, 32, and 64 workers, respectively. TAO also achieves the lowest TTFT at 16 workers; cache-aware scheduling has lower TTFT at 32 and 64 workers. Figure 9 shows the 21

Preprint

Prefix-cache hit rate (%)

accompanying reuse advantage: TAO maintains the highest prefix-cache hit rate throughout, even as hit rates decrease with concurrency for all four methods.

Dynamo vLLM RR

vLLM CA TAOS

40 50 Workers

60

60 40 20 0

20

30

Figure 9: InferenceX prefix-cache hit rate on 2 nodes with 4×H100 per node. TAO maintains higher cache reuse across the tested worker counts.

G

S ENSITIVITY TO LONGER TOOL - RETURN DELAYS

We examine how longer tool-return delays affect scheduling using a replay of SWE-Lite sessions recorded with MiniSWE. The replay fixes the source prompts, 6,995 model-request records, and 6,942 tool records, while model outputs are generated live. Each session receives a fixed additional tool-return delay drawn from 1–5 s; multipliers of 1, 2, and 4 produce additional delays of 1–5, 2–10, and 4–20 s, respectively. DeepSeek-V4 Flash runs on TP4×2 with 32 workers, a 33,024-token context limit, and a 512-token per-session reserve. Table 8 compares three configurations: the historical TAO w/o readiness reference, TAO with structured-history tool-duration predictions, and Oracle with the replay’s known effective tool duration. The historical reference also disables cached-token decay and shared-token refresh, which both current configurations enable; comparisons with that reference therefore measure the combined configuration change. Acting-token decay remains disabled in both current configurations. All six current cells pass the replay validator, covering all 6,942 tool records. Each cell is a single exploratory run with batch-invariant execution disabled and live model generation. TAO sustains gains as tool delays grow. Relative to the historical configuration, TAO reduces makespan by 20.72%, 16.42%, and 19.28% and mean session latency by 21.57%, 20.57%, and 17.91% at scales 1, 2, and 4, respectively. These gains span additional tool-return delays from 1–5 s to 4–20 s. Longer delays make readiness relevant to more placement decisions. For TAO, removing readiness from the same captured state changes 11/668 (1.65%), 53/1,345 (3.94%), and 200/2,521 (7.93%) of the exact DP plan comparisons at multipliers 1, 2, and 4. The rate increases by 6.29 percentage points from scale 1 to scale 4, indicating that tool-return timing affects a growing fraction of placement decisions as delays lengthen. These same-state comparisons isolate readiness’s effect on individual placement decisions within the TAO runs. History-based predictions approach oracle runtime performance. Providing the known replay duration reduces mean absolute prediction error by 99.53–99.69%, to 0.025–0.030 s. At scale 2, it also reduces makespan by 2.24% and mean session latency by 0.71% relative to TAO. Across all three scales, the signed makespan improvements range from −0.10% to +2.24%, and mean session-latency improvements range from −0.36% to +0.71%. Thus, the history-based predictor achieves runtime performance close to the accurate-timing variant in this sweep, even as longer delays increase the frequency of readiness-sensitive decisions. 22

Preprint

Table 8: Tool-return delay scaling on SWE-Lite replay sessions. Scales 1, 2, and 4 add 1–5, 2–10, and 4–20 s per tool return, respectively. Oracle uses accurate timing supplied by the known replay tool duration. MAE is logged duration-prediction error, including predictions unused by the no-readiness arm. Decision change is the percentage of exact snapshot comparisons whose plan changes when readiness is removed. Rel. Imprv. compares TAO with TAO w/o readiness: positive percentages denote makespan and mean E2E reductions; the last column gives the decision-change difference in percentage points (pp). Oracle results are a timing-information reference. Scale

H

Makespan (s)

Mean E2E (s)

MAE (s)

Decision change (%)

1 1 1

TAO w/o readiness TAO Oracle Rel. Imprv.

Prediction arm

1,919.906 1,522.076 1,523.532 +20.72%

229.714 180.167 180.030 +21.57%

6.531 9.490 0.030 –

0.000 1.647 4.167 +1.65 pp

2 2 2

TAO w/o readiness TAO Oracle Rel. Imprv.

2,816.496 2,354.050 2,301.282 +16.42%

352.502 279.986 278.011 +20.57%

2.593 6.315 0.030 –

0.000 3.941 4.059 +3.94 pp

4 4 4

TAO w/o readiness TAO Oracle Rel. Imprv.

5,090.251 4,108.865 4,110.209 +19.28%

599.155 491.829 493.595 +17.91%

4.512 5.837 0.025 –

0.000 7.933 7.398 +7.93 pp

J OINT P LACEMENT C ASE S TUDIES

Reconstructed snapshots from the 32-worker DeepSeek replay show how joint placement preserves more cache value and admits more sessions than a short-first heuristic (Section 4.3). The figures compare scheduler-level assignment decisions and their cache credit; vLLM manages the physical KV cache. The short-first heuristic is evaluated offline on the same snapshots. A representative cache-credit snapshot is shown in Figure 10. DP retains 16,791 tokens of cache credit, compared with 13,624 for the short-first heuristic, a 23.2% increase with the same total safe-token remainder. A separate reconstructed snapshot in Figure 11 shows the selection effect: DP admits seven sessions, including long-context session 177, while the heuristic admits six and defers that session.

I

S CHEDULER OVERHEAD A NALYSIS

Across the four-replica Qwen runs at 96–160 workers, planning submissions rise from 98 to 1,681. At 160 workers, completed assignment solving averages 1.59 ms with a P95 of 3.68 ms; the two measured parts together account for 4.03 s (0.112% of the run). Thus, the recorded scheduling work remains below 0.12% at four replicas and up to 160 workers. Table 9 expands the scheduling work and planner outcomes summarized above. State preparation runs synchronously in the scheduler, while the planner worker solves assignments asynchronously. Result delay measures dispatch, solving, and delivery together. The two- and four-replica deployments exercise this design under different models and concurrency levels. The state budget bounds DP search, with best-fit-decreasing placement handling exhausted searches. Fallback occurs in 5.24% of submissions at 64 DeepSeek workers and 3.06–8.98% in the Qwen sweep. Measured work covers state preparation and completed solves; exhausted searches and fallback execution are untimed, so work/run is a partial accounting rather than a measure of CPU utilization or request-latency overhead. Discards and partial applications record how live revalidation handles state changes before application. The 16-worker DeepSeek run submits one plan, so its timing entry describes that single solve.

J

E XTENDED R ELATED W ORK

Capacity-constrained optimization. Dynamic programming and exact multiple-knapsack algorithms provide established tools for selecting items and assigning them to capacity-limited resources (Bellman, 1957; Kellerer et al., 2004; Pisinger, 1999). Node-dependent assignment values 23

Preprint

Cache hit Cold footprint

Safe limit P89

P109 P111

P123 P137

P143

(a) Short-first affinity greedy

Cache: 13.6k Limit 51.3k

Node 1 slack: 8.4k

Limit 62.2k

Node 2 slack: 20.1k

(b) Dynamic programming

Cache: 16.8k (+3.2k, +23.2%) Limit 51.3k

Node 1

slack: 1.4k

Limit 62.2k

Node 2 slack: 27.1k

0

10

20 30 40 50 Admission weight (thousand tokens)

60

70

Figure 10: Cache utilization in one placement snapshot. Each program has a distinct hue; the translucent segment is its admission weight and the saturated overlay represents retained cache credit (shown as token labels when space permits). DP retains 16,791 cache-credit tokens versus 13,624 for short-first placement (23.2% more), with the same total safe-token remainder. Short-first affinity greedy | 6 programs; long program 177 deferred | cache: 10,397 Node 1

195

Node 2

190

193

189

187

37.8k

188

27.2k

Dynamic programming | 7 programs; program 177 admitted | cache: 14,949 189

Node 1

195

Node 2 0

187

190

5

193

177

188

37.8k

27.2k

10 15 20 25 30 35 Admission weight (thousand tokens; includes 512-token reserve)

40

Figure 11: A reconstructed replay snapshot in which DP admits seven sessions while the short-first affinity-greedy heuristic admits six. Candidate widths represent admission weights, including the 512-token reserve. DP admits the long-context session 177; the heuristic leaves it deferred after filling capacity with shorter programs. place our formulation within the multiple-knapsack/generalized-assignment family. Approximation algorithms address the computational cost of this family (Chekuri & Khanna, 2005; Shmoys & Tardos, 1993), while online resource-allocation algorithms analyze decisions under specified arrival models (Li et al., 2020). TAO uses this classical optimization machinery to connect agentic session evaluation with joint admission and node placement. Its additive, lexicographic objective makes cache opportunity, tool readiness, and token footprint precedence explicit, and its asynchronous 24

Preprint

Table 9: Scheduler overhead on SWE-Lite (300 tasks per run). TP4×2 uses two replicas; TP2×4 uses four. State preparation is the time to collect the current candidates and capacities; assignment solving is the completed asynchronous planning time; result delay includes dispatch and delivery. Measured work sums state preparation and completed solving, excluding failed attempts and BFD execution. Work fractions divide by run duration. Outcome entries are count (%): fallback/discard rates divide by submissions, partial rates by application events (including empty plans). DeepSeek-V4 Flash, TP4×2 Workers

Qwen3-32B, TP2×4

16

32

64

96

128

160

Run duration (s) State preparation total (ms) Assignment solving total (ms) Assignment solving mean (ms) Assignment solving p95 (ms) Result delay mean (ms) Result delay p95 (ms)

2321.1 1.37 0.55 0.55 0.55 337.49 337.49

1514.2 30.58 48.97 1.48 2.08 20.83 22.83

1199.0 394.95 3163.36 12.50 88.84 27.86 105.09

3151.2 50.67 52.77 0.56 0.62 6.07 5.29

3077.9 1024.09 1351.51 1.04 2.43 4.23 9.03

3583.0 1588.17 2437.68 1.59 3.68 4.89 9.99

State preparation/run (%) Assignment solving/run (%) Measured scheduler work/run (%)

0.00006 0.00002 0.00008

0.00202 0.00323 0.00525

0.03294 0.26384 0.29678

0.00161 0.00167 0.00328

0.03327 0.04391 0.07718

0.04433 0.06804 0.11236

Plan submissions Completed plans Plan applications Nonempty applications BFD fallback Discarded Partial applications

1 1 1 1 0 (0.00) 0 (0.00) 0 (0.00)

33 33 33 33 0 (0.00) 0 (0.00) 0 (0.00)

267 253 252 240 14 (5.24) 1 (0.37) 2 (0.79)

98 95 95 89 3 (3.06) 0 (0.00) 0 (0.00)

1395 1301 1299 678 94 (6.74) 2 (0.14) 0 (0.00)

1681 1530 1527 687 151 (8.98) 3 (0.18) 0 (0.00)

executor revalidates assignments against live state. Section 3.5 and Appendix B.2 specify snapshot optimality and the bounds for asynchronous application. KV locality and agent scheduling. PagedAttention improves physical KV allocation and sharing, while SGLang’s RadixAttention reuses prefixes within the serving runtime (Kwon et al., 2023; Zheng et al., 2024). Preble co-optimizes distributed prefix reuse and computation balance; Mooncake combines a distributed KV store with disaggregated serving (Srivatsa et al., 2025; Qin et al., 2025). Dynamo and production gateways expose cache- and load-aware request scheduling (NVIDIA, 2025; vLLM Production Stack Team, n.d.; SGLang Team, n.d.), and llm-d provides pluggable scoring components (Guy et al., 2025). SMetric exploits session structure by balancing first-turn requests and assigning later turns to nodes with reusable cache, using a global cache tier to limit the cost of redistribution (Wang et al., 2026a). TAO builds on these locality and load-balancing principles with joint optimization of assignments for waiting agent sessions under shared capacity. This joint view accounts for how one admission consumes capacity needed by other session–node assignments. Program-level schedulers also preserve information across model calls. Autellix uses attained service and critical-path information to prioritize single-threaded and distributed programs (Luo et al., 2025). ThunderAgent introduces the program abstraction, state-aware pausing, elapsed-time discounting of tool execution programs’ effective token demand, and a global waiting queue (Kang et al., 2026). TAO adds node-specific cache values and readiness to the joint placement objective. Tool-gap cache management addresses a complementary decision: InferCept chooses among preservation, swapping, and recomputation by modeling resource waste (Abhyankar et al., 2024); Continuum selects a cacheretention TTL using historical tool-duration distributions, reconstruction cost, and returning programs’ queueing cost, together with TTL-aware program ordering (Li et al., 2025). These systems control physical KV retention. TAO coordinates scheduler-level admission opportunities, with predicted readiness distinguishing plans that have equal higher-precedence cache value. The serving engine controls the resulting cache residency. Workflow, prediction, and tool execution. Workflow structure provides information beyond an isolated request. Parrot exposes inter-request dependencies through Semantic Variables, and Helium compiles workflow query plans for proactive reuse and cache-aware execution (Lin et al., 2024; 25

Preprint

Wadlom et al., 2026). KVFlow uses an Agent Step Graph and steps-to-execution to guide eviction and prefetching (Pan et al., 2025); CacheScout learns execution transitions online without a predefined graph (Zhang et al., 2026). Their predictions guide context residency and prefetching; readiness supplies a binary near-term admission-opportunity signal derived from tool timing. SAGA combines workflow graphs, session-affinity batching, and work stealing (Guo et al., 2026). TOPAS also uses heuristic state searching: it scores prefix-residency and request-admission states using remaining workflow paths, movement costs, and downstream reuse, with bounded candidate search and greedy packing (Ni et al., 2026). Its task-dependent utility and engine-level residency decisions differ from our fixed additive value contract and exact multi-node scheduling problem. Prediction also supports latency-oriented scheduling. ELIS predicts response length, while Kairos combines workflow-aware priority with memory-aware dispatch (Choi et al., 2025; Chen et al., 2025). Recent work separates already-ready turns from their release, using a mean–CVaR objective and an adaptive budget for committed inference work (Feng et al., 2026). This addresses tail-aware release after dependencies complete; TAO additionally evaluates near-term return opportunities during external tool execution. Complementary mechanisms change the execution timeline itself: Conveyor exposes partial tool execution, Sutradhara overlaps tool work and prefill through orchestrator–engine cooperation, and PASTE speculates on future tools while coordinating tool execution with returning model agent sessions (Xu et al., 2024; Biswas et al., 2026; Sui et al., 2026). SmoothAgent prepares transformed context on a lookahead stream, and Pie exposes programmable generation and KV handlers (Pan et al., 2026; Gim et al., 2025). TAO operates in the scheduler, using timing evidence to coordinate admission and node placement. Deployment and workloads. AgentSysBench documents substantial non-LLM latency and deployment-dependent resource demands (Chang et al., 2026). MARS explicitly coordinates GPU inference, CPU tool pressure, admission, and KV retention in heterogeneous serving (Wang et al., 2026b). In our CGDC scenario, distributed executors manage tool CPU and memory, and their resource pressure reaches the GPU scheduler through tool-return timing. TAO captures this interaction through readiness when allocating GPU admission capacity. OpenHands supports separate execution environments (Wang et al., 2025). Coding and terminal workloads, together with InferenceX/AgentX trace replay, provide complementary evidence about task completion and serving behavior (miniSWE-agent Contributors, 2026; Jimenez et al., 2024; Merrill et al., 2026; SemiAnalysis, 2026b;a), with workload-specific timing metrics defined in the evaluation.

26

Related documents

• TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving#1468834
• UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities#820473
• HexAGenT: Efficient Agentic LLM Serving via Workflow- and Heterogeneity-Aware Scheduling#200468
• NexForge: Scaling Agent Capabilities through Requirement-Driven Task Synthesis for LLMs#381815
• Efficient Benchmarking in Production: A Study of an Evolving LLM Agent#1035675