Reducing Batch Makespan for LLM Agent Workloads through Inference-Environment Coordination

Reducing Batch Makespan for LLM Agent Workloads through Inference–Environment Coordination Wei Feng, Meiling Chen, Boxiao Chen, Lu Liu, Bin Zi, Libin Ye, Xingyu Lu∗ , Lei Xu, Jun Zhou Ling Team, Ant Group

arXiv:2611.16661v1 [cs.DC] 12 Nov 2026

∗ Corresponding author

Efficient batch execution of agent tasks is critical to LLM training and development workflows, including benchmark evaluation, synthetic-data generation, and RL rollouts. Since these tasks alternate between LLM inference and environment interaction, we present coordinated optimizations to reduce batch makespan, the time required to complete all tasks in a batch. For hybrid LLMs, a workload-based resource model guides GPU-memory allocation between key-value (KV) cache and recurrent state and provides a reference sample concurrency to exploit parallelism while limiting recomputation. Trajectory-aware routing and scheduling promote cache reuse while balancing worker load. Under constrained environment capacity, we decouple sample admission from environment execution and introduce Chain-Advancing Action Scheduling (CAAS), which prioritizes long chains using observed progress to shorten the batch tail. Controlled replay combines actual inference with recorded environment responses and durations for ablations of individual optimizations. Live evaluations with Ling-3.0-Flash and Ling-3.1-Flash on the BFCL, SWE-bench Multilingual, WideSearch, and MCP-Atlas benchmarks demonstrate makespan reductions of 10.4%–65.0%, while evaluation scores remain aligned. Furthermore, these methods have been successfully applied to training and evaluation workflows for the Ling-3.x model series, accelerating the model-development cycle. Date: November 11, 2026 Correspondence: sing.lxy@antgroup.com

1

Introduction

The training and development of agentic LLMs rely on workflows such as benchmark evaluation, synthetic-data generation, and RL rollouts. Repeated across model-development iterations, these workflows often execute agent tasks in batches, with downstream stages awaiting the entire batch. Batch completion time therefore affects development efficiency, motivating our focus on reducing batch makespan, the time required to complete all tasks in a batch, rather than the request-level latency typically optimized in conventional serving, such as time to first token (TTFT) and time per output token (TPOT) (Kwon et al., 2023). We refer to an agentic task instance as a sample and its complete execution as a trajectory. Figure 1 illustrates how a batch of such trajectories is executed, each alternating between LLM inference and environment interaction. Tasks are admitted under a sample-concurrency limit, and their inference requests are routed to workers and ordered by local schedulers. Actions produced by the model are queued as environment jobs and dispatched under an execution limit. Model responses and environment results are appended to the task’s context before the next inference turn. A final 1

Batch of agentic tasks

Inference workers

Completed tasks

Task 1, Task 2, . . . , Task N

Worker 1

Batch complete when all tasks finish

Workload-aware cache allocation KV cache

Slots held until task completion

Active tasks Task i: context

Input | Prior turns

Task j: context .. .

Routing-prefix scheduling LBCH routing

Shared environment

Recurrent state

Waiting requests

Inference response

LLM inference Worker 2

···

Worker W

Yes

Final response?

Interactions

Sample admission: CS

No

(Pending) jobs Execution limit: CE CAAS

If action terminates task

response or terminal environment action completes the task and releases its admission slot. The batch finishes when all tasks complete.

Environment interaction Tools / code / services

Append inference response and environment results to the same task’s context; continue its next turn

Figure 1 Illustration of batched execution of multi-turn LLM agent tasks. Active tasks are those admitted but not yet completed, including tasks waiting for inference or environment interaction. Each maintains its own context while sharing inference and environment-execution resources. Purple labels indicate where our methods are applied. Capacity separation allows the sample-admission limit CS and environment-execution limit CE to be configured independently. Our workload-aware capacity model guides the GPU memory allocation between KV cache and recurrent state at deployment; the allocation remains unchanged during batch execution.

Successive requests within a trajectory share growing input prefixes, creating opportunities for cache reuse. Hybrid models such as Ling-3.0-Flash (Ling Team, 2026a), Qwen3.8-Max (Qwen Team, 2026), and Kimi K3 (Kimi Team, 2026) require both KV cache and recurrent state for cross-turn reuse. For a given model, KV-cache memory grows with the number of cached tokens, whereas each recurrent state has a fixed size. Thus, longer contexts increase KV-cache demand per trajectory, while admitting more concurrent trajectories requires retaining more recurrent states as well as their associated KV cache. SGLang manages these states in separate pools within a shared cachememory budget. If a default or empirically chosen allocation ratio does not accurately reflect the target workload’s cache-demand characteristics, one pool can be exhausted while substantial capacity remains unused in the other. This imbalance limits the number of trajectories whose caches can be retained together. Our workload-aware capacity model quantifies the target workload’s recurrent-state and KV-cache requirements to balance memory allocation between the two pools at deployment and provide a reference sample concurrency, enabling greater parallelism while preserving cross-turn cache reuse. The resulting allocation remains fixed during inference serving. Even with suitable allocation, reuse depends on where and when requests execute. With multiple inference workers, keeping a trajectory’s requests on the same worker favors reuse, but strict affinity can concentrate load and lengthen queues. Even on a worker holding reusable state, a request can wait long enough for that state to be evicted. In addition, model responses and environment results accumulate across turns, potentially producing very long agent contexts. Identifying cache-reuse opportunities through token-prefix matching can then incur substantial computational overhead. To reduce this overhead, we use a shared trajectory label to identify requests from the same trajectory at low cost, giving the router an affinity key and letting the worker screen waiting requests before querying the prefix cache for scheduling priority. We implement this approach through two complementary methods. Load-bounded consistent hashing (LBCH) routing uses the labels to preserve cross-turn worker affinity while accounting for worker load. Routing-prefix (RP) 2

scheduling uses them to prioritize requests from recently served trajectories and promote timely cache reuse at low scheduling cost. Environment resources may also be constrained by CPU budgets or external-service quotas. A single sample-concurrency limit, as used in the agent implementations of MCP-Atlas and τ 3 -bench (Bandi et al., 2026; Shi et al., 2026), couples the two stages. When the environment supports less concurrency than the inference service, raising this shared limit to keep inference busy can overload environment execution. Conversely, lowering it to protect the environment can leave inference workers without ready requests while active samples await environment-interaction results. We therefore decouple sample admission from environment execution, allowing more samples to remain active while an independent execution limit protects the environment. This improves stage overlap but also makes environment-side queuing more likely as more admitted samples submit jobs to the limited execution slots. Their dispatch order determines when each trajectory can resume inference. Repeated waits can defer a long trajectory’s later stages until most other samples have finished, leaving few ready inference requests and prolonging the batch tail. When multiple environment jobs await execution, Chain-Advancing Action Scheduling (CAAS) uses observed progress to prioritize those from potentially long trajectories. This aims to expose their subsequent stages sooner and shorten the batch tail without requiring knowledge of remaining trajectory lengths. Our contributions are summarized as follows. 1. Workload-based and trajectory-aware inference optimization. A hybrid-cache capacity model guides allocation and sample concurrency, complemented by LBCH routing for affinity–load balance and routing-prefix scheduling for timely cross-turn reuse. 2. Inference–environment coordination. Independent sample and environment concurrency controls improve stage overlap, while CAAS effectively advances long execution chains to shorten the batch tail. 3. Controlled replay and live end-to-end validation. Trace-driven replay combines actual inference with recorded environment responses and durations for controlled comparisons and ablations. Live evaluations with Ling-3.0-Flash and Ling-3.1-Flash across four well-known benchmarks demonstrate substantial reductions in batch makespan, while evaluation scores remain aligned (Section 5.4). The proposed methods are also applied in Ling-3.x model development, where batched agent workloads such as benchmark evaluation, synthetic-data generation, and RL rollouts play a key role. By accelerating these workloads, the methods substantially improve overall development efficiency. The remainder of this paper presents the methods and experimental evaluations supporting these contributions. Section 2 reviews related work. Sections 3 and 4 present inference-side optimizations and inference–environment coordination, respectively. Section 5 evaluates the proposed methods through controlled replay and live execution. Section 6 discusses future directions, and Section 7 concludes the paper.

2

Related Work

Prefix caching. Prefix caching avoids repeated computation over shared prompt prefixes. Serving systems support prefix reuse through mechanisms such as SGLang’s RadixAttention (Zheng et al., 2024) and vLLM’s automatic prefix caching (vLLM Project, 2026). Mooncake, Tutti, and CacheFlow extend cache management across distributed or multi-tier settings (Qin et al., 2024; Qiu et al., 2026; Nian et al., 2026). Li et al. (2026) survey these broader cache-management techniques. BatchLLM

3

exploits globally available prefix structure and throughput-oriented batching for large inference batches (Zheng et al., 2026), whereas we target cross-turn reuse within agent trajectories. Cache management for hybrid LLMs. Cache reuse in hybrid LLMs requires retaining both KV cache and recurrent state. AVMP manages separate KV-cache and state-space-model (SSM) state pools through a unified virtual page table (Nguyen, 2026). It can transfer capacity between pools at runtime when allocation fails in one pool and the other has enough spare capacity. Marconi selects recurrent states for caching based on their reuse potential and guides eviction using recency and computation saved relative to the combined footprint of the KV cache and recurrent state (Pan et al., 2025). Our workload-aware capacity model uses trajectory profiles to select a static partition of the cache budget and a reference sample concurrency before serving, with the goal of increasing joint trajectory-cache residency. Cache-aware routing and scheduling. Preble uses global prefix-cache information and computation estimates to route requests across workers (Srivatsa et al., 2024). Its local scheduler groups requests by the fraction of cached prompt tokens and selects requests from each group to form a batch, assigning larger shares to higher-priority groups. DualMap hashes a prompt prefix to two candidate workers and uses a time-to-first-token (TTFT) service-level objective to balance affinity and load (Yuan et al., 2026). PEEK maintains an incremental radix tree over pending requests to identify shared prefixes even before they are cached (Xie et al., 2026). It uses these relationships to schedule prefix-sharing requests and protect cache entries needed by queued requests. These methods use prompt-prefix information to guide routing or scheduling. In contrast, our LBCH routing and RP scheduling use trajectory labels to identify and exploit potential cache-reuse opportunities. Agent-aware inference scheduling. Autellix schedules LLM calls using observed program-level attained service and uses program locality in cross-engine routing (Luo et al., 2025). Helium represents workflows as query plans and optimizes workflow makespan through rewriting, caching, and cost-based scheduling (Wadlom et al., 2026). SAGA uses predicted workflow structure for cache management and workflow-level scheduling (Guo et al., 2026). ConServe uses observable conversation state to place the first-turn prefill and maintains decoder affinity across turns (Ding et al., 2026). RP links successive turns of an agent trajectory through shared labels and recent serving activity, without tracking program-level attained service or requiring workflow graphs. Environment-aware serving and action scheduling. SAGA uses expected tool-call durations to guide cache retention (Guo et al., 2026), while MORI uses relative tool-call idleness to guide cache placement across GPU and CPU memory (Xia et al., 2026). Both use environment behavior to inform inference-side cache decisions. ARL-Tangram directly schedules external actions, using first-come, first-served (FCFS) to select candidates and elastic resource allocation to reduce the sum of action completion times (Xiao et al., 2026). We instead order pending environment jobs under a fixed execution limit with separate sample admission. CAAS uses observed trajectory progress to advance long execution chains and shorten the batch tail without changing individual job service times.

3

Optimizing Inference Efficiency

This section presents our methods for improving inference efficiency, which are implemented in SGLang (Zheng et al., 2024). We first examine cache allocation for hybrid models, then present cross-worker routing and in-worker scheduling to improve cross-turn reuse.

4

3.1

Workload-Aware Hybrid-Cache Allocation

Hybrid models such as Ling-3.0-Flash (Ling Team, 2026a) combine Multi-head Latent Attention (MLA) with Kimi Delta Attention (KDA), requiring KV cache and recurrent state, respectively. Each inference worker maintains separate pools for these states within a shared cache-memory budget. Cache reuse requires both types of state for the reused prefix to remain available, so the more constrained pool limits how many trajectories can retain reusable state. KV-cache demand grows with cached context length, whereas recurrent state occupies a fixed number of slots per request. As a trajectory extends its prefix, the latest reusable recurrent state summarizes that prefix and supports subsequent computation without retaining states from earlier turns. We therefore retain one recurrent state per trajectory, keeping its recurrent-state memory requirement fixed as the number of turns increases. At low sample concurrency, performance is relatively insensitive to the cache ratio when both pools have sufficient capacity. Higher concurrency can shorten makespan, but also increases cache demand, so an imbalanced allocation can cause eviction and recomputation that offset parallelism gains. We therefore use a workload-aware capacity model to guide cache allocation and sample concurrency, aiming to sustain high cross-turn reuse while increasing parallelism. Model development often involves repeated execution of batched agent workloads, including benchmark evaluation, synthetic-data generation, and RL rollouts, across successive checkpoints. For similar task distributions, the mean context length often changes little between nearby checkpoints, allowing historical trajectory profiles to provide reusable estimates for guiding cache allocation and sample concurrency. 3.1.1

Joint Capacity Model

Let M be the GPU memory budget available to the KV-cache and recurrent-state pools after model weights and non-cache runtime reservations. Let Mkv and Mstate denote the memory allocated to the two pools, respectively, and define r = Mstate /Mkv . The resulting partition is Mstate (r ) = M

r , 1+r

Mkv (r ) = M

1 . 1+r

(1)

For workload D , we combine historical trajectory profiles with model and runtime parameters to estimate the per-trajectory cache footprints as Fkv (D) = Lkv (D)ckv ,

Fstate = Sstate cstate ,

(2)

where Lkv (D) is the average number of KV-cache entries required by a complete trajectory in D , measured at its final inference turn, Sstate is the number of recurrent-state slots required by each inference request under the deployment configuration, and ckv and cstate are their respective unit memory costs. The quantities Sstate , ckv , and cstate are fixed for a given model and runtime configuration. Because the accumulated context generally grows across turns, the final-turn KVcache allocation provides a conservative workload estimate, although it is not treated as a strict upper bound. Based on the pool allocations in Equation 1 and the workload-derived cache footprints in Equation 2, we approximate the number of trajectories whose cache state can be retained in each pool simultaneously using the following continuous estimates: Ckv (r; D) =

Mkv (r ) , Fkv (D)

Cstate (r ) = 5

Mstate (r ) . Fstate

(3)

Under this per-trajectory footprint approximation, we estimate the joint residency capacity as the minimum of the two pool-specific estimates: C (r; D) = min{Ckv (r; D), Cstate (r )} .

(4)

The joint capacity C (r; D) estimates how many trajectories can simultaneously retain both KV cache and recurrent state on a worker at ratio r, providing a reference for sample concurrency. The analytical model treats the available cache memory as continuously divisible and abstracts away allocator-level details such as recurrent-state slot granularity, KV-page granularity, non-contiguous memory placement, and runtime fragmentation. 3.1.2

Capacity-Balancing Estimator

Under the continuous capacity model, joint residency is limited by the pool that can support fewer trajectories. Reallocating memory from the other pool to this limiting pool increases the joint capacity until both pools support the same number of trajectories. Setting Ckv (r; D) = Cstate (r ) gives Mkv (r ) Mstate (r ) = , Fkv (D) Fstate

(5)

which yields the balancing target rbal (D) =

Fstate Sstate cstate = . Fkv (D) Lkv (D)ckv

(6)

At this balancing ratio, the two pools support the same number of trajectories, and the joint capacity simplifies to M C (rbal (D); D) = . (7) Fkv (D) + Fstate This balancing ratio may not be optimal in practice, since the workload-wide mean KV-cache footprint may not reflect the cache demand of the trajectories active at a given time. Optional local calibration can identify a better-performing nearby ratio. We use r̂ to denote the selected cache ratio, which can be either the model-derived value or a locally calibrated value. For deployment, the continuous capacity estimate C (r̂; D) is rounded down to obtain the integer reference concurrency ⌊C (r̂; D)⌋ per worker. For W homogeneous workers, we use the aggregate reference concurrency C ⋆I (r̂; D) = W ⌊C (r̂; D)⌋ .

(8)

The selected ratio r̂ stays fixed during inference serving. The reference concurrency does not impose a hard limit, since timely cache reuse may support higher concurrency. The results in Section 5.2.1 show that the model-derived cache ratios and reference concurrency settings provide effective operating points for the evaluated workloads. 3.2

Cross-Worker Trajectory-Aware Routing

We consider a distributed inference deployment comprising a gateway and multiple homogeneous workers. Each worker is a model-serving replica that independently schedules its assigned inference requests and maintains its own device-local KV-cache and recurrent-state pools. The gateway routes each inference turn to one of these workers. We focus on worker-local cache reuse without

6

cross-worker cache transfer, so a later turn can reuse its trajectory cache only on the worker that currently holds the corresponding cached state. Routing decisions based solely on load balancing may send successive turns of the same trajectory to different workers, missing opportunities for cache reuse. Conversely, prioritizing trajectory affinity without considering worker load can concentrate requests on a few workers. Requests may then queue at these workers while others remain idle, reducing overall inference efficiency. We adapt consistent hashing with bounded loads (Mirrokni et al., 2016) to per-turn inference routing. LBCH uses a shared trajectory label for affinity and current in-flight request counts for load control. This routing method requires every inference request in trajectory k to carry the same trajectory label κk . Let W denote the worker set and W = |W | its size. These workers are organized on a hash ring R, and κk is hashed to a position on R. The first worker clockwise from that position is defined as the home worker h(κk ) of trajectory k. Let L(w) denote the number of in-flight inference requests assigned to worker w that are either queued or running, and let Ltot = ∑w∈W L(w). A worker is eligible when Ltot + 1 α, (9) W where the +1 term accounts for the request currently being assigned, and α ≥ 1 is the load-bound factor. Values above 1 permit controlled load skew in favor of trajectory affinity, whereas α = 1 gives the tightest eligibility threshold. The router selects h(κk ) if it is eligible. Otherwise, it traverses the ring clockwise with the successor operator succR (w), which returns the next worker clockwise from w on R, and selects the first eligible successor. If no successor is eligible, it falls back to the home worker. Therefore, the resulting procedure, summarized in Algorithm 1, relies only on trajectory labels and worker loads, without inspecting prompt text or predicting cache contents. The experiments in Section 5.2.3 demonstrate that LBCH reduces request-load imbalance across workers while maintaining high cache reuse. L(w) ≤

Algorithm 1 Trajectory-aware routing with load-bounded consistent hashing (LBCH)

Input: trajectory label κk , hash ring R, loads { L(w)}w∈W , load-bound factor α Output: selected worker 1: w0 ← h (κ k ) L +1 2: τ ← tot W α 3: if L ( w0 ) ≤ τ then 4: return w0 5: end if 6: w ← w0 7: repeat 8: w ← succR (w) 9: if w ̸= w0 and L(w) ≤ τ then 10: return w 11: end if 12: until w = w0 13: return w0 3.3

In-Worker Routing-Prefix Scheduling

Worker affinity alone cannot guarantee cache reuse. At high sample concurrency, a request may wait in the worker’s queue long enough for its trajectory’s cached state to be evicted. When 7

the request is eventually served, the worker must recompute the affected prefix. Routing-prefix prioritizes queued requests from recently served trajectories to promote timely reuse while limiting prefix-matching overhead. For each worker, let Q = (q1 , . . . , qn ) be its waiting queue, A its trajectory-label set with capacity CA , and C its radix prefix cache. The trajectory-label set is updated using a least recently used (LRU) rule. Algorithm 2 summarizes the policy in three stages: scoring the waiting requests, building the next batch, and updating A. Each request q has a trajectory label κ (q), a cached prefix-hit length ℓ(q), and a queue-entry time tqueue (q). It also carries a binary prefix-query flag δq ∈ {0, 1}, initialized to 1 when the request is created. During scoring, requests whose labels belong to A form the first tier. The operator PrefixHit(C , q) returns the maximum prefix-match length for q in C , measured in tokens. For each first-tier request with δq = 1, the scheduler queries C via PrefixHit(C , q), records the resulting value as ℓ(q), and sets δq to 0; when δq = 0, the recorded ℓ(q) is reused. For requests with κ (q) ∈ / A, the scheduler skips the prefix-cache query. The priority key is defined as ( (0, −ℓ(q), tqueue (q)), κ (q) ∈ A, p(q) = (1, 0, tqueue (q)), κ (q) ∈ / A. ↑ The operator Sortlex (Q, p) orders the requests in Q by ascending lexicographic order of their priority keys p(q). In this ordering, the first component places requests with labels in A ahead of the others, the second ranks those requests by decreasing prefix-hit length, and the third preserves queue-entry order when the preceding components are tied.

Algorithm 2 Routing-prefix policy

Input: waiting queue Q, label set A, capacity CA , radix prefix cache C Output: admitted batch B , remaining queue Q, updated label set A 1: for all q ∈ Q do 2: if κ (q) ∈ A then 3: if δq = 1 then 4: ℓ(q) ← PrefixHit(C , q) 5: δq ← 0 6: end if 7: p(q) ← (0, −ℓ(q), tqueue (q)) 8: else 9: p(q) ← (1, 0, tqueue (q)) 10: end if 11: end for ↑ 12: Q ← Sort (Q, p ) lex 13: (B , Q) ← BuildBatch(Q) 14: for all q ∈ B do 15: A ← LRUUpdate(A, κ (q), CA ) 16: end for 17: return B , Q, A After this reordering, the existing batch builder scans Q in order and admits requests subject to the current resource limits. This operation is denoted by (B , Q) ← BuildBatch(Q), where B is the admitted batch and Q is the remaining queue. For admitted requests only, A is updated using their labels according to
LRUUpdate(A, κ, CA ) = tailCA (A \ {κ }) ∥ ⟨κ ⟩ , 8

where ∥ appends κ at the most-recent position and tailCA retains the CA most recent labels. The label set A records recent batch admissions, not GPU cache residency. Its LRU updates are independent of GPU cache eviction, so labels and cached state can become inconsistent in two ways. First, a label may remain in A after the corresponding cached state has been evicted. If eviction occurs before the prefix lookup, the resulting hit length may be short or zero, potentially lowering the request’s rank within the first tier. If eviction occurs after the lookup, the stored hit length may become stale because subsequent priority calculations reuse it without another lookup. Second, a label may be removed from A while corresponding cached state remains on the worker: removing a label does not evict that state. A request with this label skips the priority-related prefix lookup and enters the second tier, potentially missing an opportunity to prioritize reuse of the remaining cached state. The capacity CA therefore controls label coverage, not GPU cache allocation. Larger sets offer more reuse opportunities at higher prefix-query cost, while smaller sets reduce this cost but may miss reuse opportunities. We use CA = ⌊C (r̂; D)⌋ as a model-informed default, matching the set capacity to the estimated number of trajectories whose KV cache and recurrent state one worker can retain under the selected cache ratio. Section 5.2.2 evaluates whether this RP design preserves cache reuse and limits priority-calculation cost under cache pressure. 3.4

Integration with SGLang

At deployment, we apply the selected cache ratio r̂ through SGLang’s native mamba_full_memory_ratio parameter. Since each trajectory requires only its latest reusable recurrent state, as explained in Section 3.1, we use SGLang’s native path-based Mamba-state eviction with mamba_max_states_per_path=1 to prune older states as the trajectory advances. We add LBCH as a routing policy in SGLang’s router and RP as a scheduling policy in each worker. Both policies use the same trajectory label carried as request metadata. LBCH selects a worker using this label and in-flight request counts. RP reorders the waiting queue while reusing SGLang’s existing prefix-cache lookup and resource-constrained batch builder. Sample admission remains controlled by the agent execution layer, as described in Section 4.

4

Coordinating Inference and Environment Interaction

In the agent execution layer, we decouple sample admission from environment execution by setting their concurrency limits independently, and use CAAS to dispatch pending environmentinteraction jobs. Trajectory k has Nk inference stages Ik,n , with Ek,n denoting the corresponding environment-interaction stage, when present. Execution follows
Ik,1 → Ek,1 → Ik,2 → · · · → Ik,Nk → Ek,Nk (10) and completes after the final inference stage or, if present, the corresponding environment stage. Completing a nonterminal environment stage releases the next inference turn, so each trajectory has at most one outstanding environment stage. 4.1

Separating Sample Admission from Execution Capacity

Although all samples are eligible at batch start, admitting them all at once can overload inference and environment services. The sample-admission limit CS bounds the number of active samples. Each sample holds a slot from admission to completion, including while waiting for inference or environment interaction. Waiting samples are admitted in submission order as slots become available. Each environment-interaction stage is modeled as a non-preemptive job comprising one 9

or more operations. The environment-execution capacity CE limits the number of concurrently running jobs. In practice, agent-evaluation harnesses such as MCP-Atlas (Bandi et al., 2026) and τ 3 -bench (Shi et al., 2026) commonly use a single task-level concurrency limit. We model this as a unified concurrency gate that constrains sample admission by both inference and environment capacity, giving CSunified = min{C ⋆I (r̂; D), CE } , (11) where C ⋆I is the inference-side reference concurrency from Section 3.1. When CE < C ⋆I , this coupling can underutilize GPUs while samples await environment results. We instead default to decoupled

CS

= C ⋆I (r̂; D)

(12)

and independently cap running environment-interaction jobs at CE . Pending jobs retain sample slots but no environment-execution slots. Figure 11 demonstrates the benefits of improved inference–environment overlap. 4.2

Chain-Advancing Action Scheduling

Capacity separation allows more active trajectories under the same environment-execution limit, which can increase queueing for environment jobs. Their dispatch order determines which trajectory resumes inference next. In practice, a few long trajectories often prolong batch completion after most samples have finished, further reducing GPU utilization. CAAS therefore aims to identify and advance potentially long trajectories earlier through environment-job scheduling to shorten batch makespan. We consider a homogeneous execution pool where any pending job can use any available slot. When a slot becomes available, CAAS selects the next job without changing the agent’s actions or job service times. For job j, let a j be its slot-request time, n j the number of completed inference turns in its trajectory, including the turn that generated job j, and τm the observed duration of its mth inference turn. With hyperparameters B > 0 and ∆ I > 0, we define   aj gj = , (13) B n

1 j τm , τ̄j = n j m∑ =1   c j = τ̄j /∆ I .

(14) (15)

Here g j groups jobs by arrival time, while c j discretizes their observed mean inference duration. When job j joins the pending queue, CAAS assigns it the priority key
p E ( j) = g j , −n j , c j , e j , (16) where e j is a globally increasing entry sequence. Each job’s key remains fixed while it waits, and pending jobs are sorted in ascending lexicographic order. The ordering first protects jobs in earlier arrival bands through g j , then prioritizes deeper revealed chains through −n j . Within the same band and depth, c j provides a coarse tie breaker favoring a shorter estimated inference return, with e j resolving remaining ties in FCFS order. A smaller B favors arrival order, whereas a larger B gives chain depth more influence. Duration bucketing reduces noise in the inference-duration tie breaker.

10

CAAS makes online dispatch decisions using only trajectory state observed in the current run, without requiring sample-specific historical profiles or predictions of total or remaining execution time. With future trajectory lengths unknown, CAAS uses completed inference turns as a proxy for potentially long trajectories and prioritizes deeper chains within each arrival band. This allows subsequent stages to be revealed and executed sooner and reduces the risk that long chains are left to dominate batch completion. Arrival bands prevent returning trajectories from repeatedly overtaking shallower jobs waiting in earlier bands, as can happen under depth-only ordering. Section 5.3 shows that CAAS reduces batch tail and makespan over FCFS after capacity separation.

5

Experimental Evaluation

Our evaluation examines how the proposed optimizations reduce batch makespan, both individually and in combination. Section 5.1 first describes the replay framework and experimental setup. Using controlled replay, we evaluate how cache allocation, sample concurrency, scheduling, and routing affect inference efficiency (Section 5.2), and isolate the effects of capacity separation and CAAS under constrained environment capacity (Section 5.3). Live experiments then evaluate their combined effects across models of different sizes (Section 5.4). 5.1

Controlled Replay Framework and Setup

Real-world environment interactions vary with network conditions, provider load, rate limits, retries, and response drift, making repeatable comparisons across system configurations difficult. To support controlled comparisons, we combine actual model inference with recorded environment responses and service durations from historical agent trajectories. Replay runs within each comparison use the same initial messages, stage sequences, per-turn target output lengths, and recorded environment responses and durations. Figure 2 illustrates the turn-by-turn replay of a trajectory. At each inference turn, the input consists of the initial user message followed, in execution order, by model responses generated in preceding replay turns and recorded environment responses released earlier in the replay. A deployed SGLang service performs actual inference, generating the same number of output tokens as the corresponding historical turn. If an environment stage follows, the trajectory waits for an available execution slot and then pauses for the recorded job duration. The framework subsequently appends the recorded environment response to the context, making the next inference turn ready. No environment actions execute during replay. Stage dependencies and environment durations remain fixed, while inference latency and cache reuse arise from actual serving. The replay timeline includes measured inference times, recorded environment durations, and queueing delays under the selected admission limits, execution capacity, and scheduling policies. Batch makespan is the elapsed time until all trajectories complete. Replay therefore measures execution efficiency rather than semantic correctness, as generated text need not reproduce the recorded action. All replay experiments use Ling-3.0-Flash with thinking enabled, served by SGLang on H20 GPUs with four-way tensor parallelism (TP4) per worker. The model combines KDA and Multi-head Latent Attention layers at a 5:1 ratio with a sparse MoE backbone (Ling Team, 2026a). Inferenceonly replay covers SWE-bench Multilingual (SWE) (Khandpur et al., 2025), BFCL-v4 (BFCL) (Patil et al., 2025), and WideSearch (Wong et al., 2025), bypassing environment durations and capacity limits. Coordination replay uses MCP-Atlas (Bandi et al., 2026) with recorded environment stages. Within each replay comparison, we keep the checkpoint, sampling parameters, workload, hardware budget, submission order, and target output lengths fixed. Appendix A.1 provides the inferenceonly replay workload details. 11

Historical trajectory dataset stage sequence · target output lengths · environment responses · execution durations

selected trajectories

initial user message

recorded output-token count 2

Sample admission capacity CS 1

measured inference duration

Assemble context Model inference

initial user message + generated assistant responses

actual model inference

Replay event timeline

+ recorded environment responses

inference and job durations job queueing delays

generated assistant response

next inference turn

if an interaction stage follows 3

Release recorded environment responses after interaction stage completion

execution flow

Job queue recorded-duration pause capacity CE 1 job ordering

data flow

recorded job duration and replay-derived queueing delay

Estimated batch makespan last trajectory completes

optimization location

Figure 2 Overview of the controlled trace-driven agent replay framework. The numbered markers indicate optimization 1 Capacity separation: the sample-admission limit CS is set independently of the environment-action locations. ⃝ 2 Inference optimizations: cache allocation, routing-prefix scheduling, and LBCH routing. concurrency limit CE . ⃝ 3 CAAS for job ordering. ⃝

5.2 5.2.1

Inference-Side Evaluation Cache allocation and admission operating point

We evaluate the joint effect of cache allocation and sample admission on a single TP4 worker with FCFS scheduling. Using Equation 6, we obtain rbal = 0.09 for SWE, rbal = 2.26 for BFCL, and rbal = 0.10 for WideSearch. The corresponding reference concurrency, Cref = ⌊C (rbal ; D)⌋, is 98, 859, and 98, respectively. We perform a grid search over cache ratios and sample-concurrency limits around these model-derived reference points. Figure 3 summarizes the resulting makespans. Joint operating region. Under FCFS scheduling, the model-derived operating points yield makespans close to the best measured values across all three workloads, leaving only modest room for further improvement through local search. The analytical capacity model abstracts away allocation granularity and runtime fragmentation. Nevertheless, these results show that it provides a simple yet effective heuristic for jointly selecting the cache ratio and sample concurrency by accounting for trajectory token footprints and the number of trajectories that can remain resident. The sweeps around the model-derived operating points show that the benefit of increasing concurrency depends on the cache ratio. At a fixed ratio, raising concurrency from below Cref can initially shorten makespan, but further increases may lead to sharp slowdowns or, on SWE, six-hour timeouts. Conversely, at a fixed concurrency, makespan can decrease and then increase as the ratio grows. Furthermore, at sufficiently low concurrency, makespan is relatively insensitive to the ratio, whereas ratios near rbal generally yield shorter makespans at higher concurrency. We next examine cache reuse to explain these trends. Cache reuse and the cost of excess concurrency. This limited sensitivity at low concurrency is consistent with both cache pools having sufficient capacity to retain reusable state across a broad range of ratios. At higher concurrency, an imbalanced partition can make one pool a bottleneck, whereas ratios near rbal balance the two residency capacities and help preserve cross-turn reuse.

12

(a) SWE

(b) BFCL

(c) WideSearch

bal

bal

r bal = 0.1, Cref = 98

Cache ratio

r

= 0.09, Cref = 98

r

0.045 148.3 TO 0.9 TO TO TO TO 0.5× default 0.09 150.8 136.6 131.9 TO TO TO 1.13 1× 0.5× 0.135 147.4 137.6 131.7 125.7 122.4 236.6 1.5× 2.26 1× 0.18 151.5 138.4 132.6 127.2 171.8 TO 2× 3.39 0.45 147.7 138.1 131.7 TO TO TO 1.5× 5× 4.52 0.9 151.1 138.5 TO TO TO TO 2× default 49 74 98 123 147 157 0.5× 0.75× 1× 1.25× 1.5× 1.6×

= 2.26, Cref = 859

25.01

25.35

35.14

35.82

36.47

0.05 0.5×

97.23

225.26 219.12 217.49 216.19

21.40

20.75

32.67

33.76

34.21

0.1 1×

97.17

89.53

85.79

166.43 179.16

22.12

21.66

22.57

35.25

36.98

0.15 1.5×

97.21

89.61

85.95

82.43

139.90

27.98

32.08

35.75

36.70

38.00

0.2 2×

97.07

89.54

86.00

88.67

155.56

32.59

35.75

37.52

37.74

38.65

430 644 0.5× 0.75×

859 1×

1074 1289 1.25× 1.5×

0.9 100.82 132.91 203.49 205.16 209.53 default 49 74 0.5× 0.75×

98 1×

123 147 1.25× 1.5×

Sample concurrency CS 120

160

200

240

20

25

30

35

40

80

120

160

200

Batch makespan (min)

Figure 3 Cache allocation and sample-concurrency sweeps on (a) SWE, (b) BFCL, and (c) WideSearch. Cell values are complete-batch makespans in minutes; the panels use separate color scales. Second tick-label lines indicate multiples of rbal or Cref , except for the default ratio. Magenta outlines mark the best measured points. TO denotes a six-hour timeout and is excluded from the continuous color scale.

Figures 4 and 5 relate the concurrency trends to cache reuse and system throughput at each workload’s best measured cache ratio. Cache reuse measures the fraction of input tokens served from cache, while new-prefill work counts input tokens processed without cached reuse. We also report the time-averaged number of requests in service, including waiting, and output throughput, measured as successful output tokens per second. Cached input tokens (left)

(b) BFCL 200

100 80

150

60

100

40

100 80

30

60

20

40 50

20 0

(c) WideSearch 40

49

74 98 123 147 157 Sample concurrency

0

10

20 0

430

644 859 1074 1289 Sample concurrency

0

100

100

80

80 60

60

40

40

20

20

0

49

74 98 123 147 Sample concurrency

0

New-prefill tokens (million)

Cached input tokens (%)

(a) SWE

New-prefill tokens (right)

Figure 4 Cache reuse and cumulative new-prefill work versus sample concurrency at the best measured ratio for each workload: (a) SWE, r = 0.135; (b) BFCL, r = 1.13; (c) WideSearch, r = 0.15. Blue circles use the left axis and orange squares the right axis. Every point represents a complete batch. Concurrency configurations are equally spaced on the horizontal axis. Shading highlights the concurrency ranges where cache reuse drops sharply and prefill recomputation increases (SWE: 147–157; BFCL: 644–859; WideSearch: 123–147).

Within the efficient operating region where cache reuse remains high and stable, increasing concurrency leaves new-prefill work largely unchanged while reducing makespan. The mean number of requests in service and output throughput increase together (Figure 5). This pattern is consistent with higher concurrency improving GPU utilization through increased inference parallelism, 13

thereby raising system throughput without substantial additional prefill work. Mean requests in service (left)

(b) BFCL

(c) WideSearch

1000 500

150 125

800

400

100

600

300

400

200

75

1400 150 1200 125 1000 100 800 75 600

50 25 0

49

74 98 123 147 157 Sample concurrency

200

100

0

0

430

644 859 1074 1289 Sample concurrency

400

50

200

25

0

0

1000 800 600 400 200 49

74 98 123 147 Sample concurrency

0

Output throughput (tokens/s)

Mean requests in service

(a) SWE

Output throughput (right)

Figure 5 Mean requests in service (left) and output throughput (right) versus sample concurrency at the same cache ratios as Figure 4: (a) SWE; (b) BFCL; (c) WideSearch. Concurrency configurations are equally spaced on the horizontal axis. Shading highlights the concurrency ranges where output throughput drops despite increasing requests in service (SWE: 147–157; BFCL: 644–859; WideSearch: 123–147).

The shaded regions in Figures 4 and 5 highlight where this relationship reverses. Increasing CS from 147 to 157 for SWE, from 644 to 859 for BFCL, and from 123 to 147 for WideSearch causes cache reuse to decline and new-prefill work to rise to approximately eleven, three, and nine times its previous level, respectively. More requests reside in the engine, yet output throughput falls and makespan increases. Higher concurrency increases the demand for cached state from concurrently active trajectories. The resulting memory pressure can cause reusable cache entries to be evicted, reducing prefix reuse and increasing prefill recomputation. Therefore, different cache ratios support different concurrency ranges for effective cross-turn cache reuse. Within these ranges, higher concurrency can improve inference parallelism and throughput; beyond them, prefix recomputation can outweigh the gains from additional parallelism. The cache ratio and sample concurrency should be selected jointly. 5.2.2

In-worker scheduling

We compare FCFS, SGLang’s native longest prefix match (LPM) and depth-first search weighting (DFS-weight), and routing-prefix scheduling on a single TP4 worker using the same inference-only workloads. We fix the cache ratio at the model-derived balanced allocation and vary sample concurrency to evaluate scheduling performance under low, moderate, and high cache pressure. Policy settings are provided in Appendix A.1. Performance under cache pressure. Figure 6 compares complete-batch makespans. At low concurrency, makespans are generally close across scheduling policies, consistent with limited cache competition and little queueing, leaving relatively few opportunities to benefit from reordering requests. At the two highest tested concurrency levels, all three cache-aware policies achieve shorter makespans than FCFS across the three workloads. Together with the cache-reuse measurements discussed below, this contrast highlights the importance of identifying and prioritizing requests with reusable cached state to accelerate multi-turn agent inference under cache pressure. Among the three cache-aware policies, routing-prefix achieves shorter makespans than LPM and DFS-weight at most tested concurrency settings. 14

(a) SWE

(b) BFCL

(c) WideSearch

FCFS

150.8

131.9

747.4

>13 h

24.5

26.1

35.8

35.9

100.9

86.5

180.2

192.9

LPM

150.0

134.0

129.9

505.0

24.0

24.9

32.9

34.0

100.9

86.6

85.8

85.7

DFS-weight

154.4

144.7

146.0

152.2

24.3

23.3

24.0

24.0

97.6

86.7

86.1

86.0

routing-prefix

150.0

134.3

127.9

128.2

23.0

20.6

19.8

19.8

97.2

86.1

85.2

85.1

49

98

196

289

325

650

1300

1950

49

98

150

200

Sample concurrency CS

125

250 500 Makespan (min)

Sample concurrency CS

750

20

25 30 Makespan (min)

Sample concurrency CS

35

80

120 180 Makespan (min)

220

Figure 6 In-worker scheduling: (a) SWE; (b) BFCL; (c) WideSearch. Cell values are complete-batch makespans in minutes, with separate logarithmic color scales. Magenta outlines mark the minimum makespan for each workload before rounding. The hatched cell marked > 13 h denotes a timeout for SWE FCFS at CS = 289 under the 13-hour limit.

Cache reuse and scheduling overhead. Figure 7 relates these makespan differences to cache reuse and new-prefill work. At the lowest concurrency, the cache metrics are close across policies. As concurrency increases, routing-prefix continues to achieve high cache reuse with little additional prefill work. (a) SWE

(b) BFCL

60 40

1000

100

10

49

98 196 289 Sample concurrency CS FCFS

80 70 60

98 196 289 Sample concurrency CS

New-prefill tokens (M)

New-prefill tokens (M)

49

90

Cache reuse (%)

80

(c) WideSearch 100

30

20

10 325 650 1300 1950 Sample concurrency CS LPM

DFS-weight

80 60 40 20

325 650 1300 1950 Sample concurrency CS

New-prefill tokens (M)

Cache reuse (%)

Cache reuse (%)

100

49

98 150 200 Sample concurrency CS

49

98 150 200 Sample concurrency CS

150 100 50 0

routing-prefix

Figure 7 Cache reuse (top) and cumulative new-prefill work (bottom) from the scheduling runs under the configurations in Figure 6. Each point represents a complete batch; the SWE FCFS timeout at CS = 289 is omitted. The SWE new-prefill axis is logarithmic. The LPM, DFS-weight, and routing-prefix curves nearly overlap on WideSearch.

On SWE, routing-prefix and DFS-weight maintain similarly high cache reuse and low new-prefill work across the tested concurrency levels, whereas LPM shows a sharp drop in reuse and a corresponding rise in new-prefill tokens at the highest concurrency. On BFCL, routing-prefix achieves higher cache reuse and fewer new-prefill tokens than LPM and DFS-weight at the two highest concurrency levels, with a much greater advantage over LPM. On WideSearch, all three 15

cache-aware policies maintain similarly high cache reuse and low new-prefill work, with their curves nearly overlapping. Together, these results highlight routing-prefix’s consistently high cache reuse and low new-prefill work across the evaluated workloads and concurrency levels. Cache reuse alone, however, does not fully explain why routing-prefix achieves shorter makespans even when cache metrics are similar across policies. We therefore examine scheduling overhead. Table 1 reports cumulative priority-calculation time at the highest tested effective sample concurrency for each workload. Routing-prefix incurs less overhead than DFS-weight on all three workloads and less than LPM on SWE and WideSearch. LPM repeatedly matches prefixes for queued requests when computing priorities, while DFS-weight additionally traverses the cache tree. Routing-prefix limits this work through selective prefix matching and reuse of recorded hit lengths. LPM has lower overhead on BFCL because it frequently falls back to FCFS when its waiting queue exceeds SGLang’s built-in threshold of 128 requests. This fallback is intended to limit prefix-matching and sorting overhead on long queues, but bypasses cache-aware prioritization. Its lower calculation cost therefore does not by itself imply more efficient inference execution. Table 1 Cumulative wall-clock time spent computing scheduling priorities over a complete batch at the highest tested effective sample concurrency for each workload. Bold values mark the minimum in each row.

Workload

CS

LPM (s)

DFS-weight (s)

routing-prefix (s)

SWE BFCL WideSearch

289 1950 200

735.00 7.65 62.82

1704.25 172.15 144.08

303.05 10.91 39.97

By prioritizing queued requests from recently served trajectories, routing-prefix promotes reuse before their cached state is evicted. Trajectory-aware scheduling complements cache allocation and sample admission. An appropriate cache ratio and concurrency setting can preserve crossturn reuse and achieve short makespans even with FCFS. Routing-prefix helps sustain reuse with low scheduling overhead as cache competition intensifies, making performance less sensitive to increased concurrency. 5.2.3

Cross-worker routing

We compare SGLang’s native cache-aware routing (CA), native consistent hashing (CH), and LBCH on two TP4 workers using the same inference-only workloads. All workers use routing-prefix scheduling, and the cache ratio and sample-concurrency limit are kept fixed across routing policies within each workload. The workload-specific configurations and policy parameters are listed in Appendix A.1. Batch makespan and cache reuse. Figure 8 shows that LBCH achieves the shortest measured makespan on all three workloads, with reductions of 1.5%–13.6% relative to CA and 1.9%–3.8% relative to CH. Figure 9 compares cache reuse and new-prefill work. Cache reuse rates are similar across policies within each workload (98.4%–98.8% on SWE, 86.2%–87.2% on BFCL, and 95.2%– 95.7% on WideSearch), and LBCH does not consistently minimize new-prefill work. Therefore, similar cache reuse does not necessarily translate into similar makespans, motivating a closer examination of how routing policies distribute requests across workers. Load distribution. To assess load distribution across workers, we use two complementary metrics. Let Nw (t) denote the number of in-flight inference requests, including queued and executing requests, on worker w at time t.

16

(a) SWE Makespan (min)

88

(b) BFCL

84.1

16.8

(c) WideSearch 54.0

16.8

52.5

16.2

75.3

16.0

72.7

15.6

72

CH

51.9

51.0 49.5

15.2 CA

52.9

52.7

80

LBCH

CA

CH

CA

LBCH

CH

CA

CH

LBCH

LBCH

Figure 8 Cross-worker routing on two TP4 workers: (a) SWE; (b) BFCL; (c) WideSearch. Bars show measured batch makespan in minutes; the panels use separate vertical scales that do not start at zero. New-prefill work (bars, left axis)

30

98.7%

(b) BFCL 98.8%

98.4%

100

20 50 10

18

87.2%

(c) WideSearch 86.2%

86.5%

100

0 CA

CH

95.6%

95.2%

50

100

50 5

0

LBCH

95.7%

10

12 6

0

15

0 CA

CH

LBCH

0

Cache reuse (%)

New-prefill tokens (M)

(a) SWE

Cache reuse (line, right axis)

0 CA

CH

LBCH

Figure 9 Cache reuse and new-prefill work for the runs in Figure 8: (a) SWE; (b) BFCL; (c) WideSearch. Bars show new-prefill work on the left axis; the purple line shows cache reuse on the right axis. Workload panels use separate scales for new-prefill work.

Let T span from the start of the first request to the completion of the last request, and let J = {t ∈ [0, T ] : N1 (t) + N2 (t) > 1} denote the intervals with more than one in-flight request across both workers. We denote the time-averaged request-count imbalance over these retained intervals by β, defined as Z 1 d(t) dt, (17) β= | J| J where | J | is the total retained duration and d(t) is the instantaneous request-count imbalance, d(t) =

| N1 (t) − N2 (t)| , N1 (t) + N2 (t)

t ∈ J.

(18)

This metric gives equal weight to each unit of retained elapsed time. We also measure output-token imbalance, denoted by γout , to capture differences in generation workload, since request counts alone do not capture differences in output length. Let Dw be the total number of output tokens generated by requests executed on worker w over the batch: Dw =

∑ Oi ,

(19)

i ∈Rw

where Oi is request i’s actual output length and Rw contains requests actually executed on worker w. We define γout as | D1 − D2 | γout = . (20) D1 + D2 17

Time-averaged request-count imbalance (%) ↓ Output-token imbalance (%) ↓

(a) SWE 60

(b) BFCL

57.8

40

60

60

48.9

40

32.2 19.9

20

(c) WideSearch

0

40

CH

LBCH

30

35.7 23.9

20

15.3

12.6

CH

LBCH

0 CA

30.7

20 0

CA

CA

30

30

20

20

CH

LBCH

22.7

20 10.9

10

10.7 7.8

0

10

10 3.8

3.8

CH

LBCH

0 CA

CH

LBCH

5.8

7.2

5.4

0 CA

CA

CH

LBCH

Figure 10 Routing load distribution on (a) SWE, (b) BFCL, and (c) WideSearch. Top: time-averaged request-count imbalance over intervals with more than one in-flight request across both workers; each unit of retained time has equal weight. Bottom: output-token imbalance over the complete batch. Each row uses a common percentage scale. Lower values indicate more even distributions of requests over time and generated tokens across workers, respectively.

A lower β indicates more balanced in-flight request counts across workers over time, while a lower γout indicates a more balanced distribution of generated tokens across workers over the complete batch. Note that these metrics characterize load distribution rather than directly measuring computational cost. Figure 10 shows that LBCH achieves lower request-count imbalance than CA and CH across all three workloads. Its advantage over CA is particularly pronounced on SWE and BFCL, where output-token imbalance is also substantially lower. Compared with CH, LBCH improves both metrics on SWE and WideSearch, while on BFCL it achieves lower request-count imbalance with nearly identical output-token imbalance. Together with the cache-reuse results, these observations show that LBCH maintains high reuse while improving load balance, even though it does not always minimize new-prefill work. These results highlight the importance of jointly considering cache reuse and load distribution when routing multi-turn agent requests. 5.3

Inference–Environment Coordination

Using the replay framework in Section 5.1, we evaluate capacity separation and CAAS under constrained environment-execution capacity on historical trajectories from MCP-Atlas (Bandi et al., 2026), whose tasks rely on external MCP services that may degrade under high concurrency or encounter upstream rate limits. The inference deployment uses two TP4 workers with routingprefix scheduling and LBCH. Guided by the capacity model in Section 3.1, we use CS = 50 as the inference-side admission setting. To isolate the effect of capacity separation, we compare unified admission (CS = CE ) with decoupled admission at CS ∈ {10, 25, 50} for environment-execution limits of CE ∈ {3, 5}, both with FCFS job scheduling. We then compare CAAS with FCFS at CS = 50 under the same CE , changing only job ordering while keeping the inference configuration and recorded workload unchanged. Configuration details and supporting background on environment constraints are provided in Appendix A.2. Capacity separation. Figure 11 shows the effect of capacity separation at fixed environmentexecution capacity. 18

Unified baseline (CS = CE)

FCFS

(b) CE = 5

250

160

225

140

200 120 175

Mean in-flight inference requests

Makespan (min)

(a) CE = 3

6

6

4

4

2

2

0

0 3

10

25

50

5

Sample concurrency CS

10

25

50

Sample concurrency CS

Figure 11 Capacity separation under FCFS job scheduling: batch makespan (top) and mean in-flight inference requests (bottom). The first point uses unified admission CS = CE . In-flight requests are measured from client request launch to response completion and averaged over the replay makespan.

Increasing sample admission from the unified baseline to CS = 10 reduces makespan by approximately one third at both environment-execution capacities. Capacity separation thus provides substantial gains even at modest sample concurrency. Further increases bring smaller improvements, with makespan fluctuating within a narrow range at CS = 25, 50. The mean number of in-flight inference requests helps explain this pattern. It rises substantially even when sample admission is increased only to CS = 10, but barely changes when CS doubles from 25 to 50 at either environment-execution capacity. By supplying more concurrent inference work, capacity separation can improve GPU utilization and thereby increase overall system throughput. Admitting more samples allows inference and environment-interaction stages to overlap across trajectories, yet the limited environment-execution capacity constrains the release of subsequent inference work. Additional samples may then spend more time waiting for environment interaction rather than supplying concurrent inference work, so the makespan gains level off. At matched sample concurrency, CE = 5 supports more in-flight requests and shorter makespans than CE = 3, consistent with earlier release of subsequent inference work. CAAS and the completion tail. We evaluate CAAS at CS = 50 after capacity separation, where it reduces makespan relative to FCFS by 1.9% at CE = 3 and 3.5% at CE = 5. To examine how CAAS affects completion of the last few tasks, we measure the duration of the batch tail. Measuring time from the first task admission, let tq denote when the completed fraction first reaches or exceeds q%, for 0 < q ≤ 100; t100 is the batch makespan. The final-p% tail is the interval from t100− p to t100 , with duration L p = t100 − t100− p , 0 < p < 100. (21) We use p = 5 to analyze the replay completion tail below. Figure 12 shows that CAAS shortens the tail interval relative to FCFS by 19.0% at CE = 3 and 21.4% at CE = 5. Although CAAS reaches 95% completion later, it finishes the batch earlier. Prioritizing jobs from trajectories with greater revealed depth can delay other trajectories, but allows subsequent stages of deeper chains to execute earlier, shortening the final completion tail. This behavior supports its design goal of exposing and advancing later stages of long execution chains before they become batch-ending bottlenecks, using only trajectory progress observed during the current run. These gains come from changing job dispatch order, not service times. 19

FCFS

Completed (%)

(a) CE = 3

CAAS

(b) CE = 5

100

100

97.5

97.5

95 140

95 145

150

155

160

165

85

Time (min)

90

95

100

105

110

Time (min)

Figure 12 Completion of the final 5% of trajectories at CS = 50. Both policies use the same absolute replay time axis within each panel. The final-5% tail duration is denoted by L5 .

5.4

End-to-End Validation

We conduct live evaluations on BFCL, SWE, WideSearch, and MCP-Atlas using Ling-3.0-Flash (124B total and 5.1B active parameters) (Ling Team, 2026a) and Ling-3.1-Flash (560B total and 25B active parameters) (Ling Team, 2026b). We omit BFCL-v4 for Ling-3.1-Flash, as scores are nearly saturated and we no longer use this benchmark in model development. The two models use two TP4 workers on eight H20 GPUs and two TP8 workers on sixteen H20 GPUs, respectively. Within each model–workload comparison, we fix the checkpoint, task set, sampling parameters, and hardware, varying only cache allocation, sample concurrency, inference scheduling and routing, and, for MCP-Atlas, capacity separation and environment-job scheduling. Live makespan includes all samples and retry time, with no exclusion of samples that are retried or ultimately fail. For BFCL, SWE, and WideSearch, we tune sample concurrency for the original baselines with the default cache ratio, FCFS scheduling, and cache-aware routing fixed, targeting GPU saturation without substantial inference queueing. The optimized configuration combines cache allocation and sample concurrency settings guided by the workload-aware capacity model with RP scheduling and LBCH routing. For each model, SWE includes a Baseline-2 configuration that uses the same sample concurrency as the optimized configuration while retaining the baseline cache allocation and inference policies. Detailed configurations and operating points are provided in Appendix A.3 and Table A2. Table 2 Live inference-side results. Each metric is calculated per run and averaged equally over three runs per configuration. Input token counts include cached tokens. Token totals are in millions. Imbalance metrics follow Section 5.2.3. Makespan

Tokens (M)

Reuse

Imbalance (%)

Model

Workload

Config.

(min)

Input

Output

H (%)

β

γout

Score

3.0

BFCL

Baseline Opt Baseline-1 Baseline-2 Opt Baseline Opt

68.4 49.2 166.9 197.7 149.5 186.7 116.7

141.1 141.6 1286.2 1371.4 1347.9 261.9 311.1

3.9 3.9 8.5 8.7 8.7 10.9 10.0

84.2 84.5 98.3 87.5 98.6 79.9 81.7

10.6 17.0 17.2 17.2 16.4 11.5 15.6

2.9 3.1 13.0 7.7 3.5 1.4 4.6

73.0 75.65 72.4 71.3 72.6 73.6 72.7

Baseline-1 Baseline-2 Opt Baseline Opt

234.1 304.1 185.1 459.9 359.1

1073.3 1102.8 1045.7 415.7 430.0

11.1 11.5 11.0 25.0 25.1

98.2 80.4 98.3 83.2 80.7

15.1 15.9 16.2 8.8 17.6

11.6 12.0 3.3 1.7 2.5

87.16 86.0 88.33 78.6 78.1

SWE

WideSearch 3.1

SWE

WideSearch

20

Table 2 reports mean makespan, token counts, cache reuse, worker-load imbalance, and evaluation scores over three runs per configuration for the first three workloads, where environment-execution capacity is sufficient and does not constrain sample admission. For Ling-3.0-Flash, the optimized configuration reduces mean makespan by 28.1% on BFCL, 10.4% on SWE, and 37.5% on WideSearch relative to their original baselines. For Ling-3.1-Flash, the corresponding reductions are 20.9% on SWE and 21.9% on WideSearch. These results demonstrate that the proposed optimizations scale effectively to larger models while still achieving substantial makespan reductions. Stochastic model sampling and environment variability cause trajectories and token counts to differ across live runs. The makespan reductions, even with higher input-token volumes on some workloads, support the effectiveness of the optimizations under realistic execution variability. On BFCL and WideSearch, optimized runs achieve shorter makespans despite higher worker-load imbalance, while maintaining comparable cache reuse. These results suggest a more effective tradeoff among concurrency, cache reuse, and worker-load balance, where the benefits of sustaining cache reuse at higher concurrency can outweigh a slight deterioration in worker-load balance. The SWE comparison further shows that increased concurrency alone does not explain these gains. For Ling-3.0-Flash, cache reuse falls from 98.3% in Baseline-1 to 87.5% in the higher-concurrency Baseline-2. This decline is consistent with increased cache pressure and eviction, leading to additional recomputation that can offset parallelism gains. At the same concurrency as Baseline-2, however, the optimized configuration maintains 98.6% reuse and reduces makespan by 24.4%. Ling3.1-Flash shows a similar pattern: with the same sample concurrency, Opt maintains 98.3% cache reuse versus Baseline-2’s 80.4% and reduces makespan by 39.1%. Consistent with the controlled experiments in Sections 5.2.1–5.2.3, these results demonstrate the benefits of coordinated cache allocation, scheduling, and routing in preserving cache reuse at higher concurrency and reducing batch makespan. The latency measurements further clarify how these batch-level gains relate to individual request performance. Compared with the concurrency-matched Baseline-2, Opt reduces TTFT by 24.9% and 50.2%, and TPOT by 51.1% and 40.2%, for Ling-3.0-Flash and Ling-3.1Flash, respectively. Relative to Baseline-1, Opt uses higher concurrency, increasing competition for inference resources; this is consistent with higher per-request TTFT and TPOT despite shorter batch makespan, reflecting the distinction between batch efficiency and per-request latency. Appendix A.3 provides a detailed analysis of the SWE results. Table 3 Live MCP-Atlas results, averaged equally over three 500-task runs per configuration. Tail durations L5 and L10 are measured from each run’s own 95% and 90% completion points to its final task termination, including both successful and failed tasks. Mean in-flight is the time average of recorded inference calls over the full task-execution window, computed from native trajectory start/end timestamps and averaged equally across runs.

Makespan (min)

Mean in-flight

L5 (min)

L10 (min)

Score

Baseline Opt-FCFS Opt-CAAS

466.1 168.2 163.0

1.2 7.3 7.9

38.9 67.6 66.0

73.7 79.2 73.4

65.5 66.6 65.6

Baseline Opt-FCFS Opt-CAAS

333.5 140.5 128.5

1.6 12.5 11.5

26.1 49.8 35.3

43.5 54.4 38.6

81.0 80.9 81.3

Model

Configuration

3.0

3.1

For MCP-Atlas, Table 3 reports mean results over three 500-task runs per configuration, evaluating the integrated inference and environment optimizations under constrained environment capacity. Relative to the unified-admission baseline, inference-side optimizations and capacity separation (Opt-FCFS) reduce mean makespan from 466.1 to 168.2 minutes (63.9%) for Ling-3.0-Flash and from 333.5 to 140.5 minutes (57.9%) for Ling-3.1-Flash, with environment capacity unchanged for 21

each model. Mean in-flight inference calls increase from 1.2 to 7.3 for Ling-3.0-Flash and from 1.6 to 12.5 for Ling-3.1-Flash, consistent with more inference work being available while other trajectories await environment interaction. Compared with Opt-FCFS, Opt-CAAS further reduces mean makespan by 3.1% for Ling-3.0-Flash and 8.5% for Ling-3.1-Flash. Using the tail durations defined in Equation 21, the corresponding reductions in L5 and L10 are 2.4% and 7.3% for Ling-3.0-Flash, and 29.1% and 29.0% for Ling-3.1Flash. These live results are consistent with the benefits of CAAS observed in the controlled replay experiments in Section 5.3, although live trajectories vary across runs. Although Opt-CAAS completes the batch much earlier, its final 5% takes longer than the baseline’s. This apparent discrepancy can be explained by differences in the tasks that finish last. Under limited admission, the baseline’s final tasks are largely those admitted last, rather than necessarily those with the longest trajectories. Broader admission lets shorter trajectories finish while longer ones continue, so the optimized tail is more concentrated in long trajectories. The final 25 tasks under Opt-CAAS average 27.9 and 15.6 turns for Ling-3.0-Flash and Ling-3.1-Flash, respectively, compared with 10.1 and 7.9 under the baseline. When matched by task ID, the Opt-CAAS tail tasks also require more turns in baseline runs. This difference helps explain why Opt-CAAS can finish the batch much earlier while taking longer to complete its final 5%, so L5 alone should not be used to compare overall execution efficiency across different admission policies. Across the evaluated model–benchmark combinations, evaluation scores remain aligned between the baseline and optimized configurations, indicating that the efficiency gains do not come at the expense of response quality.

6

Discussion and Future Work

Although this work focuses on batch makespan in LLM training and development workflows, its underlying principles may extend to other agent workloads. For example, in personal-agent services, workload-aware cache allocation, RP scheduling, and LBCH routing could preserve cross-turn reuse while supporting concurrent users. These mechanisms could reduce prefix recomputation and shorten the prefill component of TTFT. Reducing redundant prefill may also help reduce TPOT by leaving more capacity for ongoing decoding. Separating sample admission from environment-execution capacity could let ready trajectories use GPUs while others wait for tools or external services. In workflows with parallel subagents, the chain-advancing principle behind CAAS could inform tool-job scheduling to advance subagents that delay downstream aggregation or overall task completion. Many directions remain for improving agent trajectory execution efficiency and further reducing makespan. These include extending workload-aware cache allocation and trajectory-aware scheduling to hierarchical caching while accounting for cache capacity and transfer costs, and investigating cache eviction policies for hybrid models in agent workloads. Another important direction is makespan optimization subject to application service-level objectives (SLOs), such as TTFT and TPOT targets, which is relevant to datacenters seeking to improve task throughput while meeting user-facing latency requirements.

22

7

Conclusion

We present coordinated inference and environment optimizations to reduce batch makespan for multi-turn agents served by hybrid LLMs. A workload-aware capacity model guides cache allocation and sample concurrency, while RP and LBCH promote timely cross-turn reuse at low scheduling cost and balance trajectory affinity with worker load. Under constrained environment resources, capacity separation improves inference–environment overlap, and CAAS advances long execution chains to shorten the batch tail. Controlled replay evaluates individual mechanisms, while live experiments with Ling-3.0-Flash and Ling-3.1-Flash demonstrate batch makespan reductions of 10.4%–65.0% across four well-established benchmarks without compromising model performance. Their successful application in developing the Ling-3.x model series demonstrates the practical value of these optimizations in improving model-development efficiency.

References C. Bandi et al. MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers. arXiv:2602.00933, 2026. Code: https://github.com/scaleapi/mcp-atlas. D. Guo, J. Wu, and S.-M. Yiu. SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusters. arXiv:2605.00528, 2026. Kimi Team. Kimi K3: Open Frontier Intelligence. arXiv:2607.24653, 2026. W. Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP, 2023. J. Li, T. Wang, and Y. Chen. From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving. arXiv:2607.02574, 2026. Ling Team. Ling-3.0-flash: Intelligence through Efficiency. Technical blog, July 24, 2026. https://developer.ant-ling.com/zh-CN/blogs/ling-3.0-flash-release. Official model card, 2026a. https://huggingface.co/inclusionAI/Ling-3.0-flash. M. Luo et al. Autellix: An Efficient Serving Engine for LLM Agents as General Programs. arXiv:2502.13965, 2025. A. X. Nguyen. Asymmetric Virtual Memory Paging for Hybrid Mamba-Transformer Inference. arXiv:2605.22416, 2026. S. Nian et al. CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restoration. arXiv, 2026. R. Pan et al. Marconi: Prefix Caching for the Era of Hybrid LLMs. MLSys, 2025. S. G. Patil et al. The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models. ICML, 2025. BFCL-v4: https://gorilla.cs.berkeley.edu/leaderboard.html. R. Qin et al. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. arXiv:2407.00079, 2024. S. Qiu et al. Tutti: Making SSD-Backed KV Cache Practical for Long-Context LLM Serving. arXiv:2605.03375, 2026. Qwen Team. Qwen3.8-2.4T-A95B. Official model card, 2026. https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B. Q. Shi et al. τ-Knowledge: Evaluating Conversational Agents over Unstructured Knowledge. arXiv:2603.04370, 2026. Code: https://github.com/sierra-research/tau2-bench. V. Srivatsa et al. Preble: Efficient Distributed Prompt Scheduling for LLM Serving. arXiv:2407.00023, 2024.

23

vLLM Project. Automatic Prefix Caching. vLLM Documentation. https://docs.vllm.ai/en/stable/design/prefix_caching/. N. Wadlom, J. Shen, and Y. Lu. Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective. arXiv:2603.16104, 2026. T. Xia et al. Idleness is Relative: Exploiting Tool-Call Idle Windows for Offloading in Agentic Systems with MORI. arXiv:2606.00866, 2026. B. Xiao et al. ARL-Tangram: Unleash the Resource Efficiency in Agentic Reinforcement Learning. arXiv:2603.13019, 2026. B. Xie et al. PEEK: Predictive Queue-Informed KV Cache Management for LLM Serving. arXiv:2607.02525, 2026. Y. Yuan et al. DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Serving. arXiv:2602.06502, 2026. L. Zheng et al. SGLang: Efficient Execution of Structured Language Model Programs. NeurIPS, 2024. J. Ding et al. Observation, Not Prediction: Conversation-Level Disaggregated Scheduling for Agentic Serving. arXiv:2606.01839, 2026. R. Wong et al. WideSearch: Benchmarking Agentic Broad Info-Seeking. arXiv:2508.07999, 2025. V. Mirrokni, M. Thorup, and M. Zadimoghaddam. Consistent Hashing with Bounded Loads. arXiv:1608.01350, 2016. K. Khandpur, K. Lieret, C. E. Jimenez, O. Press, and J. Yang. SWE-bench Multilingual. Official benchmark description, 2025. https://www.swebench.com/multilingual.html. Ling Team. Ling-3.1-flash. Official model card, 2026b. https://huggingface.co/inclusionAI/Ling-3.1-flash. Z. Zheng et al. BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batching. MLSys, 2026.

24

A

Experimental Configuration Details

A.1

Inference-Side Experiments

Table A1 lists the inference-only workloads. The SWE replay set contains 289 trajectories because only successful historical executions were collected; failed executions were excluded. Selected SWE trajectories have long contexts and deep chains; BFCL lengths vary substantially and include single-turn trajectories to evaluate our methods on short trajectories; WideSearch repeatedly invokes inference for broad information seeking (Khandpur et al., 2025; Patil et al., 2025; Wong et al., 2025). Replay measures execution efficiency for these recorded workloads; it does not evaluate semantic correctness or cover all failures and retries. Table A1 Historical inference-only replay workloads. Output totals are from the routing replay runs. Live task counts are reported separately in Table A2.

Workload

Recorded trajectories

Inference requests

Output tokens

289 5,106 200

25,416 17,029 5,672

6,000,784 1,441,547 4,736,147

SWE BFCL WideSearch

Cache allocation and admission. The experiments in Section 5.2.1 use one TP4 worker with FCFS scheduling. The cache is flushed before each timed run, and complete-batch makespan excludes deployment and warmup. The tested cache ratios and sample-concurrency limits are shown in Figure 3; TO denotes a six-hour timeout. In-worker scheduling. In Section 5.2.2, the cache ratio is fixed at the model-derived balanced allocation: 0.09 for SWE, 2.26 for BFCL, and 0.10 for WideSearch. LPM and DFS-weight use SGLang’s default policy parameters. Figure 6 reports the tested sample-concurrency limits; the SWE FCFS run at the largest effective concurrency of 289 is marked as a timeout under the 13-hour limit. Cross-worker routing. The replay comparison in Section 5.2.3 uses two TP4 workers with routing-prefix scheduling. The fixed (r, CS ) values are (0.09, 196) for SWE, (2.26, 2000) for BFCL, and (0.10, 200) for WideSearch. CA and CH use their native default parameters, and LBCH uses α = 1.25. Request imbalance is averaged over the intervals with more than one in-flight request across both workers, as defined in Equation 17. Output imbalance uses the generated token totals assigned to each worker.

A.2

Inference–Environment Coordination

Environment-capacity settings. MCP-Atlas (Bandi et al., 2026) tasks depend on external MCP services, which may degrade under high concurrency or encounter upstream rate limits such as HTTP 429 errors. The runner configuration referenced for these experiments defaulted to five concurrent tasks.1 This motivates evaluating constrained environment-execution settings, rather than assuming that environment capacity grows with inference capacity. We report results at CE ∈ {3, 5}. Replay configuration. Section 5.3 uses 495 successful historical trajectories with recorded environment responses and job service times. Two TP4 workers use a cache ratio of r = 0.14, routing-prefix scheduling, and LBCH routing. CAAS uses a time-band width of B = 30 seconds and an inference-duration bucket width of ∆ I = 1 second. Compared replay runs keep the model, hardware, and recorded workload fixed.

A.3

Live Execution Settings

Both models enable thinking mode and use a maximum context length of 256K tokens in the live experiments. Table A2 summarizes the task sets and operating points. For MCP-Atlas, the baseline uses unified admission with CS = CE = 3. Both Opt variants use the same inference configuration and decoupled admission at CS = 50 while retaining CE = 3. They differ only in FCFS versus CAAS environment dispatch. For the live evaluations in Section 5.4, to use the same load-balance criteria for CA and LBCH, CA sets balance_abs_threshold=0 and balance_rel_threshold=1.6666667, while LBCH uses α = 1.25. Live CAAS uses the same parameters as the replay experiments: B = 30 seconds and ∆ I = 1 second. 1 See the official MCP-Atlas runner configuration, tool-serving guidance, and Slack API rate limits.

25

Table A2 Live task sets and operating points. All baseline cache ratios are 0.9. B1 and B2 denote SWE Baseline-1 and Baseline-2. The two MCP-Atlas Opt variants share the listed settings. A dash in CE indicates that environment execution does not constrain sample admission. Sample concurrency CS Model

Workload

Tasks

Baseline/B1

B2

Opt

Opt ratio

CE

3.0

BFCL SWE WideSearch MCP-Atlas

5,106 300 200 500

100 100 40 3

– 250 – –

300 250 100 50

1.5 0.1 0.25 0.14

– – – 3

3.1

SWE WideSearch MCP-Atlas

300 200 500

50 15 3

100 – –

100 40 50

0.2 0.27 0.34

– – 3

Live WideSearch also uses the evaluated model for summarization, which can introduce additional uncached input and help explain its lower aggregate cache reuse relative to replay. Additional SWE diagnostics. Table A3 provides a closer comparison of the tuned baseline, concurrencymatched baseline, and Opt. Table A3 Live SWE diagnostics under the tuned baseline, concurrency-matched baseline, and Opt. New-prefill counts uncached input tokens. TTFT and TPOT are averaged across requests within each run, then across runs. Mean API in-flight averages outstanding API requests, including those waiting, over each live run.

Metric Sample concurrency Makespan (min) Cache reuse (%) New-prefill (M) Mean API in-flight TTFT (s) TPOT (ms/token)

Ling-3.0-Flash

Ling-3.1-Flash

Baseline-1 Baseline-2

Opt Baseline-1 Baseline-2

Opt

250 149.5 98.58 18.98 76.7 7.57 41.81

100 185.1 98.27 17.93 63.7 5.88 36.37

100 166.9 98.3 22.2 40.9 2.90 28.6

250 197.7 87.46 170.90 105.2 10.08 85.55

50 234.1 98.2 19.3 35.5 2.54 24.3

100 304.1 80.41 211.81 71.0 11.81 60.84

We also include Baseline-LowConcurrency as a reference configuration. Baseline-LowConcurrency uses CS = 20 with the default cache ratio, FCFS scheduling, and cache-aware routing. Its live SWE runs yield mean makespans of 208.1 and 263.0 minutes for Ling-3.0-Flash and Ling-3.1-Flash, respectively, over three runs per configuration. Tuning sample concurrency under this baseline configuration yields Baseline-1 at CS = 100 and 50, respectively, reducing makespan to 166.9 and 234.1 minutes (19.8% and 11.0%) while maintaining cache reuse above 98%. This is consistent with the concurrency benefits observed in Section 5.2.1. Increasing sample concurrency from Baseline-1 to Baseline-2 lowers cache reuse and increases new-prefill work, request latency, and makespan for both models. At the same sample concurrency as Baseline-2, Opt restores cache reuse to above 98% and reduces new-prefill tokens by 88.9% and 91.5%, respectively. TTFT falls by 24.9% and 50.2%, and TPOT falls by 51.1% and 40.2%. Together with makespan reductions of 24.4% and 39.1%, these results show that Opt maintains high cache reuse at high concurrency, benefiting from both parallelism and cache reuse to shorten makespan. At the same sample concurrency as Baseline-2, Opt has fewer mean in-flight requests, lower TTFT and TPOT, and shorter makespan. Together, these observations are consistent with faster request completion and task progress. Relative to Baseline-1, Opt uses higher concurrency, increasing competition for inference resources; this is consistent with higher per-request TTFT and TPOT despite shorter batch makespan, distinguishing batch efficiency from per-request latency.

26

Related documents

• SWE-Serve: Benchmarking Agentic Engineering For Production Inference Serving#1062702
• SWE-Serve: Benchmarking Agentic Engineering For Production Inference Serving#1043123
• Humanize MLE Agents: Multi-Agent Session Alternation Beyond Single-Agent Limits#1297846
• Humanize MLE Agents: Multi-Agent Session Alternation Beyond Single-Agent Limits#1291861
• GoodServe: Globally-coordinated Routing of Agentic LLM Inferences for High Goodput#1585708