Koi: Evidence-Guided, Self-Calibrating Planning for Multi-Cloud, Heterogeneous LLM Inference Fleets

Koi: Evidence-Guided, Self-Calibrating Planning for Multi-Cloud, Heterogeneous LLM Inference Fleets Mankeerat Sidhu1,* , Gangmuk Lim2,* , Hetarth Chopra1 , Maximillian Eisenberg1 , Quan Hao Ng1 , Aryan Gosaliya1 , Shu Liu3 , Luka Govedič4 , 5 Jiaxin Shan , Hongkuan Zhou6 , Vikram Sharma Mailthody6 , P. Brighten Godfrey2 , Ion Stoica3

arXiv:2610.24355v1 [cs.DC] 17 Oct 2026

1 Tandemn

2 University of Illinois Urbana-Champaign 4 New York University

5 ByteDance

3 UC Berkeley

6 NVIDIA

{mankeerat, hetarth, max, quan, aryan}@tandemn.com, {gangmuk2, pbg}@illinois.edu {lshu, istoica}@berkeley.edu, lg3769@nyu.edu jiaxin.shan@bytedance.com, {hongkuanz, vmailthody}@nvidia.com USA

Abstract Deploying a large language model (LLM) requires choosing the right accelerator, parallelization strategy, engine configuration, and replica count amongst ∼140 other variables across clouds, regions, and markets. These decisions are intertwined, and the best cluster-wide plan continually changes as jobs contend for shared capacity on shifting workloads. We present Koi, a self-calibrating cluster planner for heterogeneous, multi-cloud fleets. Koi uses a Recursive Language Model (RLM) hierarchy to propose hypotheses and reason over deployment while deterministic components predict serving behavior, score candidates, and control placement. It represents inference behavior as a fixed causal graph linking decisions (e.g., accelerator, parallelism), runtime mediators (e.g., KV-cache pressure, communication overhead), and outcomes (e.g., latency, throughput). Each deployment strengthens or weakens scoped causal relationships, building a comprehensive playbook for inference that the planner employs to guide future decisions. Job-wise configurations are composed into cluster-wide plans and evaluated against serving objectives, resource constraints, SLO risks, and cost. Across benchmarks, Koi achieves 1.18× greater goodput than the strongest baseline while reducing $/token by 20% in a real heterogeneous cluster. Ablations isolate the contribution of Koi’s architecture: a pure language model attains a composite score of 35.3, the same model equipped with Koi’s deterministic tools reaches 51.6, and the full Koi planner reaches 70.8.

1

Introduction

An LLM inference provider serves a changing mix of jobs on a shared accelerator fleet that spans clouds, regions, markets (reserved, on-demand, or spot), and hardware generations. Users specify each job’s model, workload profile * Co-first authors.

(input/output lengths and arrival process), class (online or batch), and serving target (a latency bound for online jobs or a completion deadline for batch jobs). The provider decides how every job runs, including the environment and accelerator, the parallelism and engine configuration, and the number of replicas, and must revise these decisions as jobs arrive and complete and as available capacity changes. Throughput

max. 717 tok/s 100%

Memory BW max. 61.6%

75%

Cost

50%

max. $38.28/M

25%

SM utilization

GPU count

max. 99.0%

TP4/PP2 TP4/PP4

max. 32

TP8/PP1 TP8/PP2

TP8/PP4

Figure 1. Parallelism configurations expose conflicting performance, cost, and utilization trade-offs. It used Llama 3 70B on AWS A10G GPUs with 2,048/512 input/output tokens. Each axis is normalized independently. Even for a single job, this choice is consequential. For a profiled 70B dense model on one accelerator family, five feasible tensor- and pipeline-parallel configurations span 4.4× in throughput and 7.5× in cost per token and trade off performance, cost, and utilization differently (Figures 1 and 2). TP4/PP2 across two four-GPU instances serves the workload at 180 tok/s for $0.97, whereas TP8/PP1 on one eight-GPU instance reaches only 162 tok/s for $2.16. The best configuration also depends on workload shape, because prefill-heavy traffic stresses compute while decode-heavy

0

67

2 1 0

600

400

200

Total Time (s)

3

Total Cost ($)

4

$2.94

e e e e e e rg rg rg rg rg rg x l a P2 x l a P4 x l a P2 x l a P4 x l a P1 x l a P1 D .12 , P .12 , P .48 .48 , P .12 , P .48 , P g5 TP4 x g5 TP4 x g5 TP4 g5 TP8 x g5 TP8, x g5 TP8 2 4 2 4

across a heterogeneous, multi-cloud fleet, while recalibrating those choices as observed behavior drifts. Joint planning is hard for two reasons. First, the space is vast. Each job’s deployment sets about 140 configuration knobs (Appendix A), including accelerator, parallelism, engine, quantization, batching, and KV-cache settings, which yields at least 2140 configurations per job. A cluster plan chooses one deployment for every job under shared capacity, so the number of candidate plans grows exponentially with the number of jobs and cannot be enumerated. Second, the planner must predict the performance of never-run deployments for every job and candidate. Analytical models and profile-based simulators [2, 29] rest on fixed assumptions about memory, communication, and engine behavior that break as workloads, hardware, and software change. In our measurements, even a roofline model calibrated with measured peak compute and memory bandwidth errs by up to 4.7×. Language models encode broad systems knowledge but not calibrated, deployment-specific quantities. Two frontier models, Claude Opus 4.8 and GPT-5.6 Sol, overpredict throughput by 4× and 9×. A planner therefore cannot trust predictions unless it corrects them from observed behavior. Search-based methods instead deploy candidate configurations, observe their end-to-end performance, and use it to choose the next candidates. Bayesian optimization fits a statistical model to these observations [5, 8], and LLM-guided evolutionary search asks a language model to propose new candidates from high-scoring ones [6, 17, 27, 28]. These methods have two fundamental limitations. First, because they learn only from end-to-end outcomes, they cannot tell why a candidate performed as it did, for example whether low throughput came from pipeline bubbles or KV-cache pressure, and therefore which fix applies or which other jobs the evidence should inform. Second, every trial is a live deployment that occupies capacity other jobs need and can breach a running job’s SLO. Evolutionary search needs many such trials, and Bayesian optimization restarts for each new workload. LLM-guided evolutionary search pairs a language model with measurement only loosely, because the model sees only a scalar score and measurements never confirm or refute its reasoning. A tighter pairing is difficult because the model’s knowledge is qualitative while telemetry is numeric. Koi pairs them tightly through a fixed causal graph on which both can operate. The graph links deployment decisions (e.g., accelerator, parallelism), runtime mediators (e.g., KV-cache pressure, communication overhead), and serving outcomes (e.g., latency, throughput). Scoped mechanisms compose its edges into testable hypotheses, which a language model can propose and telemetry can support or contradict. For example, one mechanism states that raising pipeline parallelism lowers throughput through pipeline bubbles for a given model and hardware. Each edge and mechanism carries a confidence and an uncertainty updated from runtime evidence, so confident relationships guide exploitation and

Total Time (s)

2m42s

381

268

$2.09 3m49s

$2.16 7m54s 130 32

$0.97

200

✗ OOM 144

400

574

600

143 $0.49 1m47s

7m6s

800

Total Cost ($)

95

Output tokens/sec

36

Throughput (tok/s)

Input tokens/sec

0

Figure 2. Parallelism substantially changes inference performance and cost. Done on AWS A10G instances with Llama 3 70B model. The workload has 2,048 input and 512 output tokens per request.

traffic stresses memory bandwidth and KV-cache capacity. Neither the workload nor the serving stack is stationary. A job’s realized input/output distribution drifts, and engine and library updates change throughput for the same deployment, so a configuration profiled once does not stay optimal and the planner must keep recalibrating from what it observes. These choices are also coupled across jobs that compete for the same finite accelerators, quota, and budget. Most schedulers decide one job at a time, giving each arriving job its best-ranked feasible placement, so scarce capacity goes to whichever job arrives first. This is suboptimal whenever an earlier job has a workable second choice that a later job lacks. For example, an online chat job and a batch evaluation job both prefer the cluster’s only free H100 node, but only the batch job can meet its target on A100s. If the batch job arrives first, a greedy scheduler gives it the H100, and the online job must miss its SLO or wait. Even a correct per-job ranking cannot prevent this, because avoiding it requires weighing the opportunity cost of giving a scarce device to one job rather than another, which only a cluster-wide decision can do. Existing systems address parts of this problem. Serving engines optimize request execution within a given deployment [3, 14, 31]. Multi-cloud orchestrators place workloads across clouds and markets but leave the accelerator type and configuration to the user [18, 30]. Configuration optimizers and heterogeneous serving planners choose parallelism, placement, or GPU types for a single model or a fixed set of models [11, 15, 19, 29]. Request schedulers decide how requests are queued and served on existing deployments [10, 22], and cluster schedulers allocate resources among competing jobs [7, 12, 25], but neither chooses the deployment configuration of an inference job. No existing system jointly chooses deployment configurations and allocates shared capacity for all online and batch inference jobs 2

uncertain ones identify deployments worth testing. Koi runs no separate trials, since every deployment both serves its job and tests the relationships it exercises, and the plan score values both. Bayesian optimization also updates a model after each trial, but that model maps configurations to end-to-end outcomes for a single workload. Koi instead updates its beliefs about mechanisms, so evidence from one job informs every job within the mechanism’s scope. Koi turns this evidence into a cluster plan by alternating between language-model proposals and deterministic evaluation. For each job, the language model proposes candidate deployments from the causal graph, the cluster snapshot, the job’s workload, and the matching mechanisms. The graph lets it focus on the decisions that affect the job’s targets, so Koi evaluates a few candidates per job instead of enumerating the space. Deterministic components predict each candidate’s behavior with a profile-based performance model whose error is tracked against observed outcomes, score it against the job’s weighted objectives, and compose candidates into a cluster-wide plan. The plan score balances serving value, the value of testing uncertain relationships, SLO risk under prediction error, and reconfiguration cost, subject to shared capacity. Because candidates for all jobs compete within one plan, this score exposes the opportunity costs that per-job optimization misses. Only deployment evidence updates causal confidence, and the language model only proposes and cannot override measured evidence or cluster constraints. We evaluate Koi on a real 104-GPU cluster spanning three regions and on simulated 400- and 800-GPU clusters, all heterogeneous, against nine baselines including the approaches above (three on the real cluster). Against the strongest baseline, Koi delivers 1.18× the goodput at 20% lower $/token on the real cluster and 45% more goodput at 31% lower $/token in the 3-hour simulation. Ablations show that these gains come mainly from the structure around the language model rather than from the model itself. This paper makes the following three contributions. • We design and implement Koi, a provider-side control plane that self-calibrates the cluster-wide deployment plan for mixed online and batch jobs across heterogeneous clouds, and release it as open source* . • We develop an evidence-guided agentic planning algorithm around a causal DAG that links deployment decisions, runtime mediators, and outcomes. Our RLM [33] hierarchy proposes candidates and scoped hypotheses, deterministic evaluation selects the cluster plan, and telemetry updates the evidence that guides subsequent intervals. • We evaluate Koi on heterogeneous benchmarks spanning ten model architectures and nine baselines, where it improves goodput by up to 45% and lowers $/token

by up to 31% while admitting 96% of online and 100% of batch jobs over 12 hours.

2

Background

2.1

Where deployment planning sits

LLM serving spans three control layers. Serving engines admit and batch requests [3, 14, 31], routers distribute requests among running replicas [10, 22], and a deployment controller chooses the hardware, parallelism, engine configuration, replica count, and placement within which those layers operate. Koi composes with serving engines and routers while operating at the third layer, where it plans and controls deployment. The separation is operational as well as temporal. Engine and router decisions are made per request and are reversible within milliseconds, whereas a deployment decision commits model weights to devices. Changing one requires loading weights, warming caches, and draining in-flight work, which bounds how many configurations a controller can try in a given period and requires damping against churn. 2.2

What determines deployment performance

The deployment objective is not to maximize raw throughput or GPU utilization. Rather, the controller should select the least-cost configuration that satisfies the workload’s service objectives: online workloads must meet TTFT, TPOT, and throughput targets, while batch workloads must complete by their deadlines. Operating closer to saturation can reduce cost, but it also reduces queueing headroom and can sharply degrade tail latency under bursty demand. The lowest-cost feasible configuration is workload-dependent because different workload characteristics expose different hardware bottlenecks. Input-heavy requests make prefill compute the limiting resource and first manifest as elevated TTFT. Long-context and output-heavy requests increase KVcache residency and HBM-bandwidth pressure, which can degrade TPOT and reduce the capacity available to concurrent requests. Arrival rate and burstiness determine the replication and queueing headroom required to preserve these objectives. Consequently, a model identifier and mean request rate alone are insufficient to determine the appropriate GPU type, model-parallel configuration, or replica count. Configuration choices affect end-to-end objectives indirectly through runtime mechanisms. For example, pipeline parallelism can reduce per-GPU memory pressure but introduce pipeline bubbles and stage imbalance; similarly degraded throughput can instead arise from KV-cache pressure or interconnect contention. These mechanisms require different corrective actions. Thus, a controller must use runtime evidence to identify the binding bottleneck and select the

* https://github.com/Tandemn-Labs/tandemn-koi

3

least-cost feasible intervention, rather than relying on repeated configuration trials guided only by endpoint metrics. This motivates Koi’s evidence-conditioned control loop.

3

Related Work

3.1

Deployment configuration and placement

programs and uses them to guide subsequent mutations [28]. EvoX further co-evolves candidate solutions and the search strategy that selects and varies them, allowing the exploration policy itself to adapt during optimization [17]. These systems evolve executable artifacts by repeatedly proposing and scoring candidate programs.

Per-job optimizers evaluate alternative serving configurations against workload and SLO objectives. AIConfigurator predicts configurations across parallelism strategies, serving modes, and engine-specific settings, then ranks them by derived latency and throughput [29]. Other systems extend deployment decisions to concurrent jobs. AlpaServe places and parallelizes multiple models to improve serving efficiency under bursty workloads [15]. Prism improves multi-model serving through demand-aware cross-model GPU-memory sharing, model placement, and request arbitration [32]. Heterogeneous serving planners optimize a single model across mixed hardware: Mélange computes a minimum-cost heterogeneous GPU allocation for a given workload and latency target [11], while Helix jointly optimizes model placement and request scheduling over heterogeneous GPUs and network links [19]. Cloud systems operate at the infrastructure layer: SkyPilot places workloads across clouds [30], while SkyServe distributes spot and on-demand replicas across regions and clouds [18]. 3.2

4

Feedback-driven and causal optimization

Measurement-driven optimizers refine decisions from observed outcomes. CherryPick uses Bayesian optimization to test cloud configurations and select subsequent trials from observed execution [5]. SCOOT applies Bayesian optimization to inference-engine parameters, using learned constraints to avoid invalid configurations while optimizing SLOs [8]. Cluster schedulers also adapt from runtime feedback: Pollux models how resource allocation and batch size affect training goodput [25]; Cilantro learns online resource-toperformance mappings under application-level objectives [7]; and Tiresias schedules distributed training jobs from observations and prior knowledge of job duration [12]. Causal methods instead use structure to determine which interventions or relationships should generalize. Invariant causal prediction identifies predictors whose relationship with a target remains stable across environments [24], while causal Bayesian optimization selects interventions on a supplied causal graph to optimize a target with fewer experiments [1]. 3.3

Problem Formulation

At each 𝑛-minute tick, Koi selects a deployment configuration for every job in the cluster. At tick 𝑡, the cluster contains active jobs A𝑡 and pending jobs Q𝑡 . Each job 𝑖 has a workload profile 𝑊𝑖 and target 𝑆𝑖 : a latency bound for online jobs or a completion deadline for batch jobs. Cluster capacity is represented by a resource map 𝑅𝑡 over environments 𝑒. A deployment is organized into the following hierarchy: Chain. A chain 𝑐 = (𝑒, 𝑥) is one model copy deployed in environment 𝑒 under one deployment configuration. Rank. A rank (𝑐, 𝑛) consists of 𝑛 homogeneous replicas of the same chain, where 𝑛 is its data-parallel size. Ladder. A ladder 𝐿𝑖 = {(𝑐𝑖1, 𝑛𝑖1 ), . . . , (𝑐𝑖𝑘 , 𝑛𝑘𝑖 )} is the set of ranks assigned to job 𝑖. Plan. A cluster plan 𝑃𝑡 = {𝐿𝑖 }𝑖 ∈ A𝑡 ∪Q𝑡 assigns one ladder to every active and pending job in the cluster. For an active job, 𝐿𝑖 = 𝐿𝑖(𝑡 −1) retains its current deployment; for a pending job, 𝐿𝑖 = ∅ defers admission. Each ladder 𝐿𝑖 produces five outcomes ($/tok, p99 TTFT, p99 TPOT, throughput, and SLO margin) collectively denoted 𝑦𝑖 (𝐿𝑖 ) ∈ R5 . The first three are minimized and the latter two maximized. Each job 𝑖 specifies Í objective weights 𝑤𝑖 = (𝑤𝑖1, . . . , 𝑤𝑖5 ), with 𝑤𝑖 𝑗 ≥ 0 and 𝑗 𝑤𝑖 𝑗 = 1, over 𝑦𝑖 (𝐿𝑖 ). These weights encode the user’s relative priorities for that job’s serving objectives. Koi reduces these five objectives to a single scalar using an augmented Tchebycheff scalarization [20]. Let 𝑧𝑖★ be a reference point for job 𝑖 that weakly dominates every outcome the job can achieve, maintained per job and updated from realized outcomes (§5.4), and let 𝜎𝑖 𝑗 be a normalizing constant per objective. The
normalized gap along objective 𝑗 is 𝑔𝑖 𝑗 (𝐿𝑖 ) = 𝑦𝑖 𝑗 (𝐿𝑖 ) − 𝑧𝑖★𝑗 /𝜎𝑖 𝑗 if 𝑗 is minimized and 𝑧𝑖★𝑗 −
𝑦𝑖 𝑗 (𝐿𝑖 ) /𝜎𝑖 𝑗 if 𝑗 is maximized, and the ladder’s exploitation score is h i ∑︁5 𝐽𝑖 (𝐿𝑖 ) = − max 𝑗 ∈ [1,5] 𝑤𝑖 𝑗 𝑔𝑖 𝑗 (𝐿𝑖 ) + 𝜌 𝑤𝑖 𝑗 𝑔𝑖 𝑗 (𝐿𝑖 ) . 𝑗=1 (1) where 𝜌 > 0 is an augmentation constant. The max term ensures that no single objective is neglected on non-convex Pareto frontiers; the weighted sum breaks ties among candidates with the same maximum gap. 𝐽𝑖 (𝐿𝑖 ) ≤ 0 always, and 𝐽𝑖 (𝐿𝑖 ) = 0 only when every objective the job weights attains its reference. Let Switch(𝐿𝑖(𝑡 −1) , 𝐿𝑖 ) denote the reconfiguration cost defined in §5.3.4, with Switch(𝐿, 𝐿) = 0 for an unchanged ladder. Koi selects a cluster-wide plan by maximizing aggregate

LLM-guided search

FunSearch uses an LLM to modify a target function and an evaluator to score each candidate [27]. AlphaEvolve extends this evolutionary search to larger programs and codebases, retaining evaluated programs to guide subsequent proposals [6]. OpenEvolve maintains a diverse set of high-scoring 4

ladder value minus this cost: max {𝐿𝑖 }

∑︁ 𝑖 ∈ A𝑡 ∪Q𝑡

𝐽𝑖 (𝐿𝑖 ) − 𝜆𝑡

(hardware, engine, parallelism, quantization, batching, routing), 𝑉 contains telemetry-derived runtime mediators (KVcache utilization, pipeline bubbles, communication latency, memory pressure, SM utilization), and 𝑌 contains the serving outcomes defined in §4. Only 𝑋 → 𝑉 and 𝑉 → 𝑌 edges are permitted, ensuring every modeled effect passes through telemetry; the registered edge set 𝐸 ⊆ (𝑋 × 𝑉 ) ∪ (𝑉 × 𝑌 ) is fixed at initialization. Each edge 𝑒 ∈ 𝐸 maintains a belief 𝑝𝑒 ∼ Beta(𝛼𝑒 , 𝛽𝑒 ), where 𝛼𝑒 and 𝛽𝑒 accumulate supporting and contradictory evidence, and 𝑐𝑒 = 𝛼𝑒 /(𝛼𝑒 + 𝛽𝑒 ) is its confidence. Because the same confidence can reflect different amounts of evidence (supporting and contradictory deployment behavior), Koi also tracks normalized posterior uncertainty

  Switch 𝐿𝑖(𝑡 −1) , 𝐿𝑖 𝑖 ∈ A𝑡 (2)

∑︁

subject to C1 Resource feasibility. The chains requested across all ladders must fit within the available capacity under 𝑅𝑡 . C2 Physical feasibility. Each chain must fit available VRAM; TP must divide the attention and KV head counts; PP must partition the model by layers. C3 SLO feasibility. Each ladder must satisfy 𝑆𝑖 under prediction uncertainty, with 𝑃 (breach) ≤ 𝜏. C4 Swap budget. At most 𝐵𝑡 active jobs may change ladders during one tick: |{𝑖 ∈ A𝑡 : 𝐿𝑖 ≠ 𝐿𝑖(𝑡 −1) }| ≤ 𝐵𝑡 . C5 Job policy. Each job must remain within its capacity quota, and higher-priority jobs must take precedence over lower-priority jobs competing for the same resources. A pending job is deferred when no ladder satisfies C1–C3, or when job priority, quota, or resource contention favors reserving capacity for another job. Equation (2) defines the decision Koi would make if 𝑦𝑖 (𝐿𝑖 ) were known for every candidate ladder. In practice, 𝐿𝑖 ↦→ 𝑦𝑖 (𝐿𝑖 ) is unknown and changes over time. Because evaluating a ladder requires deployment, consuming cluster capacity and risking SLO violations, Koi instead estimates 𝑦𝑖 from available evidence, deploys, observes the outcome, and updates its estimate for the subsequent tick.

5

Koi

5.1

Overview

𝑈 (𝛼𝑒 , 𝛽𝑒 ) =

𝑈 (1, 1) = 1.

Exploration (§5.3.2) prioritizes edges with high 𝑈 , where additional deployment evidence is most informative. Edges compose into scoped hypotheses called mechanisms. A mechanism 𝑚 = (𝐸𝑚 , 𝑆𝑚 , 𝜂𝑚 , 𝐵𝑚 ) pairs a connected subgraph 𝐸𝑚 ⊆ 𝐸 with an applicability scope 𝑆𝑚 over models, hardware, workloads, and conditions, a natural-language statement 𝜂𝑚 , and Beta state 𝐵𝑚 . The LLM proposes mechanisms, which are admitted only if they form a valid, nonduplicate 𝑋 → 𝑉 → 𝑌 path over registered edges and are initialized with 𝐵𝑚 = Beta(1, 1). Both 𝑉b and 𝑌b, the predicted mediator and outcome values, come from Koi’s planner-internal deterministic surrogate. The surrogate builds on AIConfigurator [29]. Given a job’s workload profile and one candidate ladder, it composes operator-level latency profiles with a replay simulator to predict TTFT, TPOT, and throughput point estimates. The remaining mediators and outcomes are derived from those predictions, the job’s targets, and the allocation price carried by 𝑅𝑡 . When a model has no measured profile, it substitutes the nearest profiled architecture on that hardware and rescales by its measured prefill and decode speed ratios. This surrogate is used during planning to score candidates and is distinct from LLMServingSim 2.0, which is used only as the execution environment for the simulated evaluation 6. The surrogate is an approximation, so Koi carries prediction uncertainty into planning and rejects ladders with 𝑃 (breach) > 𝜏 under C3. A deployed ladder tests every mechanism whose scope it satisfies by comparing the predicted mediator and outcome trajectories, 𝑉b and 𝑌b, against the observed 𝑉obs and 𝑌obs . For either trajectory 𝑂 ∈ {𝑉 , 𝑌 }, Koi b𝑘 and defines the residual at observation 𝑘 as 𝑟𝑘 = 𝑂 obs,𝑘 − 𝑂 accumulates it in a two-sided cumulative sum (CUSUM) [21] detector:

Each tick, Koi executes an eight-stage closed loop that updates evidence from the cluster and searches for a better cluster-wide plan (Figure 3). States S0–S3 refresh the planner state by snapshotting jobs, placements, and capacity (S0), collecting per-rank telemetry (S1), comparing observations against stored predictions (S2), and updating the evidence state (S3). Planning begins at S4, where the agent proposes a cluster-wide action for each job (place, reconfigure, retain, or defer). Deterministic evaluation at S5 scores the proposal against job-specific performance, expected evidence, SLO risk, and reconfiguration cost, and validates constraints C1–C5. Invalid or non-improving plans return to S4 with scores and diagnostics. Once the search budget is exhausted, Koi commits the best feasible cluster-wide plan (S6) and closes the loop (S7). This design relies on two assumptions: near-optimal proposal coverage (Assumption 5.3) and convergence under stationary conditions (Assumption 5.4). 5.2

12𝛼𝑒 𝛽𝑒 , (𝛼𝑒 + 𝛽𝑒 ) 2 (𝛼𝑒 + 𝛽𝑒 + 1)

Mechanisms and Predictions

To make deployment behavior testable and transferable, Koi represents inference as a directed acyclic graph (DAG), 𝑋 → 𝑉 → 𝑌 , where 𝑋 contains planner-controlled decisions

𝑆𝑘+ = max(0, 𝑆𝑘+−1 + 𝑟𝑘 − 𝛿) and 𝑆𝑘− = min(0, 𝑆𝑘−−1 + 𝑟𝑘 + 𝛿) 5

Figure 3. End-to-end Koi architecture. Each planning tick snapshots cluster state, observes runtime telemetry, updates causal confidence, proposes deployment candidates, and deterministically evaluates and selects a cluster-wide plan. with slack 𝛿 = 0.5b 𝜎 , declaring divergence when 𝑆𝑘+ > ℎ − or 𝑆𝑘 < −ℎ for threshold ℎ = 4b 𝜎 , where b 𝜎 is the empirical residual scale. Accumulating the residual rather than thresholding each sample separates persistent prediction error from isolated deviations. CUSUM assigns each mechanism evaluation to one of four quadrants: Q1 (𝑉obs ≈ 𝑉b, 𝑌obs ≈ 𝑌b), Q2 (𝑉obs ≈ 𝑉b, 𝑌obs 0 𝑌b), Q3 (𝑉obs 0 𝑉b, 𝑌obs ≈ 𝑌b), and Q4 (𝑉obs 0 𝑉b, 𝑌obs 0 𝑌b). Q1 adds supporting evidence, Q4 contradictory evidence, and Q2–Q3 update edges according to their invariant causal prediction (ICP) result [24]. §5.4 uses the Q1 rate to adapt exploration.

5.3.1 Exploitation. Exploitation is the per-job score 𝐽𝑖 (𝐿𝑖 ) of Equation (1), evaluated on the ladder’s aggregated outcomes. High-confidence causal relationships guide exploitative candidates, while 𝐽𝑖 (𝐿𝑖 ) favors those whose predicted outcomes best match the job’s weighted priorities. 5.3.2 Exploration. Exploitation alone favors deployments the surrogate already predicts to be good, so relationships that have never been exercised stay untested. Exploration rewards a plan for testing uncertain edges and mechanisms: ∑︁ ∑︁ Expl(𝑃) = 𝐴𝑒 (𝑃) 𝑈 (𝛼𝑒 , 𝛽𝑒 ) + 𝜔𝑚 𝐴𝑚 (𝑃) 𝑈 (𝛼𝑚 , 𝛽𝑚 ), 𝑒 ∈𝐸

5.3

𝑚∈ M

(3) where 𝐴𝑒 (𝑃) indicates whether any ladder in 𝑃 exercises edge 𝑒, and analogously for mechanisms. 𝑈 is the normalized posterior variance from before. Mechanisms, weighted by 𝜔𝑚 , are scored separately because their scope can fail even when each constituent edge is well supported.

Scoring a Plan

Because Koi scores predicted rather than known outcomes, it augments Equation (2) with two terms: risk from prediction error and exploration value from testing uncertain causal relationships. Exploitation, risk, and switching cost sum over jobs; exploration is scored once per plan to avoid doublecounting repeated tests. The plan score relies on a corresponding coverage assumption: Assumption 1. At each tick, the planner generates 𝐾𝑃 candidate plans. Let 𝑃𝑡★ be the optimal feasible plan and Φ(·) the plan score. We assume that, with probability at least 𝜂, at least one candidate 𝑃𝑘 has score

5.3.3 Risk. A surrogate is most likely to be wrong when conditions have shifted, which is also when a plan is most likely to breach an SLO. Koi therefore evaluates each prediction against a band rather than a point. Let 𝑟 𝑗 be the historical residuals for objective 𝑗, observed minus predicted as in §5.2, and b 𝜎 𝑗 their empirical standard deviation. The band is upper 𝑗 = 𝑦b𝑗 + 𝑄 0.95 (𝑟 𝑗 ) + 2𝜖b 𝜎𝑗, lower 𝑗 = 𝑦b𝑗 + 𝑄 0.05 (𝑟 𝑗 ) − 2𝜖b 𝜎𝑗 .

Φ(𝑃𝑘 ) ≥ Φ(𝑃𝑡★) − 𝜀𝑃 ,

(4)

where 𝑄𝑞 is the empirical 𝑞-quantile. The quantile terms carry the observed error distribution and the 2𝜖b 𝜎 𝑗 terms inflate it, with 𝜖 in the role of a Wasserstein ambiguity radius around that distribution [13] rather than as the solution

where 𝜀𝑃 is the tolerated gap from the optimum. 6

𝑊 ticks that fall in Q1, and with 𝑞★ = 1 let the regret signal be 𝑡 1 ∑︁ max(0, 𝑞★ − 𝑞𝑘 ), 𝑠𝑡 = 𝑊

to a distributionally robust program. Koi calibrates 𝜖 from realized coverage in the slow loop (§5.4), widening it when outcomes fall outside the band more often than the target rate and tightening it as calibration improves. From the band, Koi estimates the probability 𝑝 brk 𝑗 that each SLO-constrained objective is violated, and combines them as Ö
𝑝𝑖brk (𝐿𝑖 ) = 1 − 1 − 𝑝 brk , (5) 𝑗

𝑘=𝑡 −𝑊 +1

which increases as predictions become less reliable. Regret is derived from the Q1 rate: a high regret signal indicates poor configurations and motivates continued exploration. The exploration weight is given by
𝛽𝑡 +1 = clip 𝛽𝑡 exp(𝜂 𝛽 (𝑠𝑡 − 𝑠 ★)), 𝛽 min, 𝛽 max ,

𝑗 ∈ S𝑖

where S𝑖 is the set of objectives constrained by 𝑆𝑖 . Treating those objectives as independent is conservative, since latency violations are positively correlated in practice, so the product understates the joint satisfaction probability and overestimates risk.

with target regret 𝑠 ★ and adaptation rate 𝜂 𝛽 . The same signal raises the swap budget 𝐵𝑡 when predictions are unreliable, so more active jobs may be reconsidered. The switching penalty 𝜆𝑡 instead follows the observed reconfiguration rate, rising as churn grows to damp thrashing. The radius 𝜖𝑡 follows realized coverage, widening when outcomes fall outside the band more often than the target rate and tightening otherwise. The ideal reference tracks the best outcomes a job has achieved,   𝑦𝑟,𝑗 + Δ 𝑗 , 𝑗 maximized,  𝑟max  ∈ H𝑖,𝑡 ★ 𝑧𝑖,𝑗,𝑡 =  𝑦𝑟,𝑗 − Δ 𝑗 , 𝑗 minimized,  𝑟min  ∈ H𝑖,𝑡 where H𝑖,𝑡 is job 𝑖’s deployment history and Δ 𝑗 a margin that preserves dominance. The self-calibration loop relies on a corresponding convergence assumption: Assumption 2 (Convergence). If jobs, workloads, available capacity, and serving conditions remain stable for sufficiently long, we assume that Koi converges to a stable operating point. At this point, predictions remain accurate, exploration no longer finds materially better plans, reconfiguration diminishes, and the learned causal and robustness state changes only marginally.

5.3.4 Switching Cost. The reconfiguration cost for job 𝑖 is decomposed into four components: Switch𝑖 = 𝐶 cold + 𝐶 parallel + 𝐶 kill + 𝐶 risk . All terms are zero for an unchanged ladder: 𝐶 cold captures loading and warm-up, 𝐶 parallel concurrent execution, 𝐶 kill draining and termination, and 𝐶 risk transient underperformance. The penalty 𝜆𝑡 adapts to the observed reconfiguration rate (§5.4) to discourage unnecessary migrations. Separately, the swap budget 𝐵𝑡 imposes the hard C4 cap on how many active jobs may change ladders in one tick. 5.3.5 Cluster Score. The four terms in Sections 5.3.1–5.3.4 combine into a single plan score: i ∑︁ h Φ(𝑃) = 𝐽𝑖 (𝐿𝑖 ) − 𝛾 𝑝𝑖brk (𝐿𝑖 ) − 𝜆𝑡 Switch𝑖 (𝐿𝑖(𝑡 −1) , 𝐿𝑖 ) (𝑖,𝐿𝑖 ) ∈𝑃

+ 𝛽𝑡 Expl(𝑃). (6) Koi aims to select the highest-scoring feasible plan, 𝑃𝑡 ∈ arg max Φ(𝑃), 𝑃 ∈ F𝑡

5.5 (7)

The search space for a cluster deployment is vast and coupled. Each chain sets about 140 knobs (Table 7), giving at least 2140 configurations per chain before replica counts and multirank ladders are considered. A plan chooses one ladder for every job, so the number of candidate plans grows exponentially with the number of jobs, and the choices are coupled because jobs with different workloads and SLOs compete for the same capacity. Koi therefore never enumerates this space. Budget books and backward DAG queries, described next, narrow it for each job, an LLM proposes a small set of ladders within it, and deterministic code evaluates, scores, and selects among them. A root planner receives the frozen cluster snapshot and assigns each job a budget book, a soft upper bound on which GPU types and instance pools the job may consider. Budgets can overlap in resources selected, and contention is resolved later by a capacity-constrained solver that assigns at most one ladder per job to the available GPUs. The root planner

where F𝑡 is the set of plans satisfying C1–C5. Exhaustive maximization over F𝑡 is infeasible and the per-tick search is budgeted, so Koi commits the highest-scoring feasible plan it finds rather than the maximizer (§5.5). The exploration weight 𝛽𝑡 and the switching penalty 𝜆𝑡 are adapted by the slow loop (§5.4); the risk weight 𝛾 is fixed. 5.4

Agentic Planner

Self-Calibration

Koi adapts the quantities that govern planning from how well its predictions have held. Edge and mechanism confidence update every tick. On a slower schedule, Koi adjusts the exploration weight 𝛽𝑡 , the swap budget 𝐵𝑡 , the switching penalty 𝜆𝑡 , the robustness radius 𝜖𝑡 , the ideal point 𝑧𝑡★, and the CUSUM thresholds, each from accumulated residuals. Most of these follow one signal. Let 𝑞𝑡 be the fraction of decided evidence–mechanism pairs over a rolling window of 7

6.1

invokes per-job specialist sub-agents to propose candidate ladders within each job’s budget. Each specialist sub-agent receives the job’s model architecture, workload profile, current deployment, budgeted hardware, and all mechanisms whose scope matches the job. It can query the DAG backward: given a target SLO, the sub-agent traces which 𝑋 decisions affect it through which 𝑉 mediators, focusing on the decisions that matter (Section 5.2). This division lets the root reason about cluster-wide tradeoffs (which scarce accelerators to reserve, which workloads can share resources) while sub-agents focus on the configuration details of individual models. These proposals are advisory: Koi first validates each candidate for physical feasibility, then invokes the surrogate to predict and score its performance. The resulting ladders compete for finite capacity under the joint selector, which assigns at most one ladder per job to maximize cluster-wide gain (Section 5.3) while accounting for unserved demand, improvement over active deployments, and switching cost. The root planner may inspect the result and propose additional candidates, but cannot override deterministic scores, capacity checks, or constraints C1–C5.

5.6

Benchmarks. We evaluate Koi in one real-hardware setting and three simulated settings: a 3-hour deployment on a 104-GPU heterogeneous GCP cluster, 3-hour and 12-hour heterogeneous simulations, and a 12-hour homogeneous-cluster simulation. Each setting defines a benchmark consisting of a shared resource pool and a concurrent mix of online and batch workloads. For each benchmark, Koi and each of the ten baselines independently generate the placements and configurations used to serve the same workload. Each benchmark mixes 25 online and batch workloads across ten model architectures: Llama-3.2-1B, Llama-3.3-70B, Qwen2.5-Coder-32B, Phi-4, Mixtral-8x7B, Kimi-K2-Instruct, GLM-4.5-Air, Qwen3-Next-80B, DeepSeek-V3 and Qwen3235B-A22B-Instruct, spanning 1B to 235B parameters and including dense, mixture-of-experts, and frontier models. Online workloads distribution range from short moderation (256/8 tokens, TTFT < 100 ms) to long-context summaries (24,000/1,200 tokens, TTFT < 3s) and decode-heavy agentic tasks (1,400/4,096 tokens). Six batch jobs carry fixed deadlines ranging from 20 to 135 minutes. Request traffic follows Poisson, constant, and gamma arrival processes parameterized from production profiles to each job type. The workloads used in simulation and real cluster are summarized in Figure 7 and Table 6 in the Appendix. Real-Hardware Execution. Each experiment is conducted with three Google Cloud Kubernetes clusters across three distinct regions: us-central1, us-east5 and europe-west2. There are 104 GPUs in total, consisting of a heterogeneous mix of H100-80GB, A100-80GB, A100-40GB, and RTXPRO6000. Due to cost constraints, non-evolutionary algorithm based baselines were omitted from real experiments. Simulation Setup. For the simulation environment, we used LLMServingSim 2.0 [9], a discrete event simulator (DES) as the execution environment for the simulated clusters. Unlike Koi’s internal surrogate model, LLMServingSim 2.0 does not provide point estimates per placement, but rather simulates request execution and queueing semantics for all requests routed to a deployed configuration. After a plan is selected by Koi, LLMServingSim 2.0 executes the corresponding configuration and provides the realized performance observations that would otherwise come from hardware telemetry. It simulates using per-operator compute profiles across GPU, model, and tensor-parallel configurations, and an ASTRA-SIM [26] backend for collective communication. To support all model–GPU combinations in our benchmark, we augmented LLMServingSim 2.0 with layer-wise profiles for the additional LLM models and GPU types used in the evaluation. Baselines. Koi is compared against ten schedulers, grouped by the capability each baseline is intended to test. Non-learning baselines. FCFS+First-Fit admits jobs in arrival order and assigns the least-VRAM feasible placement.

Implementation

Batch and online inference jobs are executed through two different pathways. For batch jobs, large prompt files are sharded and a custom shard manager coordinates shard assignment and completion for each worker. This ensures that every rank and chain planned by Koi make progress on the same job without conflicts. Online jobs are deployed through the Nvidia Dynamo inference deployment framework. Koi decides on the exact chain configuration and desired number of maximum replicas, while Dynamo autoscales within that maximum replica bound in response to request load and desired SLAs. Per-rank telemetry from all running jobs is continuously streamed from the different worker clusters to a unified cluster state store. At each five-minute tick, Koi reads a snapshot of this store to use for the next plan.

6

Experimental Setup

Evaluation

Our evaluation asks three questions. First, how does Koi compare with static, classical, outcome-learning, evolutionary, and language-model planners in goodput, SLO attainment, admission, and cost (§6.2)? Second, do joint optimization, evidence accumulation, and health-triggered recovery operate as intended, and how do these effects change with deployment scale and duration (§6.3)? Third, what do deterministic planning tools, the causal evidence graph, and cross-tick learning contribute, and how sensitive are the results to the language-model backbone (§6.4)? 8

Table 1. Simulated benchmark results for 3-hour (400-GPU) and 12-hour (800-GPU) heterogeneous workloads. Bold marks the best value and underline the second-best; ties are marked for all matching methods

Method

Batch Hrs prog. (%)

Admit online (%)

Admit batch (%)

$/token (10 −4 )

TTFT met (%)

TPOT met (%)

Goodput (tok/s)

Overall score

3 12 3 12 3 12

57.71 38.69 57.71 24.54 57.71 38.62

77.78 78.26 66.67 69.57 77.78 78.26

57.14 50.00 57.14 57.14 57.14 50.00

2.95 6.09 2.91 29.65 2.94 6.09

3.90 2.45 5.24 7.63 4.05 2.36

6.89 6.27 13.61 13.55 7.27 6.21

1830.84 1772.01 1857.31 364.03 1833.16 1771.49

49.13 31.46 50.24 17.59 49.27 31.40

3 12 3 12 3 12 3 12 3 12 3 12 3 12

41.99 14.99 44.42 30.52 42.55 15.35 57.71 50.71 56.70 25.01 52.04 17.54 84.55 49.61

72.22 78.26 66.67 65.22 77.78 73.91 66.67 65.22 66.67 78.26 77.78 69.57 94.44 95.65

57.14 64.29 57.14 50.00 57.14 64.29 57.14 64.29 57.14 64.29 71.43 71.43 71.43 100.00

6.69 40.18 5.43 14.16 7.06 57.66 2.69 3.37 4.74 26.24 6.31 16.86 1.85 3.39

3.00 2.89 8.09 9.56 2.80 1.94 9.90 11.96 5.22 9.19 3.98 3.02 28.79 27.95

15.88 16.83 27.08 23.06 5.16 4.69 22.10 24.02 23.41 16.14 18.98 17.29 41.80 35.80

807.03 268.57 993.08 762.20 764.66 187.18 2004.06 3012.00 1139.00 411.25 855.60 640.25 2910.64 3184.15

31.90 15.83 37.49 24.89 29.06 11.14 54.34 50.01 41.74 19.84 35.28 21.34 70.83 63.13

3 12

+26.84pp −1.10pp

+16.66pp +17.39pp

+0.00pp +28.57pp

+31.23% −0.59%

+18.89pp +15.99pp

+14.72pp +11.78pp

+45.24% +5.72%

+30.35% +26.23%

Non-learning baselines FCFS + first-fit Best fixed configuration EDF + first-fit

Placement and search baselines Rolling CP-SAT Contextual bandit PRISM Rolling EvoX OpenEvolve GPT-Only (non-agentic) Koi

Δ vs. best baseline

Table 2. Real-hardware results for the 3-hour, 104-GPU heterogeneous benchmark. Bold marks the best value and underline the second-best.

Method

Batch prog. (%)

Admit online (%)

Admit batch (%)

$/token (10 −4 )

TTFT met (%)

TPOT met (%)

Goodput (tok/s)

Overall score

Contextual bandit Rolling EvoX OpenEvolve Koi

50.00 50.00 75.00 100.00

53.33 80.00 100.00 100.00

100.00 100.00 100.00 100.00

0.45 0.44 0.30 0.24

22.23 28.08 65.66 79.05

33.66 35.48 91.41 99.11

2291.03 2348.15 3497.43 4119.51

51.05 54.90 84.44 94.50

Δ vs. best baseline

+25.00pp

+0.00pp

+0.00pp

+20.00%

+13.39pp

+7.70pp

+17.79%

+11.91%

EDF [16] applies the same placement rule but prioritizes earlier deadlines. Best Fixed uses one static hardware and parallelism configuration per job, selected offline by GPT-5.6Sol with access to the complete workload trace; at runtime, it reuses these configurations and falls back to VRAM-based placement when the selected configuration is unavailable. Per-job optimizers. Prism [32] estimates memory and KVcache pressure to place preconfigured model-parallel groups, reallocating memory at runtime but never choosing GPU type, parallelism, or replica count. To adapt Prism to cluster scheduling, we enumerate policy-compliant GPU-pool and TP/PP placement candidates using workload-derived VRAM and KV-cache demand. Prism then admits waiting jobs by assigning high-KV-demand workloads to the least-pressured feasible pool under global capacity constraints. Global optimizers without mechanism learning. Rolling CPSAT [23] solves ILP at each tick over the same candidate catalog and constraints as Koi. It enumerates policy-valid

placements using checked-in maximum VRAM requirements and resource-map capacity, then optimizes lexicographically for priority-weighted admission, lower estimated reservation cost, and lower GPU footprint. But it never learns from outcomes. Contextual Thompson sampling [4] maintains a Bayesian linear reward model over public job and candidateplacement features, updating only from terminal deployment outcomes. Its bounded reward combines online TTFT, TPOT, and throughput attainment with weights 0.4, 0.4, and 0.2, respectively; for batch jobs, it combines throughput attainment and deadline completion with weights 0.4 and 0.6. LLM-guided planners. GPT-only (non-agentic) receives a frozen cluster snapshot, containing the same public job, resource, policy, and model-catalog information available to Koi, and generates a placement plan without tools or crosstick state. Its prompt requires a complete policy-compliant action for each job, specifying feasible GPU pool and TP/PP/DP 9

6.2

configuration while respecting current capacity. Agentic additionally uses a REPL and the same read-only inspection and placement-validation tools available to Koi’s agent. Evolutionary search. Rolling EvoX [17] evolves a deterministic placement function at each tick, adapting its search strategy when progress stalls. OpenEvolve [28] retains a MAP-Elites database of diverse, high-scoring placement programs across ticks. In each baseline, we used the similar objective function as Koi to make it comparable. EvoX’s each reasoning round takes ≈ 30 seconds. Considering reasonable scheduling overhead, we set it to max 8 rounds or 5 minutes whatever returns first. Again, we provided the same information with both EvoX and OpenEvolve baselines that was given to Koi. For the main performance evaluation, we used the same LLM model (GPT-5.6-Luna) for the policies that uses LLM. Additionally, we did LLM-backbone sensitivity study with different LLM models for Koi. To standardize the comparison, each baseline is given the same cluster snapshots, candidate catalog, and safety checks. Differences are limited to the plan-search policy and, for Koi alone, cross-tick mechanism learning. Metrics. A single evaluator computes every number from per-run traces. All denominators belong to the workload (offered requests, submitted jobs, full campaign sizes, a common clock 𝑇 ), so no planner can improve a metric by refusing, dropping, or narrowing work. Stochastic planners report the mean and range over at least three runs. We collapse results into an overall score with three equally weighted pillars: Quality (TTFT SLO met, TPOT SLO met, deadline met), Coverage (online admission, batch admission, batch progress), and Efficiency (goodput and cost per SLOtoken, each normalized to the best in each comparison). The score is their geometric mean; no pillar compensates for another. The problem formulation uses the same instantaneous outcome vector for online and batch jobs (Section 4). The evaluation, however, judges batch jobs by completion progress and deadline attainment, which capture finite-work completion rather than instantaneous serving quality. SLO met is the fraction of offered online requests meeting both TTFT and TPOT p99 targets, averaged per job so one high-volume tenant cannot dominate. Deadline met is the share of batch jobs finishing before their deadline. Admission is the share of submitted jobs whose ranks materialized without a fatal interruption. Batch progress is request-weighted completion over all submitted jobs. Goodput is SLO-honoring output tokens per second: online tokens passing both latency gates plus the projected tokens of deadline-meeting jobs. Cost per SLO-token divides the fleet capacity cost by total SLOtoken output; idle GPUs count as waste, so the cheapest planner produces the most SLO output from the same sunk fleet. Planner LLM cost is reported separately.

End-to-End Performance

End-to-End Performance overview. Table 1 reports the end-to-end results. Over three hours, Koi achieves 2,911 goodput tok/s, 45.24% more than the strongest baseline, while reducing cost per token by 31.23%; it also improves batch progress by 26.84% and TTFT/TPOT SLO attainment by 18.89/14.72%. Over twelve hours, Koi admits 95.65% of online jobs and all batch jobs, improves goodput by 5.72%, and leads TTFT/TPOT attainment by 15.99/11.78%, while batch progress is 1.10 points below EvoX and cost is 0.59% higher. Thus, Koi’s gain does not come from maximizing one workload class: it sustains more admitted work and substantially better online latency while remaining competitive on batch progress and cost. 6.3

Design Evaluation

Joint optimization. In the real-cluster run, a Qwen3235B job waits at tick 18 while ten services are running. The only reclaim candidate would reduce a Phi-4 service from two TP2 RTXPRO6000 replicas to one. This would halve its modeled capacity from 1,572 to 786 tok/s, below its 834 tok/s requirement, while its observed p99 TTFT already exceeds the 500 ms target at 645 ms. A full preemption would be worse, reducing that service’s availability. The joint selector rejects the reclaim as not Pareto-better than keeping Phi-4, defers Qwen for two ticks, and admits it at tick 20 after another batch finishes naturally. The ten-minute delay protects an existing service from a predicted SLO regression without sacrificing Qwen’s eventual admission. Deployment evidence makes planning more calibrated. In the 12-hour simulation on heterogeneous cluster, the mean causal uncertainty 𝑈¯ falls from a first-quarter median of 0.0215 to 0.00053 in the final quarter, a 41× reduction. The fraction of predictions receiving a learned residual correction rises from 2.3% to 10.8%, while their median absolute log error falls from 0.965 to 0.823. During warm-up, the regret signal raises the exploration weight from 0.49 to 2.0 by tick 23. As evidence accumulates, the median DRO radius tightens from 0.379 to 0.260, then widens again after the late workload shift since the uncertainty increases again with it. Thus, Koi becomes more confident where observations repeat while retaining the ability to increase exploration and robustness when conditions change. Swap. In the real cluster experiment, a Mixtral service reaches critical health at tick 15 after only one of its two A10080GB replicas materializes and observed p99 TTFT reaches 2,299 ms against a 700 ms target. The per-job sub-agent proposes one physically launchable TP8 A100 rank, and the deterministic scorer assigns a 25.44 gain over keeping the failed deployment, including a 25.00 critical-rehabilitation credit. The joint selector commits only this swap and keeps the other active jobs unchanged. The request-level TTFT 10

pass rate rises from 2.4% before recovery to 99.9% on the relaunched rank, and the job is healthy by tick 18. This example connects telemetry, candidate generation, deterministic scoring, and cluster-wide commitment in one decision. Learning and performance improvement. We distinguish fleet scale from elapsed learning time using two comparisons. Over roughly three hours, the 104-GPU real cluster commits 20 deployments and selects eight mechanisms, whereas the 800-GPU simulation commits 80 deployments and selects 17 mechanisms. At a larger cluster scale (800 GPUs for 12 hours vs. 104 GPUs for 3 hours), Koi gathered more evidence, increasing from 407 to 2,272 pieces of evidence, and ran more CUSUM tests, increasing from 18,705 to 101,420. The share of decisive mechanism tests that support their predicted mediator–outcome path rises from 1.9% to 19.1%, and ICP acceptance for the same workload-tomediator edge rises from 11.0% to 63.3%. These runs show that more concurrent deployments activate learning sooner and that a longer horizon strengthens its evidence.

Figure 4. Real-cluster allocation, SLO attainment, and adaptation over three hours. (a) Plan-declared logical GPU allocation for each policy, with Koi’s realized swap marked; (b) equal-job mean TTFT (solid) and TPOT (dashed) attainment among served online jobs; (c) Koi’s calibrated DRO radius; and (d) active/applicable and cumulatively discovered mechanisms.

Evidence changes both allocation and planner state. Figure 4 shows Koi increasing its plan-declared allocation above 90% as jobs arrive while sustaining high served-job TTFT and TPOT attainment. As observed outcomes expose prediction-band misses, Koi widens its DRO radius from 0.15 to 0.191 rather than continuing to trust an under-covering bound. In parallel, the cumulative mechanism set grows to 64 as new workload contexts arrive. Balance between exploitation and exploration. Figure 5 shows long stable placement intervals under Koi, with changes concentrated on a subset of degraded services. EvoX and the contextual bandit instead produce dense swap sequences across nearly every online workload, while OpenEvolve and GPT-only perform no explicit swaps and largely preserve their initial choices; OpenEvolve’s additional transitions are failure-driven relaunches. Across the complete trace, Koi issues 201 swaps and 2,076 keeps, compared with 1,464 swaps for EvoX and 1,928 for the bandit. The timeline therefore places Koi between two failure modes: repeatedly perturbing productive deployments and never correcting a poor initial placement. 6.4

Ablation Study

Koi sub-component ablation. Figure 6 and Table 4 compare progressively more complete planners. Giving the language model deterministic prediction, scoring, and validation tools increases goodput from 856 to 1,827 tok/s, raises TTFT attainment from 3.98% to 25.74%, and reduces cost per token from 6.31 × 10−4 to 2.95 × 10−4 . The complete system reaches 2,911 tok/s; removing the causal graph reduces goodput to 1,770 tok/s, while freezing online updates reduces it to 1,621 tok/s. Full Koi also completes 84.55% of batch work, versus

Figure 5. Online workload scheduling across methods. Per-job GPU assignments for nine workloads over 12 hours under Koi, EvoX, OpenEvolve, the contextual bandit, and GPT-only. Colors denote GPU families; markers indicate placement, swap, and rank relaunch events. 59.42% without the graph and 63.01% without online learning, despite admitting fewer batch jobs than either variant. The result separates two roles: deterministic tools ground each planning attempt, while persistent mechanism evidence improves which valid plans are retained and revisited across ticks. LLM backbone ablation. The backbone ablation localizes the Cohere–GPT-5.6 Sol difference to a single 7,200-request Llama-70B batch campaign. Both planners admit it at 00:15 and never reconfigure it, but Cohere selects one eight-GPU MI300 TP8 chain and finishes at 01:07 with 82 minutes of deadline slack, whereas Sol selects four TP4 A100 replicas

11

(b) SLO met ratio

20

1000

OL

)

Ko i

8

Ko iw /o

DA

Ko iw /o

en

tic

c) T-o nly GP

(ag

nti

T-o nly GP

(no n-a ge

T-o nly

G

0 Ko i

(no n-a ge GP nti T-o c) nly (ag en tic ) Ko iw /o DA G Ko iw /o OL

0 Ko i

behavior the stack already reports, and evaluations too expensive to run exhaustively. Extending the planner to admit them would let one controller allocate a fleet across all of these workloads rather than inference alone.

2000

GP

(no n-a ge GP nti c) T-o nly (ag en tic ) Ko iw /o DA G Ko iw /o OL

0

(c) Goodput

T-o nly

20

3000

TPOT

Figure 6. Ablations over the 3-hour heterogeneous simulation. Overall score, TTFT and TPOT SLO attainment, and goodput across five planner variants.

using 16 GPUs and reaches 7,115 requests by the 02:30 deadline, missing by 85. Because batch goodput credits the campaign only when it meets its deadline, this single decision contributes 8.64 million tokens and explains 99.3% of their trace-derived goodput gap; their online contributions differ by only 57,921 tokens. The insight is that deterministic validation bounds the action space but cannot make all valid candidates equally useful: a small number of initial proposals can dominate the final result. This single-run comparison does not isolate accelerator type, parallelism, replication, or batching limits, which change together. Another insight is that the achieved outcomes (TTFT, TPOT etc.) of all the jobs, in all ablations are well within three standard deviations of each other - indicating a model agnostic behavior.

7

Conclusion

Heterogeneous LLM inference requires more than a better predictor. A useful controller must allocate scarce devices globally, act safely under prediction error, and decide which expensive deployments should improve its future knowledge. Koi addresses these requirements by attaching falsifiable, mediator-grounded mechanisms to deployment plans; using a sandboxed LLM only to propose candidate plans; and keeping scoring, validation, and side effects deterministic. Telemetry closes a calibrated slow loop, while a stateless execution plane keeps fast serving and recovery independent of the planner. The result is a cluster control architecture that can improve with use without making availability depend on an agent’s correctness.

GP

40

40

TTFT

Goodput

60

SLO met (%)

Overall score

(a) Overall score

9

Acknowledgements

Generative AI tools, specifically ChatGPT, Claude Code and DeepSeek, were used to assist with generating sections of this work, including text, tables, graphs and code.

References [1] Virginia Aglietti, Xiaoyu Lu, Andrei Paleyes, and Javier González. 2020. Causal Bayesian Optimization. In Proceedings of the TwentyThird International Conference on Artificial Intelligence and Statistics (Proceedings of Machine Learning Research, Vol. 108). PMLR, Palermo, Sicily, Italy, 3155–3164. https://proceedings.mlr.press/v108/aglietti20a. html [2] Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav S. Gulavani, Ramachandran Ramjee, and Alexey Tumanov. 2024. Vidur: A Large-Scale Simulation Framework for LLM Inference. In Proceedings of Machine Learning and Systems, Vol. 6. 351–366. https://proceedings.mlsys.org/paper_files/paper/2024/hash/ b74a8de47d2b3c928360e0a011f48351-Abstract-Conference.html [3] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, 117–134. https://www.usenix.org/conference/osdi24/presentation/agrawal [4] Shipra Agrawal and Navin Goyal. 2013. Thompson Sampling for Contextual Bandits with Linear Payoffs. In Proceedings of the 30th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 28). PMLR, 127–135. [5] Omid Alipourfard, Hongqiang Harry Liu, Jianshu Chen, Shivaram Venkataraman, Minlan Yu, and Ming Zhang. 2017. CherryPick: Adaptively Unearthing the Best Cloud Configurations for Big Data Analytics. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17). USENIX Association, 469– 482. https://www.usenix.org/conference/nsdi17/technical-sessions/ presentation/alipourfard [6] Matej Balog, Alexander Novikov, Ngan Vu, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies,

Discussion, Limitations and future work

Koi benefits from a larger fleet as it runs more concurrent jobs, exercising more relationships per tick and covering more scopes. Evidence accumulates faster and applies to a wider set of future jobs. Additional headroom also makes exploration affordable without putting an SLO at risk, and planner cost is charged per tick and amortized over the jobs in it. Utilization rises 24.5% between the first and last quarter of the 12-hour run, but the run is too short to reach the convergence conditions. New decision axes enlarge 𝑋 without changing Koi’s per-tick cost, which is bounded by the number of candidates evaluated rather than by the size of the space. Expert parallelism, prefill–decode disaggregation, and engine-level tuning are therefore straightforward to admit. Each requires only that telemetry report at least one mediator through which its effect passes, since an axis with no measured mediator can be scored but not tested. Enumerative and black-box search have no such property and require more samples as the space grows. New accelerators, including AMD and TPU can then be additional environments in 𝑅𝑡 and need no change to the planner. Training, post-training, and fine-tuning jobs have the same three properties Koi relies on: a configuration space that can be enumerated, runtime 12

Sebastian Nowozin, and Pushmeet Kohli. 2025. AlphaEvolve: A Coding Agent for Scientific and Algorithmic Discovery. Google DeepMind technical report. https://storage.googleapis.com/deepmindmedia/DeepMind.com/Blog/alphaevolve-a-gemini-poweredcoding-agent-for-designing-advanced-algorithms/AlphaEvolve.pdf [7] Romil Bhardwaj, Kirthevasan Kandasamy, Asim Biswal, Wenshuo Guo, Benjamin Hindman, Joseph E. Gonzalez, Michael I. Jordan, and Ion Stoica. 2023. Cilantro: Performance-Aware Resource Allocation for General Objectives via Online Feedback. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23). USENIX Association, 623–643. https://www.usenix.org/conference/osdi23/ presentation/bhardwaj [8] Ke Cheng, Zhi Wang, Wen Hu, Tiannuo Yang, Jianguo Li, and Sheng Zhang. 2024. SCOOT: SLO-Oriented Performance Tuning for LLM Inference Engines. arXiv preprint arXiv:2408.04323 (2024). [9] Jaehong Cho, Hyunmin Choi, Guseul Heo, and Jongse Park. 2026. LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructure. In 2026 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). 1–14. doi:10.1109/ISPASS69572.2026.00012 [10] Kanishk Goel, Jayashree Mohan, Nipun Kwatra, Ravi Shreyas Anupindi, and Ramachandran Ramjee. 2026. QoServe: Breaking the Silos of LLM Inference Serving. In Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. 1492–1507. doi:10.1145/3779212. 3790206 [11] Tyler Griggs, Xiaoxuan Liu, Jiaxiang Yu, Doyoung Kim, Wei-Lin Chiang, Alvin Cheung, and Ion Stoica. 2024. Mélange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity. arXiv:2404.14527 [cs.DC] https://arxiv.org/abs/2404.14527 [12] Juncheng Gu, Mosharaf Chowdhury, Kang G. Shin, Yibo Zhu, Myeongjae Jeon, Junjie Qian, Hongqiang Liu, and Chuanxiong Guo. 2019. Tiresias: A GPU Cluster Manager for Distributed Deep Learning. In 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19). USENIX Association, 485–500. https://www.usenix.org/ conference/nsdi19/presentation/gu [13] Daniel Kuhn, Soroosh Shafiee, and Wolfram Wiesemann. 2025. Distributionally Robust Optimization. arXiv:2411.02549 [math.OC] doi:10.48550/arXiv.2411.02549 [14] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles. 611–626. doi:10.1145/3600006.3613165 [15] Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23). USENIX Association, 663–679. https://www.usenix.org/conference/osdi23/ presentation/li-zhouhan [16] C. L. Liu and James W. Layland. 1973. Scheduling Algorithms for Multiprogramming in a Hard-Real-Time Environment. J. ACM 20, 1 (1973), 46–61. doi:10.1145/321738.321743 [17] Shu Liu, Shubham Agarwal, Monishwaran Maheswaran, Mert Cemri, Zhifei Li, Qiuyang Mang, Ashwin Naren, Ethan Boneh, Audrey Cheng, Melissa Z. Pan, Alexander Du, Kurt Keutzer, Alexandros G. Dimakis, Koushik Sen, Matei Zaharia, and Ion Stoica. 2026. EvoX: Meta-Evolution for Automated Discovery. arXiv:2602.23413 [cs.LG] doi:10.48550/arXiv.2602.23413 [18] Ziming Mao, Tian Xia, Zhanghao Wu, Wei-Lin Chiang, Tyler Griggs, Romil Bhardwaj, Zongheng Yang, Scott Shenker, and Ion Stoica. 2025. SkyServe: Serving AI Models across Regions and Clouds with Spot Instances. In Proceedings of the Twentieth European Conference on

Computer Systems. 159–175. doi:10.1145/3689031.3717459 [19] Yixuan Mei, Yonghao Zhuang, Xupeng Miao, Juncheng Yang, Zhihao Jia, and Rashmi Vinayak. 2025. Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1. 586–602. doi:10.1145/3669940.3707215 [20] Kaisa Miettinen. 1999. Nonlinear Multiobjective Optimization. International Series in Operations Research & Management Science, Vol. 12. Kluwer Academic Publishers. doi:10.1007/978-1-4615-5563-6 [21] E. S. Page. 1954. Continuous Inspection Schemes. Biometrika 41, 1–2 (1954), 100–115. doi:10.1093/biomet/41.1-2.100 [22] Archit Patke, Dhemath Reddy, Saurabh Jha, Haoran Qiu, Christian Pinto, Chandra Narayanaswami, Zbigniew Kalbarczyk, and Ravishankar Iyer. 2024. Queue Management for SLO-Oriented Large Language Model Serving. In Proceedings of the 2024 ACM Symposium on Cloud Computing. 18–35. doi:10.1145/3698038.3698523 [23] Laurent Perron, Frédéric Didier, and Steven Gay. 2023. The CP-SAT-LP Solver. In 29th International Conference on Principles and Practice of Constraint Programming (CP 2023) (Leibniz International Proceedings in Informatics (LIPIcs), Vol. 280). Schloss Dagstuhl – Leibniz-Zentrum für Informatik, 3:1–3:2. doi:10.4230/LIPIcs.CP.2023.3 [24] Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. 2016. Causal Inference by using Invariant Prediction: Identification and Confidence Intervals. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 78, 5 (2016), 947–1012. doi:10.1111/rssb.12167 [25] Aurick Qiao, Sang Keun Choe, Suhas Jayaram Subramanya, Willie Neiswanger, Qirong Ho, Hao Zhang, Gregory R. Ganger, and Eric P. Xing. 2021. Pollux: Co-adaptive Cluster Scheduling for GoodputOptimized Deep Learning. In 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21). USENIX Association, 1–18. https://www.usenix.org/conference/osdi21/presentation/qiao [26] Saeed Rashidi, Srinivas Sridharan, Sudarshan Srinivasan, and Tushar Krishna. 2020. ASTRA-SIM: Enabling SW/HW Co-Design Exploration for Distributed DL Training Platforms. In 2020 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). 81–92. doi:10.1109/ISPASS48437.2020.00018 [27] Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M. Pawan Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, and Alhussein Fawzi. 2024. Mathematical Discoveries from Program Search with Large Language Models. Nature 625 (2024), 468–475. doi:10.1038/s41586-023-06924-6 [28] Asankhaya Sharma. 2025. OpenEvolve: An Open-Source Evolutionary Coding Agent. https://github.com/algorithmicsuperintelligence/ openevolve. Open-source evolutionary coding agent. [29] Tianhao Xu, Yiming Liu, Xianglong Lu, Yijia Zhao, Xuting Zhou, Aichen Feng, Yiyi Chen, Yi Shen, Qin Zhou, Xumeng Chen, Ilya Sherstyuk, Haorui Li, Rishi Thakkar, Ben Hamm, Yuanzhe Li, Xue Huang, Wenpeng Wu, Anish Shanbhag, Harry Kim, Chuan Chen, and Junjie Lai. 2026. AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving. arXiv preprint arXiv:2601.06288 (2026). doi:10.48550/arXiv.2601.06288 [30] Zongheng Yang, Zhanghao Wu, Michael Luo, Wei-Lin Chiang, Romil Bhardwaj, Woosuk Kwon, Siyuan Zhuang, Frank Sifei Luan, Gautam Mittal, Scott Shenker, and Ion Stoica. 2023. SkyPilot: An Intercloud Broker for Sky Computing. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). USENIX Association, 437–455. https://www.usenix.org/conference/nsdi23/presentation/ yang-zongheng [31] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX 13

Association, 521–538. https://www.usenix.org/conference/osdi22/ presentation/yu [32] Shan Yu, Yifan Qiao, Mingyuan Ma, Yangmin Li, Shuo Yang, Xinyuan Tong, Yang Wang, Zhiqiang Xie, Yuwei An, Shiyi Cao, Ke Bao, Deepak Vij, Xiaoning Ding, Yichen Wang, Qingda Lu, Zhong Wang, Gao Gao, Harry Xu, Junyi Shu, Jiarong Xing, and Ying Sheng. 2025. Prism: CostEfficient Multi-LLM Serving via GPU Memory Ballooning. Accepted to OSDI 2026. arXiv:2505.04021 [cs.DC] https://arxiv.org/abs/2505.04021 [33] Alex L. Zhang, Tim Kraska, and Omar Khattab. 2026. Recursive Language Models. arXiv:2512.24601 [cs.AI] https://arxiv.org/abs/2512. 24601

and TP/PP/DP configuration. For each waiting job, it resolves that mapping against the current resource map and greedily places the job only when the required target still has capacity. If a mapping is absent or no longer fits, the implementation uses a VRAM-feasible fallback; otherwise it defers the job. The fallback lets the control produce a valid Store plan when a named benchmark configuration cannot be bound to the current fleet. It must be reported as a fallback, not as a development-selected fixed configuration.

A

Rolling CP-SAT.. CP-SAT receives the static candidates and capacity constraints above. It creates one Boolean variable for each job–candidate pair, constrains each job to at most one selected candidate, and constrains every environment and pool footprint to its available capacity. The solver first maximizes priority-weighted job admission. With that value fixed, it minimizes hourly price, GPU footprint, and a stable candidate-order tie break. It uses one worker, seed zero, and a declared time limit. We use a rolling solve because capacity and the waiting queue change each tick; the re-solve is necessary to make the static optimizer operate on the actual current resource map rather than an offline snapshot.

Deployment Configuration Space

Table 7 groups the configuration knobs that a chain sets into families, with representative knobs for each.

B

Baseline Implementation Details

This section describes the non-Koi baselines that were implemented. It excludes Koi and its ablations. B.1

Common Store-to-plan contract

Each implemented driver independently reads one frozen Store snapshot at a planning tick: the resource map and the running, waiting, paused, and finished job groups for the evaluated user. It subtracts the footprint of running jobs from the resource map before making new assignments. It then emits exactly one lifecycle action for every visible job: keep for running jobs, place or defer for waiting jobs, defer for paused jobs, and preempt for finished jobs. A placement is serialized as a normal Store plan and is checked against the concrete environment or pool capacity before it is recorded. CP-SAT, contextual Thompson sampling, Prism, and EvoX each locally construct their candidate set from the same kinds of public facts, rather than reading Koi state. For every waiting job, they read the model, token and SLO fields, priority, and an explicit or benchmark VRAM requirement. They enumerate available resource targets and policy-permitted TP values, choose the smallest PP sufficient for the VRAM requirement with DP fixed to one, and record each candidate’s resource keys, GPU footprint, price, model-weight estimate, and KV-demand estimate. The drivers reject candidates that violate the configuration policy or nested environment/pool capacity. This gives the four methods the same job and resource-map interface while leaving their selection rules independent of Koi’s predictions, mechanisms, and objective. The contextual Thompson-sampling and EvoX drivers are currently registered in the simulator CLI. The best-fit, CPSAT, AIConfigurator, and Prism drivers are implemented and unit-tested Store adapters, but still need factory registration before they can be selected through the CLI. B.2

Contextual Thompson sampling. The contextual-bandit baseline uses a Bayesian linear reward model over 11 public action features: an intercept; job priority and queue depth; average input and output lengths; SLO tightness; candidate GPU count, memory, KV-cache pressure, price, and requested capacity share. Jobs are processed in priority order. For each job, the planner samples a weight vector from its current linear-Gaussian posterior, scores the feasible candidates with that sample, and chooses the highest-scoring candidate before reserving its capacity. After the simulator returns an outcome, the driver applies the corresponding rank-one posterior update. Online reward is the fraction of offered output tokens that meet both latency targets, minus a normalized price penalty; batch reward is zero when the deadline is missed. We use Thompson sampling rather than LinUCB, epsilongreedy, a neural bandit, or a tree/forest bandit because the action set is discrete, the context is a small explicit feature vector, and the controller needs to act from few observations. Sampling from the posterior performs probability matching: a candidate is explored when uncertainty gives it a plausible chance of being best, instead of using a fixed random-action rate or a separately tuned UCB bonus. The implementation is deliberately a linear Thompson sampler, not Bayesian optimization and not a claim that it dominates the other bandit families on these workloads. Its default posterior-draw scale is 0.25 and its random seed is zero for reproducibility.

Implemented baseline policies

Best fixed configuration. The static control uses a checkedin mapping from benchmark job identity to hardware target 14

Prism port. The Prism adapter translates a memory-pressure placement rule to the aggregate resource map. It orders jobs

Table 3. Mixed online/batch homogeneous results. All runs use the 12-hour workload horizon on the simulator.

Method

Batch progress (%)

Admit (%) online / batch

$/token (10 −4 )

SLO met (%) TTFT/TPOT

Goodput (tok/s)

Overall score

Best Fixed Configuration Contextual Bandit Rolling EvoX Openevolve GPT-Only (non agentic) Koi

37.58 1.89 14.92 14.92 40.06 36.62

60.87/42.86 43.48/7.14 65.22/14.29 65.22/14.29 65.22/57.14 78.26/57.14

3.56 14.04 4.76 6.76 4.29 3.51

24.38/31.09 10.32/10.95 29.61/37.10 20.26/32.46 18.15/33.62 31.37/39.29

2147.77 544.34 1606.39 1130.77 1783.61 2175.63

50.69 14.59 40.37 33.18 49.17 56.68

Δ vs. best baseline

−3.44pp

+8.11pp

+1.40%

+1.76pp / +2.19pp

+1.30%

+11.82%

Table 4. Agentic and Koi ablations. All runs use the 3-hour workload horizon on the simulator.

Variant

Batch progress (%)

Admit (%) online / batch

$/token (10 −4 )

SLO met (%) TTFT/TPOT

Goodput (tok/s)

Overall score

GPT-only (non agentic) Opencode SDK (GPT-5.6-Luna) Koi without DAG/mechanisms Koi without online learning Full Koi

52.04 54.77 59.42 63.01 84.55

77.78/71.43 88.89/57.14 94.44/100.00 94.44/85.71 94.44/71.43

6.31 2.95 3.05 3.33 1.85

3.98/18.98 25.74/32.62 21.66/38.33 16.49/37.74 28.79/41.80

855.60 1826.64 1770.03 1621.48 2910.64

35.28 51.55 58.34 54.79 70.83

Δ vs. best baseline

+21.54pp

−8.00pp

+37.29%

+3.05pp

+59.34%

+21.41%

Table 5. Koi backbone LLM ablation. All runs use the 3-hour workload horizon on the simulator.

Table 7. Configuration knobs per chain, grouped by family. Family

Base model

Batch progress (%)

Admit (%) Planner online / batch $/tick

Overall score

GPT-5.6 Sol DeepSeek-V4 Pro Cohere Command-A

67.31 48.76 67.37

94.44/85.71 94.44/71.43 94.44/85.71

0.75 0.30 0.44

62.12 50.44 68.84

Koi (default, GPT-5.6 84.55 Luna)

94.44/71.43

0.057

70.83

by decreasing maximum estimated KV demand and then priority. For each job, it filters out candidates that do not leave room for the model weights or exceed remaining nested capacity, then chooses the target with the least accumulated KV demand divided by residual memory. Price, GPU footprint, and candidate ID are deterministic tie breaks. After selection, the target’s residual memory and KV-demand state are updated before placing the next job. This preserves the memory-centric placement idea while using Store-expressible, exclusive shapes and the common safety checks. It is a documented heuristic port, not a claim that the full Prism artifact, including its complete sharing and scheduling runtime, is executed unchanged. Rolling EvoX.. At each tick, EvoX receives the static candidate payload and nested capacities. SkyDiscover evolves a deterministic Python function, compute_placement(payload), that returns one candidate ID or a defer decision for every waiting job. The evaluator rejects malformed or infeasible programs and scores valid outputs by priority-weighted admission, followed by hourly price and GPU-footprint tie breaks. The initial program is a simple priority-first greedy 15

Knobs

Environment

[?]

Parallelism Engine and runtime

[?] [?]

Batching and scheduling

[?]

Memory and KV cache

[?]

Quantization

[?]

Speculative decoding

[?]

Disaggregation

[?]

Total

Representative knobs market, cloud, region, availability zone, GPU type, instance type, nodes per chain, interconnect TP, PP, DP, EP, SP, CP engine and version, runtime image, attention backend, CUDA graphs, torch.compile max sequences, max batched tokens, max model length, chunked prefill and chunk size, scheduling, preemption, and router policies GPU memory utilization, block size, KV-cache dtype, prefix caching, KV offloading, KV transfer method weight and activation dtypes, weight quantization method and bits, activation quantization method method, draft model, speculative tokens, acceptance threshold prefill/decode split, prefill and decode worker counts

∼140

policy; the best program from one tick seeds the next tick’s search. The winning program is rerun on the frozen payload,

Benchmark Overview

Benchmark Metric Distributions

Requests/s 94.7

Median Mean Min–max

TTFT 10 TPOT 1 ISL 0.1 OSL Tok/s 0h

0h

1h

1h Elapsed time

2h

2h

3h

0

101

Figure 7. 3-hour simulation benchmark schedule and traffic patterns.

102

103 Value (log scale)

104

Figure 8. 3-hour simulation benchmark workload distribution characteristics.

Workload

Model

Mode

ISL

OSL

SLO / deadline

Free multiturn chat Premium chat Code autocomplete Safety moderation Small-model chat Phi-4 long-context chat Small-model chat Support assistant Nightly evaluation Phi-4 quantization probe Code morning burst Enterprise chat Product-launch chat burst Moderation launch burst Agentic decode-heavy reasoning GLM frontier reasoning Small-active MoE reasoning Phi-4 calibration probe P0 safety replay P0 safety reasoner Long-context meeting summary Sales-report generation Legal review Held-out frontier evaluation MoE exploration probe

Llama-3.2-1B Llama-3.3-70B Qwen2.5-Coder-32B Llama-3.2-1B Llama-3.2-1B Phi-4 Llama-3.2-1B Mixtral-8x7B Llama-3.3-70B Phi-4 Qwen2.5-Coder-32B Phi-4 Mixtral-8x7B Llama-3.2-1B Kimi-K2-Instruct GLM-4.5-Air Qwen3-Next-80B Phi-4 Llama-3.3-70B Phi-4 Qwen3-Next-80B Qwen3-Next-80B Llama-3.3-70B DeepSeek-V3 Qwen3-Next-80B

Online Online Online Online Online Online Online Online Batch Batch Online Online Online Online Online Online Online Batch Batch Online Online Batch Online Batch Batch

900 1,200 1,800 256 256 9,000 256 1,400 1,800 1,000 2,200 1,500 850 320 1,400 1,800 1,200 1,400 2,200 900 24,000 18,000 3,000 1,800 1,000

300 500 120 8 64 220 64 350 1,200 512 96 420 280 8 4,096 2,500 900 512 700 160 1,200 1,400 700 1,600 800

TTFT <700 ms; TPOT <55 ms TTFT <600 ms; TPOT <38 ms TTFT <180 ms; TPOT <45 ms TTFT <100 ms; TPOT <30 ms TTFT <300 ms; TPOT <40 ms TTFT <1.5 s; TPOT <55 ms TTFT <300 ms; TPOT <40 ms TTFT <700 ms; TPOT <50 ms Completion deadline: 135 min Completion deadline: 105 min TTFT <160 ms; TPOT <45 ms TTFT <500 ms; TPOT <42 ms TTFT <650 ms; TPOT <50 ms TTFT <90 ms; TPOT <30 ms TTFT <2.5 s; TPOT <45 ms TTFT <2.2 s; TPOT <50 ms TTFT <1 s; TPOT <45 ms Completion deadline: 85 min Completion deadline: 60 min TTFT <350 ms; TPOT <40 ms TTFT <3 s; TPOT <65 ms Completion deadline: 55 min TTFT <1 s; TPOT <45 ms Completion deadline: 35 min Completion deadline: 20 min

Table 6. Workload overview of 3-hour simulation benchmark.

its choices are revalidated, and valid choices become Store placement actions. EvoX is the search procedure in this baseline; there is no second optimizer. Candidate construction and feasibility checks stay in the simulator so evolved code cannot invent hardware or bypass capacity rules. The rolling driver adapts to the current queue and resource map, but its EvoX evaluator currently uses static placement facts rather than simulated serving outcomes as its fitness signal. The frozen-snapshot EvoX table uses the same candidate encoding and evaluator without the cross-tick lifecycle loop.

B.3

(London, UK). The 3 clusters combine to a fleet of 104 heterogeneous GPUs, consisting of a mix of Nvidia H100-80GB, RTX PRO 6000, A100-80GB and A100-40GB. The H100s and A100s are placed in 8-GPU instances, and the RTX PRO 6000 are placed in both 8-GPU instances and 4-GPU instances.

Real hardware configuration

Each of the real experiments are conducted on 3 Google Kubernetes Engine (GKE) clusters, in the following regions: us-central1 (Iowa, USA), us-east5 (Ohio, USA), europe-west2 16

Related documents

• Koi: Evidence-Guided, Self-Calibrating Planning for Multi-Cloud, Heterogeneous LLM Inference Fleets#1388921
• Agent-Editing World Model: Rethinking World Modeling for LLM Agents#1065019
• The Illusion of Local Privacy: Confidentiality Boundary Failures in Consumer LLM Serving Systems#1012707
• The Illusion of Local Privacy: Confidentiality Boundary Failures in Consumer LLM Serving Systems#965336
• OptiCom : A Unified Framework for State-Conditioned Composition in LLM-Driven Optimization#1132422