Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs Yixuan Mei Carnegie Mellon University
arXiv:2605.04357v1 [cs.DC] 5 May 2026
Shiqi Pan Carnegie Mellon University
Zikun Li Carnegie Mellon University Mengdi Wu Carnegie Mellon University
Zhihao Jia Carnegie Mellon University
Xupeng Miao Peking University
K. V. Rashmi Carnegie Mellon University
Abstract The usage of large language models (LLMs) has grown increasingly fragmented, with no single model dominating. Meanwhile, cloud providers offer a wide range of mid-tier and older-generation GPUs that enjoy better availability and deliver comparable performance per dollar to top-tier hardware. To efficiently harness these heterogeneous resources for serving multiple LLMs concurrently, we introduce Coral, an adaptive heterogeneity-aware multi-LLM serving system. The key idea behind Coral is to jointly optimize resource allocation and the serving strategy of each model replica across all models. To keep pace with shifting throughput demand and resource availability, Coral applies a lossless two-stage decomposition that preserves joint optimality while cutting online solve time from hours to tens of seconds. Our evaluation across 6 models and 20 GPU configurations shows that Coral reduces serving cost by up to 2.79× over the best baseline, and delivers up to 2.39× higher goodput under scarce resource availability.
1
Zixuan Chen Carnegie Mellon University
GPU
A G R
Perf. Per Cost Rel. Mem BW TF Cost (GB) (TB/s) LOPS Mem BW TF
H100 A100 L40S L4 A10G
✓ ✓ ✓ ✓ ✓
7.6 3.5 2.2 1.0 1.2
✓ ✓ ✗ ✓ ✗
✓ ✓ ✓ ✓ ✗
80 80 48 24 24
3.35 2.04 0.86 0.30 0.60
989 312 362 121 70
10.5 22.8 21.5 24.0 19.7
0.44 129.8 0.58 88.9 0.39 162.3 0.30 121.0 0.49 57.4
Table 1: GPU specs and availabilities on AWS (A), GCP (G), and RunPod (R). Relative cost is mean hourly price normalized to L4. “Perf. Per Cost” divides metrics by relative cost.
clouds are heterogeneous, offering GPUs that span multiple generations and hardware specifications. Top-end GPUs (e.g., B200, H100) deliver the highest per-GPU capability but are often supply-constrained. Older and mid-tier GPUs (e.g., A100, L4) are more widely available and substantially cheaper, and they offer comparable or even better cost efficiency per unit of compute or memory than the latest generation. Together, model fragmentation and cloud GPU heterogeneity motivate a new problem: cost-efficient, adaptive multiLLM serving on heterogeneous cloud resources. In this problem, a serving system is given a set of models with latency service-level objectives (SLOs), the current per-model throughput demand, and the real-time price and availability of different GPU configurations across cloud regions. From these inputs, it must jointly produce two outputs. The first is a resource allocation that decides which nodes to provision. The second is a model placement that decides how to partition each model across those heterogeneous nodes. The goal is to satisfy every model’s throughput demand and latency SLO at minimum total cost. Furthermore, because demand, availability, and prices shift over time, the system must re-solve the problem periodically to adapt the cluster to these changes. Prior work tackles this problem only in pieces. Existing heterogeneity-aware allocation systems target orthogonal goals, such as spot preemption resilience via cross-region replica spreading [28] and mixed interactive/batch workloads
Introduction
Large language models (LLMs) are being deployed across an ever-widening range of tasks, including interactive chatbots [7, 41], automated code generation [5, 39, 55], and agentic workflows [47, 50]. No single model dominates this landscape. Models from different families excel at different tasks [6, 22, 26], and within each family, models of different sizes target different cost–quality trade-offs [18, 30, 49]. Recent industry data further illustrates this fragmentation: no single model handles more than a quarter of queries [37]. As a result, providers must serve many models concurrently. For example, Perplexity serves 46 models behind its LLM-powered answer engine [37], and Microsoft Office 365’s AI features are backed by seven different LLMs from three families [17]. To avoid the prohibitive capital expenditure of dedicated AI hardware, organizations increasingly turn to public clouds to host these diverse models [3, 33]. As Table 1 shows, these 1
Criterion
Coral
SkyS. [28]
Sage. [17]
Cauchy [53]
Helix [29]
HexG. [21]
Resource Alloc. Model Placement
Joint Opt.
✓ ✗
✓ ✗
✓ ✗
✗ ✓
✗ ✓
Latency SLO Multi-LLM
✓ ✓
✗ ✗
✓ ✓
✓ ✓
✗ ✗
✓ ✗
and cached. This decouples placement from allocation without sacrificing joint optimality. We call this artifact a Serving Template. The space of node combinations is unbounded in principle, so Coral enumerates a principled subset. As we show in Sec. 6.7, this subset covers the cost-efficient regime with negligible loss. Accordingly, Coral adopts a two-stage design. Offline, a Serving Template generator enumerates node combinations for each (model, SLO) pair. For each combination, it uses an ILP to compute the throughput-optimal placement under latency SLO, yielding a library of reusable Serving Templates. Online, a lightweight ILP deploys templates across regions to meet throughput demand at minimum cost, under real-time availability and pricing. Lifting model placement search off the online critical path lets the online solve complete in tens of seconds. This keeps the cluster adaptive to shifts in demand and resources. We implement Coral as a runtime built on ZeroMQ [14] and NCCL [34], with vLLM [23] as the per-node execution engine. To enable large-scale experiments where running on real hardware is cost-prohibitive, we also build a high-fidelity event-based simulator. We evaluate Coral across 6 models, 5 GPU types, and 20 GPU configurations under varying resource availability and throughput demand. Coral reduces serving cost by up to 2.79× over the best baseline, and under scarce resource availability delivers up to 2.39× higher goodput. Contributions of this paper include: • A formulation of heterogeneity-aware multi-LLM serving as a joint optimization over resource allocation and model placement under per-model latency SLOs. • A lossless decomposition that moves model placement search offline via reusable Serving Templates, reducing the online problem to tens of seconds. • ILP-based solvers for both offline placement and online allocation that scale to clusters of thousands of nodes. • An end-to-end implementation of Coral and a simulator validated against real system for large-scale evaluation. • Evaluation across diverse models, workloads, and cluster scales, demonstrating substantial cost savings.
Table 2: Comparison with prior work on heterogeneous LLM serving. Existing systems optimize either resource allocation [17, 28, 53] or model placement [21, 29], but never jointly. Coral is the first to co-optimize both for multiple models under per-model latency SLOs on heterogeneous hardware. via forecast-driven replica scaling [17]. They share a common limitation: each model replica is treated as a black box on homogeneous hardware, which collapses model placement out of the optimization space. Cauchy [53] relaxes this for PrefillDecode disaggregated serving by letting prefill and decode use different GPU configurations, but each phase remains internally homogeneous. As both the examples in Sec. 2.2 and our evaluation results in Sec. 6 show, the result is not only substantial cost savings left on the table, but outright infeasibility under constrained resource availability. Helix [29] and HexGen [21] take the opposite stance, optimizing placement for a single model on a given heterogeneous node set. Wrapping them in an outer enumeration over allocations is intractable. The allocation space is exponential, and each inner placement solve is already expensive. Helix alone reports four hours for a single model on just 24 nodes. As Table 2 summarizes, no existing system addresses the problem end-to-end. Solving the heterogeneity-aware multi-LLM serving problem end-to-end poses two key challenges. First, resource allocation and model placement are tightly coupled, and each has an exponential solution space. The throughput and SLO attainable on a node set depend on how the model is placed across it. Conversely, the optimal model placement can only be determined once the node set is fixed. Neither dimension can be pruned without committing to the other. Brute-forcing the joint problem is equally infeasible, as the combined search space is exponential in both dimensions. Second, the solution must be produced in minutes, not hours. Both GPU availability [43, 46] and throughput demand [42] shift quickly. A solution that arrives too late is invalidated by the very changes it was meant to adapt to. To address these challenges, we present Coral1 , an adaptive, heterogeneity-aware multi-LLM serving system. Coral builds on a key observation: given a model and its latency SLO, the throughput-optimal model placement on any node combination depends only on that combination. It is independent of how the rest of the cluster is allocated. The optimal placement is therefore a reusable artifact that can be pre-computed offline
2 2.1
Background Distributed LLM Serving
Architecture- and Phase-Dependent GPU Affinity Modern LLMs are largely built on Transformer-based backbones, but they vary substantially in the structure of their attention and feed-forward layers, including dense full-attention models [11], hybrid-attention models that replace some fullattention computation with more efficient sparse patterns such as sliding-window attention to reduce long-context memory cost [2, 44], and mixture-of-experts (MoE) models [2, 10, 18, 49]. These architectural choices induce different inference-time execution characteristics by shifting the
1 Coral stands for Cost-efficient Orchestration of Resources for Adaptive
LLM-serving
2
balance among computation, memory footprint, and memoryaccess cost. For example, unlike a dense feed-forward network (FFN), an MoE layer activates only a sparse subset of experts for each token, enabling much larger model capacity without a proportional increase in per-token FLOPs [10, 18]. LLM serving also consists of two distinct phases: prefill, which processes the input prompt, and decode, which generates output tokens autoregressively [51]. Even for the same model, these phases stress hardware differently: prefill exposes substantial parallelism and can more effectively utilize compute throughput, whereas decode is much more sequential at the single-request level and is often bottlenecked by memory bandwidth, particularly KV-cache access [36, 54]. As a result, the most cost-effective GPU choice depends jointly on model architecture and serving phase, rather than following a single uniform rule across all LLM workloads [29, 36, 54].
L40S× 1 [6, 14)
L40S× 1 [14, 22)
H100× 2 [22, 46)
H100× 2 [46, 70)
H100× 2 [70, 94)
(a) A mixed L40S/H100 pipeline serves Qwen-3 235B prefill more cost-efficiently than any pureH100 setup (SLO = 1800 ms).
(b) Normalized throughput CDF for GPT-OSS 120B decode plans. Heterogeneous combinations fill gaps left by homogeneous ones.
Figure 1: Opportunities brought by heterogeneity. Throughput & Avail. GPU A
GPU A [0, 2) GPU B [0, 1)
Parallelism Strategies for Heterogeneous LLM Serving Distributed LLM serving commonly relies on four forms of parallelism: data parallelism (DP) [8], pipeline parallelism (PP) [16], tensor parallelism (TP) [40], and expert parallelism (EP) [24]. DP replicates the model, or a model partition, across devices and splits requests among them, whereas PP partitions the model into sequential layer blocks placed on different devices and forwards activations between stages [8, 16]. TP instead shards individual operators across devices, while EP, used in MoE models and often composed with other strategies, places experts on different devices and routes tokens to the selected experts [24, 40]. DP and PP are more natural building blocks for heterogeneous serving because they preserve coarse-grained work partitions: DP balances load across replicas, and PP can use uneven stage sizing to match different device capabilities, though it must avoid bottlenecks at the slowest stage [29]. By contrast, TP and EP require fine-grained, tightly synchronized communication, including all-reduce or all-gather in TP and all-to-all token exchange in EP [24, 40]. As a result, their performance is highly sensitive to device and link imbalance, making them less natural to scale across heterogeneous hardware.
2.2
L40S× 1 [0, 6)
GPU A [2, 4)
GPU B [1, 2)
GPU B [2, 3)
M1
GPU B [0, 1)
M2
GPU B [0, 1)
M1
GPU A [2, 3)
M2
GPU B [0, 1)
M1 M2
Throughput Demand
GPU B M1 M2
M1 M2
GPU A [1, 4)
①. Each model greedily runs on its most efficient GPU.
Throughput Demand
M1 M2
②. Resource coordination enabled by joint optimization.
Figure 2: Joint optimization across models. Greedy per-model 1 causes contention; mixed-GPU replicas proallocation (⃝) 2 satisfy both demands. duced by joint optimization (⃝) substitute for top-tier ones while improving cost efficiency. The benefit is especially pronounced when top-tier supply is scarce and helps ease contention among models. We observe similar patterns across other models and GPU combinations. Fig. 1b reveals a second benefit. With only homogeneous node sets (gray), achievable throughputs are discrete and leave large gaps between plans, forcing allocators to over-provision when demand falls between steps. Heterogeneous node sets (orange) instead yield a near-continuous spectrum of throughputs. This lets the system match per-model demand more tightly and avoid wasted capacity. Together, these two effects drive the up to 2.79× cost reduction over the best baseline that we report in Sec. 6.
Challenges and Opportunities
Joint Optimization across LLMs. The second challenge is coordinating decisions across models rather than solving each in isolation. Fig. 2 illustrates why a per-model greedy strategy is insufficient. Two models, M1 and M2, share a constrained pool of 2 GPU-A nodes and 3 GPU-B nodes, with the per-GPU throughputs shown on the left. If each model 1 independently claims its most efficient GPU (⃝), M1 takes both GPU-A nodes and over-serves its demand, while M2 is left with only GPU-B nodes and falls short. Joint optimization 2 resolves this contention (⃝): M1 yields a GPU-A node and accepts a GPU-B node in its pipeline, which frees the GPUA node for M2 to combine with GPU-B nodes in a mixed replica. Both models now meet demand from the same pool,
GPU Heterogeneity. Serving a single model replica across heterogeneous GPUs poses a new challenge: model placement and resource allocation become tightly coupled and must be jointly optimized, each over an exponential search space. This added complexity, however, unlocks new opportunities. Fig. 1a shows the most cost-efficient strategy for the prefill phase of Qwen-3 235B [49] under a 1800 ms latency SLO, drawn from the five GPU types in Table 1. The winner is a mixed pipeline of three single-GPU L40S nodes and three dual-GPU H100 nodes, with a non-uniform layer partition across stages. Notably, it beats every pure H100 setup (526 vs. 481, 460, 476 Tok/s/USD). This shows that mid-tier GPUs can 3
with no idle resources. Capturing these gains requires jointly optimizing allocation and model placement across all models, which is the key problem Coral addresses.
3
spaces. This makes the approach intractable at scale, where pools may span thousands of nodes across tens of GPU configurations and multiple regions. As a concrete data point, Helix [29] requires a 4-hour search budget to find a throughputoptimal placement for a single model on just 24 nodes, even without a latency SLO constraint. The latency SLO further compounds the already combinatorial spaces of both resource allocation and model placement, since feasibility depends on end-to-end latency that can only be evaluated after a full placement is fixed. As a result, computing even a single global solution is impractical, let alone re-solving online to adapt the cluster to fluctuating throughput demands and resource availability.
The Multi-LLM Serving Problem
We formulate the problem of serving multiple LLMs on a pool of heterogeneous GPU nodes as a joint optimization over resource allocation and model placement. Let G denote the pool of GPU nodes available for provisioning, and M the set of models to serve. Each node g ∈ G has a provisioning cost p(g) ≥ 0. For each model m ∈ M , let Tm be its required throughput under a given latency SLO. A serving strategy is specified by two functions. For each model m, the resource allocation Φ chooses a replica count nm ≥ 1 and assigns each replica a disjoint subset of nodes: Φ(m) = G1m , . . . , Gnmm , Gim ⊆ G ,
Key Insight. To jointly optimize Φ and Ψ while keeping the solve time short enough for repeated online execution, we exploit a key substructure of the problem. Once the model and latency SLO are fixed, the throughput-optimal placement Ψ∗ (G ′ ) on a set of nodes G ′ depends only on G ′ itself, independent of how the remaining nodes are allocated. We can therefore enumerate node combinations for each model, precompute offline the throughput-optimal model placement that satisfies the SLO, and cache the result for the online resource allocator to query. We call each such cached, reusable artifact a Serving Template. Theoretically, this decomposition is lossless: any valid Φ and Ψ can be expressed if all possible node combinations are cached. Because the actual space of combinations is infinite, Coral approximates the joint space by enumerating only a principled subset (Sec. 4.2). We will show in Sec. 6.7 that this approximation has negligible performance impact. Crucially, the decomposition moves model placement search off the online critical path, allowing the resource allocator to finish in tens of seconds rather than hours.
j
Gim ∩ Gm′ = ∅ for (m, i) ̸= (m′ , j). The model placement Ψ, for each replica Gim , specifies a hybrid pipeline- and data-parallel layout: the number of pipeline stages, the partition of the model’s layers across stages, and the assignment of each node g ∈ Gim to a stage. Multiple nodes may map to the same stage. In this case, they hold an identical set of layers and share the load, i.e., data-parallel replication within the stage. Following prior work [21, 29], we restrict inter-node parallelism to PP and DP, since tensor and expert parallelism shard work symmetrically and run at the speed of the slowest participating device, making them unsuitable for heterogeneous GPU mixes. Within a node, where GPUs are homogeneous and share high-bandwidth interconnects, TP and EP remain available. We write T Ψ(Gim ) for the throughput of replica i of model m under model placement Ψ. The multi-LLM serving problem jointly optimizes Φ and Ψ to minimize total provisioning cost subject to per-model latency and throughput requirements:
Two-Stage Workflow. Following this insight, Coral decomposes multi-LLM serving into two stages (Fig. 3). Offline, the Serving Template generator (Sec. 4.2) enumerates node combinations for each (m, ℓ) pair and uses integer linear programming (ILP) to find the throughput-optimal model placement satisfying ℓ on each combination. The resulting templates form the Serving Template Library. Online, the resource allocator (Sec. 4.3) queries this library to select and instantiate templates across regions, meeting throughput demand at minimum cost subject to resource availability. The offline stage is amortized across all subsequent online allocations, which run in tens of seconds and let the cluster adapt to workload and resource changes.
nm
min Φ, Ψ
s.t.
4 4.1
∑ ∑ ∑i p(g)
m∈M i=1 g∈Gm nm T Ψ(Gim ) ≥ Tm , i=1
∑
∀m ∈ M .
Optimization Formulation in Coral
4.2
Solution Overview
Serving Template Generation
As Fig. 3 shows, the Serving Template generator takes as input the set of models to serve, each model’s latency SLO, and the GPU configurations under consideration. It produces the Serving Template Library as described in Sec. 4.1.
Naive Enumeration is Intractable. A naive approach to finding the optimal serving strategy (Φ, Ψ) enumerates all resource allocation plans, computes the optimal model placement for each replica in each plan, and selects the lowest-cost result. However, both subproblems have exponential search
Serving Template. Formally, a Serving Template for model m 4
Offline / One-Time
Online / Periodic
Models & SLOs GPU Configurations
Throughput Price & Availability
Serving Template Generator
Resource Allocator
Serving Template Library
R1
R1
(𝑚2 , ℓ2 ): R1
R2
(𝑚1 , ℓ1 ):
Nodes with Different GPU Configurations
R1 = Region 1 R2 = Region 2
Cluster Setup
R2
Data Flow between Stages
…
Figure 3: Two-stage workflow of Coral. The offline Serving Template generator (Sec. 4.2) pre-computes a Serving Template Library from the provided models, SLOs, and GPU configurations. The online resource allocator (Sec. 4.3) queries the library to produce a cluster setup across regions that meets throughput demand at minimum cost. The offline stage runs once; the online stage runs periodically to adapt to workload and resource changes. Stage 1 Layer [0, 9)
Stage 2 Layer [9, 21)
2×A100
2×H100
1×L40S
Stage 3 Layer [21, 36)
each node is assigned to exactly one stage:
2×L40S
L
S
2×L40S
∑ xs j = 1 ∀s,
∑ ysk = 1 ∀k.
j=1
s=1
2×L40S
The stage layer counts must sum to the full model: Figure 4: Illustration of a Serving Template. Each box represents a node, and arrows represent data movement. Nodes in the same pipeline stage hold the same set of layers and share the load. Each request runs on one node per stage and loops back for auto-regressive generation.
S
L
∑ ∑ j · xs j = L.
s=1 j=1
End-to-end throughput is bounded by the slowest pipeline stage. Because nodes assigned to the same stage act as dataparallel replicas, their individual throughputs add up. Hence, for every stage s, we have:
under latency SLO ℓ is a tuple τ = (m, ℓ, G ′ , Ψ∗ (G ′ )). Here G ′ denotes a set of nodes from a single region. Templates do not span multiple regions, because inter-region network latency (typically tens to hundreds of milliseconds depending on geography) is prohibitive even for pipeline-parallel communication in LLM serving. For a given m and ℓ, two templates are equivalent if the number of nodes of each GPU configuration is the same. Ψ∗ (G ′ ) is the throughput-optimal model placement for serving m on G ′ subject to ℓ, from which the template’s throughput T (τ) := T Ψ∗ (G ′ ) follows directly. Fig. 4 shows an example with a node combination G ′ of 6 nodes spanning 4 GPU configurations (1×L40S, 2×L40S, 2×A100, 2×H100), and a model placement that partitions the model into three pipeline stages of varying layer counts, with data-parallel replication within each stage.
′ L |G |
T (τ) ≤ ∑ ∑ xs j · ysk · T̂ j (gk ) j=1 k=1
where T̂ j (gk ) is the maximum throughput of node gk when it holds j layers under a per-stage latency budget of ℓ/S. We obtain T̂ j (gk ) from a one-time offline profiling run for each GPU configuration. Linearization. To remove the quadratic term xs j · ysk , we introduce an auxiliary binary variable zs jk and enforce zs jk ≤ xs j ,
zs jk ≤ ysk ,
zs jk ≥ xs j + ysk − 1,
so that zs jk = 1 iff both xs j = 1 and ysk = 1. The throughput constraint becomes, for each stage s,
Optimal Model Placement via ILP. Given a node set G ′ , a model m with L layers, and a latency SLO ℓ, we formulate the search for Ψ∗ (G ′ ) as an ILP parameterized by the number of pipeline stages S. Decision variables. We introduce two groups of binary variables. Variable xs j (s ∈ [1, S], j ∈ [1, L]) equals 1 iff stage s holds j consecutive layers, and variable ysk (s ∈ [1, S], k ∈ [1, |G ′ |]) equals 1 iff node gk ∈ G ′ is assigned to stage s. Constraints. Each stage picks exactly one layer count, and
′ L |G |
T (τ) ≤ ∑ ∑ zs jk · T̂ j (gk ) j=1 k=1
Objective. The ILP maximizes the end-to-end throughput T (τ). For a given S, solving the ILP yields a candidate model placement: xs j specifies the layer partition across the S pipeline stages and ysk specifies the node-to-stage assignment. The throughput-optimal placement Ψ∗ (G ′ ) is the candidate achieving the highest throughput across S ∈ [1, |G ′ |]. 5
The ILP contains O (S · L · |G ′ |) binary variables (including auxiliaries) and O (S + |G ′ |) primary constraints, plus O (S · L · |G ′ |) auxiliary constraints from linearization. With modern solvers such as Gurobi [13], solving the ILP for a given S takes seconds, so enumerating S over the full range remains inexpensive.
type-c nodes it requires. As a shorthand, we write m pr (τm i ) := ∑ pr (c) ·Uc (τi ) c∈C
for the total provisioning cost of instantiating τm i in region r. Decision variables. For each template τm i and region r, a non-negative integer variable νr (τm i ) specifies the number of Serving Instances to deploy. Constraints. For every region r and configuration c, the number of nodes consumed cannot exceed availability:
Generating the Serving Template Library. As discussed in Sec. 4.1, our two-stage decomposition preserves the original search space defined in Sec. 3 if all node combinations are pre-computed and cached. However, since the space of node combinations is unbounded, we must restrict enumeration to a principled subset. Specifically, for each model, we evaluate only combinations containing at most Nmax nodes and a total GPU memory capacity below ρ times the model size. Combinations exceeding these thresholds suffer from higher communication overheads and are typically dominated by dividing those same resources into multiple smaller replicas. As a result, the resource allocator rarely selects them, and as we demonstrate in Sec. 6.7, this pruning strategy has a negligible impact on the most cost efficient Serving Template we can find. The resulting library yields thousands to tens of thousands of templates per (m, ℓ) pair. Because each ILP is small and independent, library generation is highly parallelizable, taking a few minutes for smaller models and tens of minutes for larger ones under moderate Nmax and ρ (Sec. 6.7).
4.3
Nm
∑ ∑ Uc (τmi ) · νr (τmi ) ≤ Ar (c).
m∈M i=1
For every model m, aggregate throughput across regions must meet its demand: Nm
∑ ∑ T (τmi ) · νr (τmi ) ≥ Tm .
r∈R i=1
Objective. The formulation in Sec. 3 targets a static snapshot, whereas the online allocator must steer the cluster through fluctuating demand and availability. Each newly provisioned instance incurs non-trivial setup overhead (e.g., node startup and weight loading), while tear-down is graceful: the runtime drains in-flight requests without user impact (Sec. 5.1). We therefore charge an initialization penalty only on newly added instances. Let ν′r (τm i ) denote the number of currently running instances of τm i in region r, a constant known at solve time, and let Ir (τm ) i ≥ 0 be a continuous auxiliary variable bounded by m ′ m m Ir (τm i ) ≥ νr (τi ) − νr (τi ) · pr (τi ) · K,
Online Resource Allocation
As illustrated in Fig. 3, the online resource allocator takes as input the pre-computed Serving Template Library, the current throughput demand for each model, and the real-time availability and pricing of GPU configurations across regions. It solves a second ILP to determine the optimal cluster allocation, minimizing total provisioning and initialization costs subject to throughput and availability constraints.
where K is a hyperparameter set to the ratio of node initialization time to cluster adjustment interval. Because Ir (τm i ) ≥ 0, the bound is active only when νr > ν′r , so scaling down contributes nothing to the objective. This penalty discourages churn between allocations of comparable cost but different composition. The allocator minimizes the sum of provisioning cost and initialization penalty: i Nm h m m min ∑ ∑ ∑ νr (τm i ) · pr (τi ) + Ir (τi ) .
Serving Instance. To map offline templates to online allocations, we define a Serving Instance as the physical instantiation of a Serving Template within a specific region. Each instance provisions the exact node combination G ′ specified by its template and serves a single model replica using the pre-computed optimal model placement Ψ∗ (G ′ ). By treating Serving Instances as the fundamental building blocks, the allocator abstracts away the complexity of model placement, reducing the online problem to simply choosing which templates to instantiate and where.
r∈R m∈M i=1
Tractability. The ILP has |R | · ∑m∈M Nm integer decision variables and an equal number of continuous penalty variables, subject to |C | · |R | + |M | capacity and throughput constraints and |R |· ∑m∈M Nm penalty bounds. Although the total variable count can reach the millions, the optimal solution is extremely sparse: a cluster of hundreds of nodes is typically assembled from only tens of active templates. As a result, only a few dozen νr (τm i ) variables take non-zero values. The millions of remaining variables stay zero, rendering their associated penalty bounds trivially satisfied. As shown in Sec. 6, Gurobi [13] solves this ILP in under 1 minute on average even with millions of variables and constraints, which is well within our online budget.
Optimal Resource Allocation via ILP. Let R denote the set of available cloud regions and C the set of GPU configurations. For each r ∈ R and c ∈ C , let Ar (c) ≥ 0 be the current number of available nodes, and pr (c) the per-node provisioning cost. For each model m ∈ M with throughput requirement Tm under latency SLO ℓm , the prior offline stage produces Nm valid Serving Templates; we denote the i-th template as τm i , let m T (τm i ) be its throughput, and let Uc (τi ) be the number of 6
Request Dispatch / Prefill Finish Prefill Instance ×M
2 2
Resource Allocator Request Router
3
Instance Life-Cycle. Coral invokes the resource allocator periodically to adapt the cluster to current throughput demand, resource availability, and node prices. Each invocation produces a new target allocation, and the runtime reconciles the running cluster to it as follows. Scaling down. For each template whose target count drops below the current count, Coral gracefully terminates the excess instances, starting with the one at lowest load. A terminating instance stops accepting new requests and shuts down once its in-flight requests complete (i.e., connection draining). Because each shutdown is scheduled in advance and drains within roughly a minute in practice, draining is preferable to request migration. Migrating in-flight decode requests would require transferring their entire KV cache, which is far more expensive than letting them finish. Scaling up. For each template whose target count grows, Coral provisions the additional nodes and initializes new instances according to the template’s model placement. This process typically takes several minutes, dominated by node startup, weight loading, and CUDA graph compilation. This is exactly the overhead captured by the initialization penalty in the allocator’s objective (Sec. 4.3). Once a new instance becomes ready, its scheduler notifies the coordinator, and the router begins dispatching requests to it.
Decode Instance ×N
Coordinator
Scheduler 1
Request Dispatch / Decode Finish
Coordinator
Scheduler 1
2
P→D KV Transfer Control Flow
Coordinator
Data Flow KV Transfer
Scheduler
CPU Nodes
Nodes with Different GPU Configs (Number is PP Stage)
Figure 5: Overview of the Coral runtime system. A central coordinator hosts the resource allocator and request router, which dispatches each request to a prefill Serving Instance and later to a decode Serving Instance. Within an instance, the scheduler assigns heterogeneous engine nodes (colored circles) per pipeline stage. KV caches are transferred directly between prefill and decode engine nodes.
5
Coral Runtime
5.1
System Design
As shown in Fig. 5, Coral’s runtime system consists of a single CPU coordinator node, a pool of prefill Serving Instances, and a pool of decode Serving Instances. The coordinator hosts the resource allocator (Sec. 4.3) and the request router, which dispatches incoming requests to the appropriate instances. Each Serving Instance contains a CPU scheduler node and a group of GPU engine nodes. The scheduler handles intra-instance request scheduling and communication with the coordinator, while the engine nodes execute model inference following the template’s model placement. We describe Coral in the PD-disaggregated setting, but the same design applies to PDaggregated serving as well.
5.2
Implementation
We implement the Coral runtime in 53K LoC of Python and C++. The control plane runs over ZeroMQ [14], on top of which we build Serving Instance life-cycle management and request dispatching. For intra-instance communication between engine nodes, we build a high-bandwidth, low-latency pipeline-parallel framework on top of NCCL [34]. This framework supports arbitrary pipeline- and data-parallel configurations and transparently enables Remote Direct Memory Access (RDMA) and GPU-Direct RDMA [35] whenever the hardware permits. For KV cache transfers between prefill and decode instances, we use GLOO [31] over CPU-based RDMA. Routing the transfer through the CPU isolates KV cache movement from the GPUs, ensuring it does not contend with model inference or pipeline-parallel communication for GPU resources. We use vLLM [23] as the per-node execution engine; however, the runtime is engine-agnostic, and any compatible inference engine can be substituted without changes to the surrounding system. To evaluate Coral at scales of hundreds of nodes—where running on real hardware is cost-prohibitive—we additionally build an event-based simulator in 5K LoC of Python. The simulator’s cost model is fitted from offline profiling data collected across every GPU configuration in our pool. Using this model, the simulator advances execution at the granularity of individual pipeline stages on each engine node. As demonstrated in our fidelity study (Sec. 6.2), the simulator reproduces the real system’s per-request prefill and decode
Request Life-Cycle. When a request arrives at the coordinator, the router selects a prefill instance for the target model via weighted round-robin, with each instance weighted by its template throughput T (τ). The request is forwarded to that instance’s scheduler, which then assigns one engine node per pipeline stage via weighted round-robin. These weights correspond to each node’s expected throughput under the template’s chosen model placement. The scheduler dispatches the request to the first pipeline stage, and the engine nodes propagate intermediate activations down the pipeline (standard pipeline-parallel inference). Once the final stage finishes prefill, the result returns to the scheduler, which notifies the coordinator. The router then selects a decode instance, after which the prefill and decode schedulers coordinate the direct node-to-node transfer of the request’s KV cache. Once the transfer completes, auto-regressive generation begins on the decode instance. 7
Phi4 14B GPT-OSS 20B Qwen3 32B Llama3 70B GPT-OSS 120B Qwen3 235B
MoE ✗ ✓ ✗ ✗ ✓ ✓
Hyb. Attn.
L
✗ ✓ ✗ ✗ ✓ ✗
40 24 64 80 36 94
Prefill (ms)
Decode (ms)
Trace
1200 900 1600 1500 1000 1800
60 30 100 80 40 120
AzureConv. AzureCode BurstGPT BurstGPT AzureConv. AzureCode
Simulation
1.00 0.75
1.00
Decode
0.75 0.50
0.25
0.25
200 400 Latency (ms)
6000.000
20 40 60 Latency (ms)
80
Figure 6: Simulator fidelity. Prefill and decode latency CDFs of Phi4 14B closely match between real system and simulator.
Workload Setup. Request lengths and arrival patterns are drawn from three datasets—Azure Code [42], Azure Conversation [42], and BurstGPT [45]—which we assign evenly across the models under test. Latency SLOs are set per model based on size and architecture (whether the model uses MoE [10] or hybrid attention mechanisms [4]), following the typical SLOs reported in AdaServe [25]. Table 3 lists the dataset and SLO of each model. By default, all models share the same average arrival rate (10 req/s for the core setup and 25 req/s for the extended setup), which we achieve by uniformly scaling each trace. Sec. 6.5 studies how imbalanced arrival rates across models affect Coral, where we assign 80% of total requests to large and small models respectively.
latencies to within 5.6% and 7.2% on average, validating it as a faithful proxy for our large-scale experiments.
Evaluation
We evaluate Coral to answer the following questions: • Does Coral reduce serving cost across diverse model sets and GPU pools? (Sec. 6.3) • Does Coral mitigate resource contention and sustain goodput under tight resource availability? (Sec. 6.4) • How does Coral’s performance vary with the throughput demand distribution across models? (Sec. 6.5) • Does Coral reduce cost relative to Helix in the singlemodel regime? (Sec. 6.6)
6.1
Real System
0.50 0.000
Table 3: Model characteristics and serving metrics used in our evaluation. First three columns: whether a model uses MoE [10], whether it uses a mix of sliding-window attention and full attention across layers, and its number of layers (L).
6
Prefill
CDF
Model
Resource Setup. Resource availability follows the production GPU-cluster trace from Alibaba [9]. By default, we scale the trace so that availability is high enough for every method to find a feasible solution. For the low-availability study (Sec. 6.4), we instead scale it to a tight but feasible level: 25% above estimated demand for the core setup and 75% above estimated demand for the extended setup. Node prices reflect real AWS US-East-2 and AP-Northeast-2 rates for the core setup, with GCP US-Central-1 added as the third region in the extended setup.
Experiment Setup
Model and GPU Setup. We evaluate Coral across a diverse set of models and GPU types. The core setup uses three models (Qwen-3 32B [49], GPT-OSS 20B [2], Phi4-14B [1]) on a pool with 12 different GPU configurations (L40S, L4, and A10G, each with 1, 2, 4, or 8 GPUs) spanning two cloud regions. The extended setup extends this with three additional models (Qwen-3 235B [49], GPT-OSS 120B [2], Llama-3 70B [11]) and eight additional configurations (H100 and A100 with 1, 2, 4, or 8 GPUs) across a third cloud region. Table 3 summarizes the diverse properties of these models. The core setup requires 20–40 GPUs to serve, while the extended setup requires 100–300 GPUs depending on the method. We evaluate the core setup on real hardware for Sec. 6.3 and Sec. 6.4, and in simulation for Sec. 6.5. The extended setup runs exclusively in simulation, as provisioning hundreds of GPUs on real hardware would cost thousands of dollars per hour. To validate the simulator (Sec. 6.2), we additionally re-run the two real-hardware experiments in simulation and compare. The simulator’s prefill and decode latencies deviate from the real system by only 5.6% and 7.2% on average, confirming it as a faithful proxy for our simulation-based experiments.
Evaluation Duration. Each experiment runs for 30 minutes, with the cluster reconfigured every 6 minutes. We define each 6-minute interval as one epoch. We cap the duration here due to the cost of real-hardware evaluation. Because production clusters reconfigure far less frequently, we amortize initialization cost over a 60-minute adjustment interval (i.e., divide by 10). Sec. 6.7 examines sensitivity of Coral to this interval. Coral Setup. Offline Serving Template generation runs on an AWS c8i instance with 384 cores, with the per-template node cap set to Nmax = 6 and the total memory cap to ρ = 12× the model size. This is a one-time process per setup and completes in a few hours for all models. The online resource-allocation ILP is solved on a c8i instance with 32 cores. Baselines. We compare against two baselines in the end-toend evaluation. Homo assumes each model replica is served 8
3 4 Epoch
5
1050
1000 500 01
0 Qwen3 GPT-OSS Phi4 32B 20B 14B
2
3 4 Epoch
5
700 350 0
Coral Cauchy Homo Prefill Decode
Qwen3 235B
P: 93.6 D: 77.1 P: 82.5 D: 49.5 P: 61.2 D: 41.3 P: 55.0 D: 13.7 P: 56.2 D: 14.7 P: 20.6 D: 17.1
2
1400
Homo
P: 141.6 D: 30.9
15
1500
P: 4.6 D: 6.3 P: 5.9 D: 7.5 P: 4.5 D: 6.3 P: 4.8 D: 4.6 P: 6.9 D: 2.3 P: 4.3 D: 2.9
25
P: 5.3 D: 17.2
30
Coral Cauchy
Avg. Provision Cost
45
2000
Coral Cauchy Homo Prefill Decode
Hourly Cost (USD/h)
60
50
01
75 Avg. Provision Cost
75
Homo
P: 761.9 D: 88.0 P: 677.1 D: 27.4
Coral Cauchy
P: 27.4 D: 17.9 P: 25.0 D: 17.3
Hourly Cost (USD/h)
100
GPT-OSS 120B
Llama3 70B
(a) Hourly cost comparison in (b) Per-model average provision (c) Hourly cost comparison in (d) Per-model average provision cost core setup. cost breakdown in core setup. extended setup. breakdown in extended setup.
Figure 7: Hourly cost comparison under default settings across the two model and GPU setups. (a, c) Hourly cost per epoch. (b, d) Per-model average provisioning cost broken down into prefill (P) and decode (D). The extended-setup breakdown shows only the three largest models, which dominate total cost. Cauchy
Homo
75 50 25 01
2
3 4 Epoch
(a) Core setup.
5
1200
Coral
Cauchy
Coral
Homo
2e3
900
Goodput (Tok/s)
Coral
Hourly Cost (USD/h)
Hourly Cost (USD/h)
100
600 300 01
1e3
2
3 4 Epoch
5
Cauchy
Homo
400
2e3
200
1e3
01 2 3 4 5 01 2 3 4 5 01 2 3 4 5
(b) Extended setup.
Qwen3-32B
Figure 8: Hourly cost under scarce resource availability. Baselines appear cheaper only because they fail to meet throughput demand (Fig. 9 and Fig. 10).
GPT-OSS-20B
Phi4-14B
Figure 9: Decode goodput across epochs under scarce resource availability (core setup). Prefill follows the same trend.
6.2
Simulator Fidelity
To validate the simulator, we re-run the real-hardware experiments in Sec. 6.3 and Sec. 6.4 in simulation and compare the two. We use per-request latency as the primary fidelity metric. Higher-level metrics such as goodput and SLO attainment are derived from latency and the SLO threshold, so matching latency distributions implies matching goodput across the full range of SLOs, whereas matching goodput at a single threshold does not. Averaged across all setups, the simulator’s prefill and decode latencies deviate from the real system by 5.6% and 7.2%, respectively. Fig. 6 shows a representative comparison for Phi-4 14B: the prefill and decode latency CDFs align closely across the full distribution, including the tail. This confirms that the simulator is a faithful proxy for the experiments in the remainder of Sec. 6.
on homogeneous hardware, but permits heterogeneity across replicas—the same assumption adopted by SkyServe [28] and SageServe [17]. It greedily selects the most cost-efficient (highest goodput per USD) homogeneous strategy for each model. Cauchy, adapted from the PD-disaggregated serving system of the same name [53], retains their cost-efficiency model and resource-allocation algorithm, but extends their GPU-combo definition so that a single prefill replica can feed multiple decode replicas, yielding more flexibility under multiLLM serving. Both baselines run within the Coral runtime for a fair comparison. Heterogeneity-aware model-placement systems such as Helix [29] do not address resource allocation, so we compare against Helix separately in Sec. 6.6. Evaluation Metrics. Our primary metric is hourly cost in USD/h, which is the sum of machine provisioning cost and amortized initialization cost (under the 60-minute interval discussed above). For the experiments in Sec. 6.4, where methods differ in how much demand they actually satisfy, we additionally report goodput, defined as the number of generated tokens per second that satisfy the latency SLO.
6.3
Cost Efficiency in Diverse Setups
This section evaluates whether Coral reduces serving cost compared with existing systems. Fig. 7 reports the results. In the core setup (Fig. 7a), Coral reduces the average hourly cost by 1.62× over Homo and 1.60× over Cauchy, while the resource allocation ILP solves in only 0.24 seconds on aver9
Coral 1e3
4e3
Cauchy
2e4
6e3
Goodput (Tok/s)
800 600 400
4e3 2e3
1e4 2e3
200 0
Homo 1e3
4e3
800 600 400
2e3
200
1 2 3 4 5 01 2 3 4 5 01 2 3 4 5 01 2 3 4 5 01 2 3 4 5 01 2 3 4 5 Qwen3-235B GPT-OSS-120B Llama3-70B Qwen3-32B GPT-OSS-20B Phi4-14B
Figure 10: Decode goodput across epochs under scarce resource availability (extended setup). Prefill follows the same trend.
6.4
age. The per-model breakdown in Fig. 7b shows that Qwen-3 32B accounts for roughly 60% of the total cost across all methods. This is expected: among the three models, Qwen3 32B has the most parameters and, unlike GPT-OSS 20B, lacks MoE or hybrid attention to reduce compute and memory demand. Coral achieves its largest cost reductions on this model (2.02× over Homo and 1.88× over Cauchy), with most of the savings coming from the prefill side. Inspecting the cluster setups chosen for Qwen-3 32B prefill, we find that both baselines assemble the cluster from homogeneous single-node replicas—one replica per 2×L40S node or per 4×L4 node—and vary the mix of the two across epochs to track throughput demand. In contrast, Coral selects Serving Templates that combine L4 and L40S nodes to serve a single replica (e.g., one 1×L4 node plus three 1×L40S nodes), with non-uniform layer partitioning across pipeline stages and data-parallel replication within selected stages.
Goodput Under Scarce Resources
This section evaluates whether Coral sustains goodput under tight resource availability. If a method cannot find a solution meeting every model’s throughput demand, we uniformly scale down the per-model arrival rate until one exists. This preserves the balanced-demand assumption from Sec. 6.3. In the core setup, Fig. 8a reports hourly cost and Fig. 9 reports per-model decode goodput; we omit prefill because it follows the same trend. Coral matches Homo in hourly cost while delivering 1.24× higher average goodput. Against Cauchy, Coral reduces cost by 1.51× and maintains roughly the same average goodput across models. The allocation ILP solves in 0.11 seconds on average. Compared to its Sec. 6.3 solution, Coral’s cluster setup here is 10% more expensive on average. The tightest epoch is epoch 1. In this epoch, Coral redirects three higher-end L40S GPUs from Phi-4 to Qwen-3—the heavier model that needs them more—and falls back to A10G and L4 for Phi-4. It also broadens its use of heterogeneous Serving Templates from 1 to 4. The baselines lack such cross-model coordination: Homo greedily selects the most cost-efficient template per model in isolation, and Cauchy encodes per-model cost efficiency directly in its ILP objective. Both designs drive every model to contend for the scarce L40S GPUs, and none obtain enough to meet demand. In the extended setup, Fig. 8b reports hourly cost and Fig. 10 reports per-model decode goodput. Against Homo, Coral is 1.52× more expensive but delivers 7.64× higher goodput. Against Cauchy, Coral reduces hourly cost by 1.25× and improves average goodput by 2.39×. The allocation ILP solves in 41 seconds on average. Compared to its Sec. 6.3 solution, Coral’s cluster setup here is 25% more expensive on average. The tightest epoch is epoch 3, which is also the hardest to solve (2 minutes) and incurs the largest cost increase (53%). In this epoch, the number of heterogeneous Serving Instances jumps from 9 to 19. For the three largest models (Qwen-3 235B, GPT-OSS 120B, Llama-3 70B), nearly all throughput is served by 16 distinct heterogeneous instances, up from 6 in Sec. 6.3 where only Qwen-3 235B relied heavily on them. This complex
In the extended setup (Fig. 7c), Coral reduces the average hourly cost by 3.14× over Homo and 2.66× over Cauchy, with an average ILP solving time of 9.68 seconds. The cost breakdown in Fig. 7d shows that Qwen-3 235B dominates total cost: despite its MoE design, its sheer parameter count makes it the most expensive model to serve. Coral again achieves its largest reductions on this model (4.93× over Homo and 4.08× over Cauchy), with most savings from prefill. The cluster setups reveal the same underlying pattern as in the core setup. Both baselines serve each Qwen-3 235B replica with a single 8×A100 or 8×H100 node, a natural choice given the model’s memory and compute footprint. Coral instead uses templates that combine multiple smaller A100 and H100 nodes (1–2 GPUs each) into one replica with non-uniform PP and data-parallel replication. Together, these results show that intra-replica heterogeneity is essential for cost-efficient serving, as mixing GPU types within a replica lets Coral tailor each instance to the workload’s latency SLO and throughput demand (as discussed in Sec. 2.2). Across all methods and both setups, the amortized initialization cost remains under 1% of the hourly cost, confirming that the ILP’s initialization penalty effectively discourages churn. 10
Homo
90 60 30 01
2
3 4 Epoch
(a) Large-Heavy, core setup.
5
2500 Coral 2000 1500 1000 500 01 2
Cauchy
60
Homo
Coral
Cauchy
800
Homo
45
(b) Large-Heavy, extended setup.
01
Cauchy
Homo
2
3 4 Epoch
400
15
5
Coral
600
30
3 4 Epoch
Hourly Cost (USD/h)
Cauchy
Hourly Cost (USD/h)
Coral
Hourly Cost (USD/h)
Hourly Cost (USD/h)
120
200
2
3 Epoch
4
5
(c) Small-Heavy, core setup.
01
5
(d) Small-Heavy, extended setup.
Figure 11: Hourly cost under imbalanced demand, where the top third of models (Large-Heavy) or bottom third (Small-Heavy) receive 80% of requests. Coral’s cost advantage grows when large models dominate.
Coral
cross-model coordination explains the long solve time. The baselines again fail for the same reason as in the core setup: all models compete for the scarce A100 and H100 nodes and none obtain enough. Together, these results yield two takeaways. First, joint optimization across models is essential for satisfying aggregate throughput demand under scarce resources. Second, heterogeneous Serving Templates give the solver the flexibility it needs to resolve cross-model contention.
Helix 0
802.5 93.36
50 100 Hourly Cost (USD/h)
637
0
500 1000 Goodput (Tok/s)
Figure 12: Comparison with Helix on Helix’s "High GPUHeterogeneity Cluster" setup.
6.6 6.5
73.8
Comparison with Helix
This section compares Coral with Helix [29], which uses an ILP to optimize model placement for a single model on a fixed heterogeneous node set. We adopt Helix’s largest experiment setup (“High GPUHeterogeneity Cluster”) and deliberately configure the comparison in Helix’s favor. The resource pool Coral allocates from contains exactly the same GPU mix as Helix’s cluster (4× A100 40G, 6× V100 16G, 16× L4 24G, and 38× T4 16G), and both systems serve the 70B Llama model. We use the node prices from AWS US-East-2. We set the arrival rate to 4 req/s, exceeding the throughput Helix reports, and impose prefill and decode latency SLOs on Coral of 2090 ms and 730 ms—the median latencies reported in Helix’s online-serving experiments. Helix itself runs unconstrained. Because provisioning 64 GPUs on real hardware is cost-prohibitive, we run Coral in our high-fidelity simulator and compare against the numbers reported by Helix, which were likewise obtained from their own simulator. The goal is to measure Coral’s cost savings under constraints strictly tighter than Helix’s. As Fig. 12 shows, Coral reduces cost by 21% and improves throughput by 26% over Helix while satisfying both latency SLOs. Coral assigns the A100 nodes to prefill and partitions the remaining hardware into three decode Serving Instances, each built from L4 and T4 nodes, leaving 6 V100 nodes and 1 T4 node unused. The gains stem from how each system exploits the resource pool. Helix consolidates all 64 GPUs into a single monolithic pipeline via PP and DP, paying substantial cross-stage communication overhead. Coral instead decomposes the pool into multiple smaller Serving Instances, each running its own throughput-optimal placement and avoiding
Robustness to Imbalanced Demand
This section studies how the imbalanced throughput demand across models affects Coral. We consider two imbalanced settings. In Large-Heavy, the top 1/3 of models by size receive 80% of the requests: Qwen-3 32B in core setup, and Qwen-3 235B together with GPT-OSS 120B (split equally) in extended setup. In Small-Heavy, the bottom 1/3 receive 80%: Phi-4 14B in core setup, and Phi-4 14B together with GPT-OSS 20B (split equally) in extended setup. Large-Heavy. Fig. 11a and Fig. 11b report hourly cost for core setup and extended setup. In extended setup, Coral reduces the average hourly cost by 2.79× over Homo and 3.47× over Cauchy. In core setup, the corresponding reductions are 1.82× and 1.78×. The large models dominate spending in this setting, consuming ∼80% of total hourly cost. Because they require multiple GPUs per replica, Coral has ample room to exploit intra-replica heterogeneity, which drives the savings. Small-Heavy. Fig. 11c and Fig. 11d report hourly cost for core setup and extended setup. In extended setup, Coral reduces the average hourly cost by 2.34× over Homo and 1.64× over Cauchy. Despite their low arrival rate, Qwen-3 235B and Llama-3 70B are large enough to still account for 40% and 20% of total cost, respectively, so most of Coral’s savings come from these two models. In core setup, all three methods yield nearly identical total cost, with Coral only ∼10% cheaper on average. This is expected: Phi-4 14B, which dominates cost in this setting, fits on 1–2 GPUs, leaving little room for intra-replica optimization. 11
2e3 1e3
637
(3, (4, 6) (5, 8) (6, 10) (7, 12) 14 )
2 8 83
1e5 5e4 0
53178 14828 605 3467
3910 3910 3576 3010 3e3 2587
PD-aggregated serving alike (Sec. 5.1).
4e3
Resource Allocation for LLM Serving. A second line of work provisions and scales LLM serving resources under shifting demand, prices, and availability [17, 28, 32]. SpotServe [32] exploits preemptible instances to reduce cost while tolerating preemption, SkyServe [28] allocates replicas across regions and clouds with spot/on-demand autoscaling, and SageServe [17] combines workload forecasting with scaling and routing to reduce reconfiguration overhead. These systems optimize cluster-level provisioning, replica allocation, or request routing, but treat each replica as a fixed, internally homogeneous configuration. Coral is complementary: it jointly decides which heterogeneous resources to allocate and how each model is placed across them, unlocking intra-replica heterogeneity that these systems leave on the table.
2e3 1e3
(3, (4 6) (5, , 8) (6, 10) (7, 12) 14 )
3e3
168695
1.5e5
Best Goodput / USD (Tok/$/s)
3358
(3, (4 6) (5, , 8) (6, 10) (7, 12) 14 )
Solve Time (s)
# Valid Templates
2e5
4e3
Figure 13: Sensitivity of Serving Template generation to the pruning parameters (Nmax , ρ). Solving time and template count grow exponentially, while the best template’s cost efficiency plateaus at (6, 12).
the overhead of one large pipeline.
6.7
Heterogeneous LLM Serving. Prior work exploits heterogeneous GPUs for LLM serving along several axes. Mélange [12] shows that the most cost-efficient GPU type depends on workload characteristics and SLOs, but selects a single homogeneous type per deployment. A recent study [19] jointly optimizes GPU composition, placement, and workload assignment, but minimizes offline batch makespan under a fixed budget on a static resource pool—the dual of Coral’s problem, and without online reconfiguration. Helix [29] and HexGen [21] optimize model placement over a fixed heterogeneous pool, leaving resource selection out of scope; as discussed in Sec. 4.1, wrapping them in an outer allocation loop is intractable. BOute [20] jointly routes and places across heterogeneous LLMs and GPUs, but still assumes a fixed device set and focuses on selecting among model variants to trade off latency and quality. Coral is the first to jointly select heterogeneous resources and place models across them for multiple LLMs under per-model latency SLOs.
Sensitivity Analysis
This section analyzes the sensitivity of Coral’s Serving Template generator to the two pruning parameters introduced in Sec. 4.2: the per-template node cap Nmax and the memory cap ratio ρ. Because adding nodes to a template also inflates its aggregate memory, we sweep the two parameters jointly. As a testbed, we use the prefill phase of GPT-OSS 120B and run the generator on an AWS c8i instance with 384 cores. For each (Nmax , ρ) pair, we record the number of valid templates produced, the total solving time, and the cost efficiency (Goodput/USD) of the best template found. Fig. 13 reports the results. Both the template count and the total solving time grow exponentially with (Nmax , ρ), yet the best template’s cost efficiency plateaus at (Nmax , ρ) = (6, 12). This is expected: larger node sets incur higher inter-node communication overhead and are dominated by splitting the same resources into multiple smaller replicas. Enumerating beyond the plateau therefore drives up offline cost without unlocking cheaper serving strategies. This confirms the principledsubset argument in Sec. 4.2 and justifies our default choice of (Nmax , ρ) = (6, 12) during the end-to-end evaluation.
7
Multi-Model LLM Serving. Several systems target efficient concurrent serving of multiple LLMs. Prism [52] enables GPU sharing with dynamic memory redistribution across colocated models, Aegaeon [48] performs token-granularity autoscaling for effective GPU pooling, and FlexPipe [27] multiplexes models in fragmented serverless clusters via dynamic pipeline refactoring. These systems improve sharing, pooling, and scheduling on a given resource pool, but do not decide which heterogeneous resources to provision or how to place each model across them. Coral complements them by solving this upstream joint optimization under per-model latency SLOs; their runtime mechanisms could be layered on top of the Serving Instances Coral produces.
Related Work
Prefill-Decode (PD) Disaggregated LLM Serving. PD disaggregation exploits the distinct performance characteristics of prefill and decode by provisioning each phase independently [15, 36, 38, 53, 54]. Splitwise [36] and DistServe [54] established this split, and Cauchy [53] extends it by selecting different GPU configurations for the prefill and decode pools. Heterogeneity in these systems, however, is confined to the phase boundary: each replica still runs on a single homogeneous configuration. Coral is complementary and admits heterogeneity within a replica via hybrid pipeline- and dataparallelism, while jointly optimizing allocation and placement across all models. It applies to PD-disaggregated and
8
Conclusion
This paper presents Coral, an adaptive heterogeneity-aware system for cost-efficient multi-LLM serving. Coral jointly optimizes resource allocation and model placement through a 12
lossless two-stage decomposition: offline ILP-based Serving Template generation and online template selection across regions. Lifting placement search off the critical path shrinks the online solve from hours to tens of seconds, letting the cluster continuously adapt to shifting demand and availability. Across 6 models and 20 GPU configurations, Coral reduces serving cost by up to 2.79× and delivers up to 2.39× higher goodput under scarce resource availability.
and Ion Stoica. Chatbot arena: an open platform for evaluating llms by human preference. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024. [7] Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.
Acknowledgment
[8] Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc’aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, et al. Large scale distributed deep networks. Advances in neural information processing systems, 25, 2012.
This research is partially supported by NSF awards CNS2211882 and CNS-2239351, Sloan Foundation faculty fellowships, and research awards from Amazon, Cisco, Google, Jane Street, Meta, NVIDIA, Oracle, Qualcomm, and Samsung. We also thank the members and companies of the PDL consortium (Bloomberg LP, Everpure, Google, Jane Street, LayerZero Labs, Meta, Microsoft Research, Oracle Corporation, Salesforce, Uber, Western Digital) for their interests, insights, feedback, and support.
[9] Jiaang Duan, Shenglin Xu, Shiyou Qian, Dingyu Yang, Kangjin Wang, Chenzhi Liao, Yinghao Yu, Qin Hua, Hanwen Hu, Qi Wang, Wenchao Wu, Dongqing Bao, Tianyu Lu, Jian Cao, Guangtao Xue, Guodong Yang, Liping Zhang, and Gang Chen. Gfs: A preemptive scheduling framework for gpu clusters with predictive spot management. In Proceedings of the 31th ACM International Conference on Architectural Support for Programming Languages and Operating Systems,ASPLOS 2026. ACM, 2026.
References [1] Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905, 2024.
[10] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022.
[2] Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss-120b & gptoss-20b model card. arXiv preprint arXiv:2508.10925, 2025.
[11] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
[3] Amazon Web Services. AWS and Anthropic announce strategic collaboration to advance generative AI. https: //press.aboutamazon.com/2023/9/amazon-and -anthropic-announce-strategic-collaborati on-to-advance-generative-ai, September 2023. Accessed: 2026-04-20.
[12] Tyler Griggs, Xiaoxuan Liu, Jiaxiang Yu, Doyoung Kim, Wei-Lin Chiang, Alvin Cheung, and Ion Stoica. M\’elange: Cost efficient large language model serving by exploiting gpu heterogeneity. arXiv preprint arXiv:2404.14527, 2024.
[4] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.
[13] Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2026.
[5] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
[14] Pieter Hintjens. ZeroMQ: messaging for many applications. O’Reilly Media, Inc., 2013. [15] Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, et al. Inference without interference: Disaggregate llm inference for mixed downstream workloads. arXiv preprint arXiv:2401.11181, 2024.
[6] Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios N. Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, 13
[16] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019.
[25] Zikun Li, Zhuofu Chen, Remi Delacourt, Gabriele Oliaro, Zeyu Wang, Qinghan Chen, Shuhuai Lin, April Yang, Zhihao Zhang, Zhuoming Chen, et al. Adaserve: Accelerating multi-slo llm serving with slo-customized speculative decoding. arXiv preprint arXiv:2501.12162, 2025.
[17] Shashwat Jaiswal, Kunal Jain, Yogesh Simmhan, Anjaly Parayil, Ankur Mallick, Rujia Wang, Renee St. Amant, Chetan Bansal, Victor Ruhle, Anoop Kulkarni, Steve Kofsky, and Saravan Rajmohan. Sageserve: Optimizing llm serving on cloud data centers with forecast aware auto-scaling. Proc. ACM Meas. Anal. Comput. Syst., 9(3), December 2025.
[26] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022. [27] Yanying Lin, Shijie Peng, Chengzhi Lu, Chengzhong Xu, and Kejiang Ye. Flexpipe: Adapting dynamic llm serving through inflight pipeline refactoring in fragmented serverless clusters. arXiv preprint arXiv:2510.11938, 2025.
[18] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024.
[28] Ziming Mao, Tian Xia, Zhanghao Wu, Wei-Lin Chiang, Tyler Griggs, Romil Bhardwaj, Zongheng Yang, Scott Shenker, and Ion Stoica. Skyserve: Serving ai models across regions and clouds with spot instances. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys ’25, page 159–175, New York, NY, USA, 2025. Association for Computing Machinery.
[19] Youhe Jiang, Fangcheng Fu, Xiaozhe Yao, Guoliang He, Xupeng Miao, Ana Klimovic, Bin Cui, Binhang Yuan, and Eiko Yoneki. Demystifying cost-efficiency in llm serving over heterogeneous gpus. arXiv preprint arXiv:2502.00722, 2025. [20] Youhe Jiang, Fangcheng Fu, and Eiko Yoneki. Boute: Cost-efficient llm serving with heterogeneous llms and gpus via multi-objective bayesian optimization. arXiv preprint arXiv:2602.10729, 2026.
[29] Yixuan Mei, Yonghao Zhuang, Xupeng Miao, Juncheng Yang, Zhihao Jia, and Rashmi Vinayak. Helix: Serving large language models over heterogeneous gpus and network via max-flow. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, ASPLOS ’25, page 586–602, New York, NY, USA, 2025. Association for Computing Machinery.
[21] Youhe Jiang, Ran Yan, Xiaozhe Yao, Yang Zhou, Beidi Chen, and Binhang Yuan. Hexgen: generative inference of large language model over heterogeneous environment. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024.
[30] Meta AI. The llama 4 herd: The beginning of a new era of natively multimodal AI innovation. https://ai.m eta.com/blog/llama-4-multimodal-intellige nce/, April 2025. Accessed: 2026-04-20.
[22] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024.
[31] Meta Platforms, Inc. Gloo: A collective communications library. https://github.com/pytorch/gloo, 2017.
[23] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
[32] Xupeng Miao, Chunan Shi, Jiangfei Duan, Xiaoli Xi, Dahua Lin, Bin Cui, and Zhihao Jia. Spotserve: Serving generative large language models on preemptible instances. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, pages 1112–1127, 2024.
[24] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.
[33] Microsoft Corp. Microsoft and OpenAI extend partnership. https://blogs.microsoft.com/blog/2023/ 01/23/microsoftandopenaiextendpartnership/, January 2023. Accessed: 2026-04-20. 14
[34] NVIDIA. NVIDIA Collective Communication Library (NCCL). https://github.com/NVIDIA/nccl, 2015.
[44] Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Robert BusaFekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, Iain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan-Thorsten Peter, Danila Sinopalnikov, Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky, Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alex Feng, Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic, Amit Vadi, András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang, Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Choo, CJ Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng, Dimitris Paparas, Divyashree Shivakumar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Klimczak-Plucińska, Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hussein Hazimeh, Ian Ballantyne, Idan Szpektor, Ivan Nardini, Jean Pouget-Abadie, Jetha Chan, Joe Stanton, John Wieting, Jonathan Lai, Jordi Orbay, Joseph Fernandez, Josh Newlan, Ju yeong Ji, Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Greff, Linhai Qiu, Marcella Valentine, Marina Coelho, Marvin Ritter, Matt Hoffman, Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma, Nabila Babar, Natasha Noy, Nathan Byrd, Nick Roy, Nikola Momchev, Nilay Chauhan, Noveen Sachdeva, Oskar Bunyan, Pankil Botarda, Paul Caron, Paul Kishan Rubenstein, Phil Culliton, Philipp Schmid, Pier Giuseppe Sessa, Pingmei Xu, Piotr Stanczyk, Pouya Tafti, Rakesh Shivanna, Renjie Wu, Renke Pan, Reza Rokni, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Girgin, Shariq Iqbal, Shashir Reddy, Shruti Sheth, Siim Põder, Sijal Bhatnagar, Sindhu Raghuram Panyam, Sivan Eiger, Susan Zhang, Tianqi Liu, Trevor Yacovone, Tyler Liechty, Uday Kalra, Utku Evci, Vedant Misra, Vincent Roseberry, Vlad Feinberg, Vlad Kolesnikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow, Yuvein Zhu, Zichuan Wei, Zoltan Egyed, Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Kat Black, Nabila Babar, Jessica Lo, Erica Moreira, Luiz Gus-
[35] NVIDIA. Developing a Linux Kernel Module using GPUDirect RDMA. NVIDIA Corporation, 2026. [36] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative llm inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pages 118–132. IEEE, 2024. [37] Perplexity Team. Inside the rise of enterprise ai modelswitching. https://www.perplexity.ai/hub/blog /inside-the-rise-of-enterprise-ai-model-s witching, Feb 2026. Accessed: 2026-04-20. [38] Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Heyi Tang, Feng Ren, Teng Ma, Shangming Cai, Yineng Zhang, Mingxing Zhang, et al. Mooncake: A kvcachecentric disaggregated architecture for llm serving. ACM Transactions on Storage, 2024. [39] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023. [40] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. [41] Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. Openai gpt-5 system card. arXiv preprint arXiv:2601.03267, 2025. [42] Jovan Stojkovic, Chaojie Zhang, Íñigo Goiri, Josep Torrellas, and Esha Choukse. Dynamollm: Designing llm inference clusters for performance and energy efficiency. In 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA), pages 1348– 1362. IEEE, 2025. [43] Foteini Strati, Zhendong Zhang, George Manos, Ixeia Sánchez Périz, Qinghao Hu, Tiancheng Chen, Berk Buzcu, Song Han, Pamela Delgado, and Ana Klimovic. Sailor: Automating distributed training over dynamic, heterogeneous, and geo-distributed clusters. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles, SOSP ’25, page 204–220, New York, NY, USA, 2025. Association for Computing Machinery. 15
tavo Martins, Omar Sanseviero, Lucas Gonzalez, Zach Gleicher, Tris Warkentin, Vahab Mirrokni, Evan Senter, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel, Noam Shazeer, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Jean-Baptiste Alayrac, Rohan Anil, Dmitry, Lepikhin, Sebastian Borgeaud, Olivier Bachem, Armand Joulin, Alek Andreev, Cassidy Hardin, Robert Dadashi, and Léonard Hussenot. Gemma 3 technical report, 2025.
[52] Shan Yu, Jiarong Xing, Yifan Qiao, Mingyuan Ma, Yangmin Li, Yang Wang, Shuo Yang, Zhiqiang Xie, Shiyi Cao, Ke Bao, et al. Prism: Unleashing gpu sharing for cost-efficient multi-llm serving. arXiv preprint arXiv:2505.04021, 2025. [53] Yihui Zhang, Han Shen, Renyu Yang, Di Tian, Yuxi Luo, Menghao Zhang, Li Li, Chunming Hu, Tianyu Wo, Chengru Song, and Jin Ouyang. Cauchy: A costefficient llm serving system through adaptive heterogeneous deployment. In Proceedings of the 2025 ACM Symposium on Cloud Computing, SoCC ’25, page 881–893, New York, NY, USA, 2026. Association for Computing Machinery.
[45] Yuxin Wang, Yuhan Chen, Zeyu Li, Xueze Kang, Yuchu Fang, Yeju Zhou, Yang Zheng, Zhenheng Tang, Xin He, Rui Guo, et al. Burstgpt: A real-world workload dataset to optimize llm serving systems. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 5831–5841, 2025.
[54] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. {DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 193–210, 2024.
[46] Zhanghao Wu, Wei-Lin Chiang, Ziming Mao, Zongheng Yang, Eric Friedman, Scott Shenker, and Ion Stoica. Can’t be late: Optimizing spot instance savings under deadlines. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), pages 185–203, 2024.
[55] Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931, 2024.
[47] Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. Science China Information Sciences, 68(2):121101, 2025. [48] Yuxing Xiang, Xue Li, Kun Qian, Yufan Yang, Diwen Zhu, Wenyuan Yu, Ennan Zhai, Xuanzhe Liu, Xin Jin, and Jingren Zhou. Aegaeon: Effective gpu pooling for concurrent llm serving on the market. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles, pages 1030–1045, 2025. [49] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. [50] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations, 2022. [51] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for {Transformer-Based} generative models. In 16th USENIX symposium on operating systems design and implementation (OSDI 22), pages 521– 538, 2022. 16