arXiv CSopen access

Frontier: Towards Comprehensive and Accurate LLM Inference Simulation

Frontier: Towards Comprehensive and Accurate LLM Inference Simulation Yicheng Feng1

Xin Tan1

Yangtao Deng1

Yimin Jiang2

arXiv:2605.21312v1 [cs.DC] 20 May 2026

1 The Chinese University of Hong Kong

Abstract

2 Anuttacon

Hong Xu1

3 StepFun

the disaggregated architectures. Most notably, Prefill-Decode Disaggregation (PDD) decouples the compute-bound prefill from the memory-bound decode and places them on separate GPU pools connected via KV-cache transfers [3, 51]. Pushing this further, Attention-FFN Disaggregation (AFD) separates the attention and feed-forward computations during the decode phase, routing activations between distinct hardware pools [40, 52]. Alongside these architectural shifts, deployments are heavily augmented with complex parallelism strategies (tensor (TP) [38], pipeline (PP) [35], data (DP), and expert (EP) [29] parallelism) and diverse runtime optimizations—continuous batching [18], PagedAttention [28], CUDA Graphs [10], prefix caching [50], chunked prefill [19], and speculative decoding [30]. The serving workloads are also becoming more diverse at the same time. Sparse MoE [31, 33, 39] models are widely used that activates a different subset of experts for each token at each layer. Reasoning models produce long hidden reasoning chains before yielding the final answer [26]. Agentic systems introduce complex tool calls and multi-turn session states [42, 46]. Furthermore, RL rollouts for post-training create workload bursts, long decode tails, and shifting phase mixes [41, 48]. Consequently, the interplay of complex workloads, advanced runtime optimizations, and disaggregated serving architectures has made production deployments exceedingly difficult to model, reason, and optimize. As a result, simulation as an essential performance modeling approach has received increasing attention [17, 20, 21, 45]. It provides a cheap and fast way to characterize a serving system without the need for expensive real deployment (at scale) [17, 20, 45]. Existing simulators, however, fall short in two dimensions that we believe make them fundamentally ineffective as a performance modeling tool in many practically crucial scenarios. First, architectural completeness. Most simulators (e.g., Vidur [17], APEX [32]) lack support for disaggregated serving and many of the runtime optimizations discussed above, and these capabilities are not easy to retrofit. The root cause is architectural: they assume a set of homogeneous, monolithic replicas executing uniform operations. That abstraction is fundamentally mismatched to disaggregated systems, where different GPU

Modern LLM serving is no longer homogeneous or monolithic. Production systems now combine disaggregated execution, complex parallelism, runtime optimizations, and stateful workloads such as reasoning, agents, and RL rollouts. Simulation is attractive for exploring this growing design space, yet existing simulators lack the architectural completeness and decision-grade fidelity it demands. Their monolithicreplica abstractions are ill-suited to disaggregated serving, while average-case analytical proxies can distort SLA predictions and even reverse optimization conclusions. We present Frontier, a discrete-event simulator for modern LLM inference serving. Frontier features a disaggregated abstraction. It captures the structure and dynamics of modern serving systems by modeling co-location, Prefill-Decode Disaggregation (PDD), and Attention-FFN Disaggregation (AFD) with role-specific cluster workers, incorporating key runtime optimizations (e.g., CUDA Graphs, speculative decoding) within the scheduler-batch-engine loop, and supporting stateful requests for emerging workloads. It further provides accurate and generalizable predictions of computation, communication, and memory costs across diverse serving scenarios with complex workload compositions. On 16-H800 GPU testbed, Frontier achieves an average throughput error below 4%. Compared with state-of-the-art simulators, it reduces end-to-end latency error from 44.9% to 6.4% under co-location and from 51.7% to 2.6% under disaggregation. It scales to over 1K GPUs on commodity CPUs and enables new use cases such as SLA-dependent Pareto frontier exploration, heterogeneous disaggregated allocation, agentic reasoning scheduling validation, and RL post-training reconfiguration.

1

Yibo Zhu3

Introduction

Inference has become a first-order production cost for LLM service providers amid the explosive growth of token use from not just chatbots, but also coding and computer use agents, which is particularly evident since 2026 [20, 45]. This pressure has driven the evolution of serving systems in multiple fronts: The serving architecture has moved beyond the traditional co-located paradigm, where both the prefill and decode phases execute within a single monolithic engine, to 1

pools execute distinct roles in parallel and interact through explicit data transfer and synchronization. Likewise, runtime optimizations are entangled with scheduler-visible state and therefore must be modeled as first-class behavior rather than analytical patches. Approaches such as AIConfigurator [45] can therefore incur large errors and even predict the wrong optimization trend (§2.2). More broadly, because serving workloads are stateful and phase-changing [42, 46], treating inference as an open-loop sequence of regular operator executions is architecturally incomplete. Second, decision-grade fidelity. Existing simulators usually deliver coarse average-case accuracy, but production use demands predictions that remain faithful under realistic serving dynamics [17]. As discussed, LLM serving workloads are dynamic and stateful: continuous batching, variable-length requests, KV-cache growth, prefill/decode asymmetry, scheduling interference, and memory pressure jointly shape latency, memory usage, and queue evolution. In this closed loop, even modest per-operator or per-batch errors can become decision-changing: they can shift time-to-first-token (TTFT)/time-per-output-token (TPOT) or memory estimates across service level agreement (SLA) boundaries, distort the Pareto frontier, and lead to the wrong configuration choice (§2.2). The core problem is that existing inference simulators smooth these dynamics into average-case models, especially for operator runtime and KV-cache budgeting: token-count proxies can cause 32.6% errors on FlashAttention and 21.0% on MoE GroupedGEMM, while “total minus weights” memory models overestimate KV-cache budget by up to 27.2% and throughput by up to 32.4% (§2.2). We build Frontier, a discrete-event simulator that addresses both limitations through two complementary design principles: (1) for architectural completeness, Frontier abandons the monolithic-replica abstraction. A control plane compiles a serving specification into role-specific cluster workers that explicitly model the domain layout and synchronization dependencies of co-located, PDD, and AFD architectures. Runtime optimizations are captured as feature-specific adapters inside the scheduler–batch-engine loop, directly updating scheduler-visible state rather than applied as analytical afterthoughts. To represent increasingly diverse workloads, Frontier introduces a stateful request abstraction where each request can carry thinking rounds, tool-call delays, perround token plans, and phase transitions, naturally modeling reasoning, agentic, and RL-rollout workloads. (2) for decisiongrade fidelity, Frontier introduces a fidelity plane that replaces coarse average-case proxies with calibrated, hardwareaware predictors. Operator runtimes, collective costs, transfer delays, and KV-cache budgets are each resolved through profiled models grounded in actual CUDA kernel behavior and memory footprints, closing the error gaps that make average-case approximations decision-changing. We implemented Frontier with roughly 70K LoC in Python. On a 16-GPU H800 testbed, we evaluate Frontier across dense

Frontier

AC

LS

VD

AP

Architecture Co-located Serving PDD Serving AFD Serving

✓ ✓ ✓

✓ ✓ ✗

✓ ✓ ✗

✓ ✗ ✗

✓ ✗ ✗

Parallelism PP / TP / DP MoE / EP DP Attention

✓ ✓ ✓

✓ ✓ –

✓ ✓ ✗

✓ ✗ ✗

✓ – ✗

Runtime Optimizations Speculative Decoding / MTP CUDA Graph Prefix Caching Quantization Chunked Prefill Hierarchical Caching

✓ ✓ ✓ ✓ ✓ ✓

– ✗ – ✓ – ✗

✗ ✗ ✓ ✗ ✗ ✓

✗ ✗ ✗ ✗ ✓ ✗

✗ ✗ ✗ ✓ ✗ ✗

Simulation Features Op/Mem. Fidelity Ex-situ Simulation Thinking/Reasoning Modeling

✓ ✓ ✓

– ✓ ✗

– ✓ ✗

– ✗ ✗

– ✓ ✗

Table 1. Comparison with representative simulators. AC: AIConfigurator [45], LS: LLMServingSim2.0 [21], VD: Vidur [17], AP: APEX [32]. ✓: supported, ✗: unsupported, –: partial/limited.

and MoE models, serving architectures, and varied workloads. Frontier achieves average throughput errors below 4% in both co-located and disaggregated settings with modern runtime optimizations enabled. On the SharedGPT trace, Frontier reduces the average end-to-end (E2E) error from 44.9% to 6.4% under co-location, and from 51.7% to 2.6% under PDD/AFD, compared with state-of-the-art simulators. Beyond fidelity, Frontier scales to large-scale clusters (e.g., 1K+ GPUs) even on commodity CPU machines, making it practical to study deployment questions that prior simulators cannot reproduce as configured. Under traditional non-reasoning batch serving, Frontier enables SLA-aware Pareto-frontier analysis across co-location, PDD, and AFD, and evaluates heterogeneous PDD/AFD allocations by separating nominal cost reduction from SLA-safe efficiency. Under emerging stateful workloads, Frontier supports mechanism studies for multi-round agentic reasoning schedulers and quantifies rollout-tail reconfiguration opportunities for RL post-training (§6).

2

Background and Motivation

2.1

Why Inference Simulation Matters

The LLM serving has pushed industry toward aggressive system–model–hardware co-design [26, 40]. The design space is exploding. For a 200B-parameter MoE model, plausible configurations easily reach 1k+ combinations [45]. On a 64GPU H100 cluster at current cloud pricing, sweeping 100 configurations takes 12,800–25,600 GPU-hours and costs up 2

Inflation

Co-location

7K 100K 29K 28K

22.6% 38.7% 45.8% 22.6%

-42.4%

-32.3%

10

-36.0% 15

-46.5%

10

5

-37.1% -59.7%

-50.2%

-40.7%

2048/256 256/2048 512/512 1024/1024

14K 111K 37K 52K

42.6% 42.5% 57.2% 40.0%

5 2048/256 256/2048 512/512 1024/1024

0

2048/256 256/2048 512/512 1024/1024

Workload (ISL/OSL)

Workload (ISL/OSL)

PDD

Figure 1. Measured vLLM TPOT with and without CUDA Graph under differ- Table 2. CUDA Graph ent workloads (64 requests per work- decode padding overhead. load, mean ISL/OSL, tested on 8×A800- Padding: wasted slots from SXM GPUs). Left: co-location. Right: batch alignment. Inflation: PDD. Percentages show reduction. extra vs. useful tokens.

Co-located Serving +48.5%

+34.6%

15 10

-43.7%

-46.4%

+61.0%

-31.8%

vLLM CUDA Graph

Vidur

AIConfigurator

-46.4%

10

-27.8%

-46.5%

-29.7%

-41.3%

256/2048

512/512

Workload (ISL/OSL)

1024/1024

0

1.0

0.6

0.0

0.6

1

2

1.0

60

8

Batch Size

40

0.9 0.7

0.6

0.5

16

32

256/2048

512/512

1024/1024

3.8

3.4

3.3

3.1

3.0 2.0

1.6

1.5

60

2.8

1.5

1.4

80

1.7

20 1.0 0

0.0

40 20

1

2

8

Batch Size

KV Budget

Linear ops GroupedGEMM Attention

0.50

0.00

2048/256

1.0

0.25

5 2048/256

4.0

1.4

0.75

15

5 0

20

1.7

Acceptance Ratio: 80%

80

1.00

PD Disagg. Serving +41.8%

vLLM AIConfigurator Error

16

32

0

Figure 3. Qwen3-30B MoE MTP decode speedup (verify_tokens=4). Bars (left axis): vLLM vs. AIConfigurator. Dashed line: prediction error. Red ×: sign-mismatch (measured speedup > 1, predicted slowdown < 1). Ground-truth acceptance rates and fixed batch sizes are used to isolate other modeling errors.

CDF

TPOT (ms)

20

Acceptance Ratio: 20% 2.0

Error (%)

Padding

2048/256 256/2048 512/512 1024/1024

Workload (ISL/OSL)

Linear: med 0.8%, mean 3.2% Grouped: med 19.8%, mean 21.0% Attention: med 22.3%, mean 32.6%

0

50

100

Error (%)

150

200

+8.1% 40 Groundtruth Analytical 20

0

+27.2%

Llama3.1-8B Qwen3-30B

Throughput Normalized Value

ISL/OSL

20

15

0

Mode

25

20

KV budget (GB)

TPOT (ms)

PD Disagg. Serving 30

vLLM Eager vLLM CUDA Graph

25

Decode speedup

Co-located Serving 30

1.0

+23.6%

+32.4%

0.5

0.0

Llama3.1-8B Qwen3-30B

Figure 4. Fidelity gaps caused by simplified modeling. Left: Relying on coarse proxies (total token count) fails to capture batch heterogeneity, yielding coarse-grained performance estimates. Right: Analytical KV-cache modeling overestimates effective memory budget, leading to a cascade of errors from admission control to overoptimistic throughput projection.

Figure 2. Existing simulators vs. vLLM CUDA Graph. Percentages represent error.

to $180,000 [17, 20]. In practice, most teams abandon systematic tuning and fall back on conservative defaults, leaving 3–5× throughput on the table [34, 45, 49]. A well-built simulator provides value far beyond replacing expensive hardware sweeps. By making hardware–software interactions explicit and event-level state observable, it can support Pareto search under SLA constraints, policy prototyping and diagnosis, dynamic workload analysis, and pre-deployment validation of heterogeneous placement or disaggregation plans. In practice, simulators can help one team map the frontier, another debug an agentic scheduler or RL rollout policy, and a third check whether a disaggregated deployment still behaves as intended before rollout.

activation transfers become causal edges, and MoE EP introduces routing-dependent synchronization. Existing simulators, however, are typically built around a homogeneousreplica abstraction, where each replica owns the full model path, scheduler state, and local batch execution loop. While sufficient for co-located serving, this abstraction cannot express PDD/AFD by simply adding a latency term; it requires role-specific workers, cross-cluster dependencies, and separate parallel domains as first-class execution objects. Completeness also breaks at the runtime-behavior level. Runtime optimizations are often treated as optional speedup knobs, but they change scheduler-visible state and batch execution semantics. For example, vLLM captures decode CUDA Graphs at fixed batch-size bins {1, 2, 4, 8, 16, 32, 64}; a pure-decode step between bins is replayed at the next larger bin, so 33 active requests execute as a 64-slot graph. As Figure 1 shows, CUDA Graph reduces decode TPOT by 32.3%−46.5% under co-location and 37.1%−59.7% under PDD, but its padding increases decode-token work by 22.6%−45.8% and 40.0%−57.2%, respectively (Table 2). This padding also changes runtime memory usage and KV-cache state for admission, chunking, and preemption. Ignoring CUDA Graph therefore yields an eager-style execution path and the workloadinsensitive TPOT surface in Figure 2. Analytical shortcuts fail similarly. For speculative decoding, even with vLLM’s per-step Qwen3-30B MoE multi-token prediction (MTP) [33] acceptance rates2 , an analytical MTP model deviates by

2.2 Challenges Table 1 summarizes the comparison between Frontier and existing simulators.1 Building upon the analysis of existing simulators in §1, we identify two primary challenges that must be addressed to realize a practical simulation framework. Unless specified, experiments use vLLM v0.10.2 (V1 engine) on an 8×A800 GPU server. Challenge 1: How to model the complex runtime behaviors and control flows of advanced serving systems? Completeness first breaks at the serving-architecture level. Disaggregated serving architectures (PDD/AFD) can achieve up to a 7.4× throughput improvement over traditional colocated serving [51]. This leap is driven by a fundamental shift from monolithic execution to role-specific cluster decoupling [40, 52]. These architectures change the event graph itself: requests traverse role-specific clusters, KV-cache or 1 In cases of discrepancy between the paper’s claims and the latest open-

2 Since AIConfigurator lacks native support for Qwen’s MTP, we extend it

source repository as of April. 2026, we refer to the latter.

by adapting its default MTP module to the Qwen MoE. 3

APEX candidates

APEX best

vLLM ground truth

Frozen SLA

budgeting used by most existing simulators overestimates the available KV-cache budget by 8.1% for Llama-3.1-8B and 27.2% for Qwen3-30B MoE. Since the scheduler uses this budget to bound concurrent decode slots, this initialization error inflates simulated decode throughput by 23.6% and 32.4%, respectively—enough to alter capacity planning decisions. Overall, per-component errors propagate through the closedloop dynamics into decision-grade failures. As Figure 5 shows, optimistic operator-runtime and KV-cache budget predictions can lead the simulator to select a “best” configuration that violates the SLA on the real system—precisely the failure mode simulation is meant to prevent.

8

P95 TPOT (ms)

20

vLLM violates SLA 15

6

SLA Region

12

4

APEX selects 76 APEX candidates pass SLA Best: qps=16, tp=8, dp=2, max seqs=32

7.5 25

32

40

2

APEX throughput (req/s)

Value drift (vLLM - APEX): ΔTTFT=+17.54 ms, ΔTPOT=+8.53 ms

50

P95 TTFT (ms)

Figure 5. Decision drift from fidelity gaps on Llama-3.1-8B over 16 H800 GPUs (co-location). The simulator-selected best configuration lies inside the frozen SLA region, but the corresponding vLLM ground-truth point moves outside the SLA because overly optimistic comp-op and KV-cache budget predictions underestimate request latency. Note that we maintain feature parity in this evaluation by disabling vLLM optimizations unsupported by APEX.

3

Frontier Design

Frontier is a discrete-event simulator designed for LLM serving. It features a disaggregated abstraction that separates hardware-aware prediction from the control and execution planes. This decomposition is grounded in a key observation: production serving engines (e.g., vLLM [28], SGLang [50]) share a common control-flow skeleton—admission, batching, execution, completion—yet differ mainly in the scheduling policies they apply and the operator backends they invoke. Frontier exploits this shared structure: it can integrate framework-specific scheduling policies and operator backends without modifying the simulation core.

27.2%−66.6% and predicts the opposite trend on 4 of 5 lowacceptance batches (Figure 3). Its scalar expectation formula flattens the non-linear speedup from event-driven variable commits, causing large cost overestimation under low acceptance rates. Challenge 2: How to accurately model the computation and memory dynamics within a closed-loop serving system? LLM inference serving is a closed-loop system: execution cost shapes memory state, memory constrains admission, and admission governs queueing and future batch composition. Within this loop, modest per-event errors can accumulate into different operating regimes. We find that fidelity breaks at two levels—per-operator runtime prediction and engine-level memory accounting—both of which are necessary for decision-grade simulation. At the operator level, the key challenge is generalization: batch composition is determined online by the scheduler, making the execution space infeasible to profile exhaustively. Existing simulators therefore rely on predictors fitted from primary operator profiles [17, 21, 32, 45], using coarse proxy features such as total token count while ignoring per-request variance within a batch. As shown in Figure 4 (left), this leads to substantial errors under dynamic workloads; for example, Vidur’s token-count predictor has a mean error of 32.6% for FlashAttention [23] (0.151 ms predicted vs. 0.224 ms measured), with similarly large errors for MoE GroupedGEMM due to data-dependent router load distributions. Such errors compound across layers and decode steps: even 0.1 ms per operator can add tens of milliseconds per request, exceeding the margins between competing serving configurations. At the engine level, memory fidelity determines how the scheduler admits, chunks, and preempts requests. The KVcache budget is the residual GPU memory after allocating weights, activation buffers, CUDA Graph capture regions, and framework-internal overheads; approximating it as “total minus weights” shifts the scheduler’s operating point. In our scheduler-isolation replay (Figure 4, right), the analytical

3.1

Simulator Overview

System architecture. Figure 6 shows the system architecture. Frontier is organized into four modules. ➀ Workload and Config. It defines the serving scenario—model, serving architecture, parallelism, runtime options, and request workloads. ➁ Fidelity Plan. It provides calibrated per-operator, communication, and memory-capacity (KV-cache budget) predictions that the execution plane queries on demand. ➂ Control Plane. It provides disaggregation primitives and dependency-based control flow. It translates these specifications into a functional simulation and coordinates its execution. ➃ Execution Plane. It enables feature-specific execution flows, advancing requests through scheduling, batching, runtime adaptation, and cross-cluster transfers. System workflow. The workflow involves four steps. Configuration and compilation. The user specifies the serving scenario—model, hardware, serving architecture, parallelism, runtime features, and request workloads—through CLI parameters, alongside calibrated profiles consumed by the Fidelity Plane. The Simulation Compiler in the Control Plane reads these inputs and instantiates the simulation topology. Two abstractions map the simulation to a real serving deployment: a Cluster Worker represents a logical device group serving a specific role (e.g., a prefill cluster, a decode cluster, or a co-located cluster), and each contains multiple Replica Workers, each of which models a serving replica spanning GPUs under the configured parallelism. 4

Event-driven simulation loop. Each Cluster Worker is driven by a per-cluster discrete-event engine that maintains a priority event queue and advances events in timestamp order. For disaggregated serving architectures, the per-cluster drivers run in parallel (one thread per cluster) and coordinate through inter-cluster event queues. Once initialization completes, the workload generator injects request arrival events into the cluster queues, triggering the simulation loop. Per-iteration request processing. When a request arrival fires, the Replica Worker’s Scheduler decides admission and batch composition; Runtime Adapters then reshape the batch before execution (§3.3). The Batch Engine queries the Fidelity Plane for per-operator runtime predictions (both computation and communication), producing the iteration’s completion time (§3.4). For disaggregated systems, batch completion triggers a cross-cluster transfer event; the downstream cluster resumes processing only after the transfer completes. Metrics and output. A Metric Tracker records each request’s history across its entire lifecycle—spanning admission, batching, iteration execution, and cross-cluster transfers. When the simulation finishes, the simulator outputs per-request latencies, batch traces, TTFT/TPOT breakdowns, throughput, E2E makespan, and memory utilization reports. 3.2

Workload & Config

1

Profiles

2

Workload Generator (Chat,Agent, Reasoing)

Comm. Backend

Mem. Capacity

3 Control Plane

Serving & Model Config. Infer. Runtime Config.

Fidelity Plane

Comp. Op Libs

Global Controller

Sim. Complier

Metric Tracker

Sim. Cluster Worker Event Driver

Replica Worker Replica ReplicaWorker Worker 4 Execution Plane

Scheduler

Replica Worker Model Runner KV-Cache Manager

Batch Engine

Runtime Adapter CUDA Graph Dispatcher

SpecDecode Runtime

Block-Hash Prefix Cache

Chunked Prefill

Transfer Trigger

Request Queue

Control Plane Figure 6. Frontier system architecture.

The Control Plane compiles a user-level serving specification into a runnable simulation by instantiating workers, binding communication domains, wiring cross-stage event dependencies, and setting Execution Plane capacity constraints. Input compilation. The Simulation Compiler resolves the model, hardware, serving architecture, runtime features, workload generator, and backend choices into typed simulation objects. Co-location yields a monolithic cluster; PDD yields prefill and decode clusters; AFD splits decode into attention and FFN (MoE) clusters. Table 3 lists two representative configurations. Parallel primitives. Production engines expose parallelism through different and often implicit vocabularies; a simulator that compiles all of them onto a single backend needs a parallelism expression that is both explicit—no coupling hidden inside the runtime—and expressive enough to subsume every framework’s sharding semantics without ambiguity. Frontier introduces a two-domain decomposition for this purpose: a replica is parameterized by the pipeline size pp together with an attention sharding tuple (tpattn, dpattn ) and an FFN sharding tuple (tpffn, epffn ), where epffn degenerates to dpffn on dense models. This decomposition is the unit at which the Fidelity Plane and the communication backend are queried. Each serving architecture maps these two domains onto a distinct set of cluster roles, denoted in sans-serif throughout: co-location uses a single cluster C hosting both domains; PDD uses a prefill cluster P and a decode cluster D, each independently hosting both domains; AFD uses a prefill cluster P, an attention cluster A, and an FFN cluster F, each hosting

a single domain. Whenever a role 𝑐 hosts both domains, the two sharding products must span the same device set: tpattn · dpattn = tpffn · epffn,

𝑐 ∈ {C, P, D}.

The per-replica world size of each cluster role is then ( pp · tpattn · dpattn, 𝑐 ∈ {C, P, D, A}, 𝑐 𝑊𝑅 = pp · tpffn · epffn, 𝑐 = F,

(1)

(2)

where the two branches agree on C/P/D by Eq. 1. Each role 𝑐 instantiates 𝑁𝑅𝑐 replicas (the deployment pod count), so the Í total simulated system size is 𝑐 𝑁𝑅𝑐 · 𝑊𝑅𝑐 . At initialization, every active parallel axis is registered as a domain group with the Comm. Backend, so that a later collective query (attention all-reduce, FFN all-to-all, etc.) resolves to a backend-native group of the correct shape. Dependency definition. The Control Plane materializes five classes of cross-stage dependency as explicit events, so that queueing and synchronization follow directly from the event graph. (i) Pipeline stages. Within a replica, the 𝑝𝑝 stages form a strict chain: a stage-end event on stage 𝑠 schedules stage 𝑠+1, with one microbatch of lag between adjacent stages. (ii) PDD KV-cache transfer. Prefill completion of a request emits a KVCacheTransferStart event to the decode cluster; the matching End event admits the request into decode scheduling. Transfer latency is queried from the Fidelity Plane using KV-cache size, link topology, and current transfer concurrency. (iii) AFD activation transfer. 5

Model

Chunked Speculative Serving Config Other Runtime Features ... Prefill Decoding Parallel Batch Size Max Batch GPU Mem. Token Accept Verify MoE CUDA Prefix Thinking Tool Hier. Quant. ... Param. Cap Tokens Util. Thresh. Ratio Tokens Routing Graph Cache Mode Call Cache 512×H100 P: pp16/tp8/dp2 CPU Fat-tree vLLM FlashInfer HTSim PDD 256 16K 0.90 2K 30% 16 Balanced ✓ ✓ – – FP8 ... 512×H20 D: pp16/tp8/dp6 offload P: pp8/tp8/dp8/ep64 1024×H800 p95 CPU A: pp8/tp8/dp4 Fat-tree SGLang Triton ASTRA-Sim AFD 512 8K 0.85 1K 25% 4 Skew ✓ ✓ depth 4 FP16 ... 1024×H20 200ms offload F: pp8/tp8/dp32/ep256 Hardware

Name

Param. Size

Llama 3.1

405B

Qwen3

235B

GPU

Engine Backend

Topo. Runtime

Attn. Backend

Comm. Backend

Sys. Arch.

Table 3. Representative simulator configurations. Column names follow the corresponding Frontier configuration fields.

Every decode iteration emits paired M2NTransferStart/ End events between the attention and FFN clusters. (iv) MoE EP synchronization. For every MoE layer, Frontier emits per-rank AllToAllCombineReady events and only fires the combine collective once the slowest rank has reported ready; straggler waits therefore arise from per-rank prediction rather than from a separate imbalance model. In the Appendix (§A.3), we detail the workflows and dependencies of disaggregated architectures (PDD and AFD). Agentic Reasoning. Frontier introduces a stateful request abstraction for new workloads such as reasoning, tool-using agents, and RL rollouts. The Control Plane treats multiround thinking/reasoning as a per-request event chain. Under reasoning, a request carries (a) a number of reasoning rounds to execute, (b) a tool-execution latency attached to each inter-round tool call, and (c) a per-round plan of prefill/decode token counts. Each intermediate round runs as an ordinary prefill→decode cycle; on completion it emits a ThinkingRequeue event that re-admits the request after the tool-execution delay, with session affinity to the same replica so that the KV cache populated in the previous round can be served as a prefix-cache hit. The final round produces the user-facing response. Runtime invariants. Before any batch is formed, the Control Plane fixes the capacity envelope used during execution (details in §3.4). It also records runtime contracts later used by admission, chunking, and preemption, including CUDA Graph capture sizes, prefix-cache eligibility, and speculativedecoding token allowances. Memory safety is enforced both at initialization and during execution: the KV-cache Manager aborts launch if weights do not fit or the resolved KVcache block count is zero, while the Scheduler checks block availability against a watermark before each admission and triggers preemption instead of overcommitting capacity.

3.3

Execution Plane

The Execution Plane advances the request lifecycle within each replica: admission, batch composition, and per-iteration cost resolution. It schedules workloads under control-plane constraints and supports runtime optimization via an adapter. Scheduler. Frontier implements the Scheduler as a standalone module, following production engines such as vLLM [28] and SGLang [50], while keeping their scheduling logic unchanged. Only the I/O layer is rewired— the request queue is fed by simulated arrivals, the batches are consumed by the DES event loop, and memory pressure is read from a 6

simulated KV-cache block counter. Preserving the mechanism is what lets Frontier reproduce engine-specific behavior under pressure (e.g., vLLM watermark preemption [16], SGLang’s admission rules [14]). We provide additional simulation results on large-scale clusters comparing SGLangand vLLM-v1-style schedulers in the Appendix (§B.4). Runtime adapters. Frontier carves out Runtime Adapters as an abstraction through which advanced engine features are supported. An adapter is a feature-specific rule attached to the Scheduler–Batch Engine loop. Each adapter declares how one optimization changes a well-defined slice of the loop: (i) the scheduler-visible state (e.g., which prompt blocks count as already computed), (ii) the batch shape the Batch Engine queries the Fidelity Plane with (e.g., the padded size under CUDA Graph), or (iii) per-request progress (e.g., how many decode tokens commit per step). We describe two in detail; the remaining ones follow the same pattern. CUDA Graph. Captured graphs remove per-kernel launch overhead only for captured batch shapes. The adapter pads each decode batch to the next larger captured size, then queries the Fidelity Plane with the kernel-only measurement family provided by the computation operator lib (§3.4). Non-graph executions use the launch-inclusive family instead. This models CUDA Graph’s coupled effects: padding increases compute work, while graph replay removes launch overhead. A scalar speedup factor cannot capture both. Speculative decoding (MTP). The adapter maintains perrequest state (planned, verified, accepted, committed token counts). For MTP, each decode step becomes a draft→verify →commit cycle in which the verify phase is a prefill-like forward pass that can share a batch with ordinary decode work. Per-request accounting lets different requests within the same batch carry different speculative depths and different acceptance outcomes, which is a prerequisite for capturing the batch-shape variance that shapes tail latency under speculative workloads. Other runtime features use the same interface. Prefix caching marks matched block hashes as already computed before admission and updates the cache when full blocks complete; chunked prefill caps long-prompt progress by the iteration token budget. 3.4

Fidelity Plane

The Fidelity Plane is a runtime-estimation layer queried by the Execution Plane for every batch iteration and crosscluster transfer. It has three components: a Compute Operator Library that predicts per-op runtime, a Memory-Capacity

Model that enforces batch-admission constraints, and a Comm. Backend that estimates collective-communication costs. Compute operator library. Frontier classifies every modeled operator by what determines its runtime and selects a tailored prediction strategy for each class:

the representative rank. The KV-cache Manager then derives the KV-cache block budget by subtracting parameter memory (computed analytically from the weight shard) and the profiled non-KV-cache overhead (torch peak increase plus non-torch residency, beyond weights) from the requested GPU budget. The Scheduler (§3.3) consumes this budget for admission, chunked-prefill partitioning, and watermarktriggered preemption. Comm. backend. Frontier integrates existing network simulation tools (ASTRA-Sim [43] and HTSim [4]) as pluggable backends. It dynamically selects the backend based on the domain scale to balance simulation accuracy and cost. The Fidelity Plane composes flexibly with Runtime Adapters in the Execution Plane to support a variety of runtime optimizations. For CUDA Graph, the adapter switches the Fidelity Plane’s measurement mode (kernel-only vs. launchinclusive) based on whether the current batch shape hits a capture bin, accurately reflecting the captured graph’s speedup. For chunked prefill, the Fidelity Plane’s profiling subsystem supplies per-operator performance data across input shapes parameterized by chunk length, ensuring its performance impact is modeled correctly.

(i) Token-count operators (linear op family, e.g., GEMM, element-wise, norm). Their runtime is primarily determined by the number of tokens processed within the operator’s effective TP slice. The feature set reduces to num_tokens, with the TP slice resolved based on whether the operator is replicated or sharded. For this class, Frontier fits either a linear-regression or a small random-forest predictor, which is configurable per operator. (ii) Sequence-dependent operators (attention op family). Their runtime varies non-linearly with the per-request sequencelength distribution inside a batch, rather than just aggregate totals. Frontier employs random-forest regressors over richer feature vectors. The features include aggregate counts (batch size, total tokens) and distributional statistics of per-request lengths (min, max, and percentiles of prefill and decode lengths). This design captures how kernel partitioning and tile scheduling react non-linearly to length heterogeneity, addressing why single length regressions (e.g., Vidur) diverge from measured latency under chunked prefill. (iii) Routing-dependent operators (MoE op family). Their runtime heavily depends on how tokens are dynamically assigned to experts. Handled similarly by random-forest regressors, the feature vectors for MoE ops combine loadbalance statistics (variance and max of token-to-expert counts), expert-selection ratio, expert count, and model dimensions. This ensures that workload skew at the expert level is faithfully preserved rather than averaged away.

4

Implementation

Frontier is implemented in about 70K LoC of Python code, retaining Vidur’s [17] foundational modules and DES utilities while refactoring the simulator out of its monolithic-replica execution path. The core engineering work is a cluster- and global-level execution stack: role-specific cluster workers, global orchestration, cross-cluster event queues, and concrete paths for KV-cache transfer, M2N transfer, comm. backend calls, and runtime adapters. In addition, the Sim. Compiler ingests model definitions in the HuggingFace config format [5]. The compute operator library exposes a uniform predictor interface calibrated against the runtime APIs of mainstream serving stacks, covering PyTorch [36], vLLM, and FlashInfer [47]; per-operator predictors are fitted once and reused across deployments. The comm. backend is realized as a pluggable adapter behind a collective interface, with implementations over ASTRA-Sim [43] and HTSim [4], selectable per cluster without touching the simulator core. Frontier surfaces 808 typed CLI options organized into four orthogonal groups—workload, hardware, serving and runtime, and system metrics.

We use separate profiling and training subsystems for these predictors. Profiling runs operators on real GPUs in a single-GPU sharded mode. It materializes each per-rank slice locally and stubs out collectives, making collection independent of simulated cluster scale. Training then fits the predictors. Each kernel is measured in two modes: kernelonly for GPU-side duration and launch-inclusive for host-side launch overhead. Memory capacity. Frontier estimates the per-replica KVcache block budget that admission control relies on. It does so through a vLLM-style dummy profile run: the per-rank sharded model is instantiated with dummy weights and executed for one sized forward pass, and the PyTorch allocator snapshot yields three quantities—weight memory, torch peak increase, and non-torch residency (e.g., NCCL workspace buffers [9])—all measured on a single representative rank. Because the sharded forward runs in a single-GPU process, simulating a thousand-GPU deployment requires only one GPU to obtain the per- rank footprint; under pipeline parallelism, Frontier takes the worst-case slice across PP stages as

5

Evaluation

Setup. Experiments are conducted on two servers, each hosting 8×H800-SXM GPUs (400 GB/s NVLink, 400 Gb/s NDR IB per GPU). For fidelity evaluations, we use two MoE models— Qwen3-30B MoE [13] and Step3-316B [40] (MFA, 56 layers, 48+1 experts, top-3 routing)—and one dense model, Llama3.18B [7]. We use vLLM v0.10.2 with the V1 engine as the ground 7

Ops Prediction (BF16)

Ops Prediction (FP8)

Throughput w/o CG Decode tokens / sec

CDF

1.00 0.75 0.50 0.25 0.00

Frontier Attention Ops Frontier MoE Ops Vidur Attention Ops

0

1 2 Relative Error

0.0

0.1 0.2 0.3 Relative Error

0.4

107.8k 107.6k 0 10

ΔFrontier

vLLM Frontier

27.0k +0.5k (1.89%) +7.6k (27.95%) Disagg. (2,2,2,4) (1,4,1,4) 12.0k -1 (0.01%) +4.6k (39.73%)

40

Hit Ratio

30

Arch. Frontier eager Frontier +CG

200 0

vLLMFrontier

vLLMFrontier

Co-location

PDD

Workload

ΔFrontier

E-vLLM CG-vLLM

Prefill-heavy 294.8k Co-loc. Decode-heavy 294.8k SharedGPT 69.6k

307.3k 420.5k 84.2k

-6.9k (2.25%) -10.7k (2.55%) +0.3k (0.40%)

Prefill-heavy 147.5k Disagg. Decode-heavy 147.5k SharedGPT 34.9k

150.1k 167.9k 42.9k

-1.2k (0.80%) +0.6k (0.37%) -30 (0.07%)

Co-located Serving

0.4 0.2 0 0.4

5

10

15

vLLM Frontier

Runtime seconds (s) PPD Serving

VL AR TTFT (%) TPOT (%) Thpt. (%) E2E (%) 2 0.3 0.7

5.48 2.75

4.37 1.92

0.77 1.88

5.01 2.40

8 0.3 0.7

0.84 4.43

0.00 1.26

5.12 3.53

6.45 5.57

32 0.3 0.7

4.14 6.40

6.18 3.25

4.23 2.05

9.32 11.28

Table 6. Speculative decoding (MTP) fidelity on Qwen3Runtime seconds (s) 30B MoE under SharedGPT. Figure 10. Cumulative prefix-cache Values are relative errors in hit ratio over time on Qwen3-30B percent. VL: verify length; MoE under co-location and PDD. AR: forced acceptance ratio.

truth.3 We evaluate four online workload patterns by input sequence length (ISL) and output sequence length (OSL): (1) prefill-heavy (2048/256); (2) decode-heavy (256/2048); (3) balanced (1024/1024); and (4) a SharedGPT trace [15]. System performance is assessed with request-level TTFT and TPOT (reported at P95 unless stated otherwise), throughput, and workload-level E2E makespan. We provide fidelity results for other GPU types (H20) in the Appendix (§A.1 and §A.2). Baselines. We compare Frontier with four open-source simulators as mentioned in §2: • AIConfigurator [45]: The SOTA simulator from NVIDIA, combining an op database with an analytical workflow. • Vidur [17]: Microsoft’s discrete-event simulator, lacking support for MoE and disaggregated architectures. • LLMServingSim2.0 [21]: A simulator primarily focused on heterogeneous hardware systems. • Apex [32]: An emulator searching for optimal plans, lacking support for disagg. architectures and EP (limited). 5.1

vLLM eager vLLM +CG

Prefix-Cache Hit Ratio Timeline

Table 4. Initial KV-cache block Runtime seconds (s) budget accuracy for Qwen3-30B Figure 8. Available KV-cache MoE. Parallel: (PP,TP,DP,EP). Fronblocks over time under co- tier uses profiled runtime non-KVlocation (Qwen3-30B MoE, cache memory overhead in the KVSharedGPT trace), with a max cache manager, while the analytical gap of 294 blocks (115.6 MB; baseline omits the runtime profile. ΔMB = 0.393Δ𝐵, where Δ𝐵 is Percentages are relative errors to the block gap). vLLM. 20

400

ΔAnalytical

31k +7 (0.02%) +4.4k (14.10%) Co-loc. (1,8,1,8) (4,2,1,2) 58.0k +1.0k (1.76%) +12.5k (21.38%)

Hit Ratio

Available KV blocks

Mode Parallel vLLM

108.0k

3.05× 2.86×

2.95× 2.90×

600

Table 5. Compute-participating Figure 9. Decode throughput. token accounting on Qwen3-30B +CG bars stack eager through- MoE. E: eager; CG: CUDA Graph. put with the CUDA Graph Eager ΔFrontier is +0 (0%) for all gain; top labels show total rows; CG ΔFrontier is the signed delta to vLLM. +CG/eager speedup.

Figure 7. Per-operator relative-error CDF on H800 for attention, and MoE kernels under BF16 (left) and FP8 (right). Vidur is undefined for MoE ops, and its baseline cannot be evaluated under FP8. KV Cache Blocks Over Time

800

0.2 0

2

4

6

reaches a p50/p95 of 55.4%/376.1% and is undefined for GroupedGEMM; FP8 shifts the Frontier attention p95 further down to 8.8%, a regime Vidur does not support. The residual is consistent with the classification in §3.4: attention latency is shaped by the per-request length distribution inside a batch and GroupedGEMM by expert-level routing load, so a token-aggregate regressor cannot separate compositions that diverge at the kernel’s partitioning and tile-scheduling level. Memory (KV-cache budget) accuracy. Figure 8 and Table 4 report the initial and time-varying per-rank KV-cache block budget on Qwen3-30B MoE under co-location and PDD. Frontier keeps the initial block budget within 1.89% of vLLM across all four (𝑝𝑝, 𝑡𝑝, 𝑑𝑝, 𝑒𝑝) configurations, whereas the analytical baseline shared by most prior simulators overreports the admissible budget by 14.10%–39.73%; along the SharedGPT replay the block-availability curve of Frontier overlaps the vLLM curve through admission, preemption, and release events, and the simulated makespan lies within 7.6% of the measurement. The gap stems from the KV-cache manager’s treatment of other memory overhead: parameter memory is one of several occupants of the device budget, and omitting the runtime footprint of activation scratch, NCCL workspace, and CUDA Graph capture regions inflates the admissible block count that the scheduler later consumes for admission, chunking, and watermark-triggered preemption. The Frontier overhead residual is not uniform across configurations: it concentrates on pipeline-parallel layouts where a single representative-rank profile misses the PP-worst slice,

Operator and Mem. Fidelity

Operator accuracy. Figure 7 reports the relative-error CDF of per-operator latency prediction on H800 under BF16 and FP8, covering the attention, and MoE GroupedGEMM kernels that dominate the batch runtime. Under BF16, Frontier reaches p50/p95 errors of 3.5%/14.2% on attention, 3.3%/6.4% on linear, and 1.4%/5.3% on GroupedGEMM, whereas Vidur’s token-only attention predictor, fit on the same population, 3 Due to the absence of a publicly available AFD implementation, we use an

in-house implementation for AFD calibration. 8

1.0

0.0 TTFT 0.2

1.5

x2.1

x2.1

x2.7

x2.1

x2.4

APEX SharedGPT

x2.5

x2.6x2.2

x2.4

x2.7

TPOT 0.4

Thpt 0.6 0.8E2E 1.0

Prefill-Heavy x2.5

TTFT

x2.5

TPOT

Thpt

Decode-Heavy

E2E

TTFT

x3.0

TPOT

Thpt

Balanced

E2E

TTFT

x2.2

TPOT

Thpt

E2E

x3.0

x2.7

SharedGPT

0.8 1.0 0.5 0.6 0.0

x2.6

x2.3

x3.0 x2.3

x2.3

x2.9

x2.3

x2.6

MoE

0.4 1.5

Dense

PDD Serving

1.0

x2.6 x2.3

LLMServingSim2.0 Vidur Balanced

MoE

1.5 0.8 1.0 0.5 0.6 0.0 0.4 1.5 1.0 0.2 0.5 0.0

AIConfigurator Decode-Heavy

Dense

Colocated Serving

Frontier Prefill-Heavy

1.0 0.2 0.5

0.0 0.0 TTFT 0.2

TPOT Thpt 0.8E2E 1.0 TTFT TPOT Thpt E2E TTFT TPOT Thpt E2E TTFT TPOT Thpt E2E 0.4 0.6 Figure 11. End-to-end fidelity on a 16-card H800 testbed for co-location and PDD across prefill-heavy, decode-heavy, balanced, and SharedGPT workloads. Panels report TTFT, TPOT, throughput, and E2E makespan for both Llama3.1-8B (dense) and Qwen3-30B MoE; dashed bars indicate simulators that do not support the serving architecture or model family. All data are normalized against the ground truth (vLLM), represented by the black line (y=1.0).

Speculative decoding. Table 6 reports p95 fidelity of Frontier on Qwen3-30B MoE MTP under SharedGPT for verifytoken lengths {2, 8, 32} and forced acceptance ratios {0.3, 0.7}. Across the six configurations, Frontier keeps TTFT, TPOT, throughput, and E2E p95 errors within 11.28%, with five below 10%. Among existing simulators, only AIConfigurator provides limited support for MTP. AIConfigurator’s analytical MTP path instead incurs an average 46.9% decode throughput error and flips the trend on low-acceptance (AR=0.3) batches. This gap comes from event-level MTP modeling: Frontier tracks per-request planned, verified, accepted, and committed tokens, and models MTP as a draft→verify→commit cycle sharing a batch with ordinary decode. It therefore preserves per-request speculative-depth and acceptance variance within a batch, which governs tail latency but is lost in static analytical speedup models.

so the direct tightening path is per-stage reprofiling rather than refining the analytical formula.

5.2

Runtime Optimization Fidelity

CUDA Graph. Figure 9 and Table 5 report decode throughput and compute-participating token counts for Qwen330B MoE under vLLM’s full_decode_only capture path on SharedGPT. Frontier matches vLLM’s decode speedup within 1.7% under co-location and 6.1% under PDD, preserving the 2.86–3.05× envelope; its padded token count tracks vLLM within 2.55% across prefill-heavy, decode-heavy, and SharedGPT workloads. AIConfigurator, which omits graphcapture modeling, instead produces a workload-insensitive TPOT surface that structurally diverges from measurements (§2.2). The residual reflects CUDA Graph’s coupled effects: batch quantization inflates compute-participating tokens while graph replay removes launch overhead, so the Fidelity Plane must switch to kernel-only timings in sync with padding—a coupling no scalar speedup factor can reproduce. The 6.1% PDD residual comes from the decode-side padding tail under a sparse capture-bin ladder, suggesting denser capture-bin coverage rather than a new modeling primitive. Prefix cache. Figure 10 reports cumulative prefix hit ratios on Qwen3-30B MoE under co-location and PDD with vLLM’s block-hash prefix cache enabled. Frontier matches vLLM’s final hit ratios exactly: 36.98% under co-location and 37.11% under PDD. The match comes from modeling prefix caching as a block-hash index that marks matched prefix blocks as already computed, thereby reshaping schedulervisible admission state without storing KV tensors. This replays vLLM’s block-level hit/miss sequence and admission behavior while avoiding token-level KV-cache bookkeeping, which is orthogonal to admission-shape fidelity.

5.3

End-to-End Fidelity

Co-location. Under co-location (Figure 11, left), Frontier tracks vLLM within 9.37% across 32 cases spanning Llama3.18B and Qwen3-30B MoE, with TPOT-p95 and E2E makespan inside 6.4% in all but one case, whereas Vidur (dense-only) errors range from 2.9% to 45.5%. While effective for TensorRTLLM on H200 [45], AIConfigurator prediction accuracy degrades under vLLM and H800. It produces E2E-makespan errors up to 170.0% on decode-heavy dense and 135.3% on balanced MoE. Vidur’s accuracy collapses on MoE because its operator library has no GroupedGEMM class and cannot express routing-dependent runtime, while AIConfigurator’s decode-heavy makespan over-prediction is the compounded consequence of missing CUDA-graph and speculative paths (§2.2) once they are composed across hundreds of decode steps. Frontier remains within this envelope because the event-driven execution plane keeps the closed loop 9

Throughput (tok/s)

1200

1000

1000

800

800

600

600

400

400 vLLM (Internal) Frontier

200

AFD (FFN-EP)

1400

1200

0

SLO-feasible region Min speed SLO (x=50)

200 0

vy

ll-Hea

Prefi

vy Hea odeDec

ced

Balan

Prefi

Best config (frontier ∩ SLO) Sweep candidates

Co-location PDD

TTFT ≤ 3000 ms Throughput (tokens/s)

AFD (FFN-TP)

1400

vy

ll-Hea

ode-

Dec

Hea

vy

ced

Balan

Figure 12. AFD fidelity on Step3-316B (16 H800 GPUs) against the ground truth across prefill-heavy, decode-heavy, and balanced workloads. AFD-TP and AFD-EP report throughput (decode toks/s).

PDD: 137.4k

150k

AFD

TTFT ≤ 500 ms 150k

AFD: 116.2k

AFD: 116.2k

100k Co-location: 27.7k

50k

0

PDD: 100.7k

100k

35

40

45

Co-location: 27.7k

50k

50

55

Generation Speed (tokens/s/user)

60

0

35

40

45

50

55

60

Generation Speed (tokens/s/user)

Figure 13. Frontier projects throughput–generation-speed Pareto frontiers across co-location, PDD, and AFD on 256×H800 GPUs. The dashed line marks a 50 toks/s/user SLA; stars mark the best frontier point for each architecture under that SLA.

of per-iteration cost, KV-cache block capacity, and admissionwatermark preemption intact; small per-operator errors bounded in §5.1 therefore do not amplify through the feedback path into the qualitatively different regimes that static models exhibit. PDD. Under PDD (Figure 11, right; 𝑡𝑝=4, 𝑑𝑝=2 for dense and 𝑡𝑝=4, 𝑑𝑝=2, 𝑒𝑝=8 for MoE), Frontier keeps all 32 metric entries within 10.99% of vLLM, with 29 of 32 inside 9% and 20 of 32 inside 3%; Vidur and Apex do not support PDD and therefore cannot be placed on this axis, while AIConfigurator reaches 200.0% E2E-makespan error on both dense and MoE decode-heavy. The AIConfigurator collapse is architectural: its homogeneous-replica model cannot represent the cross-cluster KV-cache transfer dependency, so a missing arc in the event graph translates into a multiplicative makespan error that no post-hoc scalar can close (§2.2). Frontier holds because the control plane materializes PDD as explicit cross-cluster KV-cache transfer events, and the fidelity plane resolves each transfer against KV-cache size, link topology, and transfer concurrency (§3.2), preserving the strict prefill→transfer→decode ordering that governs queueing. We found the worst case is MoE prefill-heavy TTFT at 10.99%, which falls in the regime where chunkedprefill residuals interact with expert-routing imbalance at admission; tightening it requires extending the wide-chunk token buckets in the operator profile rather than new controlplane primitives. AFD. We evaluate two AFD decode-FFN layouts on Step3316B with decode-attention fixed at 𝑑𝑝=8: FFN-TP shards expert weights with 𝑡𝑝=8 and no EP, while FFN-EP enables expert parallelism with 𝑒𝑝=8. On both AFD-TP and AFD-EP (Figure 12), Frontier holds TPOT within 6.4% and throughput within 7.0% across prefill-heavy, decode-heavy, and balanced workloads, and reproduces decode-heavy TPOT to within 1.0% on AFD-EP. Architectural fidelity follows from the cluster-role decomposition of §3.2: AFD is realized as a prefill cluster plus separate attention and FFN clusters with a paired attention-to-FFN transfer per decode iteration, and every MoE layer gates its combine collective on the slowest rank so routing-induced stragglers arise from per-rank prediction rather than a separate imbalance model. Because AFD was calibrated in an offline setting, we focus the main comparison on throughput-oriented metrics, where decode 10

iteration cost and cross-cluster transfer dominate sustained serving behavior.

6

Use Cases

We demonstrate Frontier through four use cases in two groups. The first group (§6.1, §6.2) is resource configuration tuning for non-reasoning batch serving: tuning the serving architecture and parallelism configuration for a given cluster (§6.1), and tuning the GPU placement across heterogeneous GPU types (§6.2). The second group (§6.3, §6.4) is what-if validation of new optimizations under emerging reasoning and RL rollout workloads: validating a new scheduling algorithm (§6.3), and validating a dynamic parallelism reconfiguration strategy (§6.4). To the best of our knowledge, none of the four cases can be reproduced as configured by any prior simulator. 6.1

Optimal Configuration via Pareto Frontiers

Problem. Given fixed cluster resources, which serving architecture and parallelism configuration is optimal under a given SLA constraint? Scenario. Pareto frontiers are the operator interface for architecture selection: given a TTFT constraint, they show the maximum throughput reachable at each generation-speed target. Measuring this surface on a live 256-GPU cluster is impractical, since each point can change the parallelism layout, cluster split, batching limit, memory budget, and load level. We run the search ex-situ for a Llama-3.3-70B [8] batch-inference workload on 256 H800 GPUs, comparing colocation, PDD, and AFD with vLLM-style scheduler, CUDA Graph, and chunked prefill. Across the three serving architectures, the full sweep contains 483,536 candidate configurations under the 256-GPU budget. Static memory filtering skips 65,190 OOM-infeasible cases before execution. The final figures summarize 496 cases that satisfy the SLA. Simulation. Figure 13 asks which serving architecture delivers the most throughput once generation speed must exceed 50 toks/s/user; the shaded region is feasible and the stars mark the best frontier point at this SLA. The three curves expose different failure modes. Co-location keeps all work in one scheduler and one KV-cache budget, so prefill and decode compete for batch slots; once p95 TPOT must stay near 20 ms

Normalized spend ratio SR(g) = Price(g0)/Price(g)

Cost efficiency CE(g)

(a) PDD (1024 × GPU) Ratio (vs. all-H800)

2.50 FAIL (G2) PASS

1.50

1.37

PASS 1.00

1.69

SR(𝑔) =

2.00

1.65

1.50

1.33

PASS

1.00

1.00

0.50 0.00

scores for a candidate allocation:

2.50

2.00

1.00

Gate 3 target CE(g)≥1.08

(b) AFD (1024 × GPU)

1.00

1.00

PASS

1.16

1.12

H20-DEC P:D=1:1

H20-DEC P:D=2:6

0.00

all-H800 (P:A:F=4:4:4)

H20-ATTN H800-FFN

1.16

1.05

𝑇 (𝑔) / Price(𝑔) , 𝑇 (𝑔0 ) / Price(𝑔0 )

where 𝑇 is aggregated token throughput. SR(𝑔) is the normalized spend ratio: higher is better, with SR(𝑔)>1 meaning 𝑔 is cheaper than 𝑔0 on the bill. CE(𝑔) is cost efficiency, i.e. throughput-per-dollar relative to 𝑔0 : higher is better, with CE(𝑔)>1 meaning 𝑔 delivers more tokens per dollar than 𝑔0 . Each valid allocation scheme must satisfy three constraints: (1) hardware–workload alignment: Frontier uses perrole stage metrics and matched GPU-type counterfactuals to classify each role’s bottleneck under the current workload; compute-bound roles must use H800, while H20 is allowed only when the role is not compute-limited. (2) SLA constraints: P95 TTFT and TPOT shall stay within the SLA thresholds; and (3) positive throughput-per-dollar ROI: even when SLA is met, a placement that merely saves cost while noticeably reducing total throughput is not deploymentworthy. We require CE(𝑔)>1.08, i.e., at least 8% more tokens per dollar than the all-H800 baseline, absorbing measurement variance and ensuring genuine operational value. These representative PDD and AFD placements lead to two findings (Figure 14). First, heterogeneity does not always pay off. Pushing the P:D capacity split toward decode in PDD amplifies the H20 hourly discount: SR(𝑔) climbs to 1.69 at P:D=2:6 and the nominal CE(𝑔) reaches 1.65 as the expanded decode pool boosts token generation throughput. However, P capacity collapses, a long queueing tail dominates P95 TTFT, and the candidate is rejected at Gate 2; the price saving is invalidated by an SLA-violating queueing tail. Second, heterogeneous savings are gated by different bottlenecks in the two disaggregated designs. PDD reaches its accepted point by downgrading the whole D role at P:D=1:1 (SR(𝑔)=1.37, CE(𝑔)=1.33); pushing the split toward decode keeps amplifying the nominal discount but starves P, so the candidate is rejected by queue-driven SLA violation. AFD, in contrast, fixes P:A:F=4:4:4 and asks which decode subrole to downgrade: placing H20 on A passes at SR(𝑔)=1.16, CE(𝑔)=1.12, while the symmetric swap (H20 on F) is rejected at Gate 1 because the FFN role is compute-bound and regresses on H20 (CE(𝑔)=1.05). A price-only or fused-decode analysis would favor rejected placements, either because they maximize bill savings or because they cannot express attention/FFN role-GPU-type mismatch. Frontier’s disaggregated cost-performance modeling ties GPU-type choice to role bottlenecks and SLA outcomes, separating raw price reduction from deployment-grade cost efficiency.

H800-ATTN H20-FFN

Figure 14. Heterogeneous Qwen3-235B-A22B [12] allocation exposes which PDD and AFD role assignments convert hardware discounts into cost efficiency.

(i.e., 50 toks/s/user), decode batches shrink and throughput is capped. PDD removes that interference by giving prefill and decode separate clusters, which makes it strong when TTFT is loose and most GPUs can be kept on decode. AFD further separates decode-attention from decode-FFN; it pays activation transfer between the two decode clusters, but isolates the latency-sensitive attention path from FFN batching and keeps the prefill path short. The best-config markers quantify this tradeoff. Under the loose TTFT constraint (TTFT ≤ 3000 ms), PDD is best: the SLA frontier point reaches 137.4K toks/s on the P:D = 112:144 split, where more GPUs are assigned to decode. AFD reaches 116.2K toks/s with P:DA:DF = 120:80:56, while colocation reaches only 27.7K toks/s. When TTFT tightens to 500 ms, the optimum shifts: AFD remains at 116.2K toks/s with p95 TTFT 115.9 ms, whereas PDD drops to 100.7K toks/s at P:D = 144:112 and p95 TTFT 211 ms because the prefill pool must grow at the expense of decode capacity. Without Frontier that jointly models disaggregation, parallelism, runtime state, and SLA filtering, this decision would be made on the wrong frontier: PDD would appear universally preferable under the loose-SLA view, but AFD is the correct choice under sub-second TTFT. 6.2

CE(𝑔) =

FAIL (G1)

0.50

all-H800 (P:D=1:1)

Price(𝑔0 ) , Price(𝑔)

Heterogeneous GPUs for Disaggregated Serving

Problem. In a disaggregated serving architecture, how does the use of heterogeneous GPUs affect the performance-cost tradeoff? Scenario. It is known that PDD and AFD can improve serving efficiency. Here we explore another degree of freedom which is common in practice: heterogeneous GPU fleets. We consider a cluster of fixed scale with 1024 GPUs, serving Qwen3-235B-A22B MoE [12] along with the runtime optimizations specified in §6.1. By assigning different GPUs to different roles (P/D/A/F), how does this further improve the cost-performance tradeoff? Í Simulation. Let Price(𝑔)= 𝑟 𝑁𝑟 𝑝 (𝑔(𝑟 )) denote the cluster’s total hourly spend under an allocation scheme 𝑔, where 𝑁𝑟 is the number of GPUs assigned to role 𝑟 and 𝑝 (·) maps each GPU type to its public cloud price (H800: $3.49/hr, H20: $1.59/hr) [2]. Against the all-H800 baseline 𝑔0 , we report two

6.3

Scheduling Algorithm Validation for Stateful Reasoning

Problem. Given a fixed resource configuration, can a new scheduling algorithm that exploits cross-round request state improve latency over existing priority policies? 11

+23.2%

1.2

+0.5%

1.0

1.0 0.8

-30.4%

0.8

Mechanism Breakdown 1.2

0.6

0.5

0.4

0.2

0.2

0.0

0.0

p95

FT

aT T

p95

OT

aTP

en Hidd

hput

ug thro

2.5

vLLM Scheduler Frontier

-18.9%

-12.1% -47.5%

pan Wait Span akes eue Drain gM g Qu refill sonin sonin vy-P Rea Rea Hea p95

Dynamic config. x2.04

2.0 1.5 1.0 0.5 0.0

Figure 15. Phase-aware scheduling for multi-round reasoning.

Static config.

x1.00

x1.00 x0.49

E2E makespan

Throughput

1.0

Fraction completed

vLLM Scheduler Frontier

Normalized Value

Normalized Value

End-to-End System Performance 1.5

Completion fraction of tail requests

0.8 0.6 0.4 Static config. Dynamic config. Switching overhead (4.52s)

0.2 0.0

0

100

200

300

Time (s)

400

500

Figure 16. Performance gains from dynamic reconfiguration.

Scenario. Multi-round agentic reasoning workloads no longer behave like a single prompt followed by a single decode stream. Taking a coding agent such as Claude Code [1] as an example, each request progresses through two phases: internal planning and final response generation. During planning, the agent runs multiple thinking rounds marked by <thinking> tags [26]; each round may issue tool calls such as code execution or calculation, whose observations seed the next round until planning terminates and the visible response is produced. The resulting traffic is long-context and long-tailed [41]: the number of thinking rounds varies sharply across sessions, and per-round prefill spans a wide range. The essential modeling requirement is therefore not merely longer contexts, but request-state and event modeling: the simulator must preserve round history, tool-call delay, requeue events, and prefix-cache continuity as part of the same session. This changes the scheduling problem from prioritizing isolated requests to scheduling stateful, multievent sessions whose final answer latency depends on hidden planning history. Since short-job-first scheduling is effective in non-reasoning serving—for example, FastServe [44] uses a skip-join multi-level feedback queue (MLFQ) [22] to prioritize short requests—we ask whether such mechanisms remain effective once the reasoning state is made explicit. We use aTTFT to denote answer-visible TTFT, measured from session arrival to completion of the final answer-round prefill, i.e., the first user-visible token boundary. We define hidden planning throughput as the hidden planning-token rate over the fixed trace. Simulation. We employ Frontier for an algorithmic exploration. We model Llama-3.1-405B-FP8 [6] on 1024×H800 GPUs under a PDD architecture, with chunked prefill, prefix caching, CUDA Graph, quantization (FP8) and CPU offloading enabled to reflect a realistic deployment. The trace is prefill-heavy and long-tailed as the scenario describes: perround prefill spans 4K–32K tokens while per-round decode stays at ∼0.2K. Plugging FastServe’s skip-join MLFQ into this setting yields only a 1.45% p95 aTTFT improvement and degrades hidden planning throughput by 2.3%. The reason is that its priority is decided from the current round’s observable size and ignores cross-round history; a heavy-tail session whose final answer round looks small is therefore treated like an ordinary short request. Guided by this diagnosis, we deploy a phase-aware scheduler that classifies

sessions by both current-round size and accumulated planning history, achieving a 30.4% p95 aTTFT reduction and 23.2% hidden-planning-throughput improvement (Figure 15). We provide more details in the Appendix (§B.1–§B.3). With Frontier, we enable rapid, low-cost design and validation of scheduling algorithms for advanced scenarios like stateful reasoning—without deploying large-scale GPU clusters or modifying a production serving stack. 6.4

Dynamic Parallelism Reconfig. for RL Rollouts

Problem. Given a fixed scheduling logic, can dynamically switching the parallelism layout mid-workload reduce endto-end makespan? Scenario. Agentic RL rollouts exhibit a phase transition that is invisible to steady-state serving benchmarks [41, 48]. At the start of a rollout batch, the trainer launches many trajectories in parallel, so the inference cluster is bound by burst throughput and rewards high replica concurrency. Once most short trajectories terminate, a heavy-tail subset continues decoding alone and dictates when the trainer can advance to the next gradient step. Recent work addresses this asymmetry by reconfiguring parallelism online as the active set shrinks [41]. The benefit depends on both when to switch and how much reconfiguration cost the switch incurs. We ask whether Frontier can prototype such reconfiguration policies and expose their system-level impact ahead of deployment. Simulation. We employ Frontier for a dynamic reconfiguration exploration. We retain the model, cluster scale, and runtime features from the previous use case (§6.3), and drive a co-located deployment with a 4,000-trajectory burst whose heavy-tail fraction is 5%. The policy is realized in Frontier as a time-sliced layout switch. The baseline pins Layout A— a high-DP configuration (DP=32, PP=16, TP=2) that maximizes batch parallelism—throughout the rollout. The dynamic policy begins with Layout A and switches to Layout B (DP=8, PP=16, TP=8) once the active-trajectory count drops below 10% of the batch, paying a profiled reconfiguration cost on transition, including weight reshard and KV-cache rematerialization. As Figure 16 shows, the dynamic policy reduces rollout makespan from 528.8 s to 259.1 s and improves effective decode throughput by 2.04×. The gain is phasematched parallelism: Layout A absorbs the dense burst, while re-sharding from replica-wide DP to wider TP after the tail 12

emerges shortens per-trajectory decode latency once spare replicas no longer translate into utilization. With Frontier, such phase-adaptive policies can be rapidly prototyped and validated before deployment, avoiding the hardware cost of repeatedly sweeping thousand-GPU layouts and the engineering cost of implementing layout switching directly in a production serving stack.

7

show that Frontier reliably predicts performance across diverse configurations and supports large-scale design-space exploration on commodity hardware, providing a practical foundation for studying modern LLM serving systems.

References [1] 2025. claude code. Website. https://code.claude.com/. [2] 2025. cloudgpu. Website. https://cloudgpu.app/. [3] 2025. dynamo. Website. https://www.nvidia.com/en-us/ai/dynamo/. [4] 2025. htsim Network Simulator. Website. https://github.com/ Broadcom/csg-htsim. [5] 2025. huggingface. Website. https://huggingface.co/. [6] 2025. Llama-3.1-405B-FP8. Website. https://huggingface.co/metallama/Llama-3.1-405B-FP8. [7] 2025. Llama-3.1-8B. Website. https://huggingface.co/meta-llama/ Llama-3.1-8B. [8] 2025. Llama-3.3-70B-Instruct. Website. https://huggingface.co/metallama/Llama-3.3-70B-Instruct. [9] 2025. NCCL workspace buffer. Website. https://docs.nvidia.com/ deeplearning/nccl/user-guide/docs/usage/bufferreg.html. [10] 2025. Nvidia CUDA Graph. Website. https://docs.nvidia.com/cuda/ cuda-programming-guide/04-special-topics/cuda-graphs.html. [11] 2025. Nvidia TensorRT-LLM. Website. https://github.com/NVIDIA/ TensorRT-LLM. [12] 2025. Qwen3-235B-A22B. Website. https://huggingface.co/Qwen/ Qwen3-235B-A22B. [13] 2025. Qwen3-30B-A3B. Website. https://huggingface.co/Qwen/ Qwen3-30B-A3B. [14] 2025. sglang admission. Website. https://github.com/sgl-project/ sglang/blob/main/docs/advanced_features/server_arguments.md. [15] 2025. SharedGPT trace. Website. https://docs.vllm.ai/en/v0.12.0/ benchmarking/cli/. [16] 2025. vllm watermark. Website. https://docs.vllm.ai/en/v0.9.0/api/ vllm/core/block_manager.html. [17] Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav S Gulavani, Ramachandran Ramjee, and Alexey Tumanov. 2024. Vidur: A large-scale simulation framework for llm inference. Proceedings of Machine Learning and Systems 6 (2024), 351– 366. [18] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming {Throughput-Latency} tradeoff in {LLM} inference with {Sarathi-Serve}. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). 117–134. [19] Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gulavani, and Ramachandran Ramjee. 2023. Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills. arXiv preprint arXiv:2308.16369 (2023). [20] Amey Agrawal, Mayank Yadav, Sukrit Kumar, Anirudha Agrawal, Garv Ghai, Souradeep Bera, Elton Pinto, Sirish Gambhira, Mohammad Adain, Kasra Sohrab, Chus Antonanzas, and Alexey Tumanov. 2026. Revati: Transparent GPU-Free Time-Warp Emulation for LLM Serving. arXiv:2601.00397 [cs.DC] https://arxiv.org/abs/2601.00397 [21] Jaehong Cho, Hyunmin Choi, Guseul Heo, and Jongse Park. 2026. LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructure. arXiv preprint arXiv:2602.23036 (2026). [22] Fernando J Corbató, Marjorie Merwin-Daggett, and Robert C Daley. 1962. An experimental time-sharing system. In Proceedings of the May 1-3, 1962, spring joint computer conference. 335–344. [23] Tri Dao. 2024. Flashattention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations, Vol. 2024. 35549–35562.

Discussion

Currently, Frontier is primarily developed and calibrated around vLLM. As outlined in §3, its design natively supports extending simulation capabilities to other mainstream frameworks (e.g., SGLang [50] and TensorRT-LLM [11]), which we leave for future work. For precision calibration, since GPU execution dominates the overall latency (∼90%) [20], we follow Vidur’s methodology to model the remaining CPU overhead by combining testbed measurements with a prediction mechanism (Fidelity Plane). However, this approach for CPU overhead prediction may still lack robustness. As CPU overhead fluctuates with system scales and architectural updates, a potentially more accurate approach for CPU simulation would employ CUDA API interception—similar to Revati [20]—to enable full CPU-side replay. Nevertheless, the usability and scalability of this approach across diverse hardware, CUDA versions, and dynamic runtime optimizations (e.g., CUDA Graphs) remain debatable. Developing a robust yet accurate simulation scheme capable of adapting to diverse inference frameworks and technological advancements represents a promising future direction. Furthermore, during calibration, we observed that beyond standard warmup, profiling tools easily introduce intrusive overhead. Therefore, we strictly isolate the measurement of fine-grained metrics (e.g., operator execution, batch scheduling) from system-level metrics (e.g., TTFT, TPOT, throughput) to prevent statistical contamination and ensure reliable calibration.

8

Related Work

We discuss related work other than those covered in §2. GPU simulation. GPU computation simulators like SCALEsim [37] and Accel-Sim [27] focus on instruction-level modeling. Frontier can integrate these into the op library to support operation modeling for future hardware. Training simulators. Simulators for LLM training [24, 25, 43] model training with a focus on optimizing parallelism strategies and resource utilization. However, they do not capture the dynamics of inference workloads and complex runtime behavior, which are critical for LLM serving.

9

Conclusion

We present Frontier, a discrete-event simulator for LLM serving. By jointly modeling architectural heterogeneity, runtime behavior, and workload statefulness, Frontier enables accurate, scalable evaluation of complex serving designs. Results 13

[39] Siddharth Singh, Olatunji Ruwase, Ammar Ahmad Awan, Samyam Rajbhandari, Yuxiong He, and Abhinav Bhatele. 2023. A hybrid tensorexpert-data parallelism approach to optimize mixture-of-experts training. In Proceedings of the 37th International Conference on Supercomputing. 203–214. [40] StepFun, :, Bin Wang, Bojun Wang, Changyi Wan, Guanzhe Huang, Hanpeng Hu, Haonan Jia, Hao Nie, Mingliang Li, Nuo Chen, Siyu Chen, Song Yuan, Wuxun Xie, Xiaoniu Song, Xing Chen, Xingping Yang, Xuelin Zhang, Yanbo Yu, Yaoyu Wang, Yibo Zhu, Yimin Jiang, Yu Zhou, Yuanwei Lu, Houyi Li, Jingcheng Hu, Ka Man Lo, Ailin Huang, Binxing Jiao, Bo Li, Boyu Chen, Changxin Miao, Chang Lou, Chen Hu, Chen Xu, Chenfeng Yu, Chengyuan Yao, Daokuan Lv, Dapeng Shi, Deshan Sun, Ding Huang, Dingyuan Hu, Dongqing Pang, Enle Liu, Fajie Zhang, Fanqi Wan, Gulin Yan, Han Zhang, Han Zhou, Hanghao Wu, Hangyu Guo, Hanqi Chen, Hanshan Zhang, Hao Wu, Haocheng Zhang, Haolong Yan, Haoran Lv, Haoran Wei, Hebin Zhou, Heng Wang, Heng Wang, Hongxin Li, Hongyu Zhou, Hongyuan Wang, Huiyong Guo, Jia Wang, Jiahao Gong, Jialing Xie, Jian Zhou, Jianjian Sun, Jiaoren Wu, Jiaran Zhang, Jiayu Liu, Jie Cheng, Jie Luo, Jie Yan, Jie Yang, Jieyi Hou, Jinguang Zhang, Jinlan Cao, Jisheng Yin, Junfeng Liu, Junhao Huang, Junzhe Lin, Kaijun Tan, Kaixiang Li, Kang An, Kangheng Lin, Kenkun Liu, Lei Yang, Liang Zhao, Liangyu Chen, Lieyu Shi, Liguo Tan, Lin Lin, Lin Zhang, Lina Chen, Liwen Huang, Liying Shi, Longlong Gu, Mei Chen, Mengqiang Ren, Ming Li, Mingzhe Chen, Na Wang, Nan Wu, Qi Han, Qian Zhao, Qiang Zhang, Qianni Liu, Qiaohui Chen, Qiling Wu, Qinglin He, Qinyuan Tan, Qiufeng Wang, Qiuping Wu, Qiuyan Liang, Quan Sun, Rui Li, Ruihang Miao, Ruosi Wan, Ruyan Guo, Shangwu Zhong, Shaoliang Pang, Shengjie Fan, Shijie Shang, Shilei Jiang, Shiliang Yang, Shiming Hao, Shuli Gao, Siming Huang, Siqi Liu, Tiancheng Cao, Tianhao Cheng, Tianhao Peng, Wang You, Wei Ji, Wen Sun, Wenjin Deng, Wenqing He, Wenzhen Zheng, Xi Chen, Xiangwen Kong, Xianzhen Luo, Xiaobo Yang, Xiaojia Liu, Xiaoxiao Ren, Xin Han, Xin Li, Xin Wu, Xu Zhao, Yanan Wei, Yang Li, Yangguang Li, Yangshijie Xu, Yanming Xu, Yaqiang Shi, Yeqing Shen, Yi Yang, Yifei Yang, Yifeng Gong, Yihan Chen, Yijing Yang, Yinmin Zhang, Yizhuang Zhou, Yuanhao Ding, Yuantao Fan, Yuanzhen Yang, Yuchu Luo, Yue Peng, Yufan Lu, Yuhang Deng, Yuhe Yin, Yujie Liu, Yukun Chen, Yuling Zhao, Yun Mou, Yunlong Li, Yunzhou Ju, Yusheng Li, Yuxiang Yang, Yuxiang Zhang, Yuyang Chen, Zejia Weng, Zhe Xie, Zheng Ge, Zheng Gong, Zhenyi Lu, Zhewei Huang, Zhichao Chang, Zhiguo Huang, Zhirui Wang, Zidong Yang, Zili Wang, Ziqi Wang, Zixin Zhang, Binxing Jiao, Daxin Jiang, Heung-Yeung Shum, and Xiangyu Zhang. 2025. Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding. arXiv:2507.19427 [cs.LG] https://arxiv.org/abs/2507.19427 [41] Xin Tan, Yicheng Feng, Yu Zhou, Yimin Jiang, Yibo Zhu, and Hong Xu. 2026. OrchestrRL: Dynamic Compute and Network Orchestration for Disaggregated RL. arXiv preprint arXiv:2601.01209 (2026). [42] Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2024. A survey on large language model based autonomous agents. Frontiers of Computer Science 18, 6 (2024), 186345. [43] William Won, Taekyung Heo, Saeed Rashidi, Srinivas Sridharan, Sudarshan Srinivasan, and Tushar Krishna. 2023. Astra-sim2. 0: Modeling hierarchical networks and disaggregated systems for large-model training at scale. In 2023 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). IEEE, 283–294. [44] Bingyang Wu, Yinmin Zhong, Zili Zhang, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xuanzhe Liu, and Xin Jin. 2023. Fast distributed inference serving for large language models. arXiv preprint arXiv:2305.05920 (2023). [45] Tianhao Xu, Yiming Liu, Xianglong Lu, Yijia Zhao, Xuting Zhou, Aichen Feng, Yiyi Chen, Yi Shen, Qin Zhou, Xumeng Chen, Ilya Sherstyuk, Haorui Li, Rishi Thakkar, Ben Hamm, Yuanzhe Li, Xue Huang, Wenpeng Wu, Anish Shanbhag, Harry Kim, Chuan Chen, and Junjie

[24] Jiangfei Duan, Xiuhong Li, Ping Xu, Xingcheng Zhang, Shengen Yan, Yun Liang, and Dahua Lin. 2024. Proteus: Simulating the performance of distributed DNN training. IEEE Transactions on parallel and distributed systems 35, 10 (2024), 1867–1878. [25] Yicheng Feng, Yuetao Chen, Kaiwen Chen, Jingzong Li, Tianyuan Wu, Peng Cheng, Chuan Wu, Wei Wang, Tsung-Yi Ho, and Hong Xu. 2024. Echo: Simulating distributed training at scale. arXiv preprint arXiv:2412.12487 (2024). [26] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025). [27] Mahmoud Khairy, Zhesheng Shen, Tor M Aamodt, and Timothy G Rogers. 2020. Accel-sim: An extensible simulation framework for validated gpu modeling. In 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA). IEEE, 473–486. [28] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles. 611–626. [29] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668 (2020). [30] Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast Inference from Transformers via Speculative Decoding. arXiv:2211.17192 [cs.LG] https://arxiv.org/abs/2211.17192 [31] Jiamin Li, Yimin Jiang, Yibo Zhu, Cong Wang, and Hong Xu. 2023. Accelerating distributed {MoE} training and inference with lina. In 2023 USENIX Annual Technical Conference (USENIX ATC 23). 945–959. [32] Yi-Chien Lin, Woosuk Kwon, Ronald Pineda, and Fanny Nina Paravecino. 2024. APEX: An extensible and dynamism-aware simulator for automated parallel execution in LLM serving. arXiv preprint arXiv:2411.17651 (2024). [33] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024). [34] Xupeng Miao, Yujie Wang, Youhe Jiang, Chunan Shi, Xiaonan Nie, Hailin Zhang, and Bin Cui. 2022. Galvatron: Efficient transformer training over multiple gpus using automatic parallelism. arXiv preprint arXiv:2211.13878 (2022). [35] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. 2019. PipeDream: Generalized pipeline parallelism for DNN training. In Proceedings of the 27th ACM symposium on operating systems principles. 1–15. [36] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019). [37] Ananda Samajdar, Yuhao Zhu, Paul Whatmough, Matthew Mattina, and Tushar Krishna. 2018. Scale-sim: Systolic cnn accelerator simulator. arXiv preprint arXiv:1811.02883 (2018). [38] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 (2019).

14

Lai. 2026. AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving. arXiv:2601.06288 [cs.LG] https://arxiv.org/abs/2601.06288 [46] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629 (2022). [47] Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, et al. 2025. Flashinfer: Efficient and customizable attention engine for llm inference serving. Proceedings of Machine Learning and Systems 7 (2025). [48] Zili Zhang, Yinmin Zhong, Chengxu Yang, Chao Jin, Bingyang Wu, Xinming Wei, Yuliang Liu, and Xin Jin. 2026. Heddle: A Distributed Orchestration System for Agentic RL Rollout. arXiv preprint arXiv:2603.28101 (2026). [49] Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P Xing, et al. 2022. Alpa: Automating inter-and {Intra-Operator} parallelism for distributed deep learning. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). 559–578. [50] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. 2024. Sglang: Efficient execution of structured language model programs. Advances in neural information processing systems 37 (2024), 62557–62583. [51] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. {DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). 193–210. [52] Ruidong Zhu, Ziheng Jiang, Chao Jin, Peng Wu, Cesar A Stuardo, Dongyang Wang, Xinlei Zhang, Huaping Zhou, Haoran Wei, Yang Cheng, et al. 2025. MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism. arXiv preprint arXiv:2504.02263 (2025).

15

H20 BF16

1.0

H20 FP8 p90

Co-location

p90

Cluster worker

0.6 0.4

Frontier Core Attention

0.2

ops GlobalLinear Controller MoE

0.0 0

10

Attention Linear ops MoE

Configuration Analyser 20

30

40

0

10

FrontierWorkload APE (%) Generator

Profiling Backend

Req. Schedul arrival -ing Replica Worker

20

30

Global Controller

CDF

0.8

40

Frontier APE (%)

Comm. Backend

FigurePerformance 17. H20 operator fidelity CDFs. show Frontier absoCollector SpaceCurves Explorer lute percentage error for attention, linear ops, and MoE on the H20 BF16 and FP8.

PD Disaggregation Cluster worker Prefill Prefill Prefill Worker Worker replica

Cluster Worker

Mean relative error (%)

8

Model Runner 6.5% 6.0%

6 4

3.6% 2.8%

H20 MoE

Cluster worker Prefill Prefill Decode Worker Worker replica

Cluster worker

Cluster worker

Cluster worker

Prefill Prefill Prefill Worker Worker replica

Prefill Prefill Decode-Attn Worker Worker replica

Prefill Prefill Decode-FFN Worker Worker replica

7.0% 5.8%

Batching Executor 5.1% 4.5% 4.0%

4.4%

3.8%

Runtime Predictor

2

3.1% 2.7% 0.5%

Figure 19. Example of three serving architectures in Frontier.

0.9% 0.9%

0

fillPre

T vy ed avy GP hea mix -he red rid/ ode Sha Hyb Dec

T vy ed avy GP hea mix -he red rid/ ode Sha Hyb Dec

A.3

fill-

Pre

Workload

A

Additional Details

A.1

H20 Operator Fidelity

PDD and AFD Scheduling Workflow

Frontier operates as a discrete-event simulator (DES). This subsection expands the control and execution-plane mechanisms into a concrete workflow-level algorithmic description. We show the example of three serving architectures of Frontier in Figure 19. Algorithm 1 illustrates the scheduling and execution pipelines for both Prefill-Decode Disaggregation (PDD) and Attention-FFN Disaggregation (AFD) serving architectures, using an MoE model simulation with thinking/reasoning modes enabled as a representative example. From a high-level perspective, the algorithm is modularly structured around role-local scheduler ticks. In the initialization phase, roles are compiled and bound to their respective parallel domains and hardware resources ({P, D} for PDD, or {P, A, F} for AFD). At runtime, an event-driven loop processes request admissions and batch formations via runtime_adapter() gates. These gates cleanly inject production optimizations (e.g., chunked prefill, speculative decoding) into the scheduler-visible state without polluting the core simulation logic. Requests then branch into role-specific execution paths: • P (Prefill): Completes prompt processing and triggers a request-granular KV-cache transfer to downstream decode roles. • D (PDD Decode): Executes unified Attention and FFN layers sequentially, emitting a global batch completion event. • A and F (AFD Decode): Pipelines per-layer activation transfers (A → F and F → A). The F role explicitly manages MoE routing, Expert Parallel (EP) dispatch, and synchronization barriers. For multi-phase reasoning workloads, requests traverse this role graph for each thinking round, re-entering the prefill

Workload

Figure 18. H20 end-to-end fidelity across dense and MoE model families. Bars show the mean relative error for co-location and PDD on the four workloads, averaging p95 TTFT, p95 TPOT, throughput, and p95 E2E within each workload-architecture pair.

We use the selected CDF rows from the H20 BF16 and FP8 operator packages, with 600 rows per family. Figure 17 reports Frontier APE CDFs for attention, linear ops, and MoE of H20 BF16 and FP8 operator. On BF16, Frontier p50/p90 APE is 3.24%/14.20% for attention, 2.95%/9.58% for linear ops, and 0.93%/3.12% for MoE. On FP8, the corresponding p50/p90 values are 2.11%/8.49%, 0.42%/2.29%, and 2.01%/8.64%. Linear ops stay the tightest curve in both packages. Attention keeps the longest BF16 tail, while MoE widens again on FP8. A.2

KV-Cache transfer

AF Disaggregation

Traffic Router Replica Worker H20 Dense 10 Request Scheduler Co-location PDD

State Logical update prefill/decode Completion check

H20 End-to-End Fidelity

Figure 18 summarizes H20 end-to-end fidelity for dense and MoE models under co-location and PDD across prefillheavy, decode-heavy, hybrid/mixed, and SharedGPT workloads. Each bar averages the relative error over p95 TTFT, p95 TPOT, throughput, and p95 E2E for one workload-architecture pair. All bars stay below 10%; dense co-location peaks at 6.5% on decode-heavy, dense PDD at 6.0% on decode-heavy, MoE co-location at 3.8% on prefill-heavy, and MoE PDD at 7.0% on decode-heavy. 16

Table 7. Multi-phase reasoning workload. Each cell is ℓ𝑟,𝑟 ′ /𝑜𝑟,𝑟 ′ (new prompt / decode tokens); round 5 is the answer-visible round.

Algorithm 1 PDD/AFD scheduling and execution workflow 1: Compile roles: Instantiate {P, D} (PDD) or {P, A, F} (AFD);

bind each 𝑐 ∈ C to domain, 𝑄𝑐 , budget, and backend. 2: Arrival/re-entry: Enqueue new requests and Thinking re-

Template

𝑟 =1

𝑟 =2

entries to 𝑄 P at time 𝑡. ⊲ Preserve session affinity 3: for each scheduler tick of role 𝑐 ∈ C do 4: Admission: Call runtime_adapter() on state. Admit requests, allocate blocks, and preempt while budget permits. 5: Execution: Form batch 𝐵, call runtime_adapter() on shape, query Fidelity Plane, and run through local PP stages. 6: if role 𝑐 = P then 7: Prefill: On completion, retain KV cache and emit request KV-cache transfer start. On end, enqueue 𝑟 to D or A. 8: else if role 𝑐 = D (PDD) then 9: PDD decode: Execute unified Attention+FFN. Emit GlobalBatchEnd on final layer/token completion. 10: else if role 𝑐 = A (AFD) then 11: AFD attention: For non-final decode layers, emit A→F transfer with (ℓ, stage, lane) metadata. 12: else if role 𝑐 = F (AFD) then 13: AFD FFN/MoE: Group A→F arrivals by stage/lane. Apply padding contracts, route tokens, build EP sub-batches. 14: if epffn > 1 then 15: Run pre-dispatch MoE work, wait for EP dispatch collective, then execute expert sub-batches. 16: EP sync: Wait for all EP lanes; combine at 𝑖 max𝑖 𝑡 ready + ΔEP . 17: else 18: Execute expert sub-batches locally without EP synchronization. 19: end if Recover raw batches and emit F→A transfers. A ad20: vances layer progress or emits GlobalBatchEnd. 21: end if 22: Completion: Commit tokens, release resources. Emit ThinkingRequeue if non-final, else export user metrics. 23: end for

Short Heavy tail

4096/96 32768/96

1024/64 16384/64

B.1

Workload

𝑟 =5 256/192 256/192

Normalized Value

1.2

+0.0%

-1.5%

1.0

-2.5%

0.8 0.6 0.4 0.2

vLLM Scheduler Skip-join MLFQ

0.0

p95

FT

aTT

p95

OT aTP

hput

roug

n th

e Hidd

Figure 20. Skip-join MLFQ on the agentic trace: p95 aTTFT improves marginally and hidden planning throughput regresses. Bluebar annotations show the relative change over vLLM.

(matching the <thinking> blocks in the scenario description) followed by one answer-visible round. Round 𝑟 ′ contributes ℓ𝑟,𝑟 ′ new prompt tokens (after same-request prefix reuse) and 𝑜𝑟,𝑟 ′ decode tokens. Following the main definition, aTTFT is the time from request arrival to completion of the answer-round prefill. Table 7 lists the two templates used in our trace. B.2

Why Skip-Join MLFQ Is Insufficient

We use Frontier to realize the skip-join MLFQ comparator against the native vLLM scheduler. It yields only marginal gains: p95 aTTFT drops from 20.84 s to 20.54 s (−1.45%), p95 aTPOT is unchanged, and hidden planning throughput regresses by 2.54% (Figure 20). The blue-bar annotations in the figure show the relative change over vLLM. The failure mode is structural: priority is decided on the current round’s observable size, but a heavy-tail request may present a small answer round indistinguishable from a true short request. Symmetrically, a heavy hidden-prefill continuation can occupy the prefill budget ahead of latency-critical answer prefills. This motivates a classifier that retains crossround history.

The key property is that disaggregation changes the event graph, not the simulator core. PDD inserts a request-granular KV-cache dependency; AFD adds per-layer activation pingpong. Both keep MoE synchronization explicit: prefill and PDD decode use in-role sync barriers, while AFD exposes decode-time FFN barriers in F before activations return to A. Runtime optimizations remain modular because adapter gates mutate scheduler-visible state, batch shape, or perrequest progress exactly where the production loop observes those effects.

Agentic Multi-Phase Reasoning

𝑟 =3

End-to-End System Performance

queue after a simulated tool delay until the final answer is generated.

B

𝑟 =4

512/64 512/64 8192/64 4096/64

B.3

H2Q-BR: History-Aware Two-Queue Scheduling with Bounded Release

Design intent. Skip-join MLFQ improves over FIFO-style admission by letting short current-round work bypass long prompt chunks. However, the previous section shows that this local view is not enough for agentic sessions: a heavytail request can later expose a tiny answer-visible round, and a running hidden-prefill continuation can keep consuming

Multi-round requests. We model each agentic request 𝑟 as a sequence of 𝑅𝑟 =5 rounds: four hidden planning rounds 17

prefill budget before short answer prefills are admitted. H2QBR (History-Aware Two-Queue with Bounded Release) keeps the skip-join intuition, but adds session history so that the scheduler can distinguish “small now” from “historically small” without using a final-round oracle. High-level idea. H2Q-BR follows three simple rules. First, it records compact session_id-scoped history: whether a session has already shown long-prompt behavior, how many new prompt tokens it has consumed, the current round’s new-prompt increment, and whether a prefill chunk spilled across scheduling iterations. Second, it separates work into a short queue 𝑄 𝑆 and a long-history queue 𝑄 𝐿 . Small requests from historically short sessions remain in 𝑄 𝑆 and are ordered by smaller current prompts first; for example, a 256-token answer prefill from a short session can bypass a 4096-token hidden prompt. A 32768-token hidden prefill enters 𝑄 𝐿 immediately, and later slices from the same session remain in 𝑄 𝐿 even if the current answer round is only 256 tokens. Third, H2Q-BR prevents starvation: long-history work normally waits behind 𝑄 𝑆 , but a one-shot carryover release lets an already started long prefill make bounded progress, and a liveness quota forces the oldest 𝑄 𝐿 slice after too many consecutive short-queue slices. The policy only changes request order before batch construction. The inherited batch builder still enforces token budgets, memory admission, chunking, and preemption; H2QBR does not replace the execution model. State update and classification. Algorithm 2 starts when a slice arrives or re-enters the scheduler. H2Q-BR computes the current-round prompt length as the increase in total prefill tokens since the previous completed round, then classifies the slice by Eq. 3. Once a session crosses the long-history boundary, the sticky flag keeps later rounds in 𝑄 𝐿 ; there is no round-class oracle and no final-round promotion. Classification. ( 𝑞𝑟 =

𝑄 𝐿 , 𝑧𝑟 =1 ∨ 𝐻𝑟 > 𝐶 ∨ ℓ𝑟 > 𝐿, 𝑄 𝑆 , otherwise.

released. This gives spilled prefills a bounded path forward without letting them jump ahead of older short requests. Ranking and batch construction. Before each scheduling pass, H2Q-BR ranks the combined running and waiting sets by  −2, 𝑖 = 𝑖 rel (release),      −1, 𝑖 = 𝑖 live (liveness),  𝜌 (𝑖) =  0, 𝑞𝑟 (𝑖 ) = 𝑄 𝑆 ,     1, 𝑞 𝑟 (𝑖 ) = 𝑄 𝐿 , 

(4)

where 𝑖 live is the oldest 𝑄 𝐿 slice forced after the short-streak counter reaches 𝐵. Within 𝑄 𝑆 , the secondary key is (ℓ𝑟 , 𝑑𝑖 , 𝑎𝑖 ): smaller prompts first, prefill before decode, then earlier arrival. Within normal 𝑄 𝐿 , decode precedes prefill to limit TPOT regression. A released or liveness-forced slice uses the negative rank above, then decode-before-prefill and arrival time as tie-breakers. After this ordering step, the existing scheduler consumes the ordered running and waiting sets exactly as before. The final procedure in Algorithm 2 then accounts executed tokens, refreshes per-session history, consumes a selected carryover credit, and updates the short-streak counter for the next pass. B.4

SGLang Scheduler vs. vLLM v1 Scheduler

Policy contrast. The two schedulers differ in what they treat as the first-class unit of service. SGLang is prefill-first: when a prompt-prefill batch can be formed, it prefers to build that batch before falling back to decode. vLLM v1 is running/decode-first: it advances running decode work first and then admits waiting requests if budget remains. For example, if a latency-sensitive answer prefill arrives while several decode continuations are already running, SGLang tries to serve the prefill side first, whereas vLLM v1 keeps the running decode stream in front of the queue. Implementation mapping. This contrast appears in sotainfer-engine. In SGLang, sglang/srt/managers/scheduler. py attempts prefill before decode fallback. In vLLM v1, vllm/ v1/core/sched/scheduler.py schedules running requests before waiting requests. In Frontier, the mirror lives under frontier/scheduler/replica_scheduler: sglang_style_ replica_scheduler.py is the prefill-first wrapper, and vllm_ v1_engine_replica_scheduler.py is the running-first baseline. Runtime setup. To isolate scheduler-policy effects from hardware and model differences, we run Frontier in online co-location mode on a simulated 256-H800 deployment of Qwen3-235B-A22B. Both schedulers use the same vLLMstyle parallel semantics (PP=2, attn_tp=8, attn_dp=16, moe_ep=128, cluster_num_replicas=1), chunked prefill, decode-only CUDA graph capture, disabled prefix caching, and Frontier’s monolithic MoE stage aggregation path. The active workload is sharedgpt_trace replay with 1,024 requests at qps64; this

(3)

Here 𝑧𝑟 is the sticky long-history flag, 𝐻𝑟 is cumulative served new tokens, 𝐶 is the service cap, and 𝐿 is the longround threshold. The strict ℓ𝑟 > 𝐿 matters: a 4096-token short round stays in 𝑄 𝑆 , while a 32768-token heavy hidden round enters 𝑄 𝐿 on arrival. Bounded release. The middle procedure of Algorithm 2 decides whether any long-history slice should temporarily outrank 𝑄 𝑆 . When chunked prefill stops before finishing a prompt, H2Q-BR marks the session as long-history, sets a one-shot carryover-release flag, and requeues the slice in 𝑄 𝐿 . On a later scheduling pass, it may release at most one such carryover request. The release is safe only if the request arrived no later than the oldest waiting 𝑄 𝑆 request; if no 𝑄 𝑆 request is waiting, the oldest pending carryover request is 18

Algorithm 2 H2Q-BR scheduling step

prompt length Avg. batch size

483

x1.03

x1.00

x1.00

x0.93

100.0 0.5

TFT

POT

T P95

P95

(c) Batch size over time

. Avg

500 200

0

250 0

120

240

360

Time (s)

480

0

600

1551

1077

923

3.0

s ch ers iter bat p it pty P95 No-o -em Non

(d) Prefill/decode mixing

1000 100 750

400

ch bat

472

0.0

t

E2E hpu oug Thr

265

0.0

184

50.0

0.2

600

75 50 25 0

0

120

240

360

480

600

Time (s)

Figure 21. Frontier online SharedGPT qps64 scheduler comparison. Panel (a) reports normalized macro metrics, and the bar labels show multipliers relative to vLLM. Panels (b)–(d) retain the microscheduling view of batch size, backlog, and prefill/decode mixing over time (s).

11: procedure BeforeSchedulingPass 12: Bounded Release: 13: if 𝑄 𝑆 = ∅ then 14: 𝑖 rel ← oldest pending carryover slice (with 𝑐𝑟 (𝑖 ) = 1) 15: else 16: 𝑖 rel ← oldest pending carryover slice (with 𝑐𝑟 (𝑖 ) = 1)

where 𝑎𝑖 rel ≤ min 𝑗 ∈𝑄𝑆 𝑎 𝑗

setting was selected because it sustains a saturated queueing regime while still allowing both schedulers to complete cleanly. Figure 21 should be read as a normalized comparison: the labels above the bars in panel (a) report multipliers relative to vLLM, rather than absolute latency scales. In this setting, SGLang remains within 1.06% of vLLM on P95 TTFT, lowers P95 TPOT by 7.15%, but increases P95 E2E by 3.01% and reduces throughput by 2.90%. The figure therefore does not support a simple scalar ranking; instead, it exposes a tradeoff between token-level service regularity and completionoriented efficiency. Interpretation. The micro panels explain this trade-off at the scheduler level. SGLang increases the average non-empty batch size from 183.703 to 264.554 and the P95 non-empty batch size from 471.5 to 923.0, while reducing non-empty decisions from 1,551 to 1,077 and increasing no-op decisions from 3 to 483. The saturation envelope remains similar: both schedulers reach a maximum of 868 waiting requests, and their maximum running-request counts stay close (1,017 for vLLM and 1,021 for SGLang). These counts indicate that the two policies operate under comparable pressure, but allocate that pressure differently across batch construction and admission pacing. The time-series view makes the phase structure more explicit. Between 390–420 s, SGLang’s average batch size is 49 with zero decode share, and between 450–480 s it falls to 15.5 while remaining decode-free; in the same windows, vLLM sustains average batch sizes of 523 and 506.5, with decode shares of 4.0% and 8.5%, respectively. The transition into a decode-dominant regime also occurs later for SGLang: at 480–510 s it reaches an average batch size of 467.450331 with a 96.3902% decode share, whereas vLLM already reaches

17: end if 18: if 𝜂 ≥ 𝐵 and 𝑄 𝐿 ≠ ∅ then 19: 𝑖 live ← arg min𝑖 ∈𝑄 𝐿 𝑎𝑖 ⊲ Force liveness for oldest 𝑄 𝐿 20: else 21: 𝑖 live ← ⊥ 22: end if 23: Rank Running ∪ Waiting by 𝜌 (𝑖) (Eq. 4). 24: Apply tie-breakers: (ℓ𝑟 , 𝑑𝑖 , 𝑎𝑖 ) for 𝑄 𝑆 , and (𝑑𝑖 , 𝑎𝑖 ) for 𝑄 𝐿 . 25: Pass the ordered queues to the batch builder. 26: end procedure 27: procedure OnBatchCompletion 28: for each scheduled slice 𝑖 do 29: 𝐻𝑟 (𝑖 ) ← 𝐻𝑟 (𝑖 ) + executed new tokens. 30: if round completes then 31: Refresh 𝑇𝑟last . (𝑖 )

end if if prefill slice 𝑖 makes partial progress but is unfinished

32: 33:

150.0

0.8

T P95

4: if 𝑧𝑟 ∨ (𝐻𝑟 > 𝐶) ∨ (ℓ𝑟 > 𝐿) then 5: 𝑞𝑟 ← 𝑄 𝐿 ⊲ Assign to long-history queue 6: else 7: 𝑞𝑟 ← 𝑄 𝑆 ⊲ Assign to short-history queue 8: end if 9: Enqueue slice 𝑖 to 𝑞𝑟 . 10: end procedure

(b) Scheduler decisions x1.00 x0.97

Decode token share (%) Max waiting reqs

1: procedure OnArrivalOrReentry(slice 𝑖, session 𝑟 ) 2: Load session state (𝑧𝑟 , 𝐻𝑟 ,𝑇𝑟last, 𝑐𝑟 ) from session_id. 3: ℓ𝑟 ← max(PrefillTokens(𝑖) − 𝑇𝑟last, 0) ⊲ Current-round

1.0

SGLang

(a) Macro metrics x0.99

Normalized to vLLM v1

streak counter 𝜂.

x1.00

vllm

Require: Session state (𝑧𝑟 , 𝐻𝑟 ,𝑇𝑟last, 𝑐𝑟 ) for each session 𝑟 . Short

then 34: 𝑧𝑟 (𝑖 ) ← 1, 𝑐𝑟 (𝑖 ) ← 1, 𝑞𝑟 (𝑖 ) ← 𝑄 𝐿 ⊲ Mark carryover 35: end if 36: if 𝑖 = 𝑖 rel ⊲ The selected release slice ran then 37: Clear its carryover flag 𝑐𝑟 (𝑖 ) ← 0. 38: end if 39: end for 40: if any 𝑄 𝐿 slice ran then 41: 𝜂←0 42: else 43: 𝜂 ← 𝜂 + (number of scheduled 𝑄 𝑆 slices) 44: end if 45: end procedure

19

92.8177% decode share in the same window and moves into full decode dominance one window earlier. This behavior is consistent with a prefill-first two-phase policy that consolidates prompt service before mixing in decode, while vLLM’s running-request-first ordering admits decode into the service stream earlier. Under the present 256-H800 SharedGPT setting, that difference explains why SGLang can improve TTFT and TPOT yet still exhibit a small E2E and throughput penalty: the policy shifts work into larger prompt-oriented batches, but postpones mixed-phase service and makes the later decode completion burstier.

20