CoRun: Padding is Simple and Efficient for Deterministic LLM Inference

C O RUN: Padding is Simple and Efficient for Deterministic LLM Inference Shiju Zhao∗ , Jiacheng Yang† , Qihang Chen† , Junhao Hu∗‡ , Jiaqi Zheng, Guihai Chen, Xusheng Chen†

arXiv:2608.14376v1 [cs.OS] 14 Aug 2026

State Key Laboratory for Novel Software Technology, Nanjing University, China † Hunyuan Team, Tencent, China ‡ School of Computer Science, Peking University, China

Abstract—Despite fixed sampling parameters and random seeds, Large Language Model (LLM) inference exhibits output inconsistency, which undermines downstream tasks such as model evaluation and reinforcement learning. A major source of this nondeterminism is batch-dependent GPU execution: dynamic input shapes change kernel tiling and floating-point reduction orders. Existing systems address this problem with batch-invariant kernels, but these kernels restrict optimized tiling and split reductions, increasing more than 2× latency and reducing serving throughput by up to 74%. This paper observes that although most kernels are not batch-invariant, they are position-invariant. Leveraging this property, we present C O RUN, a scheduling-based system that achieves deterministic inference without requiring batch invariance. C O RUN employs isolated prefill and fixedshape batched decode to handle the two stages of LLM inference, respectively, leveraging CUDA graphs for efficient execution and simplified implementation. Experiments on LLMs with diverse architectures, including Qwen and DeepSeek, show that C O RUN ensures determinism while improving throughput by 15–324% over batch-invariant approaches, reducing time-to-first-token by 51.8% and time-per-output-token by 48.6% on average.

88.4 End-to-end latency (ms)

80

(a) Design space

MatMul Attention RMSNorm

60 40

32.6

20

19.4

0

Standard

Batch-invariant

Ours: CoRun

(b) Latency breakdown

Fig. 1: C O RUN’s design objective and the latency opportunity. (a) Positioning in the determinism-throughput design space. (b) Decode-step latency breakdown for a batch of 32 requests with variable context lengths.

operations in Mean, RMSNorm, Softmax, matrix multiplication (MatMul), and attention [8], [9], [13]. Since floating-point addition is not associative, the same token can receive different low-order bits when it runs with different co-runners. Section III explains this mechanism in detail using an illustrative reduction kernel. Existing deterministic inference stacks address this problem with batch-invariant kernels [12], [15], [21], [23]. A batchinvariant kernel returns the same result for a token regardless of both the batch size and the token’s position. It achieves this property by retaining a fixed tiling and reduction order across supported shapes. The restriction prevents the kernel from selecting the most efficient decomposition for each input and can disable optimizations such as Split-K and Split-KV. Fig. 1a summarizes the resulting design space. Standard vLLM and SGLang provide high throughput but can produce batch-dependent outputs, while batch-invariant execution restores determinism at lower throughput. This work targets the upper-right region by retaining fast standard kernels and controlling the shapes on which they execute. Fig. 1b illustrates the opportunity with the latency breakdown of one variablelength decode step. Batch-invariant execution increases end-toend latency from 19.4 ms to 88.4 ms, with MatMul accounting for most of the increase. C O RUN reduces the latency to 32.6 ms by avoiding batch-invariant implementations for most operators, including MatMul and RMSNorm.

I. I NTRODUCTION Large language model (LLM) inference can produce different token sequences for the same prompt even when users fix the sampling parameters and random seed [3]–[5], [30]. The discrepancy is easy to overlook because it often begins as a small numerical perturbation, but autoregressive generation can amplify it into a different answer. In hosted models, repeated nominally deterministic runs have produced task-score gaps of up to 70% between the best and worst observed outcomes [4]. Such variation makes model comparisons sensitive to incidental serving conditions. It also complicates debugging and reproducibility, and can create rollout mismatches that harm Reinforcement Learning (RL) training [15], [28], [33]. Deterministic execution is consequently becoming an explicit requirement in model evaluation, debugging, stability analysis, and consistent post-training [12], [15]. A major source of this nondeterminism is dynamic GPU batching. Modern LLM servers admit and retire requests between model iterations to increase utilization [16], [29]. The resulting batch shape depends on request arrival times, prompt lengths, and generation progress. GPU libraries use these dimensions to choose tiling and parallel reduction strategies. Changing the shape can change the order of floating-point * This work was completed during his internship at Tencent.

1

5 0

50

100

150

Batch Size

200

250

(a) Mean, RMSNorm, SoftMax

10.07x

5.98x

6.35x

8192

18.22x

18.41x

15.12x

8.58x

5.82x

5.99x

4096

9.35x

9.46x

9.53x

9.26x

6.29x

6.62x

2048

6.15x

6.17x

6.45x

7.24x

6.52x

6.24x

1024

3.32x

3.38x

3.57x

3.93x

5.42x

6.02x

512

2.58x

2.57x

2.59x

2.61x

3.07x

5.06x

N (Output Dimension)

co-running requests, batch position, and arrival order without requiring every operator kernel to be batch-invariant. We formalize this guarantee as an end-to-end determinism contract and prove it by induction over autoregressive generation. Isolated prefill establishes an identical initial request state; fixed-shape decode and position-invariant kernels preserve that state across batch compositions and positions; request-bound RNG state then ensures that identical logits produce identical sampled tokens. We make the following contributions:

35 30 25 20 15 10 5

Latency Ratio (BI / STD)

10

17.74x

16 38 4

15

32.29x

81 92

20

36.20x

40 96

25

16384

20 48

30

Latency (μs)

Latency Ratio: BI / STD (matmul, batch_size=32)

51 2

softmax (STD) softmax (BI)

10 24

rmsnorm (STD) rmsnorm (BI)

K (Reduction Dimension)

mean (STD) mean (BI)

(b) MatMul

Fig. 2: Performance comparison between standard (STD) nondeterministic kernels and batch-invariant (BI) kernels.

We distinguish batch invariance from position invariance, characterize the invariance properties and costs of common LLM kernels, and identify execution-shape control as an alternative to making the full kernel stack batchinvariant. • We design C O RUN , which combines isolated prefill, fixed-shape CUDA-Graph decode, selective kernel constraints, and request-bound sampling state in an existing continuous-batching engine. We state its determinism contract and prove that these mechanisms produce the same token sequence across co-running requests, batch positions, and arrival orders. • We evaluate C O RUN on Qwen3-235B-A22B, DeepSeekV3, and Hy3 using production, code-editing, and synthetic workloads. C O RUN produces identical outputs in our determinism suite and improves throughput by 15– 324% over batch-invariant inference, more than doubling aggregate throughput for every evaluated model. •

The operator-level cost is also substantial. Fig. 2 compares standard (STD) and batch-invariant (BI) implementations of four common operators. Each point is the mean of 100 executions. Across batch sizes, BI increases the latency of Mean, RMSNorm, and Softmax by 2.12–5.61× (Fig. 2a). For matrix multiplication, the overhead grows with the reduction dimension and reaches 36.2× (Fig. 2b). A larger output dimension exposes more independent work and narrows the gap, but does not remove it. End to end, these slower operators reduce serving throughput by up to 74% (Section V-C). This paper observes that full batch invariance is stronger than necessary. Although the optimized kernels in our model stack are generally not batch-invariant, most are positioninvariant: a token retains the same result when it moves within a batch of the same shape. This weaker property suggests a different route to determinism: control the execution shape seen by each request, then retain the faster position-invariant kernels. We formalize the distinction and characterize the relevant kernels in Section III. Applying this idea to existing LLM serving systems is challenging because prefill and decode1 have different shape distributions. Prefill processes a variable-length prompt, so padding every request to one global shape would waste substantial computation. Batching several prompts, however, makes the shape of each prefill depend on its co-runners. Decode contributes one query token per active request, but the active set changes whenever a request arrives or completes. Sampling creates a final complication: a request must consume the same random number after it moves to another batch slot. We present C O RUN, a scheduling-based system for deterministic, high-throughput LLM inference. C O RUN isolates each request during prefill, preserving its natural prompt shape while removing co-runner dependence. During decode, it pads the active batch to the maximum concurrency and replays one fixed-size CUDA Graph. C O RUN disables attention Split-KV for decode and uses deterministic collectives for the small set of operators that are not position-invariant. The sampler binds Random Number Generator (RNG) state to requests rather than batch slots and pads its input to the same fixed shape. Together, these mechanisms make the output independent of

II. BACKGROUND A. LLM Inference and Continuous Batching An autoregressive LLM serves a request in two stages. The prefill stage processes all prompt tokens in parallel, materializes their key-value (KV) cache, and produces the first output token. The decode stage then generates one token per request in each model iteration, reusing the cached keys and values. Prefill therefore exposes substantial parallelism but may contain thousands of tokens, whereas each decode iteration contributes only one new token per active request. Modern serving systems batch requests at iteration granularity rather than waiting for an entire request to finish [16], [29]. This continuous batching policy admits newly arrived requests and removes completed requests between iterations, improving GPU utilization and avoiding head-of-line blocking. The resulting batch is dynamic: its request count, token count, sequence-length distribution, and token-to-slot mapping can all change from one iteration to the next. Systems can also mix prefill tokens and decode tokens in one iteration. Sarathi-Serve, for example, divides a long prefill into bounded chunks (named chunked prefill) and combines them with ongoing decodes to reduce generation stalls [2]. These scheduling techniques improve serving efficiency, but they make the GPU execution shape depend on the requests that happen to run together.

1 Prefill and decode are two stages of LLM inference (Section II-A).

2

Batch Size = 1

B. CUDA Graphs A conventional GPU execution path launches each kernel separately, incurring CPU, runtime, and driver overhead for every launch. A CUDA Graph records a sequence of kernels, memory operations, and their dependencies, instantiates that sequence once, and replays the entire graph with one launch operation [17]. Replay amortizes setup and dispatch overhead, which is particularly valuable for decode iterations composed of many short kernels. Graph replay expects the captured operations, tensor addresses, and launch configuration to remain valid. LLM engines therefore capture graphs for a set of sequence lengths and pad a smaller runtime batch to the smallest captured size that can contain it [24]. For example, an iteration with 31 query tokens can use a graph captured for 32 tokens by adding one masked padding token. The redundant computation is often cheaper than returning to the per-kernel eager path. Decode is well suited to graph capture because every active request contributes one query token and the batch size is bounded by the maximum concurrency. Prefill shapes vary over a much wider range and include long prompts or prompt chunks. Capturing enough prefill graphs would consume substantial memory, while padding a short prompt to a large capture size would waste significant computation. Serving engines therefore commonly use CUDA Graphs for decode and an eager path for prefill. This asymmetry suggests separating variable-shaped prefill from fixed-shape decode, an opportunity we exploit in the rest of this paper.

𝑥1 𝑎 𝑏 𝑐 𝑑

Batch Size = 4 𝑥2 𝑥3 𝑥1 𝑎 𝑏 𝑐 𝑑 𝑥4

Low Batch Parallelism: Enable Split-K

𝑥1 𝑎 𝑏

𝑥1 𝑐 𝑑

SM 1

SM 2

(𝑎 + 𝑏) + (𝑐 + 𝑑)

Sufficient Batch Parallelism: Disable Split-K

𝑥2 𝑥3

𝑥1 𝑎 𝑏 𝑐 𝑑 𝑥4

SM 1

≠

SM 2

𝑎+𝑏+𝑐+𝑑

(a) Adaptive tiling in a batch-dependent kernel

Batch Size = 1 𝑥1 𝑎 𝑏 𝑐 𝑑

Batch Size = 4 𝑥2 𝑥3 𝑥1 𝑎 𝑏 𝑐 𝑑 𝑥4

Same Tiling and Reduction Order ⟹ Identical Output

𝑥1 𝑎 𝑏

𝑥1 𝑐 𝑑

SM 1

SM 2

(𝑎 + 𝑏) + (𝑐 + 𝑑)

𝑥2 𝑥3

𝑥1 𝑎 𝑏 𝑥4

Unnecessary Partitions ⟹ Extra Reduction Overhead

𝑥2 𝑥3

SM 1

=

𝑥1 𝑐 𝑑 𝑥4

SM 2

(𝑎 + 𝑏) + (𝑐 + 𝑑)

(b) Fixed tiling in a batch-invariant kernel

Fig. 3: Adaptive and fixed tiling for a sum reduction. Adaptive tiling changes the reduction tree with batch size, whereas fixed tiling preserves the tree at the cost of extra work.

but makes the kernel batch-dependent: changing the batch size changes the partitioning and reduction tree used for the same row. The two trees are equivalent over real numbers but need not be equivalent in floating-point arithmetic. Addition is not associative:

III. M OTIVATION Continuous batching makes the execution shape depend on the requests that happen to run together, as discussed above. This section first explains why shape changes make standard GPU reductions batch-dependent. It then contrasts batch invariance with position invariance and shows why the latter creates an opportunity for a faster deterministic inference system.

fl(fl(a + b) + c) ̸= fl(a + fl(b + c)), because each operation rounds its result to a finite representation [13]. Consequently, two legal reduction trees can produce different low-order bits for x1 . A batch-invariant kernel removes this dependence by using a canonical partition and reduction tree for every supported batch size. Fig. 3b always divides each row into [a, b] and [c, d] and combines the two partial sums in the same order. The target row therefore produces (a + b) + (c + d) at both batch sizes. This fixed strategy sacrifices adaptivity: at batch size four, row-level parallelism is already sufficient, yet the kernel still creates partial sums and performs an additional reduction. A kernel could instead prohibit splitting at every batch size, but that choice would leave the GPU underutilized for a batch of one. Batch invariance must choose one shapeindependent strategy and thus cannot optimize both cases. The sum kernel represents a broader class of GPU reductions. Matrix multiplication may partition its reduction dimension K across thread blocks, a strategy called SplitK. Attention kernels similarly use Split-KV to partition a

A. Batch-Dependent GPU Reductions GPU kernels divide their inputs into tiles that thread blocks process in parallel. Kernel libraries adapt this tiling to the input shape to improve locality and expose enough parallel work to occupy the GPU. Consequently, the tiling selected for one token can depend on the other tokens in its batch. Fig. 3 illustrates this behavior with a sum kernel. Each row xi contains four values to reduce, and x1 = [a, b, c, d] is the target input whose output we compare across batch sizes. In Fig. 3a, a batch of one row provides little parallelism. The adaptive kernel therefore splits the reduction dimension across two Streaming Multiprocessors (SMs). The two SMs compute a + b and c + d independently, and then combine the partial sums as (a + b) + (c + d). With four rows, row-level parallelism is sufficient to occupy both SMs. The kernel no longer splits each row and instead assigns complete rows to the SMs; the target row is reduced locally in the order represented by a + b + c + d. This adaptive choice improves utilization,

3

+19.16

Batch Size = 1

standard batch-invariant

+18.88

Token 1

+13.97 +12.85

Token 2

Bitwise Consistent

Output 1

Token 1

Token 4

Kernel

Output 2 Output 3 Output 1 Output 4

(a) Batch-invariant kernel

_h e ad

Batch Size = 4 Token 1

lm

no rm

rm s

_p r oj

do wn

_pr oj

ga te_ up

o_p roj

on att en ti

Token 3

+0.87

+0.20

qk

Kernel

Batch Size = 4

+8.61

v_p roj

Latency (ms)

25 20 15 10 5 0

Token 2

Token 3

Token 4

Kernel

Output 1 Output 2 Output 3 Output 4

Bitwise Consistent

Batch Size = 4

Fig. 4: Per-forward total latency of different operators.

Token 2

Token 3

Token 1

Token 4

Kernel

Output 2 Output 3 Output 1 Output 4

(b) Position-invariant kernel

request’s KV sequence when the batch provides insufficient query parallelism [8]–[10]. In each case, changing the batch shape can change the tile size, partition count, thread-block assignment, and floating-point reduction order. The resulting numerical difference is usually small, but autoregressive generation repeatedly feeds each sampled token back into the model. A small logit perturbation can eventually change a token decision and cause two sequences to diverge [15], [30]. Fig. 4 quantifies this cost for one decode forward pass of Qwen3-235B-A22B. We aggregate the latency of each operator type across all model layers and compare the standard and batch-invariant implementations. Batch invariance increases the latency of every measured operator, but the projection and attention kernels dominate the overhead. The four MatMul groups add 8.61–19.16 ms each, and batch-invariant attention adds 13.97 ms. In contrast, RMSNorm and the language-model head (lm head) add only 0.20 ms and 0.87 ms, respectively. These results show that the performance loss is distributed across the forward pass but is concentrated in its computeintensive MatMul kernels. Replacing those kernels with their standard position-invariant implementations therefore offers the largest opportunity to recover deterministic inference performance. Most standard token-wise kernels satisfy the weaker property of position invariance. For a fixed shape, the kernel selects the same tile dimensions, thread-block layout, and reduction partitions for every batch position. Moving x1 to another row changes its address or program identifier, but does not change how its four values are partitioned or combined. The output therefore remains unchanged. Prior work calls this behavior a shape-consistent reduction and observes it in common LLM operators [14], [15].

Fig. 5: Illustration of batch invariance and position invariance.

Consider a kernel Fn that processes an ordered batch X = (x1 , . . . , xn ). For kernels with request-local metadata, such as KV lengths and page tables, let xi include that metadata. Definition 1: The kernel F is batch-invariant if a request receives the same output under arbitrary batch compositions and positions: xi = x′j =⇒ Fn (X)i = Fm (X ′ )j ,

∀n, m, i, j.

This property is stronger than deterministic execution at one shape: the request output must remain unchanged when either the batch size or its co-runners change. Definition 2: The kernel F is position-invariant if it is equivariant to permutations at a fixed shape. For every permutation matrix P , Fn (P X) = P Fn (X). Position invariance allows a request to move between batch slots, but requires the overall execution shape to remain fixed. Batch invariance implies position invariance, but the converse does not hold. In particular, a reduction kernel may choose a different tile decomposition for Fn and Fm while applying the same decomposition to every position within Fn . Existing deterministic inference stacks obtain batch invariance by fixing tile sizes and reduction trees across supported shapes [12], [15], [21], [23]. A fixed decomposition prevents co-runners from changing the arithmetic order, but it also prevents the kernel from selecting the fastest decomposition for each shape. It may reduce parallelism, add passes over intermediate results, or disable Split-K and Split-KV. TreeBased Invariant Kernels (TBIK) apply the same principle across tensor-parallel configurations by aligning intra- and inter-GPU reductions through a fixed hierarchical tree [31]. TABLE I summarizes the implementations used in our stack [14], [31]. Ring AllReduce and Split-KV attention remain position-dependent because tensor offsets or KV partitions can follow different reduction orders. C O RUN handles these exceptions with deterministic collectives and by disabling decode-time Split-KV, while retaining faster positioninvariant implementations for the remaining kernels.

B. Batch and Position Invariance The tiling example explains the numerical mechanism, while Fig. 5 states the two invariance properties from a serving perspective. Fig. 5a compares the same target token across different batch shapes and positions. A batch-invariant kernel must preserve its output in every case. Fig. 5b holds the batch shape fixed and only moves the target token. A positioninvariant kernel needs to preserve the output under this weaker change.

4

TABLE I: Invariance properties of the kernel implementations considered in this work. Fixed-tree AllReduce uses a fixed reduction topology.

Isolated Prefill Request

Isolated

LLM Engine

Fixed-Shape Batched Decode LLM Engine

Kernel Mean RMSNorm SoftMax MatMul Ring AllReduce Fixed-Tree AllReduce FlashAttention w/ Split-KV FlashAttention w/o Split-KV

Batch-Invariant ✗ ✗ ✗ ✗ ✗ ✓ ✗ ✓

Position-Invariant ✓ ✓ ✓ ✓ ✗ ✓ ✗ ✓

Stashed

Shape-Aligned Sampling

Padding

LM Head & Sample

(CUDA graph)

LM Head & Sample Output

Fig. 6: C O RUN overview. Each square represents a token; colors distinguish requests.

The key opportunity is therefore to make each inference stage present a repeatable shape to the model. Positioninvariant kernels can then retain their optimized, shape-specific tiling without allowing batch position or co-runners to change a request’s output. The next section turns this observation into C O RUN’s isolated-prefill, fixed-shape-decode, and shapealigned-sampling design.

B. Isolated Prefill A simple way to standardize prefill would pad every prompt to one global length. This design is inefficient because prompt lengths vary by orders of magnitude. A large target length wastes computation on short requests, while a small target length forces long prompts through many chunked-prefill iterations. C O RUN instead makes each prefill request its own batch. A prompt can retain its natural length, but its execution shape no longer depends on the lengths or arrival times of other requests. Isolation also preserves an important prefill optimization. Split-KV may choose its partitioning from the prompt length and kernel configuration. Because the same request always enters prefill alone, its prompt and chunk shapes remain unchanged across executions. C O RUN can therefore retain prefill-time Split-KV without requiring the attention kernel to be position-invariant. This choice avoids the substantial loss that would result from disabling Split-KV for long prompts. C O RUN implements isolation as a scheduling wrapper around the original vLLM scheduler. Algorithms 1 and 2 show the relevant logic. The scheduler maintains a running queue containing requests that have started execution and a waiting queue containing requests that have not started prefill. A request at the head of running may still require prefill when a long prompt has been divided into fixed-size chunks. In that case, Algorithm 1 selects the partial request before admitting new work, preserving its chunk sequence (lines 2-6). C O RUN starts a waiting request only when the number of requests currently being processed has not exceeded the maximum limit (lines 7-9) and the KV-cache capacity can accommodate its full configured sequence (line 10). This rule prevents a request from being preempted or recomputed because a later decode step exhausts capacity. If no request needs prefill, the wrapper invokes the original scheduler and allows a normal decode iteration. When Algorithm 1 selects a request, Algorithm 2 temporarily isolates it without replacing the original scheduler. The algorithm first calls Algorithm 1 to find a request requiring prefill (line 2). If no such request exists, it immediately invokes the original scheduler, which produces a normal decode iteration (lines 3–4). Otherwise, it copies both request queues so

IV. D ESIGN C O RUN builds on the opportunity identified in Section III: it controls the execution context seen by each request instead of replacing every optimized operator with a batch-invariant implementation. Fig. 6 presents the design. C O RUN isolates variable-shaped prefill work (Section IV-B), pads all decode iterations to one captured shape (Section IV-C), and aligns sampling with the same fixed request layout (Section IV-D). A. Determinism Contract Let Y (x, s; C, p, a) denote the output token sequence for prompt x and per-request sampling seed s, where C is the set of co-running requests, p is the request’s batch position, and a is the request arrival order. Given the same model weights, sampling parameters, hardware, software stack, and tensorparallel configuration, C O RUN provides Y (x, s; C, p, a) = Y (x, s; C ′ , p′ , a′ ) for arbitrary valid co-runners, positions, and arrival orders. C O RUN establishes this contract by making each target request observe the same request-local inputs and arithmetic order. Isolated prefill removes other requests from the prompt computation. Fixed-shape decode makes the launch shape independent of the number of active requests. Position-invariant kernels then tolerate slot changes within that shape. C O RUN retains a fixed tree for collective reductions and disables decode-time attention Split-KV, the two components in TABLE I that are not position-invariant. Finally, request-bound RNG state makes seeded multinomial sampling independent of the batch slot. The contract does not claim reproducibility across hardware architectures, kernel versions, numerical formats, or tensorparallel degrees. Those changes can alter the arithmetic itself and require orthogonal techniques such as cross-configuration invariant kernels [31].

5

Algorithm 1 Finding the request for isolated prefill

Algorithm 2 Scheduling isolated prefill

Input: Request queues running and waiting Output: A request requiring prefill, or none Hyperparameter: The maximum number of running requests max num reqs. 1: function FIND PREFILL REQUEST 2: if running ̸= ∅ then 3: head ← running[0] 4: if (head.num computed tokens < 5: head.num prompt tokens) then 6: return head ▷ Continue chunked prefill 7: if (waiting ̸= ∅ & 8: LENGTH (running) < max num reqs) then 9: head ← waiting.PEEK REQUEST() 10: if CAN FIT FULL SEQUENCE(head) then 11: return head 12: return none

Input: Request queues running and waiting Output: Scheduling result output 1: function SCHEDULE 2: det ← FIND PREFILL REQUEST() 3: if det == none then 4: return SCHEDULE ORIGIN() 5: stashed running ← COPY(running) 6: stashed waiting ← COPY(waiting) 7: waiting.CLEAR() 8: if det in running then 9: running ←[det] 10: stashed running.REMOVE(det) 11: else 12: running.CLEAR() 13: waiting.ADD REQUEST(det) 14: stashed waiting.REMOVE(det) 15: output ← SCHEDULE ORIGIN() 16: running ← running + stashed running 17: for request in stashed waiting do 18: waiting.ADD REQUEST(request) 19: return output

that the requests hidden during isolated prefill can be restored afterward (lines 5–6). It then clears the waiting queue to prevent the original scheduler from admitting another request into the prefill batch (line 7). Lines 8–14 expose only the selected request to the original scheduler. If the request has already started chunked prefill, it resides in running; the algorithm replaces running with a singleton queue and removes the request from the stashed copy (lines 8–10). Otherwise, the request has not started prefill. The algorithm clears running, inserts the selected request into waiting, and removes it from the stashed waiting queue to avoid restoring a duplicate entry (lines 11–14). The call to SCHEDULE ORIGIN then produces an ordinary scheduling result for this single request (line 15). Finally, the algorithm appends the stashed running requests and reinserts the stashed waiting requests in their original order (lines 16–18), then returns the isolated-prefill result (line 19). The model runner therefore executes an ordinary prefill iteration with batch size one; isolation requires no separate model-execution path.

limits the engine to 256 concurrent sequences. Since isolated prefill prevents mixed prefill-decode iterations, every decode iteration contains at most 256 decode tokens. The engine pads that iteration to the only captured size, making its launch shape exactly 256. The capture size is configurable; we use 256 to match the maximum concurrency in our evaluation. This construction reuses the engine’s existing graph padding and masking logic; C O RUN does not add a second modelexecution path. It also amortizes the cost of padding through graph replay. Padding can be expensive when only a few requests remain, but it has little cost while the engine is highly utilized. This tradeoff motivates C O RUN’s focus on batched evaluation and RL rollout workloads rather than lowconcurrency interactive serving. The fixed query count alone is insufficient for attention because requests have different KV lengths. A Split-KV implementation can vary the number and placement of KV partitions according to the ragged batch. C O RUN disables Split-KV during decode so that each request follows a request-local reduction order independent of its neighbors. Other model kernels use the same launch shape and are position-invariant, allowing requests to move among the 256 slots as earlier requests finish.

C. Fixed-Shape Batched Decode Each active request contributes exactly one query token to a decode iteration. C O RUN therefore chooses the engine’s maximum concurrency, max num reqs, as the fixed decode shape. High-throughput offline workloads usually keep the running queue near this limit, so most slots perform useful work. When fewer requests are active, masked padding tokens occupy the unused slots. C O RUN realizes fixed-shape decode through the existing CUDA Graph interface, without modifying the model runner or CUDA Graph implementation. Our configuration uses:

D. Shape-Aligned Sampling The model runner produces logits, but the sampler converts those logits into the next token. Sampling executes outside the captured CUDA Graph in our stack, so graph padding does not automatically standardize its input. During prefill, the engine selects only the final prompt token’s hidden state; isolated prefill therefore presents one sampling row. During

-cc.cudagraph_mode=full_decode_only \ --cudagraph-capture-sizes 256 \ --max-num-reqs 256 The first option restricts graph execution to decode. The second captures one graph for 256 query tokens, and the third

6

decode, C O RUN pads the sampler input to max num reqs, matching the fixed model-output shape. C O RUN supports seeded multinomial sampling by binding RNG state to a request rather than to a batch slot. Padding entries never consume a real request’s RNG state. Removing a completed request or moving a live request to another slot also leaves every other request’s RNG progression unchanged. The target request consequently receives the same random variate at each generation step. Given the identical logits produced by the model path, the same per-request random variate selects the same token and completes the determinism contract.

BurstGPT

Blazedit

0.00175

0.0030

0.00150

0.0025

Density

Density

0.00125 0.00100 0.00075

0.0015 0.0010

0.00050

0.0005

0.00025 0.00000

0.0020

0

2000

4000

6000

8000

0.0000

0

Input length (tokens)

(a) Input Length

500

1000

1500

2000

2500

3000

3500

Output length (tokens)

(b) Output Length

Fig. 7: Input and output length distribution.

E. Determinism Argument We now show that the mechanisms above satisfy the contract stated at the beginning of this section. The argument assumes the fixed hardware, software, model, and tensorparallel configuration specified by the contract. It also uses the verified properties in TABLE I: every kernel exercised by the evaluated model configurations has been verified to be position-invariant or batch-invariant [14], and each request’s model computation is isolated from the values of other requests. The latter property follows from request-local attention masks, KV caches, and token-wise model operators. Corunners may affect the execution shape, but cannot participate in the target request’s mathematical computation. Let ztr denote the state of request r before generation step t. It contains the prompt and generated prefix, the request’s KV cache, and its sampling parameters. Consider two executions, A and B, with the same prompt x and request seed s, but arbitrary co-runners, batch positions, and arrival orders. We prove by induction that ztr,A = ztr,B

and

r,A r,B y<t = y<t

where d is the request-local decode function. The equality also covers collectives and attention because C O RUN uses fixed-tree reductions and disables decode-time Split-KV. Thus, the two executions produce identical logits ℓr,A = ℓr,B and t t identical updated KV-cache values. Let R(s, t) be the random variate consumed by request r at step t, and let S be the deterministic sampling function. Since RNG state is bound to the request and padding does not consume it, r,B r,B ytr,A = S(ℓr,A t , R(s, t)) = S(ℓt , R(s, t)) = yt .

Appending the same token to identical prefixes and KVr,A r,B cache states yields zt+1 = zt+1 , completing the induction. Therefore, the complete output sequence is identical: Y (x, s; C, p, a) = Y (x, s; C ′ , p′ , a′ ). Arrival order and co-runners can change when a request executes, but under C O RUN they cannot change its sequence of model states, random variates, or sampled tokens.

∀t,

r is the generated token prefix before step t. where y<t a) Base case: Isolated prefill executes request r without co-runners. Because both executions have the same prompt and configuration, they present the same prefill shapes and values to every kernel. Each kernel therefore follows the same tiling and reduction order, producing identical logits and KV-cache states: z0r,A = z0r,B .

V. E VALUATION We evaluate whether C O RUN (1) preserves token-sequence determinism, (2) recovers the throughput lost by batchinvariant inference, and (3) remains effective across models, workloads, and concurrency levels. A. Experimental Setup

The request-bound RNG is initialized from the same seed s, so its initial state is also identical. b) Inductive step: Assume ztr,A = ztr,B . C O RUN pads both decode batches to the same captured size N = max num reqs. Request r may occupy positions pA and pB , and the remaining N − 1 positions may contain different requests or padding. Let XtA and XtB denote the two padded decode batches at step t, including their request-local tokens, KV-cache metadata, attention masks, and padding entries. Let DN denote one fixed-shape model step and πp select the output at position p. Request isolation and position invariance give

a) Models: We select three large open models with different architectures: Qwen3-235B-A22B [27], DeepSeekV3 [11], and Hy3 [22]. b) Workloads: We evaluate three workload classes and sample 300 requests from each workload. BurstGPT contains traces collected from regional Azure OpenAI services and captures the burstiness and length variation of production traffic [26]. Its length distribution is long-tailed, with a small number of prompts and responses exceeding 8 K tokens, as shown in Fig. 7. Blazedit is a code-editing workload characterized by long inputs and long outputs [7]. We also construct synthetic workloads with fixed input and output lengths to isolate how each length affects C O RUN.

d(ztr,A ) = πpA DN (XtA ) = πpB DN (XtB ) = d(ztr,B ),

7

non-determinism Throughput (tokens/s)

c) Baselines: We implement C O RUN in vLLM 0.25.0 [16]. We compare three configurations. Nondeterminism (STD) is the default vLLM execution path, which does not enforce deterministic outputs. It represents the performance ceiling of a system without determinism constraints. Batch-invariant (BI) enables vLLM’s batchinvariant execution. C O RUN uses isolated prefill, fixed-shape decode, shape-aligned sampling, and the fixed-tree collective and decode-time Split-KV policy described in Section IV-C. All configurations use the same model, tensor-parallel degree, memory limits, and request set. d) Metrics: Our primary metric is end-to-end output token throughput because model evaluation and RL rollout process a finite batch of requests and seek to minimize its total completion time. For a collection of tasks, we report aggregate throughput: P Throughputi × Timei . Aggregate Throughput = i∈TaskP i∈Task Timei

batch-invariant

CoRun

in=4096 in=1024 in=4096 out=128 out=2048 out=2048

Total

5000 4000 3000 2000 1000 0

BurstGPT

Blazedit

in=1024 out=128

Throughput (tokens/s)

(a) Qwen3-235B-A22B 3000

2000

1000

0

BurstGPT

Blazedit

in=1024 out=128

in=4096 in=1024 in=4096 out=128 out=2048 out=2048

Total

Throughput (tokens/s)

(b) DeepSeek-V3

This time-weighted average gives greater weight to tasks that occupy the system for longer. An unweighted arithmetic mean would assign the same importance to a short task and a long-running task, even though optimizing the latter has a larger effect on the total time required to finish the complete workload. We also report time-to-first-token (TTFT) and timeper-output-token (TPOT) to expose the latency effects of isolated prefill and fixed-shape decode.

5000 4000 3000 2000 1000 0

BurstGPT

Blazedit

in=1024 out=128

in=4096 in=1024 in=4096 out=128 out=2048 out=2048

Total

(c) Hy3

Fig. 8: Throughput evaluation on diverse datasets and models. “in=m, out=n” refers to a dataset with an input length of m and an output length of n.

B. Determinism Validation We verify the determinism of our method using vLLM’s batch-invariance test harness [25]. It compares isolated and batched execution under seeded multinomial sampling. For each evaluated model, the test generates 32 prompts and executes each prompt first at batch size one and then as part of a batch of 32. It uses temperature 0.6, seed 42, and generates up to 8 output tokens per prompt. The comparison checks the complete generated token sequence and the sampled-token log probability at every generation step. All 32 prompts pass for each of the three models: C O RUN returns identical token sequences and exactly matching sampled-token log probabilities between isolated and batched execution. This test changes both the co-running request set and the batch position for all prompts. Together with the request-independent execution shapes and requestbound RNG state established in Section IV-D, the result validates the batch-composition and position components of C O RUN’s determinism contract for the evaluated stack.

The gain varies with workload shape. C O RUN’s smallest improvements occur on BurstGPT. Its long tail leaves only one request active near the end of the run, so C O RUN continues to replay the fixed-size decode graph while most slots contain padding. The synthetic workloads and Blazedit maintain useful concurrency for longer and amortize that padding across more active requests. They also expose the high kernel cost of BI, producing larger relative gains. Notably, on DeepSeek-V3 with input length 4096 and output length 128, C O RUN even exceeds STD. We found that for this workload, the standard scheduler frequently combines prefill and decode requests. DeepSeek-V3 uses the FLASH ATTN MLA attention backend, which cannot use a CUDA Graph for attention when the batch mixes prefill and decode. C O RUN separates the two stages, allowing every decode iteration to replay the captured graph. This graph benefit outweighs C O RUN’s padding and isolation overhead in this configuration. Models using the FLASH ATTN backend can retain graph execution for mixed batches, so their STD configurations do not receive the same penalty. a) Throughput over time: Fig. 9 separates prefill and decode throughput over time on BurstGPT. At time zero, 256 requests arrive, so all three systems initially devote most of

C. End-to-End Throughput Fig. 8 reports throughput across the three models and six workload configurations. C O RUN improves throughput by 15– 324% over BI. Its aggregate throughput more than doubles for every evaluated model. These results show that replacing broad batch invariance with fixed-shape scheduling recovers a large fraction of the performance available to STD.

8

CoRun

non-determinism 1.0

7500 5000 2500

0.6

0.6

0.2

0.0

0.0

40

60

80

Time (s)

100

120

5000 10000 15000 20000 25000 30000 35000

in=4096 out=128

140

20000

30000

40000

TTFT (ms)

50000

0

in=1024 out=2048

0.8

0.8

0.6

0.6

0.6

0.4

0.4 0.2

0.0 20000 40000 60000 80000 100000120000140000

0.4

0.0 0

TTFT (ms)

TTFT (ms)

in=4096 out=2048

0.2

0.0 0

5000 10000 15000 20000 25000 30000 35000

1.0

0.8

3000

5000 10000 15000 20000 25000 30000 35000

0

TTFT (ms)

20000 40000 60000 80000 100000120000140000

TTFT (ms)

(a) Time to first token

2000

BurstGPT

1.0

60

80

Time (s)

100

120

140

0.8

0.8

0.8

0.6

0.6

0.6

0.4

0

(b) Decode throughput over time

1000

2000

4000

TPOT (ms)

5000

6000

0.0 25

in=4096 out=128

1.0

50

75

100

125

150

TPOT (ms)

175

200

50

in=1024 out=2048

1.0 0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.2

0.2

0.0

0.0 0

200

400

600

TPOT (ms)

800

1000

100

150

200

250

TPOT (ms)

300

350

in=4096 out=2048

1.0

0.8

CDF

CDF

3000

0.4 0.2

0.0

0.0

their capacity to prefill and shift toward decode later in the run. The client submits a new request whenever one request completes, until it has submitted all 300 requests. STD progresses through both stages fastest. It completes all prefill work in 20 seconds and all decode work in 37 seconds. BI has substantially lower throughput in both stages, requiring approximately 40 seconds to finish prefill and 124 seconds to finish decode. C O RUN instead front-loads isolated prefill: it performs no decode work during the first 30 seconds while processing prompts one at a time. This serialized prefill is its primary source of overhead. Once C O RUN enters its decodeintensive phase, however, its decode throughput approaches that of STD and reaches 83% of STD’s peak. The large decode-throughput improvement over BI accounts for most of C O RUN’s end-to-end gain and makes it particularly effective for decode-intensive workloads such as RL rollout.

0.4 0.2

0.2

Fig. 9: Prefill and decode throughput over time using the Qwen3-235B-A22B model and the BurstGPT dataset.

CDF

40

in=1024 out=128

1.0

CDF

20

Blazedit

1.0

CDF

0

CDF

1000 0

0.0 10000

1.0

0.2

4000

0.4 0.2

0

TTFT (ms)

CDF

20

0.4

0.2

1.0

(a) Prefill throughput over time

Decode Throughput (token/s)

0.8

0.6 0.4

CoRun in=1024 out=128

1.0

0.8

0

0

1.0

CDF

CDF

10000

batch-invariant Blazedit

0.8

CDF

12500

0

BurstGPT

CDF

batch-invariant

CDF

Prefill Throughput (token/s)

non-determinism

0.4 0.2 0.0

20

40

60

80

100

TPOT (ms)

120

140

20

40

60

80

100 120 140 160 180

TPOT (ms)

(b) Time per output token

Fig. 10: Latency analysis on Hy3.

non-determinism

batch-invariant

4000

3000

2000

1000

0

CoRun

120

Throughput (tokens/s)

Throughput (tokens/s)

5000

50

100

150

200

250

Concurrency

100

80

60

40

0

20

40

60

80

100

120

Padding size

Fig. 11: Scalability evaluation Fig. 12: The impact of padding under different concurrencies. in the decode phase.

D. Latency Analysis Fig. 10 shows the TTFT and TPOT distributions for Hy3. C O RUN generally lies between STD and BI. STD can batch or chunk multiple prefills and therefore provides the lowest latency, while BI pays the latency of slower invariant kernels. C O RUN retains fast kernels but serializes requests’ prefill work. Across the six workloads, C O RUN reduces mean TTFT by 51.8% and mean TPOT by 48.6% relative to BI. Isolated prefill produces a staircase in TTFT: requests wait while earlier prompts complete their prefill stages, and then advance as each isolated prompt leaves the queue. The effect is most visible on BurstGPT. Requests that arrive before a long prompt see short waits, while requests behind that prompt inherit its long prefill time. Decode padding similarly creates a TPOT tail when BurstGPT’s active set drains. Despite these

scheduling costs, C O RUN retains lower TTFT and TPOT than BI for most of the distribution because it avoids the latency of broadly batch-invariant kernels. E. Scalability with Concurrency Fig. 11 evaluates Hy3 on Blazedit while varying the maximum number of concurrent requests. C O RUN’s throughput rises with concurrency because more fixed decode slots perform useful work and isolated prefills are amortized over a longer period of batched decoding. BI improves more slowly because its invariant kernels remain the bottleneck as the batch grows. C O RUN therefore benefits from the high concurrency available in offline evaluation and rollout workloads.

9

1000 0

BurstGPT

0.91x 0.76x 0.77x 0.37x 0.36x

2000

Blazedit

in=1024 out=128

show that higher-precision accumulation reduces but does not eliminate the problem [30]. These effects matter beyond evaluation [20]. Training-inference mismatch can cause rollout tokens and training-time log probabilities to disagree, independently degrading LLM reinforcement learning [28], [33]. b) Deterministic LLM inference: Thinking Machines, SGLang, and vLLM obtain deterministic serving by replacing batch-dependent reductions with batch-invariant kernels and deterministic collectives [15], [21], [23]. DeepSeek-V4 similarly uses deterministic execution to support debugging, stability analysis, and reproducible post-training behavior [12]. TBIK extends invariance across tensor-parallel degrees by mapping reductions onto a unified hierarchical binary tree [31]. C O RUN addresses a different axis: it keeps the model and tensor-parallel configuration fixed, but prevents dynamic request batching from changing the execution of a target request. LLM-42 avoids running every token through batch-invariant kernels. It generates tokens on a fast dynamic path, periodically verifies them with a deterministic fixed-shape path, and rolls back when verification fails [14]. MarginGate reduces this verification load by triggering deterministic verification only when the decision margin indicates that a numerical perturbation could change the selected token [6]. These methods preserve flexible batching through speculation and verification. C O RUN instead constrains prefill and decode shapes before execution. It does not generate unverified tokens, invoke a separate verifier, or roll back requests. This design targets workloads in which every request requires determinism and sufficient concurrency is available to amortize fixed-shape decode. c) LLM serving and scheduling: Orca introduced iteration-level scheduling to admit and retire requests between decoding steps [29]. vLLM combines continuous batching with paged KV-cache management to sustain larger dynamic batches [16], while SGLang adds a structured runtime and cache-aware execution for complex model programs [32]. Sarathi-Serve uses chunked prefill to bound generation stalls and enable stall-free batching of prefill and decode [2]. Other systems disaggregate or independently schedule the two stages to meet throughput and latency objectives [18], [19], [34]. C O RUN deliberately restricts this scheduling freedom: it serializes prefill across requests and standardizes decode shape to make optimized position-invariant kernels deterministic.

0.87x 0.71x 0.71x

3000

CoRun

0.99x 0.84x 0.84x

4000

0.96x

0.85x 0.66x 0.66x

5000

+isolated prefill 0.93x 0.90x 0.90x

+disable splitkv

0.59x 0.44x 0.40x

Throughput (tokens/s)

non-determinism

in=4096 in=1024 in=4096 out=128 out=2048 out=2048

Total

Fig. 13: Ablation study.

F. Ablation Study C O RUN differs from STD in three performance-relevant ways: it pads decode to a fixed shape, disables attention SplitKV during decode, and isolates prefill. We isolate these costs to identify when the scheduling approach is most effective. a) Decode padding: Fig. 12 measures the worst case for fixed-shape decode. Only one request is active while we vary C O RUN’s padding size; STD and BI do not pad and therefore remain constant. C O RUN throughput decreases approximately inversely with the padding size because nearly all added tokens are redundant. When the padding size exceeds 64 in this single-request experiment, C O RUN falls below BI. This result defines the system’s intended operating region: C O RUN targets high-concurrency offline inference, where most graph slots are occupied, rather than single-request serving. b) Split-KV and isolated prefill: Fig. 13 starts from STD and adds the remaining constraints. +disable splitkv disables Split-KV only for decode attention; isolated prefill is disabled. +isolated prefill then separates prefill from all concurrent work and completes the C O RUN scheduling policy. Disabling Split-KV costs 15–41% when prompt lengths vary, because the ragged batch offers uneven attention work that benefits from additional KV partitions. The loss falls to 1– 9% when input lengths are similar. Isolated prefill accounts for the larger share of C O RUN’s gap to STD because it gives up batching across prompts and prevents prefill-decode mixing. The isolation cost decreases as inputs grow. Long prompts reduce the number of prefill requests that can profitably share an iteration, limiting the benefit of batching them in STD. The cost also decreases as outputs grow because decode occupies a larger fraction of end-to-end execution. C O RUN is therefore particularly well suited to workloads with long inputs, long outputs, or both, including the code-editing and RL rollout workloads that motivate deterministic offline inference.

VII. L IMITATIONS AND D ESIGN T RADEOFFS As discussed in the previous section, LLM-42 uses verified speculation to retain dynamic batching [14]. This design complements C O RUN in two scenarios where fixed-shape scheduling is less suitable.

VI. R ELATED W ORK a) Numerical nondeterminism in LLM inference: Empirical studies show that nominally deterministic API settings can still produce different text and task scores. Atıl et al. observe output variation across all tested hosted models and report a gap of up to 70% between the best and worst task outcomes available across repeated runs [4]. Yuan et al. trace such divergence to numerical sources within LLM inference, including precision and batch-dependent computation, and

A. Sparse Deterministic Traffic LLM-42 can apply verification only to requests that require deterministic outputs, while all other requests remain on the unmodified fast path. Its overhead therefore scales with the fraction of deterministic traffic, making it attractive when that fraction is small.

10

4000 3000

non-determinism batch-invariant CoRun LLM42 @10% LLM42 @50% LLM42 @100%

Rollback Ratio of LLM-42

Throughput (tokens/s)

5000

2000 1000 0

50

100

150

Concurrency

200

250

cution provides more opportunities for small numerical differences to affect a token decision. Qwen3-30B-A3B, a Mixtureof-Experts (MoE) model, reaches 45.41%, nearly twice the ratio of the similarly sized dense Qwen3-32B. One plausible explanation is that a small router perturbation can change the selected experts, after which the two execution paths perform different computations rather than merely accumulating different low-order bits. High rollback rates may therefore limit verified speculation on large MoE models.

45.41% 0.4 0.3 0.2

22.78%

24.25%

Qwen314B

Qwen332B

17.94%

0.1 0.0

Qwen38B

Qwen330B-A3B

(a) Throughput under mixed de- (b) Rollback ratio across model terministic traffic sizes

Fig. 14: Applicability and rollback cost of LLM-42. The verification window is 64 tokens in both experiments.

VIII. F UTURE W ORK The ablation study identifies isolated prefill as C O RUN’s largest source of overhead. We consider two directions for reducing this cost.

Fig. 14a compares throughput across concurrency levels and deterministic-request fractions. “LLM-42 @ k%” denotes a workload in which k% of requests require determinism; we set its verification window to 64 tokens. At a low deterministic fraction, LLM-42 can approach or occasionally exceed C O RUN because most requests avoid verification. As the fraction grows, verification becomes frequent and its throughput falls. When all requests require determinism, as in model evaluation and RL rollout, LLM-42 performs below both C O RUN and the BI baseline in our experiment. The verification rate explains this trend. At concurrency 256, each decode iteration produces 256 candidate tokens. With a 64-token window, an average of 256/64 = 4 requests become eligible for verification per iteration. Each verification replays a fixed-shape model step, which is substantially more expensive than generating tokens on a fast dynamic path. Frequent verification therefore dominates execution when deterministic traffic is dense. C O RUN is better suited to this all-deterministic, high-concurrency regime.

A. Batch-Invariant Kernels for Prefill C O RUN isolates prefill because mixing prefill and decode would destroy the fixed decode shape. Prefill requests could still be batched with other prefill requests, which would substantially reduce the cost of short prompts. Such batching changes the model input shape, so the prefill stage would need batch-invariant kernels. A hybrid design could use batch-invariant kernels for batched prefill and faster positioninvariant kernels for fixed-shape decode. Its performance relative to isolated prefill requires experimental study, and maintaining different kernel paths for the two stages introduces an engineering challenge. B. Prefill-Decode Disaggregation C O RUN’s separation of prefill and decode maps naturally to existing prefill-decode disaggregation systems [18], [19], [34]. Prefill and decode instances can use different parallelization and kernel configurations. Decode instances can apply C O RUN’s fixed-shape execution without modification, while the deployment can provision more prefill instances to prevent isolated prefill from limiting end-to-end throughput. Disaggregation also simplifies the hybrid-kernel design above: prefill instances can use batch-invariant kernels, while decode instances retain optimized position-invariant kernels.

B. Variable-Concurrency Online Serving LLM-42 is also better suited to online services whose request rate is unknown or changes over time. C O RUN must choose a maximum concurrency before launch because that value determines the padded decode shape. Setting it too high adds many padding tokens when traffic is light and reduces decode throughput. Setting it too low leaves requests waiting outside the running batch even when the system could exploit more parallelism. No single fixed setting is efficient across a wide range of dynamically changing arrival rates. By retaining dynamic batching on its fast path, LLM-42 adapts to variable concurrency without this configuration constraint.

IX. C ONCLUSION This paper shows that full batch invariance is unnecessary when optimized kernels are position-invariant and the serving system controls their execution shapes. C O RUN applies this insight through isolated prefill, fixed-shape CUDA-Graph decode, selective constraints on Split-KV and collective reductions, and request-bound sampling state. We formalize this guarantee through an end-to-end determinism contract and an inductive argument showing that request state and sampling decisions remain unchanged across co-runners, batch positions, and arrival orders. Across three large models and diverse workloads, C O RUN preserves deterministic token generation while improving throughput by 15–324% over batch-invariant inference. C O RUN is most effective for high-concurrency workloads in which all requests require deterministic outputs. These results establish execution-shape control as a practical alternative to making the entire LLM stack batch-invariant.

C. Rollback Cost of Verified Speculation Our experiments also reveal a limitation of LLM-42. Fig. 14b reports its rollback ratio on ShareGPT [1] with a 64-token verification window. We define the rollback ratio as the number of rejected candidate tokens divided by the total number of tokens generated on the nondeterministic fast path. A larger ratio represents more wasted computation and a higher verification penalty. The rollback ratio rises from 17.94% on Qwen3-8B to 24.25% on Qwen3-32B, suggesting that deeper model exe-

11

R EFERENCES

Y. Zhang, Y. Xu, Y. Ma, Y. Huang, Y. Li, Y. Li, Y. Xu, Y. Zhao, Y. Sun, Y. Wang, Y. Qian, Y. Shao, Y. Yu, Y. Zhang, Y. Ding, Y. Shi, Y. Wu, Y. Xiong, Y. Ma, Y. He, Y. Tang, Y. Zhou, Y. Luo, Y. Zhong, Y. Piao, Y. Wang, Y. Zhang, Y. Chen, Y. Tan, Y. Wei, Y. Ma, Y. Liu, Y. Yang, Y. Guo, Y. Wu, Y. Wu, Y. Li, Y. Cheng, Y. Ou, Y. Xu, Y. Li, Y. Wang, Y. Yang, Y. Xu, Y. Wu, Y. Meng, Y. Zou, Y. Zha, Y. Xiong, Y. Chen, Y. Lin, Y. Cao, Y. Wang, Y. Zhang, Y. Yan, Y. Lin, Y. Gu, Y. Luo, Y. You, Y. Liu, Y. Zhou, Y. Zhou, Y. Huang, Z. Wu, Z. Wang, Z. Zhao, Z. Ren, Z. Zhang, Z. Sha, Z. Fu, Z. Ju, Z. Xu, Z. Xie, Z. Zhang, Z. Gao, Z. Hao, Z. Gou, Z. Ma, Z. Yan, Z. Shao, Z. Huang, Z. Chen, Z. Wu, Z. Ren, Z. Wu, Z. Li, Z. Zhang, Z. Xu, Z. Wang, Z. Qu, Z. Gu, Z. Zhu, Z. Li, Z. Zhang, Z. Xie, Z. Gao, Z. Wan, Z. Pan, and Z. Yao, “Deepseek-v4: Towards highly efficient million-token context intelligence,” 2026. [Online]. Available: https://arxiv.org/abs/2606.19348 [13] D. Goldberg, “What every computer scientist should know about floating-point arithmetic,” ACM Comput. Surv., vol. 23, no. 1, p. 5–48, Mar. 1991. [Online]. Available: https://doi.org/10.1145/103162.103163 [14] R. Gond, A. K. Kamath, R. Ramjee, and A. Panwar, “Llm-42: Enabling determinism in llm inference with verified speculation,” 2026. [Online]. Available: https://arxiv.org/abs/2601.17768 [15] H. He and T. M. Lab, “Defeating nondeterminism in llm inference,” Thinking Machines Lab: Connectionism, 2025, https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llminference/. [16] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” in Proceedings of the 29th Symposium on Operating Systems Principles, ser. SOSP ’23. New York, NY, USA: Association for Computing Machinery, 2023, p. 611–626. [Online]. Available: https://doi.org/10.1145/3600006.3613165 [17] NVIDIA Corporation, CUDA Programming Guide: CUDA Graphs, 2026. [Online]. Available: https://docs.nvidia.com/cuda/cudaprogramming-guide/04-special-topics/cuda-graphs.html [18] P. Patel, E. Choukse, C. Zhang, A. Shah, I. Goiri, S. Maleki, and R. Bianchini, “Splitwise: Efficient generative llm inference using phase splitting,” in 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), 2024, pp. 118–132. [19] R. Qin, Z. Li, W. He, J. Cui, F. Ren, M. Zhang, Y. Wu, W. Zheng, and X. Xu, “Mooncake: Trading more storage for less computation — a KVCache-centric architecture for serving LLM chatbot,” in 23rd USENIX Conference on File and Storage Technologies (FAST 25). Santa Clara, CA: USENIX Association, Feb. 2025, pp. 155–170. [Online]. Available: https://www.usenix.org/conference/fast25/presentation/qin [20] Rainbird AI, “Deterministic graph-based inference: The key to safe ai in financial services,” https://rainbird.ai/deterministic-graph-basedinference-the-key-to-safe-ai-in-financial-services/, 2025. [21] SGLang Team, “Towards deterministic inference in sglang and reproducible rl training,” https://www.lmsys.org/blog/2025-09-22-sglangdeterministic/, 2025. [22] Tencent Hunyuan Team, “Tencent Hy3,” https://huggingface.co/tencent/ Hy3, 2026. [23] vLLM Team, Batch Invariance, https://docs.vllm.ai/en/latest/features/ batch invariance/, 2026. [24] vLLM Team, CUDA Graphs, https://docs.vllm.ai/en/latest/design/cuda graphs/, 2026. [25] vLLM Team, “test batch invariance,” https://github.com/vllm-project/ vllm/blob/main/tests/v1/determinism/test batch invariance.py, 2026. [26] Y. Wang, Y. Chen, Z. Li, X. Kang, Y. Fang, Y. Zhou, Y. Zheng, Z. Tang, X. He, R. Guo, X. Wang, Q. Wang, A. C. Zhou, and X. Chu, “Burstgpt: A real-world workload dataset to optimize llm serving systems,” in Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, ser. KDD ’25. New York, NY, USA: Association for Computing Machinery, 2025, p. 5831–5841. [Online]. Available: https://doi.org/10.1145/3711896.3737413 [27] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu, “Qwen3 technical report,” 2025. [Online]. Available: https://arxiv.org/abs/2505.09388

[1] “Sharegpt vicuna unfiltered,” https://huggingface.co/datasets/ anon8231489123/ShareGPT Vicuna unfiltered, 2023. [2] A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. Gulavani, A. Tumanov, and R. Ramjee, “Taming Throughput-Latency tradeoff in LLM inference with Sarathi-Serve,” in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). Santa Clara, CA: USENIX Association, Jul. 2024, pp. 117–134. [Online]. Available: https://www.usenix.org/conference/osdi24/presentation/agrawal [3] S. Anadkat, How to make your completions outputs consistent with the new seed parameter, https://developers.openai.com/cookbook/examples/ reproducible outputs with the seed parameter, 2023. [4] B. Atıl, S. Aykent, A. Chittams, L. Fu, R. J. Passonneau, E. Radcliffe, G. R. Rajagopal, A. Sloan, T. Tudrej, F. Ture, Z. Wu, L. Xu, and B. Baldwin, “Non-determinism of “deterministic” LLM system settings in hosted environments,” in Proceedings of the 5th Workshop on Evaluation and Comparison of NLP Systems, M. Akter, T. Chowdhury, S. Eger, C. Leiter, J. Opitz, and E. Çano, Eds. Mumbai, India: Association for Computational Linguistics, Dec. 2025, pp. 135–148. [Online]. Available: https://aclanthology.org/2025.eval4nlp-1.12/ [5] S. Chann, “Non-determinism in gpt-4 is caused by sparse moe,” https: //152334h.github.io/blog/non-determinism-in-gpt-4/, 2023. [6] K. Chu, Y. Zhou, and W. Zhang, “Margingate: Sparse margin-triggered verification for batch-invariant llm inference,” 2026. [Online]. Available: https://arxiv.org/abs/2605.30218 [7] V. Daita, X. Lian, L. Zhang, and J. Liu, “Blazing-fast code editing via multi-layer speculation,” https://github.com/ise-uiuc/blazedit, 2025. [8] T. Dao, “Flashattention-2: Faster attention with better parallelism and work partitioning,” in International Conference on Learning Representations, B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun, Eds., vol. 2024, 2024, pp. 35 549–35 562. [Online]. Available: https://proceedings.iclr.cc/paper files/paper/2024/ file/98ed250b203d1ac6b24bbcf263e3d4a7-Paper-Conference.pdf [9] T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré, “Flashattention: Fast and memory-efficient exact attention with io-awareness,” in Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 16 344–16 359. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/ 2022/file/67d57c32e20fd0a7a302cb81d36e40d5-Paper-Conference.pdf [10] T. Dao, D. Haziza, F. Massa, and G. Sizov, “Flash-decoding for longcontext inference,” https://crfm.stanford.edu/2023/10/12/flashdecoding. html, 2023. [11] DeepSeek-AI, “Deepseek-v3 technical report,” 2024. [Online]. Available: https://arxiv.org/abs/2412.19437 [12] DeepSeek-AI, A. Xu, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, C. Ling, C. Lu, C. Zhao, C. Deng, C. Hou, C. Xu, C. Shao, C. Ruan, C. Sun, D. Dai, D. Guo, D. Yang, D. Chen, D. Li, D. Ji, E. Li, F. Wei, F. Lin, F. Yuan, F. Xia, F. Dai, G. Hao, G. Chen, G. Cao, G. Meng, G. Li, H. Yu, H. Zhang, H. Xu, H. Li, H. Liang, H. Zhang, H. Luo, H. Wei, H. Yuan, H. Zhang, H. Luo, H. Chen, H. Ji, H. Zhang, H. Ding, H. Tang, H. Cao, H. Gao, H. Qu, H. Zeng, J. Yang, J. Zhu, J. Luo, J. Song, J. Yu, J. Huang, J. Cai, J. Liang, J. Zhou, J. Ye, J. Li, J. Xu, J. Hu, J. Yang, J. Chen, J. Yan, J. Chen, J. Zhou, J. Xiang, J. Yuan, J. Cheng, J. Zhou, J. Zhu, J. Yu, J. Sun, J. Ran, J. Jiang, J. Qiu, J. Li, J. Zheng, J. Song, K. Dong, K. Gao, K. Guan, K. Zhou, K. Huang, K. Yu, L. Wang, L. Zhang, L. Wang, L. Xia, L. Zhang, L. Zhao, L. Guo, L. Luo, L. Ma, L. Zhu, L. Wang, L. Cai, L. Zhang, L. Chen, M. Di, M. Xu, M. Mei, M. Wang, M. Zhang, M. Zhang, M. Tang, M. Li, M. Zhou, M. Han, N. Wang, P. Huang, P. Wang, P. Cong, P. Wang, P. Zhang, Q. Wang, Q. Zhu, Q. Li, Q. Chen, Q. Du, Q. Jiang, R. Tian, R. Xu, R. Lu, R. Xu, R. Ge, R. Zhang, R. Pan, R. Wang, R. Chen, R. Yin, R. Xu, R. Shen, R. Zhang, R. Chen, S. Liu, S. Lu, S. Sun, S. Zhou, S. Chen, S. Cai, S. Nie, S. Wu, S. Chen, S. Hu, S. Liu, S. Hu, S. Ma, S. Wang, S. Yu, S. Zhou, S. Pan, S. Yu, S. Zhou, T. Ni, T. Yun, T. Jin, T. Pei, T. Ye, T. Lin, T. Ji, T. Cui, T. Yue, T. Yu, T. Wang, W. Zhang, W. Xiao, W. Zeng, W. An, W. Zhao, W. Liu, W. Liang, W. Pang, W. Luo, W. Yao, W. Gao, W. Yang, W. Huang, W. Hou, W. Zhang, W. Ma, X. Gao, X. He, X. Wang, X. Wang, X. Bi, X. Liu, X. Wang, X. Chen, X. Zhang, X. Nie, X. Sun, X. Wang, X. Cheng, X. Liu, X. Xie, X. Liu, X. Liu, X. Yu, X. Li, X. Yang, X. Zhang, X. Chen, X. Wang, X. Su, X. Chen, X. Lin, X. Fu, Y. Yan, Y. Wang, Y. Ma, Y. Luo,

12

[28] F. Yao, L. Liu, D. Zhang, C. Dong, J. Shang, and J. Gao, “Your efficient rl framework secretly brings you off-policy rl training,” Aug. 2025. [Online]. Available: https://fengyao.notion.site/off-policy-rl [29] G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, “Orca: A distributed serving system for Transformer-Based generative models,” in 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). Carlsbad, CA: USENIX Association, Jul. 2022, pp. 521–538. [Online]. Available: https://www.usenix.org/ conference/osdi22/presentation/yu [30] J. Yuan, H. Li, X. Ding, W. Xie, Y.-J. Li, W. Zhao, K. Wan, J. Shi, X. Hu, and Z. Liu, “Understanding and mitigating numerical sources of nondeterminism in llm inference,” in Advances in Neural Information Processing Systems, D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen, Eds., vol. 38. Curran Associates, Inc., 2025, pp. 169 819–169 851. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/ 2025/file/f80094a824ba5912d4a2de169c404a40-Paper-Conference.pdf [31] Z. Zhang, X. Ding, J. Yuan, R. Liu, H. Mao, J. Xing, and Z. Liu, “Deterministic inference across tensor parallel sizes that eliminates training-inference mismatch,” in Forty-third International Conference on Machine Learning, 2026. [Online]. Available: https: //openreview.net/forum?id=5eZmlUyFpl [32] L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, C. Barrett, and Y. Sheng, “Sglang: Efficient execution of structured language model programs,” in Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, Eds., vol. 37. Curran Associates, Inc., 2024, pp. 62 557–62 583. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/ 2024/file/724be4472168f31ba1c9ac630f15dec8-Paper-Conference.pdf [33] T. Zhong, N. Ling, Y. Pi, Z. Wei, T. Yu, G. Fox, P. Wu, and X. Yu, “Diagnosing training inference mismatch in llm reinforcement learning,” 2026. [Online]. Available: https://arxiv.org/abs/2605.14220 [34] Y. Zhong, S. Liu, J. Chen, J. Hu, Y. Zhu, X. Liu, X. Jin, and H. Zhang, “DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving,” in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). Santa Clara, CA: USENIX Association, Jul. 2024, pp. 193–210. [Online]. Available: https://www.usenix.org/conference/osdi24/presentation/zhong-yinmin

13

Related documents

• The Integer Alibi: Localizing Cross-Kernel Divergence in INT8-Quantized LLM Inference#630138
• The Integer Alibi: Localizing Cross-Kernel Divergence in INT8-Quantized LLM Inference#628276
• Cached LLM Probability Retrieval for Speech Recognition#617696
• Cached LLM Probability Retrieval for Speech Recognition#617196
• GRACE:Gradient-guided Coreset Selection for LLM Unlearning#618564