ConfluxServe: Reconciling Resource Utilization and Parallelism Efficiency across Prefill and Decode in LLM Serving

ConfluxServe: Reconciling Resource Utilization and Parallelism Efficiency across Prefill and Decode in LLM Serving Wenhao Chen

Hong Zhang

University of Waterloo Waterloo, Canada wenhao.chen@uwaterloo.ca

University of Waterloo Waterloo, Canada hong.zhang@uwaterloo.ca

arXiv:2610.37810v1 [cs.DC] 27 Oct 2026

Abstract

mentarity motivates co-execution designs such as co-batching (e.g., Chunked Prefill [4]) and spatial multiplexing [10, 26], which execute the two phases concurrently in the same serving instance to improve overall hardware utilization (§3.1). From a parallelism perspective, however, prefill and decode impose fundamentally divergent parallelism requirements under their respective SLOs. Long prefills must scale out to satisfy TTFT and favor an efficient scale-out configuration, whereas decode can typically satisfy TBT on a much narrower execution group, with further scale-out introducing additional communication and reducing execution efficiency. This divergence favors phase-isolated execution, which separates prefill and decode either across machines [24, 33, 52] or over time [12, 45], allowing each to use a parallelism configuration better suited to its execution characteristics (§3.2). The two observations together create a dilemma: co-executing prefill and decode improves hardware utilization by exploiting their complementary resource demands, but typically requires them to share a common parallelism configuration, forcing at least one phase away from its efficient operating point. In contrast, phase-isolated execution allows each phase to use a more suitable parallelism configuration, but forgoes the utilization benefits of co-execution. We refer to this tension as the utilization–efficiency dilemma (§3.3). Is it possible to exploit prefill–decode resource complementarity without sacrificing phase-specific parallelism efficiency? We answer affirmatively with Conflux Parallelism (XP), a parallel execution abstraction that co-executes prefill and decode with different parallelism within the same batch. Conflux Parallelism is enabled by the observation that both the feasibility and benefit of co-batching computations with different parallelism configurations are operator-dependent. Dense linear operators benefit strongly from co-batching and remain co-batchable even when prefill and decode use different degrees of sequence parallelism (SP). Attention, in contrast, cannot be directly co-batched within a single invocation under different parallelism configurations, yet benefits much less from co-batching in the first place. This observation motivates the central design choice in Conflux Parallelism: cross-phase parallel execution is composed at the operator level rather than uniformly across the model. Concretely, Conflux Parallelism fully co-batches the linear computations of prefill and decode, gracefully accommodating their distinct SP degrees. At attention, Conflux

Prefill and decode are complementary in resource usage, but often divergent in parallelism preferences. Executing the two phases separately preserves their preferred parallelism but leaves complementary resources underutilized, while cobatching improves utilization but often requires a uniform parallel configuration that hurts execution efficiency. To resolve this utilization–efficiency dilemma, we propose Conflux Parallelism (XP), a parallel execution abstraction that co-executes prefill and decode with different parallelism within the same batch. Conflux Parallelism exploits operatorlevel Transformer structure to construct a heterogeneous execution path with different batching and parallelism strategies across operators, combining the best of co-batching and phase-specific execution. Building on Conflux Parallelism, we design ConfluxServe, an SLO-aware serving system that maximizes cluster goodput under TTFT and TBT constraints. Evaluation on production workloads shows that ConfluxServe improves serving goodput by 1.26–1.62× over the strongest baselines.

1

Introduction

Large language models (LLMs) are becoming core infrastructure for modern applications, from code generation [6, 32] and customer-support assistants [47] to scientific copilots [16]. Serving these applications requires handling large volumes of concurrent inference requests, each subject to stringent latency objectives such as time-to-first-token (TTFT) and time-between-tokens (TBT), both of which are critical to user experience. Accordingly, LLM serving systems seek to maximize goodput: the maximum sustainable request rate under TTFT and TBT SLO constraints [36, 45, 52]. LLM inference consists of two distinct phases: prefill and decode. Prefill processes the input prompt to produce the first output token and populate the KV cache, whereas decode generates subsequent tokens iteratively by reusing the cached state. Maximizing goodput across these two phases is challenging because they differ not only in their resource bottlenecks, but also in their preferred parallelism configurations under SLO constraints. From a resource-utilization perspective, prefill and decode are naturally complementary. Prefill is predominantly compute-intensive, whereas decode is constrained primarily by memory bandwidth and KV-cache capacity. This comple1

Seq sharded 𝑆/𝑃, 𝐻

Transformer Block × N Input

QKV Proj

Core Attention

O Proj

FFN

a1

1

Q K V

3

2

All-to-all

a2

Figure 1. Transformer block workflow in LLM inference.

a3 a4

Parallelism executes the two phases under distinct parallelism configurations, with decode attention remaining local and prefill attention executing across ranks with SP. This heterogeneous execution combines three benefits that are difficult to achieve simultaneously. First, Conflux Parallelism preserves efficient co-batching through the dense linear operators, exploiting resource complementarity. Second, it allows each phase to retain its preferred parallelism configuration throughout execution, avoiding the efficiency loss of uniform parallelism. Third, it enables coordinated spatial multiplexing between prefill and decode attention within a narrow, repeatable window, further exploiting complementary resources with stable cross-phase interference (§4). Building on Conflux Parallelism, we design ConfluxServe, an SLO-aware serving system that coordinates heterogeneous prefill–decode execution to maximize goodput under both TTFT and TBT constraints (§5). ConfluxServe keeps ongoing decodes localized to their assigned ranks while coexecuting high-SP prefill chunks across ranks using the residual capacity left by decode. In each execution iteration, ConfluxServe prioritizes pending prefills by TTFT urgency and dispatches their tokens across ranks to maximize admitted prefill work without violating decode TBT SLOs. Once a prefill completes, ConfluxServe selects its decode-residency rank to maximize co-execution opportunities throughout its decode lifetime. ConfluxServe comprises four key components. First, to execute heterogeneous batches efficiently, ConfluxServe retains CUDA Graph optimization for shared linear operators and limits interference during attention multiplexing through SM partitioning (§6.1). Second, to ensure heterogeneous co-execution respects TBT SLOs, we develop a latency model capturing critical-rank and critical-path dependencies, and a profile-fitted estimator for co-batched iteration latency (§6.2). Third, to maximize admissible prefill work across current and future iterations, ConfluxServe uses waterfill prefill dispatch to reduce current rank imbalance, and dual-mode decode placement to balance future execution efficiency against KV capacity (§6.3). Finally, ConfluxServe integrates TTFT-aware admission and prioritization with dispatch and placement in an end-to-end scheduler to maximize goodput (§6.4). We implement ConfluxServe on vLLM (§7) and evaluate it on representative workloads against state-of-the-art baselines (§8). Under FCFS scheduling, Conflux Parallelism sustains 1.33–1.62× higher goodput than the strongest existing designs. With its full scheduler, ConfluxServe improves goodput by 1.26–1.52× over the strongest baselines, even after

4

1 2 3 4

1 2 3 4

Seq sharded 𝑆/𝑃, 𝐻 1

2

3

O

F F … N

O

F F … N

1

4

2

Head shard 0

All-to-all

Q K V

Head sharded 𝑆, 𝐻/𝑃

3

Head shard 1 1

2

3

4

4

Figure 2. Ulysses sequence parallelism. augmenting each with enhanced scheduling. These gains remain consistent across model configurations and workloads.

2

Background

Computation Workflow in LLM Inference. Most LLMs [17, 48] consist of stacked Transformer blocks [41], as illustrated in Figure 1. Each block applies a QKV projection, performs core attention (attention hereafter), and then applies an output projection followed by a feed-forward network (FFN). QKV projection, output projection, and FFN consist primarily of dense linear operators implemented using general matrixmatrix multiplication (GEMM) kernels. In contrast, attention combines matrix multiplications, softmax, and element-wise operations, and is typically implemented using specialized fused kernels such as FlashAttention [11]. Attention accesses keys and values of preceding tokens, which inference systems retain in a KV cache to avoid recomputation [23, 35]. Prefill and Decode. LLM inference consists of two phases. During prefill, the system processes the input tokens in parallel, performs causal attention over the prompt, and materializes their KV cache. Prefill phase is typically computeintensive, and its latency is measured as time-to-first-token (TTFT). During decode, the system generates tokens autoregressively, processing one token per iteration while reading the historical KV cache. Decode phase is predominantly memory-bound [33, 35], and its inter-token latency is measured by time-between-tokens (TBT). LLM serving typically imposes SLOs on both TTFT and TBT. TTFT targets are commonly on the order of a few seconds, while TBT targets typically range from tens to a few hundred milliseconds [50, 52]. Parallelism Strategies. LLM serving systems employ different parallelism strategies to scale inference across GPUs. Tensor parallelism (TP) [39] shards model weights across ranks, reducing per-rank computation and memory usage. Pipeline parallelism (PP) [20] partitions consecutive blocks across stages, reducing per-GPU model memory but not directly reducing single-request latency, and may suffer from pipeline bubbles. Sequence parallelism (SP) [13] partitions the sequence dimension across ranks, reducing per-rank computation while replicating model weights. Two common SP designs are Ring-SP [27] and UlyssesSP [21]. Ring-SP keeps the sequence partitioned and circulates KV blocks across ranks, whereas Ulysses-SP uses All-to-All to transpose between sequence- and head- par2

Length

HBM capacity Latency (s)

75

1K

2K

4K

8K 16K 32K 1K

Context length

(a) Prefill( phase

2K

4K

8K

16K 0

Context length (b=16)

(b) Decode ( phase

1K

2K

0.5

1

2

4

Parallelism degree

Chunk size

(c) Co-batching (

8

TP

SP

1.0

Parallelism

-6%

-7%

0.75

-20% -34%

0.5

1

2

4

8

Parallelism degree

Figure 4. Prefill scaling (Qwen3-14B/H100). (a) Execution latency for 16K–64K prompts. (b) Parallelism efficiency under TP and SP for a 32K prefill. ure 3(c) shows, approximately 1K and 2.2K prefill tokens can therefore be co-batched while satisfying 50 ms and 100 ms TBT targets, increasing normalized compute utilization from about 16% to 75% and 90%, respectively. Another line of work [10, 26] exploits this complementarity through spatial multiplexing, partitioning GPU SMs between concurrently executing prefill and decode kernels.

titioned layouts around attention. We focus on Ulysses-SP because it achieves more favorable communication scaling and better supports chunked-prefill execution. Appendix A details the execution characteristics of Ring-SP, Ulysses-SP, and TP. Our analysis and heterogeneous-parallelism design are not specific to Ulysses-SP and also apply to Ring-SP. Unless otherwise stated, SP refers to Ulysses-SP in the paper. Figure 2 illustrates Ulysses-SP for sequence length 𝑆, SP degree 𝑃, and hidden dimension 𝐻 . Each rank initially holds 𝑆/𝑃 tokens with the full hidden dimension 𝐻 . The first All-to-All transposes the QKV states from sequence partitioning to head partitioning, so that each rank holds the full sequence for a subset of attention heads. After computing attention on these head shards, a second All-to-All restores the sequence-partitioned layout for subsequent operators.

3.2

The Utilization–Efficiency Dilemma

We observe a key tension in LLM serving: prefill and decode exhibit complementary resource demands (§3.1) but divergent parallelism preferences (§3.2). Together, they create a utilization–efficiency dilemma for existing designs (§3.3). 3.1

64K

TTFT 2s

2

0.1

3K

Figure 3. Prefill–decode resource complementarity (Qwen3-14B/H100): (a–b) Normalized compute, HBMbandwidth, and HBM-capacity utilization for prefill and decode. (c) Normalized compute utilization when co-batching prefill with 16 decodes at 8K context; dashed lines mark the largest chunks satisfying 50 ms and 100 ms TBT targets.

3

32K

10

0

m

s

25

m

s

50

0

16K

5

Parallelism eff.

HBM bandwidth

50

Norm. util. (%)

Compute 100

Prefill–Decode Resource Complementarity

Prefill and decode stress different GPU resources. Prefill processes prompt tokens in parallel and is primarily computeintensive, whereas decode generates one token per request per iteration while repeatedly accessing model weights and resident KV caches, making it primarily constrained by HBM bandwidth and KV-cache capacity. We validate this contrast by profiling Qwen3-14B on an NVIDIA H100 over varying prefill lengths and decode context lengths. Figure 3(a–b) shows that prefill sustains high compute utilization, while decode approaches the HBM-bandwidth limit and consumes increasing KV capacity as context grows. This complementarity creates an opportunity for co-execution. Chunked Prefill [4] exploits it by splitting long prefills into chunks and co-batching them with ongoing decodes, using the remaining TBT headroom for compute-intensive prefill work. In our profiling, a batch of 16 decodes with 8Ktoken contexts takes only about 20 ms per iteration. As Fig3

Prefill–Decode Parallelism Divergence

We observe that prefill and decode favor fundamentally different parallelism configurations with their respective SLOs. Prefill requires scale-out. A prefill request must process its entire prompt before producing the first token, making the TTFT SLO increasingly difficult to satisfy as prompt length grows. Moreover, production LLM workloads often exhibit heavy-tailed prompt length distributions. For example, P95 prompt lengths reach 26K and 49K in Mooncake-Toolagent and Mooncake-Synthetic (Table 3). Figure 4(a) profiles the latency of 16K, 32K, and 64K prefills on Qwen3-14B across different parallelism degrees. To sustain a 2 s TTFT SLO [50], 32K prompts require at least two GPUs, while 64K prompts require at least four. These degrees are only lower bounds derived from isolated execution latency. In practice, additional scale-out is needed to accommodate queueing delay under traffic variability. Figure 4(b) further examines how efficiently TP and SP realize this scale-out. Given a workload, we define the parallelism efficiency of a parallelism strategy at degree 𝑑 as 𝐸 (𝑑) = 𝑇 (1)/(𝑑 · 𝑇 (𝑑)),

(1)

where𝑇 (𝑑) denotes execution latency on 𝑑 GPUs, and 𝐸 (𝑑) = 1 corresponds to ideal linear scaling. While both TP and SP reduce prefill latency, SP maintains substantially higher parallelism efficiency, retaining 93% at degree 8 versus 66% for TP. This higher efficiency stems from SP’s better communication scaling (analyzed in Appendix A). We observe a similar trend across prompt lengths. Takeaway 1: Long-context prefill requires scale-out to satisfy TTFT, and often favors SP for higher parallelism efficiency. Decode favors narrow execution. Decode exhibits a different scaling behavior. Each iteration processes only one new token per request rather than the entire prompt as in prefill. Moreover, decode operates on batches, and its latency reflects the context-length mix of the batch rather than

4K

8K

16K Parallelism eff.

Latency (ms)

Length 25 20 15 10 5 0

1

2

4

8

Parallelism

TP

SP

Design

1.0 -35%

Phase-isolated designs

-61%

0.5

◦ Static isolation [24, 33, 52]

-70%

0.0

1

2

4

✓

✗

✓

✓

✗

Representative: MuxWise [10]

✓

✗

Conflux Parallelism (Ours)

✓

✓

◦ Elastic isolation [12, 45]

8

Parallelism degree

Representative: LoongServe [45]

Figure 5. Decode scaling (Qwen3-14B/H100). (a) Decode execution latency (batch size 16, 4K–16K contexts). (b) Parallelism efficiency under TP and SP (batch size 16, 8K context).

Co-execution designs ◦ Co-batching [4, 19, 37, 43] Representative: Chunked Prefill [4]

◦ Spatial multiplexing [10, 14, 26]

the long-context tail of an individual request. For example, median prompt lengths are 6K / 12K tokens in MooncakeToolagent / Mooncake-Synthetic, respectively, substantially below their P95 lengths. In addition, KV-cache capacity further bounds the resident decode requests and, consequently, per-iteration decode latency. Together, decode often requires only a narrow parallelism configuration to satisfy TBT SLO. Figure 5(a) profiles decode batches of 16 requests each with 4K, 8K, and 16K contexts on Qwen3-14B across parallelism degrees. Further increasing batch size or context length exceeds single-GPU HBM capacity. A single-GPU configuration already completes each iteration within 25 ms across these settings, leaving substantial margin even under a tight 50 ms TBT SLO [15]. Figure 5(b) further shows that scaling out decode is inefficient: parallelism efficiency drops to only 39% and 16% at degree 8 under TP and SP, respectively. This low efficiency stems from decode’s limited per-iteration work, which provides little opportunity to amortize the communication overhead of scale-out. We observe a similar trend across context lengths. Takeaway 2: Decode can typically satisfy TBT on a narrow parallelism configuration, with further scale-out substantially reducing parallelism efficiency.

Table 1. Utilization–efficiency dilemma in existing designs. efficiency (§3.2). As summarized in Table 1, existing designs satisfy one requirement at the expense of the other, creating what we term the utilization–efficiency dilemma. Phase-Isolated Designs. Static isolation designs [24, 33, 52] dedicate separate GPU pools to the two phases, enabling independent provisioning but potentially stranding resources when their demands are imbalanced. Elastic isolation designs [12, 45] dynamically reassign GPU instances between prefill and decode over time, thereby reducing resource imbalance compared with static isolation. Both approaches keep prefill and decode isolated in separate iterations. This enables phase-specific parallelism, but prevents the two phases from exploiting their complementary compute and HBMbandwidth demands. Co-Execution Designs. Co-batching designs [4] mix prefill chunks with ongoing decodes in the same batch; recent designs [37, 43] further adapt how prefill and decode work are partitioned and co-batched according to workload conditions and SLOs. Spatial-multiplexing designs [10, 26] instead partition a GPU’s SMs between prefill and decode so that their kernels execute concurrently. Both approaches exploit prefill–decode resource complementarity on the same GPUs. However, they typically require the two phases to share a common parallelism configuration, incurring the uniformparallelism tax identified in §3.2. Example. We illustrate the dilemma on four H100 GPUs on Qwen3-14B using Mooncake-Toolagent workload. Following §3.2, we set the prefill scale-out degree to 4 to maintain sufficient TTFT headroom, while degree 1 leaves sufficient TBT slack for decode. We maintain 96 ongoing decode requests while progressively increasing the offered prefill load, and report the maximum prefill throughput each design sustains without violating the decode TBT SLO. Figure 6 instantiates representative systems using their designed parallelism configurations.1 LoongServe [45] (dynamic isolation) preserves phase-specific parallelism by run-

Uniform-parallelism tax. The two takeaways together create an inherent tradeoff when prefill and decode are forced to share the same parallelism configuration. A low degree preserves decode parallelism efficiency but can leave long prefills unable to meet TTFT, whereas a high degree improves prefill TTFT feasibility at the cost of unnecessary communication and lower decode parallelism efficiency. Thus, any uniform configuration must compromise between prefill SLO feasibility and decode efficiency. We refer to this loss from enforcing a common parallelism configuration as uniformparallelism tax. Such a tax makes phase-isolated designs attractive: separating prefill and decode either across machines [24, 33, 52] or over time [12, 45] allows each phase to use its preferred parallelism configuration. 3.3

✗

Representative: DistServe [52]

-84%

Parallelism degree

Resource Phase-Specific Complementarity Parallelism

Limitation of existing works

The preceding observations expose two conflicting requirements: resource complementarity favors prefill–decode coexecution for hardware utilization (§3.1), whereas parallelism divergence favors phase-isolated execution for parallelism

1We omit static phase isolation because prefill requires all four GPUs for

scale-out, leaving no separate decode pool. 4

D

D (TP4) P (TP4)

D

D

D

LoongServe

Chunked Prefill

Prefill A

D P (TP4) (TP4)

P (SP4)

D (TP4) P (TP4)

D

MuxWise

Conflux Parallelism

Decode C

P (SP4)

Decode B

D

D (TP4) P (TP4)

D

LoongServe

Chunked Prefill

MuxWise

b4

Q K V

a4

a3 a4

c2

Local attention (Decode)

a3

c2

a3

a3 a4

Head shard 0 (Prefill)

a3 a4

Head shard 1 (Prefill)

O

F F … N

O

F … F N

b4

Local attention (Decode)

Phase-specific attention parallelism

a4 c2

Cross-SP co-batch

Figure 7. Operator-level execution flow of Conflux Parallelism across two ranks. Prefill and decode with different SP degrees are co-batched through the shared linear operators while retaining distinct attention-parallelism configurations.

Conflux Parallelism

50 ms 2.41 4.17 4.38 6.15 100 ms 5.17 5.92 5.43 8.27 Table 2. Prefill throughput (req./s) under different TBT SLOs.

and process them in a single GEMM. Attention exhibits the opposite profile. Attention computations can be directly co-batched only when they share a common QKV layout. Different parallelism configurations with different QKV sharding therefore cannot be co-batched within a single invocation. Yet attention gains much less from cross-parallelism co-batching in the first place: each request attends to its own KV state, so batching more requests increases computation and KV-state traffic proportionally without asymptotically increasing arithmetic intensity. Appendix B details the operator analysis. Conflux Parallelism execution flow. Figure 7 illustrates how Conflux Parallelism exploits this asymmetry to enable co-batching while preserving each phase’s parallelism configuration throughout the Transformer layer. Prefill request 𝐴 processes a chunk containing 𝑎 3 and 𝑎 4 , while decode requests 𝐵 and 𝐶 process tokens 𝑏 4 and 𝑐 2 , respectively. Conflux Parallelism partitions 𝐴’s prefill chunk across the two ranks with SP degree 2, while executing 𝐵 and 𝐶 locally, one on each rank. Before attention, each rank co-batches its local prefill and decode tokens, forming [𝑎 3, 𝑏 4 ] on rank 0 and [𝑎 4, 𝑐 2 ] on rank 1. Each rank then processes its concatenated tokens together in a single QKV-projection. At attention, the tokens follow their retained layouts concurrently: decode attention remains local, while prefill attention executes with SP using two All-to-All collectives and restores sequence partitioning afterward. After attention, both phases again expose compatible rank-local token tensors, allowing each rank to co-batch them through the remaining linear operators. KV residency and KV transfer. As illustrated in Figure 8, Conflux Parallelism preserves the KV-residency layout of each phase. Decode request 𝐵 extends its rank-local KV cache with 𝑏 4 without inter-rank communication, while prefill request 𝐴 appends 𝑎 3 and 𝑎 4 to its head-sharded KV cache under SP. Diverging and reconverging the attention paths therefore introduces no additional KV redistribution: decode KV remains local, while prefill KV remains head-sharded. This head-sharded prefill layout also naturally supports chunked prefill, allowing subsequent chunks to reuse historical KV without redistribution. The only additional KV transfer occurs when a request transitions from distributed prefill to

ning prefill at SP4 and decode at SP1, but temporally isolates the two phases, leaving substantial compute unused during decode execution. Chunked Prefill [4] (co-batching) exploits resource complementarity by co-batching prefill and decode under a uniform TP4 configuration. However, the high decode parallelism lowers parallelism efficiency and consumes more TBT headroom, limiting the admissible prefill chunk size. MuxWise [10] (spatial multiplexing) partitions GPU SMs between concurrent prefill and decode kernels using a shared TP4 configuration. It therefore still incurs the uniform-parallelism tax, while SM partitioning further limits the compute capacity available to prefill to protect decode latency. Moreover, because the two phases execute as separate kernels, their linear operators cannot be co-batched to amortize model-weight traffic, and the concurrent kernels can contend for shared memory and communication resources. Table 2 reports the prefill throughput of each design under 50 ms and 100 ms TBT SLOs, alongside Conflux Parallelism (ours), which unlocks up to 1.40× higher prefill throughput by addressing the dilemma. We show how Conflux Parallelism realizes this opportunity next.

4

Q K V

Cross-SP co-batch

Figure 6. Representative systems under their designed parallelism configurations. Each black box denotes a single batch. TBT SLO

b4

All-to-all

D (TP4) P (TP4)

All-to-all

D

Conflux Parallelism

Conflux Parallelism is enabled by an operator-level asymmetry in cross-parallelism co-batching: Linear operators and attention differ in the feasibility and benefit of co-batching computations with different parallelism configurations. Dense linear operators, including QKV and output projections and FFNs, apply the same weights independently to each token. Co-batching prefill and decode therefore amortizes model-weight traffic over more tokens, improving weight reuse and arithmetic intensity. Crucially, such co-batching only requires a shared weight matrix and feature dimension. This creates an opportunity to co-batch prefill and decode under different SP degrees: SP only changes how the sequence dimension is partitioned across ranks, while leaving the feature dimension and model weights unchanged. Each rank can therefore concatenate its local prefill and decode tokens 5

KV Cache (Rank 0)

KV Cache (Rank 0)

b1

b2

b1

b2

b3

b4

a1

a2

a1

a2

a3

a4

b3

Local attention (Decode)

…

b4

b1

…

a3 a4

a1

b2

b3

b4

b4

…

a3 a4

…

Requests

①

a3

a4

…

XP Instance M

XP Instance SLO-Aware Scheduler

Figure 8. KV residency in Conflux Parallelism, illustrated on one rank. Decode KV remains local, prefill KV remains head-sharded, and cross-layout KV transfer is required only at the prefill-to-decode transition.

P2

Execution D

Queue P1

Latency Estimator

… D

②

Admission

③

④

P2 D

Dispatcher

⑤

Placement

D

Figure 9. System overview of ConfluxServe.

localized decode, at which point its KV shards are transferred once to the selected decode rank. This transfer uses the intranode high-bandwidth interconnect and can be effectively overlapped with execution of the final prefill chunk (§7).

prefill–decode execution to maximize serving goodput under TTFT and TBT constraints. ConfluxServe keeps ongoing decode requests localized to their assigned ranks and dispatches high-SP prefill chunks across ranks according to TTFT urgency. These prefill chunks co-execute with decode, utilizing its residual capacity without violating TBT SLOs.

Conflux Parallelism over a base TP configuration. The preceding discussion assumes one GPU per logical Conflux Parallelism rank. For larger models, Conflux Parallelism composes naturally with TP by treating each fixed TP group as one logical rank: prefill applies SP across logical ranks, while each decode request remains localized within one rank. Since both phases share the same base-TP layout, their rank-local tokens can still be co-batched through TP-sharded linear operators and their associated TP collectives, while attention retains phase-specific parallelism across logical ranks. Appendix C details the base-TP execution workflow.

XP Instances. We partition the GPU cluster into independent XP instances, which serve as the basic units of scheduling and execution. Within each instance, GPUs are organized into fixed base-TP groups, with each TP group forming one logical rank. The base TP degree is chosen to accommodate model weights while reserving sufficient KV-cache capacity, and the remaining intra-node parallelism is used to maximize the SP degree. Prefill executes across these logical ranks using the fixed SP configuration, while each decode request remains localized within one rank. Although higher SP introduces additional communication, Conflux Parallelism overlaps the distributed prefill path with local decode attention, mitigating the resulting execution overhead and enabling near-linear prefill scaling when co-executed with decode (§8.3). Moreover, the fixed configuration avoids runtime resizing and KV redistribution. It also simplifies online scheduling by removing SP degree from the decision space.

Benefits. Conflux Parallelism provides three key benefits: 1. Resource complementarity via co-batching. Conflux Parallelism co-batches prefill and decode in shared linear operators, combining their complementary resource demands and amortizing model-weight traffic across the mixed batch. 2. Phase-specific parallelism via heterogeneous attention. Conflux Parallelism lets prefill use distributed attention while decode remains localized, preserving each phase’s preferred parallelism and avoiding unnecessary decode communication and the associated parallelism-efficiency loss. 3. Coordinated multiplexing via attention overlap. Conflux Parallelism enables coordinated spatial multiplexing of heterogeneous prefill and decode attention, further exploiting resource complementarity while keeping interference stable. The overlap is stable: the two paths diverge at a fixed point and reconverge immediately afterward, creating a narrow, repeatable overlap window. It is also complementary: communication- and compute-intensive prefill path overlaps with memory-bandwidth-intensive decode attention [53]. In contrast, request-level spatial multiplexing [10, 26] lets separate prefill and decode kernels progress independently throughout execution without an operator-bounded overlap window, leading to higher and less stable interference.

5

XP Instance 1

XP Instance 0

Head shard 0 (Prefill) a2

Global Router

Scheduling workflow. As illustrated in Figure 9, newly arrived requests are routed to the XP instance with the lowest normalized queued workload, measured as queued work relative to the instance’s processing capacity (①). Within each instance, ConfluxServe organizes execution into serving iterations. Each iteration advances all resident decode requests by one token on their assigned ranks, ensuring stall-free decode. The remaining capacity available for prefill without violating the TBT SLO defines the TBT-safe prefill budget. The scheduler performs TTFT-aware admission and prioritization to determine which pending prefills consume this capacity and in what order (②). Following this order, the waterfill dispatcher queries the latency estimator for candidate prefill allocations (③). Using these estimates, the dispatcher determines the chunk size for each request and its corresponding per-rank assignment to maximize the TBTsafe prefill budget (④). If a selected chunk completes the request’s prefill, the scheduler invokes decode placement to

ConfluxServe: Overview

Building on Conflux Parallelism, we design ConfluxServe, an SLO-aware serving system that coordinates heterogeneous 6

CUDA graph i

Main stream

O Proj

FFN

Eager QKV Proj

D-stream

P Attn

CUDA graph i+1 O Proj

D Attn

FFN

① Shared linear region QKV Proj

Rank 0

…

Time

Graph i

ator. Execution therefore alternates between graph-captured shared-linear segments and narrow heterogeneous-attention regions. This decomposition (1) preserves optimized CUDAgraph execution for all shared linear operators, (2) isolates heterogeneous execution to a narrow attention region, which enables coordinated attention overlap. Coordinated attention overlap. Within the custom attention operator, ConfluxServe executes the prefill and decode attention paths concurrently on separate CUDA streams. The distributed prefill path, including its All-to-All communication and attention kernel, runs on the main stream, while localized decode attention runs on the D stream. Concurrent prefill and decode attention can introduce SM contention. ConfluxServe therefore uses GreenContext [31] to partition SMs within the custom operator. We allocate decode the minimum number of SMs needed to sustain 90% of its standalone peak HBM bandwidth, leaving the remaining SMs to prefill.

High-performance heterogeneous execution. Efficient XP execution requires preserving CUDA-graph execution while limiting interference between phase-specific attention paths. ConfluxServe captures the linear operators in piecewise graph segments and uses SM partitioning to limit interference in attention-path overlap (§6.1). Latency modeling and estimation. Conflux Parallelism’s heterogeneous execution makes iteration latency depend on the slowest rank and execution path at its synchronization boundaries. We develop a latency model that captures these rank- and path-dependent bottlenecks, and construct a profile-fitted estimator to ensure TBT-safe co-batching (§6.2). Model-guided dispatch and placement. As Conflux Parallelism’s execution latency depends on rank-local load, both prefill dispatch and decode placement affect the TBT-safe prefill budget. Rank imbalance increases the bottleneck load, leaving bubbles on other ranks and reducing the admissible TBT-safe prefill work. Guided by the latency model, ConfluxServe maximizes the TBT-safe prefill budget through two coupled decisions: (1) waterfill dispatch assigns prefill tokens to balance the current combined prefill–decode load across ranks; and (2) dual-mode decode placement shapes future resident load, spreading requests to reduce execution imbalance or packing them to preserve KV capacity (§6.3). SLO-aware scheduler design. Maximizing SLO-compliant goodput requires deciding which prefills should consume the TBT-safe budget. ConfluxServe integrates TTFT-aware admission and prioritization with model-guided prefill dispatch and decode placement within the scheduling loop (§6.4).

High-Performance Heterogeneous Execution

…

Figure 11. Latency model across two logical synchronization boundaries. Rank- and path-level imbalance introduces idle time in shared linear and split attention regions.

Design Components

6.1

P D Time

select and reserve its decode rank (⑤). Preselecting this rank allows the KV transfer to overlap with the final prefill chunk, enabling decode to begin in the next iteration.

ConfluxServe: Design

… D

Rank 1

Figure 10. Piecewise CUDA-graph integration of Conflux Parallelism. Shared linear operators remain graph-captured, while phase-specific attention paths execute concurrently on separate CUDA streams in a narrow eager region, confining cross-phase contention to the custom attention operator.

6

P

Graph i

Contention window

5.1

② Split attention region

6.2

Latency Modeling and Estimation

Synchronization Boundaries. As Figure 11 shows, Conflux Parallelism execution exposes two logical synchronization boundaries per Transformer layer: one after the shared linear region (①) and one after the split attention region (②). We therefore decompose per-iteration latency across 𝐿 layers as 𝑇iter = 𝐿 (𝑇GEMM + 𝑇ATTN ) , (2) where each term is determined by the slowest rank or execution path reaching the corresponding boundary. For compactness, we express component latencies using base latency functions 𝑓GEMM , 𝑓𝐷 , 𝑓𝑃 , and 𝑓comm . We detail the functional forms of these components in Appendix D. Shared linear region. Let 𝑛𝑟𝑝 and 𝑛𝑑𝑟 denote the numbers of prefill and decode tokens processed on rank 𝑟 , respectively. Since each rank processes its local prefill and decode tokens together through the shared linear operators,  
𝑇GEMM = max 𝑓GEMM 𝑛𝑟𝑝 + 𝑛𝑑𝑟 = 𝑓GEMM max (𝑛𝑟𝑝 + 𝑛𝑑𝑟 ) , 𝑟

(3)

𝑟

where 𝑓GEMM (·) is a nondecreasing function of the ranklocal token count. Thus, the shared-linear boundary is determined by the largest combined token load across ranks. Split attention region. At the second boundary, prefill follows the distributed SP attention path, while decode at-

Piecewise graph execution. We leverage piecewise CUDA graphs [42] to efficiently realize XP’s split execution structure. As illustrated in Figure 10, shared linear operators remain within graph-captured segments, while the attention region is replaced by our custom heterogeneous attention oper7

tention remains rank-local. Let 𝑇𝑃 and 𝑇𝐷 denote their standalone latencies. Since the two paths execute concurrently, 𝑇ATTN = max {𝛽𝑇𝑃 , 𝛼𝑇𝐷 } ,

largest common combined-load target 𝑛 target satisfying the TBT SLO, then waterfilling prefill tokens toward this target. We design a waterfill dispatcher based on this insight. The dispatcher uses binary search with the latency estimator to find 𝑛 target . Under this target, rank 𝑟 has capacity max{0, 𝑛 target − 𝑛𝑑𝑟 }. For a given request, the aggregate capacity determines the TBT-safe budget 𝐵, capped by its remaining prefill length, and waterfilling these tokens toward 𝑛 target yields Ap . With 𝑁 ranks and a target-level search range of 𝐵, each invocation takes 𝑂 (𝑁 log 𝐵) time, since each latency-estimator query takes 𝑂 (𝑁 ). During scheduling, as shown in Algorithm 1, the dispatcher is invoked incrementally across prefill requests. For request 𝑖, it returns the largest TBT-safe chunk ℓ𝑖 and corresponding per-rank assignment A𝑖 , given the existing assignment Ap (Line 6). The scheduler then updates Ap with A𝑖 (Line 13). Appendix E provides the dispatcher pseudocode. Decode: dual-mode placement. When a prefill completes, ConfluxServe places it on a single rank for KV residency and decode execution. The placement aims to maximize the TBTsafe prefill budget in subsequent iterations while preserving sufficient KV capacity for new decode placements. To maximize the TBT-safe prefill-dispatch budget, the latency model suggests a simple placement principle: place each decode request on the rank with the smallest aggregate Í decode context, 𝑖 ∈ D𝑟 𝑐𝑖 . For the split attention region (Eq. 5), placing a request on the rank with the smallest aggregate Í decode context, 𝑖 ∈ D𝑟 𝑐𝑖 , minimizes decode imbalance. For the shared linear region (Eq. 3), a placement increases 𝑛𝑑𝑟 by only one on the selected rank, while the resulting imbalance can be compensated by subsequent waterfill prefill dispatch. KV capacity, however, favors tighter packing. Spreading decode requests can fragment free KV capacity across ranks, limiting future placements. ConfluxServe therefore reconciles these objectives using a KV-utilization threshold 𝜂. While any feasible rank remains below 𝜂, it uses worst-fit placement, selecting the rank with the smallest aggregate decode context. If all ranks exceed 𝜂, it switches to best-fit to reduce fragmentation. Appendix F provides the pseudocode.

(4)

where 𝛼, 𝛽 ≥ 1 capture the slowdowns caused by contention for the decode and prefill paths, respectively. For decode, each request remains localized to its resident rank, so the path latency is determined by the slowest rank:  ∑︁  𝑇𝐷 = max 𝑓𝐷 𝑐𝑖 , (5) 𝑟 𝑖 ∈ D𝑟

where D 𝑟 denotes the decode requests resident on rank 𝑟 , and 𝑐𝑖 is the context length of request 𝑖. For prefill, the distributed path consists of All-to-All communication and attention. Let P denote the scheduled prefill requests, with processed context length 𝑐𝑖 and scheduled chunk size ℓ𝑖 for request 𝑖. We model its latency as !    1 ∑︁ ℓ𝑖 + 1 𝑟 𝑇𝑃 = 𝑓comm max 𝑛𝑝 + 𝑓𝑃 ℓ𝑖 𝑐𝑖 + . (6) 𝑟 𝑁 𝑖∈P 2 The communication term is determined by the largest prefill shard, max𝑟 𝑛𝑟𝑝 , where 𝑛𝑟𝑝 denotes the number of prefill tokens assigned to rank 𝑟 . The attention term captures the aggregate prefill attention workload, which Ulysses-SP distributes evenly across the 𝑁 ranks, independent of the per-rank prefill token assignment. Estimator construction. We instantiate the model through offline profiling. We profile the shared linear operators, prefill attention, decode attention, and communication in isolation to obtain 𝑓GEMM , 𝑓𝑃 , 𝑓𝐷 , and 𝑓comm , and use heterogeneous Conflux Parallelism profiles to calibrate the contention factors 𝛼 and 𝛽. The resulting estimator closely matches measured iteration latency across diverse configurations (§8.5). 6.3

Model-Guided Dispatch and Placement

Prefill: waterfill dispatch. Given the current decode state and TBT SLO, we seek a prefill assignment Ap = {𝑛𝑟𝑝 }𝑟𝑁=1 Í that maximizes the TBT-safe prefill budget 𝐵 = 𝑟 𝑛𝑟𝑝 . The latency model provides key guidance for dispatch. For the shared linear region (Eq. 3), latency depends on the largest combined prefill–decode load, max𝑟 (𝑛𝑟𝑝 + 𝑛𝑑𝑟 ). Since 𝑛𝑑𝑟 is fixed within an iteration, this suggests assigning more prefill tokens to decode-light ranks. For the split attention region (Eq. 6), prefill communication depends on max𝑟 𝑛𝑟𝑝 , while prefill-attention computation is independent of the cross-rank prefill assignment because Ulysses-SP evenly redistributes the aggregate attention workload across ranks. Importantly, our profiling shows that shared-linear latency typically dominates prefill communication latency. Together, these observations yield an important dispatch principle: maximizing the TBT-safe prefill budget primarily requires balancing the combined token load, 𝑛𝑟𝑝 + 𝑛𝑑𝑟 , across ranks. This reduces the dispatch problem to finding the

6.4

SLO-Aware Scheduler Design

Within each XP instance, the scheduler determines which pending prefills should consume the TBT-safe budget available under the resident decode workload. As shown in Algorithm 1, it first performs TTFT-aware admission and prioritization (Lines 1–4). A Moore–Hodgson admission policy [30] (Appendix G) selects requests that can meet their TTFT deadlines under the reference budget 𝐵 0 , with admitted requests considered in EDF order; the remaining requests are ordered by SJF for opportunistic backfill. Following this order, the scheduler constructs the prefill batch incrementally (Lines 5–13). For each request, it invokes the prefill dispatcher to obtain a TBT-safe chunk and corresponding assignment under the current state. If the 8

Algorithm 1 SLO-Aware Scheduler

Workload

Prompt (P50/P95)

Input: Pending prefills R, decode state D, time 𝑡, TBT SLO 𝜏 Output: Prefill assignment Ap , decode placement Ad e ← D; Ad ← ∅ 1: P ← ∅; Ap ← ∅; D 2: 𝐵 0 ← Dispatch(𝜏, D, P, Ap, ∅) 3: (R adm, R rej ) ← MooreHodgsonAdmission(R, 𝐵 0, 𝑡, 𝜏) 4: Sort R adm in EDF order; sort R rej in SJF order 5: for each 𝑖 in R adm followed by R rej 6: (ℓ𝑖 , A𝑖 ) ← Dispatch(𝜏, D, P, Ap, 𝑖) 7: if ℓ𝑖 = 0 then break ⊲ Prefill budget exhausted 8: if ℓ𝑖 = 𝑚𝑖 then ⊲ Prefill completes e 9: 𝑔𝑖 ← DecodePlacement(𝑖, D) 10: if 𝑔𝑖 = Infeasible then ⊲ No feasible decode host 11: break e←D e ∪ {(𝑖, 𝑔𝑖 )} 12: Ad ← Ad ∪ {(𝑖, 𝑔𝑖 )}; D 13: Ap ← Ap ∪ A𝑖 ; P ← P ∪ {(𝑖, ℓ𝑖 )} 14: return Ap, Ad

Mooncake-Toolagent Mooncake-Synthetic QwenA

6,346 11,587 1,046

Output (P50/P95) 30 69 375

600 519 1,032

Table 3. Workload prompt and output length statistics.

8

Evaluation

We evaluate ConfluxServe to answer the following questions: • How does ConfluxServe compare with state-of-the-art systems across different models? (§8.2) • How do the key design components contribute to ConfluxServe’s performance gains? (§8.3) • How do workload characteristics and SLO tightness affect ConfluxServe’s performance? (§8.4) • How accurate is ConfluxServe’s latency estimator, and what runtime overhead does ConfluxServe introduce? (§8.5)

chunk completes the request’s prefill, the scheduler further invokes decode placement before admitting the transition. An infeasible placement discards the completing chunk and its tentative assignment and stops further admission. Otherwise, the selected decode rank is reserved and incorporated into the tentative decode state so that subsequent placement decisions account for earlier transitions in the same iteration. The accepted chunk is then added to the prefill batch, and its assignment is accumulated into the cumulative per-rank assignment. Let 𝑅 = |R| be the number of pending prefill requests and 𝑁 the number of ranks. Admission and ordering require 𝑂 (𝑅 log 𝑅) time, while at most 𝑂 (𝑅) dispatcher invocations each take 𝑂 (𝑁 log 𝐵). Transition validation takes 𝑂 (𝑅𝑁 ) time in the worst case. The overall per-iteration complexity is therefore 𝑂 (𝑅 log 𝑅 + 𝑅𝑁 log 𝐵).

7

26,114 49,009 8,809

8.1

Setup

Models. We evaluate ConfluxServe on three widely used open-source models spanning different sizes: Llama-3.1-8B [17], Qwen3-14B, and Qwen3-32B [48], abbreviated as Llama-8B, Qwen-14B, and Qwen-32B. Following prior works [10, 12, 50], we use TTFT/TBT SLOs of 1s/50ms for Llama-8B and 2s/100ms for Qwen-14B and Qwen-32B. Metrics. We measure request-level SLO attainment [2, 15]: for the 𝑖-th output token, the deadline relative to request arrival is TTFTSLO +(𝑖 −1)·TBTSLO . A request is SLO-compliant only if every output token meets its corresponding deadline, jointly capturing prefill and decode performance. Our primary metric is goodput: the maximum sustainable request rate at which at least 95% of requests are SLO-compliant. Testbed. We conduct experiments on NVIDIA H100 SXM5 80GB GPUs connected via NVLink. We use four GPUs for Llama-8B and Qwen-14B and eight GPUs for Qwen-32B. Each configuration forms one XP instance. Due to the available testbed scale, we evaluate a single instance and therefore do not exercise global routing. All key design choices in Conflux Parallelism and ConfluxServe are evaluated. All compared systems use the same number of GPUs for each model. Our software stack uses CUDA 12.8, NCCL 2.27.5, and PyTorch 2.9.1. We use vLLM v0.15.1 [23] and SGLang v0.5.9 [51] for the corresponding baselines. For LoongServe, we use its open-source implementation and update its CUDA, NCCL, and PyTorch dependencies to match our testbed. Workloads. We evaluate three representative workloads derived from the Mooncake [36] and Qwen [44] traces: MooncakeToolagent, Mooncake-Synthetic, and QwenA. As summarized in Table 3, Mooncake-Toolagent has long-tailed prompts and short outputs, representative of tool-and-agent workloads. Mooncake-Synthetic has substantially longer prompts, while QwenA has shorter prompts and much longer outputs. We use Mooncake-Toolagent for the main end-to-end

Implementation

We implement ConfluxServe on vLLM v0.15.1 [23] in approximately 6K lines of code. We reuse vLLM’s continuous batching, PagedAttention, and FlashAttention [38] backends, and adapt the Ulysses-SP implementation from the ArcticInference plugin [18] for Conflux Parallelism. Beyond optimizations mentioned in §6.1, ConfluxServe further implements two optimizations for phase-specific KV layouts. First, KV-cache aliasing allows head-sharded prefill and localized decode to share the same physical PagedAttention blocks, while separate block tables and address mappings expose the corresponding views to their attention kernels. This avoids KV duplication and separate KV pools. Second, during a prefill-to-decode transition, ConfluxServe preselects the decode rank before the final prefill chunk completes and pipelines KV transfer layer by layer. After layer 𝑖 finishes attention, its KV shard is transferred asynchronously while layer 𝑖 +1 executes its graph-captured GEMM segment, overlapping transfer with the prefill execution. 9

SLO attainment (%)

100

SLO attainment (%)

100

Chunked MuxWise-FCFS DistServe LoongServe ConfluxServe-FCFS

90

80

70

1

3

5

7

9

11

1

2

3

4

5

6

7

1

2

3

4

5

Chunked++ MuxWise++ DistServe++ LoongServe++ ConfluxServe

90

80

70

1

3

5

7

9

11

1

3

5

7

9

1

2

3

4

5

Arrival rate (req/s)

Arrival rate (req/s)

Arrival rate (req/s)

Llama-8B

Qwen-14B

Qwen-32B

6

7

Figure 12. SLO attainment on Mooncake-Toolagent, with FCFS scheduling (top) and enhanced scheduling configurations (bottom). Vertical lines indicate the maximum arrival rate a system sustains while meeting an SLO attainment of over 95%. evaluation and the other two for sensitivity analysis. We subsample each trace and uniformly scale inter-arrival times to obtain the target arrival rate. We cap prompt and output lengths at each model’s maximum context length. Baselines. We compare ConfluxServe against four stateof-the-art systems spanning the prefill–decode execution architectures in Table 1: Chunked Prefill [4], DistServe [52], LoongServe [45], and MuxWise [10]. For each baseline, we retain the parallelism strategy (i.e., TP or SP) prescribed by its original design and implementation. To separate execution benefits from request-scheduling effects, we evaluate two settings. First, we normalize all systems to FCFS, disabling MuxWise’s preemption mechanism (MuxWise-FCFS), to isolate the underlying prefill–decode execution architecture. We then enhance each baseline with the same SRPT-style policy, prioritizing requests with the fewest remaining prefill tokens, and denote these variants with a “++” suffix. MuxWise++ outperforms the original MuxWise scheduler under the evaluated workloads. • Chunked Prefill (co-batching). We use vLLM’s chunkedprefill execution with TP4 for Llama-8B and Qwen-14B and TP8 for Qwen-32B. For each model, workload, and TBT target, we sweep the token budget and report the best-performing configuration. • MuxWise (spatial multiplexing). We use the MuxWise implementation in SGLang with TP4 for Llama-8B and Qwen-14B and TP8 for Qwen-32B. We retain its TBTaware SM allocation, assigning just enough SMs to decode to meet the TBT SLO and the remaining SMs to prefill. • DistServe (static isolation). We use vLLM’s disaggregated serving with NCCL KV connector. We sweep feasible prefill–decode GPU allocations and use the bestperforming 1:1 split: We use TP2 per phase for Llama-8B and Qwen-14B, and TP4 per phase for Qwen-32B. • LoongServe (elastic isolation). We use the original LoongServe implementation with base TP1 and maximum SP4 for Llama-8B and Qwen-14B, and base TP2 and maximum

SP4 for Qwen-32B, where TP1 leaves insufficient KV-cache headroom after loading the model weights. We retain its execution architecture and enable piecewise CUDA graphs to align launch overhead with the other systems. For ConfluxServe, XP instances use base TP1 with SP4 for Llama-8B and Qwen-14B, and base TP2 with SP4 for Qwen32B, following the same rationale as LoongServe. 8.2

End-to-End Performance

Figure 12 reports request-level SLO attainment on MooncakeToolagent across the three models. The top row compares all systems under FCFS scheduling, while the bottom compares the scheduling-enhanced baselines with full ConfluxServe. We first examine FCFS to understand the execution benefit of Conflux Parallelism. Most systems achieve high SLO attainment at low to moderate load but degrade at markedly different rates as the arrival rate increases. DistServe and LoongServe degrade earliest because phase isolation prevents queued prefills from using compute capacity left underutilized by decode: DistServe uses a static prefill–decode partition, while LoongServe dynamically reallocates resources across phases but still executes them separately. MuxWise improves cross-phase utilization through spatial multiplexing, but separate prefill and decode kernels forgo co-batching’s weight reuse and introduce cross-kernel resource contention. Chunked Prefill performs better by co-batching the two phases, but its uniform parallelism configuration consumes decode-latency headroom and constrains the TBT-safe prefill budget. In contrast, Conflux Parallelism combines cobatching with phase-specific parallelism, allowing ConfluxServeFCFS to maintain high SLO attainment at substantially higher loads. It sustains approximately 7.9, 6.0, and 4.1 requests/s on Llama-8B, Qwen-14B, and Qwen-32B, outperforming the strongest baseline by 1.33×, 1.34×, and 1.62×, respectively. The scheduling-enhanced setting shows a similar trend, and ConfluxServe retains a clear advantage as load increases. With its SLO-aware scheduler, ConfluxServe sustains approximately 9.2, 7.9, and 5.6 requests/s on Llama-8B, Qwen10

w/o Overlap 1.23x

120 1.07x

100

1.13x

w/o Phase-specific

Normalized latency

Latency (ms)

Conflux Parallelism

w/o Co-batching 1.34x

800 700

1.11x

1.16x

600

80

500

60

400

8K prefill

Figure 13. Component-wise benefit of Conflux Parallelism with 8K- and 32K-token prefills. Each configuration progressively removes attention overlap, phase-specific parallelism, and prefill–decode co-batching. Values denote latency normalized to full Conflux Parallelism. Contention-free

Overlapped

Latency (s)

Latency (ms)

10 0

d=8

d=16

d=24

(a) Attention overlap

d=32

2 1.93x

1 0

4.17x

N=1

N=2

N=4

7.94x

N=8

(b) SP degree selection

Figure 14. Overlapping and parallelism design in Conflux Parallelism. (a) attention overlap introduces up to 1.3× speedup with highly stable performance. (b) parallelism configuration for XP instance achieves near-linear scaling. 14B, and Qwen-32B, respectively, exceeding the strongest scheduling-enhanced baseline by 1.26×, 1.52×, and 1.44×. Relative to ConfluxServe-FCFS, the SLO-aware scheduler further improves goodput by 1.16×, 1.32×, and 1.37×, respectively, demonstrating the additional benefit of our scheduler design. 8.3

1.03x

0.99x

1.0

1.39x

Attention

1.01x

0.5 0.0

Waterfill

Even

Worst-fit

Moderate

Severe

(b) Decode placement

attention execution against serialized execution and an idealized contention-free lower-bound derived from isolated component latencies. As the decode load increases, the opportunity for overlap grows: at 32 decode requests per GPU, Conflux Parallelism reduces attention-region latency from 36.5 ms to 28.0 ms, a 1.3× speedup over serialized execution, closing 58% of the gap to the idealized contention-free lower bound. Moreover, attention overlap exhibits a coefficient of variation below 3.5% across 20 runs for every configuration, indicating stable interference under coordinated overlap.

1.00x

20

1.32x

Figure 15. Effectiveness of model-guided prefill dispatch and decode placement. Latencies are normalized to our solution.

Serialized

30

1.75x

Linear

1.5

(a) Prefill dispatch

32K prefill

40

2.0

Ablation Study

We evaluate how key design components contribute to ConfluxServe. Unless otherwise specified, experiments use Qwen14B on a four-GPU XP instance (TP1–SP4). Conflux Parallelism performance breakdown. To isolate the component-wise benefits of Conflux Parallelism, we maintain 24 decode requests per GPU, each with 8K-token contexts under a 100 ms TBT SLO, and measure the completion time of an injected 8K- or 32K-token prefill request. Figure 13 progressively removes attention overlap, phasespecific parallelism, and prefill–decode co-batching from Conflux Parallelism. For the 8K prefill, disabling attention overlap increases latency to 1.07× that of Conflux Parallelism. Further removing phase-specific parallelism increases latency to 1.13×, while additionally removing co-batching raises it to 1.23×. The impact becomes more pronounced for the 32K prefill: the slowdowns increase to 1.11×, 1.16×, and 1.34×, respectively. The larger gap reflects the cumulative cost of these execution inefficiencies across multiple prefill chunks. Effectiveness of coordinated attention overlap. Figure 14(a) evaluates attention overlap by co-executing one 8K-token prefill while increasing the decode load from 8 to 32 requests per GPU. We compare Conflux Parallelism’s overlapped 11

Parallelism configuration for XP instance. Figure 14(b) validates our choice of using a fixed maximum intra-node SP degree for each XP instance. We fix the base TP degree to 1 and scale a 32K-token prefill from SP1 to SP8, while keeping the per-GPU decode load at 24 requests with 8K-token contexts under a 100 ms TBT SLO. Increasing the SP degree from 1 to 2, 4, and 8 reduces prefill completion latency by 1.93×, 4.17×, and 7.94×, respectively. Although higher SP introduces additional communication, Conflux Parallelism overlaps the distributed prefill path with local decode attention, effectively hiding much of this overhead and enabling near-linear prefill scaling when co-executed with decode. Effectiveness of prefill dispatch. Figure 15(a) evaluates waterfill under a skewed four-GPU decode load: one rank hosts 128 decode requests with 2K-token contexts, while the other three have no decodes. We co-execute a 1K-token prefill and compare against even dispatch. By assigning fewer prefill tokens to the decode-heavy rank, waterfill balances the combined load, while even dispatch incurs 1.32× the sharedlinear latency. Meanwhile, split-attention latency remains nearly unchanged (0.99×), as Ulysses-SP redistributes the prefill workload across ranks. Together, waterfill reduces the critical-rank bottleneck without penalizing prefill attention. Effectiveness of decode placement. Figure 15(b) evaluates the benefit of worst-fit decode placement when sufficient KV headroom is available. We fix 64 decode requests, each with an 8K-token context, and co-execute a 1K-token prefill. We evaluate three decode placements: worst-fit [16, 16, 16, 16], moderately skewed [8, 8, 24, 24], and severely skewed [0, 0, 32, 32]. Shared-linear latency remains nearly unchanged (1.03× and 1.01×) because waterfill prefill dispatch compensates for the uneven decode load. In contrast, split-attention latency increases to 1.39× and 1.75× under moderate and severe skew, respectively, as the most heavily loaded rank determines the

SLO attainment (%)

Chunked++ MuxWise++ DistServe++ LoongServe++ ConfluxServe

MAPE

100

Queue depth

80

← LoongServe++

1

3

5

7

13

Time (ms)

14

Arrival rate (req/s)

15

16

Arrival rate (req/s)

SLO attainment (%)

Figure 16. ConfluxServe on Qwen-14B with MooncakeSynthetic and QwenA (other setups follow §8.2).

8.5

90 80

1

2

3

4

5

6

7

Arrival rate (req/s)

8

9

1

2

3

4

5

6

Arrival rate (req/s)

7

10

50

100

500

0.36±0.05 0.47±0.05 0.56±0.05 1.04±0.05

Component Accuracy and Overheads

Latency-Estimator Accuracy. We evaluate the latency estimator using mean absolute percentage error (MAPE) on 200 randomly sampled heterogeneous batches per model, spanning prefill chunks of 32–10K tokens, decode batches of 1–128, and context lengths of 1K–16K tokens. As shown in Table 4, the estimator achieves 4.1–4.8% MAPE across the three models, showing that it accurately captures Conflux Parallelism execution latency across model sizes. Scheduler Overhead. We measure ConfluxServe’s scheduling latency with different pending queue depths. As Table 5 shows, scheduling overhead remains below 1.05 ms across all tested queue depths. Since scheduling runs asynchronously with GPU execution, most of this cost can be hidden.

100

(a) TBT SLO = 150 ms

8

(b) TBT SLO = 50 ms

Figure 17. ConfluxServe performance on Qwen-14B with different TBT SLOs (other setups follow §8.2). critical path. This validates our choice of worst-fit placement to balance decode load across ranks. 8.4

Qwen-32B 4.1%

Table 5. ConfluxServe scheduling latency. Results report mean ± standard deviation over 100 scheduling steps.

(b) qwen_traceA (b)

(a) mooncake_synthetic (a)

70

Qwen-14B 4.8%

Table 4. Latency-estimation accuracy across models.

90

70

Llama-8B 4.4%

Sensitivity Study

Workload Sensitivity. We evaluate Qwen-14B on two additional workloads, Mooncake-Synthetic and QwenA, using the same setup and SLO targets in §8.2. As summarized in Table 3, Mooncake-Synthetic is more prefill-heavy, with longer prompts and similarly short outputs, whereas QwenA is more decode-heavy with shorter prompts and longer outputs. Figure 16 shows that ConfluxServe sustains the highest load on both workloads. On Mooncake-Synthetic, ConfluxServe sustains 5.5 requests/s, 1.7× that of the strongest baseline, showing that its advantage persists under substantially longer prompts. On QwenA, ConfluxServe sustains 15.6 requests/s, 1.1× that of the strongest baseline. The smaller gain is expected as shorter prompts reduce the prefill work available for exploiting prefill–decode resource complementarity. ConfluxServe still benefits from localized decode execution and avoids the uniform-parallelism tax. TBT Sensitivity. We vary the default 100 ms TBT SLO to 150 ms and 50 ms on Qwen-14B. As shown in Figure 17, ConfluxServe sustains 8.3 requests/s at 150 ms and 7.2 requests/s at 50 ms, outperforming the strongest baseline by 1.4× and 1.5×, respectively. Relaxing the TBT SLO provides additional headroom for prefill co-execution, increasing the sustainable rate of ConfluxServe and Chunked++. The other baselines gain little since they cannot effectively use this headroom for prefill progress. Conversely, tightening the TBT SLO reduces the TBT-safe prefill budget, slightly reducing the sustainable rate of both ConfluxServe and Chunked++. ConfluxServe nevertheless retains its advantage, as its phase-specific parallelism preserves more decode-latency headroom than Chunked++.

9

Related Work

We discussed prefill–decode execution designs in §3.3; we now discuss prior work related to ConfluxServe in other areas. SLO-Aware LLM Scheduling. A large body of work studies SLO-aware scheduling to improve goodput under TTFT and TBT SLOs [1, 9, 25, 28, 34, 46, 50]. ConfluxServe shares this objective, but its scheduling design addresses the new challenges introduced by Conflux Parallelism’s heterogeneous execution: accurately modeling its latency, dispatching prefills across ranks, and placing decode requests to realize its execution-efficiency gains as end-to-end goodput. Dynamic Parallelism. Shift Parallelism [18] and Seesaw [40] dynamically adapt model-wide parallelism over time, selecting a configuration better suited to the currently dominant workload or phase. However, each execution interval still uses a single model-wide configuration, forcing co-executed prefill and decode to share the same parallelism. They therefore remain subject to the utilization–efficiency dilemma. Operator-level Heterogeneity. Prior systems exploit operatorlevel heterogeneity for different objectives. Helix [7] decouples attention and FFN sharding to accelerate ultra-longcontext decoding, while Hetis [29] maps dense computation and decode attention differently to better utilize heterogeneous GPUs. POD-Attention [22] co-executes prefill and decode attention through a fused, SM-aware kernel under a fixed parallelism configuration. These systems do not support co-executed prefill and decode with different inter-GPU layouts. Orca [49] co-batches requests through compatible 12

operators while retaining request-specific attention, accommodating heterogeneous sequence lengths under a common parallelism configuration. In contrast, ConfluxServe enables co-executed prefill and decode to retain different parallelism configurations. This parallelism heterogeneity further introduces cross-phase KV relocation and rank-aware prefill dispatch and decode placement, which ConfluxServe addresses.

10

[8] William Brandon, Aniruddha Nrusimha, Kevin Qian, Zachary Ankner, Tian Jin, Zhiye Song, and Jonathan Ragan-Kelley. 2023. Striped Attention: Faster Ring Attention for Causal Transformers. arXiv preprint arXiv:2311.09431 (2023). [9] Siyuan Chen, Zhipeng Jia, Samira Khan, Arvind Krishnamurthy, and Phillip B Gibbons. 2025. SLOs-Serve: Optimized serving of multi-SLO LLMs. arXiv preprint arXiv:2504.08784 (2025). [10] Yukang Chen, Weihao Cui, Han Zhao, Ziyi Xu, Xiaoze Fan, Xusheng Chen, Yangjie Zhou, Shixuan Sun, Bingsheng He, and Quan Chen. 2026. Towards High-Goodput LLM Serving with Prefill-decode Multiplexing. In Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (USA) (ASPLOS ’26). Association for Computing Machinery, New York, NY, USA, 2030–2047. https://doi.org/10.1145/3779212.3790236 [11] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022 (NeurIPS 2022). arXiv:2205.14135 [cs.LG] https://arxiv.org/abs/2205. 14135 [12] Jiangsu Du, Hongbin Zhang, Taosheng Wei, Zhenyi Zheng, Jiazhi Jiang, Kaiyi Wu, Zhiguang Chen, and Yutong Lu. 2026. Efficient LLM Serving on Commodity GPU Clusters with Data-Reduced CrossInstance Orchestration. In 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 26). USENIX Association. arXiv:2504.18154 [cs.DC] https://www.usenix.org/conference/osdi26/ presentation/du [13] Jiarui Fang and Shangchun Zhao. 2024. USP: A Unified Sequence Parallelism Approach for Long Context Generative AI. arXiv preprint arXiv:2405.07719 (2024). [14] Jingqi Feng, Yukai Huang, Rui Zhang, Sicheng Liang, Ming Yan, and Jie Wu. 2025. WindServe: Efficient Phase-Disaggregated LLM Serving with Stream-based Dynamic Scheduling. In Proceedings of the 52nd Annual International Symposium on Computer Architecture (ISCA ’25). Association for Computing Machinery, 1283–1295. https://doi.org/10. 1145/3695053.3730999 [15] Kanishk Goel, Jayashree Mohan, Nipun Kwatra, Ravi Shreyas Anupindi, and Ramachandran Ramjee. 2026. QoServe: Breaking the Silos of LLM Inference Serving. In Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. Association for Computing Machinery, 1492–1507. https://doi.org/10.1145/3779212.3790206 arXiv:2503.22562 [cs.LG] [16] Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, et al. 2026. Accelerating Scientific Discovery with Co-Scientist. Nature 655 (2026), 487–496. https://doi.org/10.1038/s41586-026-10644-y [17] Aaron Grattafiori et al. 2024. The Llama 3 Herd of Models. arXiv:2407.21783 [cs.AI] https://arxiv.org/abs/2407.21783 [18] Mert Hidayetoglu, Aurick Qiao, Michael Wyatt, Jeff Rasley, Yuxiong He, and Samyam Rajbhandari. 2026. Shift Parallelism: Low-Latency, High-Throughput LLM Inference for Dynamic Workloads. In Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (USA) (ASPLOS ’26). Association for Computing Machinery, New York, NY, USA, 1749–1763. https://doi.org/10.1145/3779212.3790219 [19] Connor Holmes, Masahiro Tanaka, Michael Wyatt, Ammar Ahmad Awan, Jeff Rasley, Samyam Rajbhandari, Reza Yazdani Aminabadi, Heyang Qin, Arash Bakhtiari, Lev Kurilenko, and Yuxiong He. 2024. DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference. arXiv preprint arXiv:2401.08671 (2024). [20] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. 2019. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. In Advances in

Conclusion

We identify a utilization–efficiency dilemma in existing prefilldecode execution designs. To resolve it, we propose Conflux Parallelism, a parallel execution abstraction that co-executes prefill and decode with different parallelism within the same batch. Conflux Parallelism exploits operator-level Transformer structure to construct a heterogeneous execution path with different batching and parallelism strategies across operators, combining the best of co-batching and phase-specific execution. Building on Conflux Parallelism, we design ConfluxServe, an SLO-aware serving system that maximizes cluster goodput under TTFT and TBT constraints.

References [1] Muhammad Adnan, Rohan Mahapatra, Prashant J Nair, Daniel Berger, Pantea Zardoshti, Rodrigo Fonseca, and Esha Choukse. 2026. Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving. arXiv preprint arXiv:2608.06557 (2026). [2] Amey Agrawal, Anmol Agarwal, Nitin Kedia, Jayashree Mohan, Souvik Kundu, Nipun Kwatra, Ramachandran Ramjee, and Alexey Tumanov. 2024. Etalon: Holistic Performance Evaluation Framework for LLM Inference Systems. https://arxiv.org/abs/2407.07000. arXiv:2407.07000 [cs.LG] [3] Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav S. Gulavani, Ramachandran Ramjee, and Alexey Tumanov. 2024. Vidur: A Large-Scale Simulation Framework for LLM Inference. In Proceedings of the Seventh Annual Conference on Machine Learning and Systems, MLSys 2024, Santa Clara, CA, USA, May 13-16, 2024. mlsys.org. https://proceedings.mlsys.org/paper_files/paper/2024/hash/ b74a8de47d2b3c928360e0a011f48351-Abstract-Conference.html [4] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2024, Santa Clara, CA, USA, July 10-12, 2024. USENIX Association, 117–134. https://www.usenix.org/ conference/osdi24/presentation/agrawal [5] Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023. Association for Computational Linguistics, 4895–4901. https://doi. org/10.18653/v1/2023.emnlp-main.298 [6] Anthropic. 2026. Claude Code. https://www.anthropic.com/claudecode. [7] Nidhi Bhatia, Ankit More, Ritika Borkar, Tiyasa Mitra, Ramon Matas, Ritchie Zhao, Maximilian Golub, Dheevatsa Mudigere, Brian Pharris, and Bita Darvish Rouhani. 2025. Helix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decoding. arXiv preprint arXiv:2507.07120 (2025). 13

Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada. 103–112. https://proceedings.neurips.cc/ paper/2019/hash/093f65e080a295f8076b1c5722a46aa2-Abstract.html [21] Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. 2023. DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models. arXiv:2309.14509 [cs.LG] https://arxiv.org/abs/2309.14509 [22] Aditya K. Kamath, Ramya Prabhu, Jayashree Mohan, Simon Peter, Ramachandran Ramjee, and Ashish Panwar. 2025. POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (Rotterdam, Netherlands) (ASPLOS ’25). Association for Computing Machinery, New York, NY, USA, 897–912. https://doi.org/10.1145/ 3676641.3715996 [23] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023. ACM, 611–626. https://doi.org/10.1145/3600006.3613165 [24] Cong Li, Yuzhe Yang, Xuegui Zheng, Qifan Yang, Yijin Guan, Size Zheng, Li-Wen Chang, Shufan Liu, Xin Liu, and Guangyu Sun. 2026. Tetris: Efficient Long-context LLM Serving with Chunkwise Dynamic Sequence Parallelism. In 2026 ACM/IEEE 53rd Annual International Symposium on Computer Architecture (ISCA). IEEE, 1287–1301. https: //doi.org/10.1109/ISCA66397.2026.00098 [25] Zikun Li, Zhuofu Chen, Remi Delacourt, Gabriele Oliaro, Zeyu Wang, Qinghan Chen, Shuhuai Lin, April Yang, Zhihao Zhang, Zhuoming Chen, et al. 2025. AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding. arXiv preprint arXiv:2501.12162 (2025). [26] Zejia Lin, Hongxin Xu, Guanyi Chen, Zhiguang Chen, Yutong Lu, and Xianwei Zhang. 2026. Bullet: Boosting GPU Utilization for LLM Serving via Dynamic Spatial-Temporal Orchestration. In Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (USA) (ASPLOS ’26). Association for Computing Machinery, New York, NY, USA, 290–306. https://doi.org/10.1145/3779212.3790135 [27] Hao Liu, Matei Zaharia, and Pieter Abbeel. 2023. Ring Attention with Blockwise Transformers for Near-Infinite Context. arXiv preprint arXiv:2310.01889 (2023). [28] Jiachen Liu, Zhiyu Wu, Jae-Won Chung, Fan Lai, Myungjin Lee, and Mosharaf Chowdhury. 2024. Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services. CoRR abs/2404.16283 (2024). https://doi.org/10.48550/ARXIV.2404.16283 arXiv:2404.16283 [29] Zizhao Mo, Jianxiong Liao, Huanle Xu, Zhi Zhou, and Chengzhong Xu. 2025. Hetis: Serving LLMs in Heterogeneous GPU Clusters with Finegrained and Dynamic Parallelism. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’25). Association for Computing Machinery, 1710–1724. https://doi.org/10.1145/3712285.3759784 arXiv:2509.08309 [30] J. Michael Moore. 1968. An 𝑛 Job, One Machine Sequencing Algorithm for Minimizing the Number of Late Jobs. Management Science 15, 1 (1968), 102–109. https://doi.org/10.1287/mnsc.15.1.102 [31] NVIDIA. 2025. CUDA Driver API: Green Contexts. https://docs.nvidia.com/cuda/archive/12.8.0/cuda-driverapi/group__CUDA__GREEN__CONTEXTS.html. CUDA Toolkit 12.8.0 documentation. [32] OpenAI. 2026. OpenAI Codex. https://openai.com/codex/. [33] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo

Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. In 51st ACM/IEEE Annual International Symposium on Computer Architecture, ISCA 2024, Buenos Aires, Argentina, June 29 - July 3, 2024. IEEE, 118–132. https: //doi.org/10.1109/ISCA59077.2024.00019 [34] Archit Patke, Dhemath Reddy, Saurabh Jha, Haoran Qiu, Christian Pinto, Chandra Narayanaswami, Zbigniew Kalbarczyk, and Ravishankar Iyer. 2024. Queue management for SLO-oriented large language model serving. In Proceedings of the 2024 ACM Symposium on Cloud Computing. 18–35. [35] Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2023. Efficiently Scaling Transformer Inference. In Proceedings of Machine Learning and Systems, Vol. 5. 606–624. arXiv:2211.05102 https://proceedings.mlsys.org/paper_files/paper/2023/hash/ c4be71ab8d24cdfb45e3d06dbfca2780-Abstract-mlsys2023.html [36] Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture for Serving LLM Chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST 25). USENIX Association, 155–170. https://www.usenix.org/conference/fast25/presentation/qin [37] Chaoyi Ruan, Yinhe Chen, Dongqi Tian, Yandong Shi, Yongji Wu, Jialin Li, and Cheng Li. 2026. Libra: Flexible Request Partitioning and Scheduling for Serving Unbalanced and Dynamic LLM Workloads. In 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26). USENIX Association, 1243–1258. https: //www.usenix.org/conference/nsdi26/presentation/ruan-libra [38] Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. 2024. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-Precision. In Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024 (NeurIPS 2024). arXiv:2407.08608 [cs.LG] https://arxiv.org/abs/2407.08608 [39] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2020. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv:1909.08053 [cs.CL] https://arxiv.org/abs/1909.08053 [40] Qidong Su, Wei Zhao, Xin Li, Muralidhar Andoorveedu, Chenhao Jiang, Zhanda Zhu, Kevin Song, Christina Giannoula, and Gennady Pekhimenko. 2025. Seesaw: High-throughput LLM Inference via Model Re-sharding. In Proceedings of Machine Learning and Systems, Vol. 7. arXiv:2503.06433 https://proceedings.mlsys.org/paper_files/paper/2025/hash/ cbc4ab80cd77aa0eb87da062fbcddb46-Abstract-Conference.html [41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017 (NeurIPS 2017). arXiv:1706.03762 [cs.CL] https: //arxiv.org/abs/1706.03762 [42] vLLM Team. 2026. CUDA Graphs. https://docs.vllm.ai/en/v0.15.1/ design/cuda_graphs/. vLLM v0.15.1 design documentation. [43] Chao Wang, Pengfei Zuo, Zhangyu Chen, Yunkai Liang, Zhou Yu, and Ming-Chang Yang. 2026. Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving. In The International Conference for High Performance Computing, Networking, Storage, and Analysis (SC). [44] Jiahao Wang, Jinbo Han, Xingda Wei, Sijie Shen, Dingyan Zhang, Chenguang Fang, Rong Chen, Wenyuan Yu, and Haibo Chen. 2025. KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider. In 2025 USENIX Annual Technical Conference (USENIX ATC 25). USENIX Association. 14

Throughput. In 19th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2025, Boston, MA, USA, July 7-9, 2025. USENIX Association, 749–765. https://www.usenix.org/conference/ osdi25/presentation/zhu-kan

arXiv:2506.02634 [cs.DC] https://www.usenix.org/conference/atc25/ presentation/wang-jiahao [45] Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. 2024. LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles, SOSP 2024, Austin, TX, USA, November 4-6, 2024. ACM, 640–654. https: //doi.org/10.1145/3694715.3695948 [46] Bingyang Wu, Yinmin Zhong, Zili Zhang, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xuanzhe Liu, and Xin Jin. 2026. FastServe: Iteration-Level Preemptive Scheduling for Large Language Model Inference. In 23rd USENIX Symposium on Networked Systems Design and Implementation, NSDI 2026, Renton, WA, May 4-6, 2026, Srikanth Kandula and Hakim Weatherspoon (Eds.). USENIX Association, 57–74. https://www.usenix.org/conference/nsdi26/presentation/ wu-bingyang [47] Jochen Wulf and Juerg Meierhofer. 2024. Exploring the Potential of Large Language Models for Automation in Technical Customer Service. https://arxiv.org/abs/2405.09161. arXiv:2405.09161 [cs.CL] [48] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. 2025. Qwen3 Technical Report. arXiv:2505.09388 [cs.CL] https: //arxiv.org/abs/2505.09388 [49] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. In 16th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2022, Carlsbad, CA, USA, July 11-13, 2022. USENIX Association, 521–538. https: //www.usenix.org/conference/osdi22/presentation/yu [50] Wei Zhang, Zhiyu Wu, Yi Mu, Rui Ning, Banruo Liu, Nikhil Sarda, Myungjin Lee, and Fan Lai. 2026. JITServe: SLO-aware LLM Serving with Imprecise Request Information. In 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26). USENIX Association. https://www.usenix.org/conference/nsdi26/presentation/ zhang-wei [51] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark W. Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Structured Language Model Programs. In Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024. http://papers.nips.cc/paper_files/paper/2024/hash/ 724be4472168f31ba1c9ac630f15dec8-Abstract-Conference.html [52] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. In 18th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2024, Santa Clara, CA, USA, July 10-12, 2024. USENIX Association, 193–210. https://www.usenix.org/conference/ osdi24/presentation/zhong-yinmin [53] Kan Zhu, Yufei Gao, Yilong Zhao, Liangyu Zhao, Gefei Zuo, Yile Gu, Dedong Xie, Zihao Ye, Keisuke Kamahori, Chien-Yu Lin, Ziren Wang, Stephanie Wang, Arvind Krishnamurthy, and Baris Kasikci. 2025. NanoFlow: Towards Optimal Large Language Model Serving 15

A

Parallelism Analysis

the participating ranks, limiting the usable SP degree by the available head count; this constraint becomes more restrictive for GQA and MQA models with fewer KV heads [13]. Hybrid Ulysses–Ring designs have been proposed in part to relax this head-count constraint and adapt communication to hierarchical network topologies [13].

Table 6 compares the per-rank communication costs of TP, Ring-SP, and Ulysses-SP for chunked prefill. Let 𝑃 denote the parallel degree, 𝐵 the batch size, 𝑆 the current prefill chunk length, 𝐶 the historical context length, 𝐻 the hidden dimension, and 𝐺 the query-to-KV head ratio. All three strategies have the same leading-order per-rank computation, Θ((𝐵𝑆𝐻 2 +𝐵𝑆 (𝐶+𝑆)𝐻 )/𝑃), so their primary asymptotic difference lies in communication.

Comparison with TP. For reference, standard TP [39] communicates current-chunk activations through the collective operations associated with the attention output projection and FFN. Under a ring-based All-Reduce, its per-rank communication remains Θ(𝐵𝑆𝐻 ) as 𝑃 increases. Like Ulysses-SP, TP communication is independent of the historical context length; unlike Ulysses-SP, however, increasing the parallel degree does not asymptotically reduce the per-rank communication volume.

Ring-SP versus Ulysses-SP. The key distinction between the two SP strategies is how they communicate and retain KV state. Standard Ring-SP [27] keeps KV sequence-partitioned and circulates remote K/V blocks across ranks. During chunked prefill, each rank must therefore receive remote K/V states covering both the historical context and the current chunk. Its per-rank communication consequently depends on 𝐶 + 𝑆 and remains Θ(𝐵(𝐶 + 𝑆)𝐻 /𝐺) as 𝑃 increases. Moreover, this historical-KV communication is incurred again for each subsequent prefill chunk. Ulysses-SP [21] instead uses an All-to-All to transpose the current-chunk Q/K/V activations from sequence partitioning to head partitioning. Each rank then computes its assigned attention heads for all current-chunk queries using the cached history and current-chunk K/V, after which a second All-to-All restores sequence-partitioned outputs. Historical KV remains head-partitioned and can be reused directly by subsequent chunks, so only activations associated with the current chunk are communicated. As a result, its per-rank communication is independent of 𝐶 and decreases as Θ(𝐵𝑆𝐻 (1 + 1/𝐺)/𝑃) with increasing 𝑃.

Effect of grouped-query attention. A larger query-to-KV head ratio 𝐺, as used by grouped-query attention (GQA) [5], reduces K/V communication in both SP designs. For Ring-SP, the communicated state consists of K/V blocks, so its communication decreases proportionally with 1/𝐺. For UlyssesSP, only the K/V portion of the first All-to-All decreases with 𝐺, while query and output communication remain unchanged. The leading-order attention computation is unaffected because the number of query heads does not change. Overall, Ring-SP and Ulysses-SP expose complementary trade-offs. Ring-SP enables fine-grained communication– computation overlap and does not fundamentally require head-wise partitioning, but standard execution repeatedly communicates historical KV state, retains constant per-rank communication as 𝑃 grows, and requires additional mechanisms to avoid causal load imbalance. Ulysses-SP avoids historical-KV communication during chunked prefill and reduces per-rank communication with increasing 𝑃, while preserving efficient whole-head attention computation, at the cost of global All-to-All synchronization and headcount constraints. For the intra-node parallel degrees considered in ConfluxServe, these properties favor Ulysses-SP for distributed prefill.

Practical trade-offs. Communication volume alone does not capture all differences between the two SP strategies. Ring-SP pipelines P2P KV circulation with blockwise attention computation, allowing communication to be hidden when sufficient attention work is available [27]. However, splitting attention into sequence blocks can reduce fusedkernel efficiency [13], and vanilla Ring-SP with contiguous sequence partitions suffers from substantial load imbalance under causal attention because different ranks cover different portions of the triangular attention matrix [8, 13]. Variants such as Striped Attention and load-balanced Ring Attention mitigate this imbalance by redistributing sequence positions across ranks [8, 13]. These optimizations do not, however, remove the need to communicate remote KV state in standard pass-KV Ring execution. Ulysses-SP preserves complete attention computation for each assigned head and can therefore directly use efficient fused attention kernels such as FlashAttention [11, 21]. Its two All-to-All collectives, however, introduce global synchronization points around attention and can become exposed communication overhead when the interconnect is slow or the local attention workload is small. Standard Ulysses also requires the attention heads to be partitionable across

B

Operator Analysis

Table 7 analyzes how cross-request batching affects the two dominant operator classes in LLM inference. We consider independent requests and omit constant factors and element sizes. For attention, we assume a fused exact-attention implementation [11] that avoids materializing the attention matrix in HBM. Dense linear operators. Consider an 𝐻 × 𝐻 linear transformation applied to 𝑇 active tokens. Its computation is Θ(𝑇 𝐻 2 ), while its dominant tensor traffic is Θ(𝐻 2 + 𝑇 𝐻 ): the 𝐻 2 model weights are shared across all tokens, while activation traffic grows with 𝑇 . Its arithmetic intensity is therefore Θ(1/(1/𝐻 + 1/𝑇 )) and increases with the number of co-batched tokens. For prefill, 𝑇 = 𝐵𝑆; for decode, 𝑇 = 𝐵. 16

Parallelism

Partition

Communication

Model dimensions / heads

2×All-Reduce

Ring-SP

Sequence

P2P KV circulation

Ulysses-SP

Sequence ↔ heads

2×All-to-All

TP

Per-Rank Communication Volume   𝑃 −1 𝐵𝑆𝐻 4 𝑃   𝑃 − 1 𝐵(𝐶 + 𝑆)𝐻 2 𝑃 𝐺     𝑃 −1 1 2 𝐵𝑆𝐻 1 + 𝑃2 𝐺

Scaling in 𝑃 Θ(1) Θ(1) Θ(1/𝑃)

Table 6. Per-rank communication of one Transformer block. 𝑃, 𝐵, 𝑆, 𝐶, and 𝐻 denote the parallel degree, batch size, current prefill chunk length, historical context length, and hidden dimension, respectively; 𝐺 is the query-to-KV head ratio, with 𝐺 = 1 corresponding to multi-head attention. TP communicates activations associated with the current chunk through two All-Reduce operations. Standard Ring-SP circulates sequence-partitioned K/V states from both the historical context and the current chunk, whereas Ulysses-SP communicates only current-chunk Q/K/V activations and attention outputs. Communication volumes count transmitted tensor elements per rank and exclude local transfers. The TP expression assumes a ring-based All-Reduce, and the Ring-SP expression assumes standard dense pass-KV Ring Attention. Scaling in 𝑃 holds 𝐵, 𝑆, 𝐶, 𝐻 , and 𝐺 fixed. Operator

Compute

Dominant Tensor Traffic

Prefill Dense Linear

Θ(𝐵𝑆𝐻 2 )

Θ(𝐻 2 + 𝐵𝑆𝐻 )

Prefill Attention

Θ(𝐵𝑆𝐿𝐻 )

Θ(𝐵𝑆𝐻 + 𝐵𝐿𝐻 /𝐺)

Decode Dense Linear

Θ(𝐵𝐻 2 )

Θ(𝐻 2 + 𝐵𝐻 )

Decode Attention

Θ(𝐵𝐿𝐻 )

Θ(𝐵𝐻 + 𝐵𝐿𝐻 /𝐺)

Arithmetic Intensity   1 Θ 1/𝐻 + 1/(𝐵𝑆)   1 Θ 1/𝐿 + 1/(𝐺𝑆)   1 Θ 1/𝐻 + 1/𝐵   1 Θ 1/𝐿 + 1/𝐺

vs. Larger 𝐵 Increases

Unchanged

Increases

Unchanged

Table 7. Asymptotic compute, dominant tensor traffic, and arithmetic intensity of major LLM operators. 𝐵 is the number of independent requests, 𝑆 the number of active prefill tokens per request, 𝐿 the average number of KV positions attended by each active query, 𝐻 = 𝑛𝑞 𝑑 the query hidden width, and 𝐺 = 𝑛𝑞 /𝑛𝑘𝑣 the query-to-KV head ratio. Dense-linear rows represent an 𝐻 × 𝐻 projection; QKV, output-projection, and FFN dimensions change only constant factors when they scale with 𝐻 . For attention, we assume a fused exact-attention implementation [11] that avoids materializing the attention matrix in HBM; the tensor-traffic expressions capture the Q/K/V/output tensors and abstract away implementation-dependent tiling and reloads. Fixed constants and bytes per element are omitted. Co-batching requests therefore amortizes model-weight traffic over a larger token batch, an effect also observed in prior operator-level analyses of LLM serving [53]. Importantly, this benefit depends only on sharing the same weight matrix and feature dimension. Tokens from different requests, or from prefill and decode, can therefore be concatenated along the token dimension and processed by the same GEMM. In Conflux Parallelism, co-batching the two phases increases the effective 𝑇 seen by the shared QKV projection, output projection, and FFN operators without requiring them to use the same attention parallelism.

cels: adding requests increases attention computation and request-specific KV traffic proportionally. For decode, where each request contributes one active query, this reduces to Θ(1/(1/𝐿 + 1/𝐺)). Thus, increasing the cross-request batch size does not asymptotically improve attention arithmetic intensity. This does not imply that attention receives no practical benefit from batching: larger batches can improve kernel occupancy or amortize launch overhead. Rather, attention lacks the model-weight amortization available to dense linear operators because its dominant KV state is request-specific. Prior serving systems similarly exploit this operator-level distinction through selective batching [49], while specialized kernels such as POD-Attention exploit the complementary resource demands of prefill and decode attention through concurrent execution rather than shared model-weight reuse [22].

Attention. Attention exhibits a different batching behavior because each request attends to its own KV state. For 𝐵 independent requests with 𝑆 active queries per request and average attended KV length 𝐿, attention performs Θ(𝐵𝑆𝐿𝐻 ) computation and accesses Θ(𝐵𝑆𝐻 + 𝐵𝐿𝐻 /𝐺) query/output and KV tensors. Its arithmetic intensity is therefore Θ(1/(1/𝐿 + 1/(𝐺𝑆))). Unlike dense linear operators, the factor 𝐵 can-

Implication for Conflux Parallelism. This asymmetry motivates Conflux Parallelism’s operator-level execution 17

structure. Co-batching prefill and decode through the shared linear operators captures the arithmetic-intensity benefit of amortizing model-weight traffic, whereas separating their attention paths gives up substantially less cross-request batching benefit while allowing each phase to retain its own attention parallelism. The separated paths can additionally execute concurrently to exploit the complementary resource demands of prefill and decode attention.

C

Thus, composing Conflux Parallelism with TP does not require the two phases to share the same parallelism across logical ranks. The base-TP configuration is common to both phases and its collectives remain unchanged, while Conflux Parallelism applies additional SP only to the prefill attention path across logical ranks. This preserves prefill–decode co-batching in the shared linear operators while allowing distributed prefill attention and localized decode attention to coexist over the same base-TP execution layout.

Conflux Parallelism over a Base TP Configuration

D

Latency Model Details

This section details the base latency functions used by the estimator in §6.2. Following prior operator-level latency models [3, 45, 52], we model each component as a function of its dominant workload metric and fit the corresponding coefficients from offline execution profiles. Shared linear operators. The shared QKV/O projections and FFNs process the rank-local prefill and decode tokens together. We model their per-layer latency as

The main discussion of Conflux Parallelism assumes one GPU per logical rank. For models that require TP to fit model weights or provide sufficient KV-cache capacity, Conflux Parallelism instead treats each fixed TP group as one logical rank. Figure 18 illustrates this composition with base TP2 and two logical ranks, for a total of four GPUs. Prefill applies SP across the two logical ranks, while each decode request remains localized within one logical rank. Both phases retain the same base-TP layout within each rank. Shared linear operators. At the beginning of each Transformer block, the rank-local prefill and decode tokens are replicated across the GPUs within their base-TP group. The QKV projection follows standard column-parallel TP: each TP rank applies its local QKV weight shard and produces the corresponding subset of Q/K/V heads. Prefill and decode tokens can therefore remain co-batched through the projection because they share the same rank-local input and base-TP weight partition. Phase-specific attention. The two phases diverge after the QKV projection. Decode attention remains entirely within its logical rank: each TP rank computes attention for its local decode head shard using the KV cache resident in that logical rank. Prefill additionally applies Ulysses-SP across logical ranks. For each base-TP shard, the corresponding TP ranks from different logical ranks form an independent SP group. The first All-to-All transforms the prefill activations from sequence partitioning across logical ranks into finer head partitions, allowing each physical GPU to compute its assigned prefill attention heads for all current-chunk queries using the cached history and current-chunk K/V. A second All-to-All restores the sequence-partitioned layout while preserving the base-TP partition. Rejoining the shared path. After attention, the prefill and decode paths reconverge on each TP rank and are co-batched through the TP-sharded output projection. Because the output projection is row-parallel, each TP rank produces a fullshape partial output, which is summed across the base-TP group with an All-Reduce. The resulting hidden states are again replicated within the TP group. The FFN follows the standard TP pattern: its expansion projections are columnparallel, the contraction projection is row-parallel, and a second All-Reduce reconstructs the full hidden state.

𝑓GEMM (𝑛) = 𝛾𝑛 + 𝛾 0,

(7)

where 𝑛 is the number of rank-local tokens. The fitted coefficients absorb fixed execution overheads, including communication internal to the base TP configuration when applicable. Decode attention. For localized decode attention, the dominant workload is proportional to the aggregate context length of the resident requests. We model 𝑓𝐷 (𝑊 ) = 𝑎𝐷𝑊 + 𝑎𝐷,0,

(8)

Í

where 𝑊 = 𝑖 𝑐𝑖 is the aggregate decode-attention workload on a rank. Prefill attention. For a prefill chunk of length ℓ𝑖 with existing context 𝑐𝑖 , the causal-attention workload is proportional to   ℓ𝑖 + 1 ℓ𝑖 𝑐𝑖 + . (9) 2 We therefore model the prefill-attention latency as 𝑓𝑃 (𝑊 ) = 𝑎𝑃 𝑊 + 𝑎𝑃,0,

(10)

where 𝑊 denotes the per-rank prefill-attention workload after Ulysses-SP redistributes the scheduled prefill work across ranks. SP communication. The two Ulysses All-to-All collectives communicate the QKV states before attention and the output states afterward. Their latency is modeled from the largest initial prefill shard: 𝑓comm (𝑛) = 𝑎 comm𝑛 + 𝑎 comm,0,

(11)

where 𝑛 = max𝑟 𝑛𝑟𝑝 . Equivalently, the bandwidth-dominated component scales with 𝐵 QKV + 𝐵𝑂 , (12) BW where 𝐵 QKV and 𝐵𝑂 denote the per-token communication volumes and BW the effective interconnect bandwidth. 𝑛

18

Logical rank 0 (TP2) b4

a3

a3

b4

F F N

a3

a3

O

a4

a4

c2

c2

b4

… F F N

b4

O

a3

b4

… a3

a4 c2

Q K V

a4 c2

Co-batched

Head shard 1-0 (Prefill) Head shard 0 (Decode)

a4 c2

a4

Head shard 1-1 (Prefill) a4 c2

O

a4 c2

Head shard 1 (Decode)

Phase-specific attention paths

c2

F F N

a4 c2

a4

… F F N

All-reduce

c2

Q K V

All-reduce

Replicated

Decode B

Replicated

a4

c2

All-to-all

Prefill A

Head shard 0-1 (Prefill)

b4 a3

b4

Logical rank 1 (TP2) All-to-all

Decode C

a3 a4

Head shard 1 (Decode) All-to-all

b4 a3

All-to-all

a3

O

Head shard 0-0 (Prefill) Q K V

b4

b4

b4 a3

All-reduce

a3

Head shard 0 (Decode)

b4 a3

All-reduce

Q K V

b4

c2

a4

… c2

Co-batched

Figure 18. Conflux Parallelism composed with a base TP2 configuration across two logical ranks. Each logical rank consists of two TP ranks that share the same rank-local prefill and decode tokens. The phases are co-batched through TP-sharded linear operators, diverge during attention, and reconverge before the output projection: decode remains localized within its logical rank, while prefill applies Ulysses-SP across corresponding TP ranks of different logical ranks. Row-parallel output and FFN projections produce partial outputs that are reconstructed by the base-TP All-Reduce collectives. Algorithm 2 Waterfill Dispatch

ment quantum 𝑞 determined empirically. Given 𝑛 target , the additional prefill capacity available on rank 𝑟 is max{0, 𝑛 target − 𝑛𝑑𝑟 − 𝑛𝑟𝑝 (Ap )}, yielding an aggregate additional budget

Input: TBT SLO 𝜏, decode state D, selected prefills P, existing prefill assignment Ap , optional request 𝑖, alignment quantum 𝑞 Output: Prefill budget 𝐵 if 𝑖 = ∅; otherwise, prefill chunk ℓ𝑖 and its per-rank assignment A𝑖 1: ⊲ Find the largest feasible water level 2: 𝑛 target ← BinarySearch(𝜏, D, P, Ap, 𝑖, 𝑞) Í 3: 𝐵 ← 𝑟𝑁=1 max{0, 𝑛 target − 𝑛𝑟d − 𝑛𝑟p (Ap )} 4: if 𝑖 = ∅ then return 𝐵 5: ℓ𝑖 ← min{𝑚𝑖 , 𝐵} 6: A𝑖 ← WaterfillAssign(D, Ap, ℓ𝑖 , 𝑛 target ) 7: return ℓ𝑖 , A𝑖

𝐵=

max{0, 𝑛 target − 𝑛𝑑𝑟 − 𝑛𝑟𝑝 (Ap )}.

𝑟 =1

When request 𝑖 is provided, the dispatcher selects the largest chunk that fits within this budget, ℓ𝑖 = min{𝑚𝑖 , 𝐵},

Fitting. We obtain the base functions by profiling sharedlinear execution, prefill attention, decode attention, and SP communication independently over their relevant workload ranges. We then profile heterogeneous Conflux Parallelism batches to calibrate the contention factors 𝛼 and 𝛽 used in Eq. (4). This separates the standalone component costs from the slowdown introduced by concurrent execution.

E

𝑁 ∑︁

where 𝑚𝑖 is the request’s remaining prefill length. WaterfillAssign then distributes these ℓ𝑖 tokens across ranks toward 𝑛 target , starting from the existing assignment Ap , and returns the incremental per-rank assignment A𝑖 . The scheduler commits A𝑖 only after any required prefill-to-decode transition is validated. When 𝑖 = ∅, the dispatcher omits the candidate-specific prefill workload and returns 𝐵 as a reference residual budget under the current decode and prefill state. The SLO-aware scheduler uses this mode to obtain 𝐵 0 for Moore–Hodgson admission.

Waterfill Dispatch

Algorithm 2 details the waterfill dispatcher used by ConfluxServe. The dispatcher takes the current decode state D, the selected prefill chunks P, their existing per-rank assignment Ap , and an optional request 𝑖. It uses binary search with the latency estimator to find the largest feasible combinedload target 𝑛 target under the TBT SLO 𝜏. When request 𝑖 is provided, this target is request-specific because the latency contribution of the candidate chunk depends on its current context length. To account for kernel quantization effects, ConfluxServe restricts 𝑛 target to a hardware-dependent align-

F

Dual-Mode Decode Placement

Algorithm 3 details the dual-mode decode-placement policy used by ConfluxServe. When request 𝑖 completes prefill, ConfluxServe first identifies the set of ranks G that can accommodate its KV state while preserving a headroom of ℎ. e including The scheduler passes its tentative decode state D, placements reserved earlier in the same iteration. If no such 19

Algorithm 3 Dual-Mode Decode Placement

the total remaining prefill work 𝐶 of the currently admitted requests. Whenever adding request 𝑖 causes 𝐶 > 𝐾𝑖 , the Moore–Hodgson algorithm [30] removes the admitted request with the largest remaining prefill length using a max-heap. Since 𝐾𝑖 is nondecreasing in EDF order, this single removal restores feasibility before proceeding to the next request. The surviving requests form R adm and retain EDF priority in the scheduler. Removed requests form R rej and are subsequently considered in SJF order for opportunistic backfill.

Input: Request 𝑖 completing prefill, decode state D, KV-utilization threshold 𝜂, KV headroom ℎ Output: Selected rank 𝑔★ or Infeasible 1: G ← ranks feasible for 𝑖 under D with KV headroom ℎ 2: if G = ∅ then 3: return Infeasible 4: Glow ← ranks in G with KV utilization below 𝜂 5: if Glow ≠ ∅ then 6: return WorstFit(Glow ) 7: return BestFit(G)

Algorithm 4 Moore–Hodgson Admission Input: Pending prefills R, per-iteration token budget 𝐵 0 , time 𝑡, TBT SLO 𝜏 Output: Admitted requests R adm , rejected requests R rej 1: Sort R in EDF order 2: 𝐶 ← 0; R rej ← ∅ 3: H ← MaxHeap(key = 𝑚𝑖 ) 4: for each 𝑖 ∈ R 5: 𝐶 ← 𝐶 + 𝑚n𝑖 ; Hj .push(𝑖) k o 6: 𝐾 ← max 0, 𝑑𝑖𝜏−𝑡 𝐵 0 ⊲ Capacity before 𝑑𝑖 7: if 𝐶 > 𝐾 then 8: 𝑗 ← H .popMax(); 𝐶 ← 𝐶 − 𝑚 𝑗 9: R rej ← R rej ∪ { 𝑗 } 10: R adm ← R \ R rej 11: return R adm, R rej

rank exists, the transition is infeasible and the scheduler does not admit the completing prefill chunk. When at least one feasible rank has KV utilization below the threshold 𝜂, ConfluxServe uses WorstFit over these low-utilization ranks. WorstFit selects the rank with the smallest aggregate decode context, which also corresponds to the lowest KV utilization. This spreads decode workload across ranks, reducing decode-attention imbalance and preserving a high TBT-safe prefill-dispatch budget. Once all feasible ranks reach the threshold, ConfluxServe switches to BestFit, selecting the feasible rank with the highest KV utilization that can still accommodate the request. This packs KV state more tightly and preserves free capacity on other ranks for subsequent placements.

G

Moore–Hodgson Admission

Algorithm 4 details the TTFT-aware admission procedure used by ConfluxServe. Let 𝑚𝑖 denote the remaining prefill length of request 𝑖, and let 𝑑𝑖 denote its TTFT deadline. The dispatcher first provides a reference per-iteration prefill budget 𝐵 0 under the current decode state. Under this fixedcapacity surrogate, the cumulative prefill capacity available before deadline 𝑑𝑖 is     𝑑𝑖 − 𝑡 𝐾𝑖 = max 0, 𝐵0 , 𝜏 where 𝑡 is the current time and 𝜏 is the TBT SLO. Requests are processed in EDF order while maintaining 20

Related documents

• ConfluxServe: Reconciling Resource Utilization and Parallelism Efficiency across Prefill and Decode in LLM Serving#1488602
• Taking Tokenization off the Host for Agentic LLM Serving#1297677
• Taking Tokenization off the Host for Agentic LLM Serving#1291773
• Resource-Efficient Speculative Decoding for Long-Context LLM Serving#1117896
• HAServe: Horizon-Aware Scheduling and Memory Co-Design for Efficient LLM Serving#1355574