DualPace: Decoupled Scheduling and Cooperative GPU--CPU Decoding for LLM Serving

DualPace: Decoupled Scheduling and Cooperative GPU–CPU Decoding for LLM Serving Kingsum Chow Ningbo Global Innovation Center, Zhejiang University Ningbo, China kingsum.chow@zju.edu.cn

CCS Concepts: • Computer systems organization → Heterogeneous (hybrid) systems; Parallel architectures; • Computing methodologies → Machine learning. Keywords: Large language model inference, GPU–CPU heterogeneous computing, KV cache offloading

1

Introduction

Large language models (LLMs), including GPT-3 [4], GPT4 [32], PaLM [26], and the Llama family [22, 41], now power conversational assistants, code generation, retrieval augmented generation, and agentic applications [19, 31, 48]. Serving these models at scale is expensive: autoregressive inference consumes scarce GPU compute while maintaining

DualPace (ours) NEO vLLM

200K 150K 100K 50K 0K

P99 Inter-Token Latency (s)

arXiv:2610.33501v1 [cs.DC] 24 Oct 2026

Abstract Large language model (LLM) serving relies on batching to improve inference throughput, but the key-value (KV) cache grows with sequence length and limits GPU-resident concurrency. CPU offloading can expand KV capacity using host memory, yet existing systems remain largely GPU-gated: CPU-resident requests make progress only when the GPU scheduler exposes suitable overlap opportunities. As a result, host memory and CPU compute cannot act as an independent serving resource, leading to bursty CPU-side progress and inflated tail inter-token latency (ITL) under load. We present DualPace, an LLM serving runtime that turns the host into an independently scheduled decode engine. DualPace separates scheduling authority across two engines: the GPU engine executes prefills and GPU-resident decodes, while an AMX-enabled CPU engine independently batches and advances CPU-resident decodes. The engines are decoupled in scheduling but cooperative in execution: the CPU can execute a complete decode iteration locally or selectively invoke GPU Dense Assist for dense stages, while offline profiles guide hardware-specific batching and routing. Our evaluations show that, in decode-oriented scenarios, DualPace improves serving throughput by 6.9–43.2% over NEO, a state-of-the-art open-source CPU-offloading system, and by 11.5–90.6% over vLLM, a widely used GPU serving system, while maintaining the same level of latency. DualPace also preserves NEO’s P99 time-to-first-token (TTFT) advantage over vLLM while sustaining 7.3–50.5% higher request rates than NEO at the same level of P99 ITL.

CPU-resident Decoded Tokens

KeQiang Wu Ningbo Global Innovation Center, Zhejiang University Ningbo, China wkq@zju.edu.cn Total Tokens Generated

YuXuan Lin Ningbo Global Innovation Center, Zhejiang University Ningbo, China cilinmengye@zju.edu.cn

0

200

400

600

800

DualPace (ours) NEO

80K 60K 40K 20K 0K

0

Time (s)

200

400

600

800

Time (s)

18 15 12

DualPace (ours) NEO

9 6 3 0

2

2.5

3

3.5

4

4.5

Request Rate (req/s)

Figure 1. Llama-3.1-8B runs on an RTX 4090 with 12 physical Intel Xeon Silver 4510 cores, serving 2,000 AzureConversation requests [23, 40]. Top left: cumulative generated tokens; DualPace sustains the steepest slope and completes the trace first. Top right: cumulative CPU-resident decoded tokens; DualPace advances continuously, whereas NEO exhibits repeated plateaus. The inset magnifies a representative progress stall. Bottom: P99 client-observed intertoken latency (ITL) across offered loads, quantifying the taillatency consequence of intermittent CPU-resident progress.

a per-request key-value (KV) cache that grows throughout generation. A serving system must therefore sustain high request rates while controlling both time to first token (TTFT) and the inter-token latency (ITL) during generation. Continuous batching amortizes GPU execution across requests [18, 47], but its effective concurrency is bounded by GPU memory. The KV cache grows with both sequence length and batch size and can occupy most available HBM under long contexts or high load. PagedAttention reduces fragmentation via fine-grained KV allocation [18], but it does not remove the capacity limit of GPU-resident KV state. In contrast, serving nodes typically provide substantially larger host-memory capacity together with increasingly capable CPUs. In particular, Intel AMX provides high-throughput BF16 matrix operations, making the host a potential compute tier rather than merely a spill space for KV state [13, 17].

YuXuan Lin, KeQiang Wu, and Kingsum Chow

Prior systems exploit this opportunity through heterogeneous pipelines and KV-cache offloading, placing request KV state in host memory and executing KV-dependent attention on the CPU while retaining dense model stages on the GPU [8, 10, 15, 24, 38]. This organization expands effective KV capacity, but it does not turn the host into an autonomous serving tier. CPU work remains a stage of a GPU-controlled pipeline, and a CPU-resident request advances only when the GPU scheduler admits it into a suitable overlap window. Thus, abundant host memory and available CPU compute do not necessarily translate into sustained decode progress. We call this coupling GPU-gated CPU offloading. NEO [15], a state-of-the-art open-source CPU-offloading system, illustrates this limitation. NEO admits CPU attention only when its predicted execution time fits within GPU work from another sub-batch and when the resulting pipeline is expected to outperform GPU-only execution. These safeguards prevent CPU attention from extending the GPU critical path, but they also make CPU-resident progress conditional on GPU-side scheduling opportunities. In other words, the CPU can execute useful work, but it does not control when its resident requests are allowed to advance. Figure 1 (top right) exposes the resulting behavior under a trace-driven workload. DualPace advances CPU-resident requests at a sustained rate, whereas NEO’s CPU-resident token curve progresses in bursts separated by pronounced plateaus. Figure 1 (bottom) shows the client-visible consequence: as load increases, these stalls translate into high P99 ITL even though host memory still contains resident decode requests. The bottleneck is therefore not simply hostmemory capacity or CPU compute capability. The fundamental limitation is that CPU-resident requests lack independent scheduling authority. This observation motivates a different design principle: scheduling authority should be decoupled from execution placement. Existing CPU offloading systems largely decide where computation executes while leaving when CPU-resident requests progress under GPU control. DualPace decouples both dimensions. We present DualPace, an LLM serving runtime built around decoupled GPU–CPU cooperative decoding. DualPace assigns scheduling authority to two independently progressing engines. The GPU engine owns prefills and GPUresident decodes, while the CPU engine owns CPU-resident decodes, maintains its own queues and scheduling clock, and forms batches over host-resident KV state. CPU-resident requests can therefore continue making decode progress without waiting for the GPU scheduler to create an overlap opportunity. Large host-memory capacity determines how many requests can remain CPU-resident, while AMXenabled BF16 operators and host PagedAttention convert that residency into useful decode throughput. Crucially, scheduling independence does not imply execution isolation. A CPU-owned request remains under CPU

scheduling control even when selected computation is executed on the GPU. For each CPU decode batch, DualPace keeps dense stages on the CPU when host execution is efficient and selectively invokes GPU Dense Assist when GPU execution is preferable. Offline profiling derives hardwarespecific batching and routing thresholds that determine when CPU-local execution or GPU Dense Assist provides the better latency–throughput tradeoff. Our evaluations show that, in decode-oriented scenarios, DualPace improves serving throughput by 6.9–43.2% over NEO, a state-of-the-art open-source CPU-offloading system, and by 11.5–90.6% over vLLM, a widely used GPU serving system, while maintaining the same level of latency. DualPace also preserves NEO’s P99 TTFT advantage over vLLM while sustaining 7.3–50.5% higher request rates than NEO at the same level of P99 ITL. In summary, this paper makes the following contributions: • We identify GPU-gated CPU offloading as a source of intermittent CPU-resident decode progress. • We design and implement DualPace, a dual-engine serving runtime with decoupled GPU–CPU cooperative decoding, which separates scheduling authority from execution placement. • We evaluate DualPace across diverse serving scenarios and show that it sustains higher request rates and lower tail latency than GPU-gated CPU offloading, while also characterizing the workload regimes in which independent CPU decoding provides the greatest benefit.

2

Background and Motivation

2.1

LLM Inference and Execution Boundaries

Decoder-only LLMs predict tokens autoregressively with stacked transformer layers [22, 41, 43]. Each request first prefills its prompt, then decodes one token per request per iteration [1, 2, 16, 18]. Prefill turns many token rows into computebound GEMMs; decode processes one current-token row per request while scanning a growing KV cache and is typically memory-bandwidth-bound [17, 35, 45]. Serving engines increase throughput through iterationlevel batching [11, 18, 27, 30, 47, 48]. Chunked prefill further mixes bounded prefill work with ongoing decodes [1, 2, 16]. Attention remains phase-specific: prefill uses IO-aware kernels such as FlashAttention, whereas decode applies one query to paged historical KV states [5, 6, 46]. Figure 2 defines the execution boundaries. For batch size 𝐵 and hidden dimension 𝐻 , let 𝐿 denote the prompt or chunk length during prefill and the accumulated context length during decode. A prefill input has shape [𝐵, 𝐿, 𝐻 ], whereas a decode step processes [𝐵, 1, 𝐻 ] and attends to 𝐵 KV histories of length 𝐿. Thus, dense operators process 𝐵𝐿 rows during prefill but only 𝐵 rows during decode. We partition inference into a pre-layer, 𝑁 transformer layers—each split into pre-attention, attention, and post-attention stages—and

Input Token ID Batch

shape [BL]

Pre-layer (Embedding) shape [BL,H]

N x Transformer layers

For decode request: Decoding Attention

101 100

CPU Page Attention GPU Page Attention

10−1

L=64

RoPE Multi-Head Self Attention Post-Attention Stage

Layer Norm LM Head

QKV Projection

Pre-Attention Stage

Residual & Layer Norm FFN

Post-layer (Output Head)

Residual

FC1: Gate/Up Projection SiLU FC2: Down Projection

Sampling Output Token ID Batch

L=256 L=1024 B=1

L=64

CPU AVX512 BF16

O Projection

Latency (ms, log scale)

For prefill request: Prefiling Attention

Layer Norm

Latency (ms, log scale)

DualPace: Decoupled Scheduling and Cooperative GPU–CPU Decoding for LLM Serving

L=64

CPU AMX BF16

Packed QKV Linear

L=256 L=1024 B = 128

GPU CUDA

FFN FC1 (Packed Gate/Up) 102

101 100

101

10−1

100 200

400

600

Number of tokens

shape [B]

Figure 2. Conventional decoder-only LLM inference and the analytical execution stages used throughout this paper.

L=256 L=1024 B = 32

800

1000

200

400

600

800

1000

Number of tokens

Figure 3. Latency of representative Llama-3.1-8B operations on a 12-core Intel Xeon Silver 4510 CPU (Sapphire Rapids) and an RTX 4090 GPU [12, 28]. Top: PagedAttention latency. Bottom: packed-QKV and FFN gate/up projection latency.

a post-layer. These stages form the CPU–GPU placement and cooperation boundaries used by DualPace. 2.2

Host Memory as an Active Decode Tier

When HBM cannot accommodate further KV-cache growth, GPU-only systems may preempt requests by swapping KV state to host memory and restoring it when space becomes available [18, 36, 50]. As context length and concurrency grow, repeated swap-out and swap-in can make CPU–GPU interconnect bandwidth a serving bottleneck [3, 14, 20, 39]. CPU attention turns host memory from passive swap space into an active decode tier. For each transformer layer, fetching host-resident KV cache for GPU attention transfers Θ(𝐵𝐿𝐻 ) data. Computing attention on the CPU keeps historical KV local and exchanges only the current-token Q/K/V activations and attention output, Θ(𝐵𝐻 ), when dense stages remain on the GPU [8, 10, 15]. This near-data execution reduces CPU–GPU interconnect traffic and allows host-resident requests to continue decoding. Near-data CPU attention is practical because decode attention has low arithmetic intensity and is memory-bandwidthbound [10, 17, 21, 45]. As a result, the CPU–GPU performance gap is driven more by memory bandwidth than by peak FLOPS. Intel’s Sapphire Rapids reaches 15–35% of H100 throughput for memory-bound GEMV, versus 4–11% for compute-bound GEMM [17]. Figure 3 (top) shows the same trend: CPU PagedAttention is comparable at 𝐵 = 1 and remains within roughly one order of magnitude for the larger tested configurations. Thus, CPU attention adds decode capacity without repeatedly moving the full KV cache. 2.3

AMX Expands the CPU Compute Envelope

Attention-only offloading still leaves CPU-resident requests dependent on the GPU for dense operators. Intel AMX, available in recent Xeon processors, adds two-dimensional tile registers and a BF16 tile-multiply unit for matrix computation [12, 13, 17]. LIA measures 20 TFLOPS on Sapphire

Rapids and 40 TFLOPS on Granite Rapids, up to 4.5× and 9× AVX512 throughput, respectively [17]. Figure 3 (bottom) compares packed QKV and FFN gate/up projections across token counts from 1 to 1,024. AVX512 can trail the RTX 4090 by nearly two orders of magnitude, whereas AMX narrows the gap to roughly one order. In decode, each request contributes one token row, so this covers CPU batches up to 1024 requests, still larger batches directly lengthen token iterations. DualPace therefore uses packed oneDNN BF16 kernels to exploit AMX [42]. AMX expands the host from an attention-only tier into a practical full-decode compute tier at serving-relevant batch sizes.

3

Performance Bottlenecks of GPU-Gated CPU Offloading

This section examines why GPU-gated CPU offloading system cannot consistently translate host KV capacity into CPUresident decode progress. We use NEO as a case study, first reviewing its asymmetric pipeline and then identify the GPUside admission gates that motivate DualPace. 3.1

NEO’s GPU-Gated Asymmetric Pipeline

Early offloading systems enlarge batch size by placing model state and KV cache in host memory, but often trade latency for throughput through coarse-grained movement across GPU, CPU, and disk [38]. FastDecode avoids repeatedly moving KV cache by executing decode attention near host memory while retaining weight-intensive computation on the GPU, but requires substantial CPU resources and a balanced heterogeneous pipeline [10]. NEO advances this direction for dynamic online workloads through local-CPU partial offloading, asymmetric pipelining, and per-iteration load-aware scheduling [15]. We therefore use NEO as a representative state-of-the-art open-source CPU-offloading system.

YuXuan Lin, KeQiang Wu, and Kingsum Chow

GPU

CPU

Prelayer

Pre-Attention Stage 0

PreAttention Stage 1

Post-Attention Stage 0

Pre-Attention Stage 0

Post-Attention Stage 1

GPU Attention 0 CPU Attention 0

Communication

CPU Attention 1

PreAttention Stage 1

Post-Attention Stage 1

Post-Attention Stage 0 GPU Attention 0

Postlayer

CPU Attention 1

CPU Attention 0

Swap out KV Cache 0

Swap out KV Cache 0

x (N – 1) transformer layer

Figure 4. NEO’s asymmetric GPU–CPU pipeline, redrawn from [15]. Subscripts 0 and 1 identify sub-batches. LLM Engine

Figure 4 redraws NEO’s asymmetric pipeline [15] using our terminology. Subscripts 0 and 1 denote sub-batch identifiers, not layer indices. Sub-batch 0 contains prefills, GPUresident decodes, and selected CPU-resident decodes; subbatch 1 contains the remaining CPU-resident decodes. The GPU executes the pre- and post-attention stages for both, whereas the CPU executes only attention over host-resident KV cache. As a result, activations cross the CPU–GPU boundary at every transformer layer. NEO overlaps CPU attention from one sub-batch with pre post GPU work from the other. Let 𝑇 𝑗lin = 𝑇 𝑗 +𝑇 𝑗 denote GPU ga pre- and post-attention stage time for sub-batch 𝑗, 𝑇0 GPU attention time for sub-batch 0, and 𝑇 𝑗ca CPU attention time. The two overlap regions in Figure 4 require

CPU Engine

GPU Engine GPU Executor GPU Coordinator Scheduler Control Schedule

CPU Scheduler

CPU Executor Coordinator Schedule Control Worker 0 (driver worker)

Worker 0 (driver worker)

Model Shard 0

Model Shard 0

KV Cache 0

KV Cache 0

…

PCIE

Worker N-1

Shared Memroy

Host

… Worker M-1

Model Shard N-1

Model Shard M-1

KV Cache N-1

KV Cache M-1

Figure 5. DualPace system overview.

ga

(1) 𝑇0lin ≥ 𝑇1ca, 𝑇1lin + 𝑇0 ≥ 𝑇0ca . The first condition requires sub-batch 0’s linear work to hide CPU Attention 1; the second requires sub-batch 1’s linear work and GPU Attention 0 to hide CPU Attention 0. NEO tentatively adds each CPU decode request and removes it when either predicted margin is negative. Even when both conditions hold, it adopts the pipeline only if its predicted request rate exceeds GPU-only execution. CPU work is thus admitted only when a GPU iteration supplies a sufficiently long and throughput-beneficial overlap window. 3.2

Intermittent CPU-Resident Decode Progress

Figure 1 (top right) show NEO advances CPU-resident requests over time. We classify a token as CPU-resident when its request retains KV state in host memory and executes decode attention on the CPU. NEO makes progress in bursts, separated by repeated stalls; the inset highlights one representative multi-second pause. After roughly 600 seconds, the CPU-resident token curve plateaus, even though NEO’s total generated-token curve in Figure 1 (top left) continues to increase. This indicates that CPU-resident requests stop making progress while GPU-side work remains. The overlap constraints in Equation 1 explain both the intermittent stalls and the final plateau. A prefill request contributes many prompt-token rows to GPU linear work, whereas each decode request contributes only one currenttoken row. Consequently, with many decodes but few prefills, 𝑇 𝑗lin becomes short, while CPU attention 𝑇 𝑗ca must scan the

accumulated KV histories of CPU-resident requests. As prefill availability and KV lengths vary over time, the overlap conditions repeatedly switch between feasible and infeasible, producing bursty CPU-resident progress. Once prefills drain, decode-dominated iterations no longer provide sufficient overlap, leading to the sustained plateau. The key limitation is therefore not CPU capability itself, but GPU-gated progress: CPU-resident requests can advance only when the GPU scheduler exposes a sufficiently favorable overlap window. This dependency motivates DualPace’s design, in which CPU-resident requests make progress under an independent CPU scheduler rather than waiting for GPU-side admission opportunities.

4

DualPace Design

DualPace addresses the GPU-gated progress problem in §3 by making CPU-resident decoding a first-class serving path rather than an opportunistic GPU sub-batch. Figure 5 shows the DualPace system architecture. A shared LLMEngine maintains global request state and coordinates two independently scheduled execution engines: a GPU engine and a CPU engine. Each engine has its own scheduler and worker group, allowing CPU-resident requests to make progress independently of GPU-side admission opportunities. Within each engine, the scheduler manages KV-cache state, and forms execution batches for its workers. The engines are decoupled in scheduling but cooperative in execution. §4.1 first describes our scheduling approach,

DualPace: Decoupled Scheduling and Cooperative GPU–CPU Decoding for LLM Serving

§4.2 explains the cooperative execution mechanism, and §4.3 explains the key system parameters and offline profileguided configuration. 4.1

Residency-Aware Dual-Engine Scheduling

In our system, the GPU and CPU engines maintain separate request queues, scheduling clocks, and KV-cache managers. At any time, each request is owned by exactly one engine. The GPU engine owns all prefills and all GPU-resident decodes and manages the HBM KV cache of GPU-resident requests. The CPU engine owns CPU-resident decodes and manages their host-resident KV cache. Unlike heterogeneous pipelines, there is no global iteration barrier requiring the two batches to start or finish together. Consequently, a CPUresident request can continue decoding even when the current GPU batch does not provide a suitable overlap window. All requests enter the system through the GPU engine because DualPace executes every prefill on the GPU. For each new request, the GPU scheduler independently determines its post-prefill KV residency. A Prefill-to-GPU request computes its prompt on the GPU, retains the generated KV cache in HBM, and subsequently joins GPU decode. If HBM cannot retain that KV cache, Prefill-to-CPU reserves its final CPU KV blocks and uses temporary per-layer GPU staging. Once a transformer layer materializes its K/V states, their copy to the reserved CPU blocks is issued asynchronously and overlaps later GPU computation when possible. The GPU scheduler can also change the residency of an existing decode request. Under HBM pressure, it may select a GPU-resident request for swap-out, reserve the corresponding host blocks, and transfer its materialized KV cache to CPU memory. The request becomes visible to the CPU scheduler only after the transfer completes. Conversely, when HBM is sufficient, the GPU scheduler may select CPUresident requests for swap-in. The decoupled execution model creates a scheduling problem absent from a single GPU-centric scheduler. The GPU and CPU engines form batches independently, but request migration couples their HBM and DRAM allocations. Scheduling the two engines in isolation can overcommit host memory, whereas aggressive migration can repeatedly move a request between devices and waste PCIe bandwidth. To address these challenges, DualPace follows several principles: Migration Minimization. Requests remain with their current engine whenever possible. For request 𝑟 , we define its finish-peak KV footprint 𝑏b𝑟 as the maximum number of KV blocks it may require before completion. Swap-out is triggered only when HBM cannot accommodate the nexttoken KV growth of GPU-resident requests. Before swapping in request 𝑟 , the GPU scheduler adds its finish-peak footprint 𝑏b𝑟 to the HBM admission budget. The swap-in proceeds only if this footprint, together with those of existing GPU-owned requests, fits within HBM capacity. This admission keeps the

request GPU-resident through completion, avoiding shortlived migrations and residency oscillation. Global KV Safety. Host DRAM is a finite execution tier rather than an unbounded spill buffer. Without coordinated admission, the GPU scheduler could continuously direct prefills to CPU memory, improving short-term TTFT but eventually exhausting the space required for active CPU decodes to grow or for GPU-resident requests to be evicted. DualPace therefore tracks four CPU KV obligations, all measured in physical KV blocks: 𝐵𝐶used + 𝐵𝐶planned + 𝐵𝐶growth + 𝐵𝐶migration ≤ 𝐵𝐶capacity .

(2)

𝐵𝐶used is materialized CPU KV; 𝐵𝐶planned covers same-round allocations not yet materialized, including Prefill-to-CPU and immediate swap-out. 𝐵𝐶growth reserves the remaining KV growth of the CPU running-decode queue Q𝐶run . For each CPU-owned running request 𝑟 , let 𝑏𝑟 be its current blocks and 𝑏b𝑟 its finish-peak requirement: ∑︁ 𝐵𝐶growth = max{𝑏b𝑟 − 𝑏𝑟 , 0}. (3) 𝑟 ∈ Q𝐶run

𝐵𝐶migration reserves host space for GPU-owned requests that may later need to be evicted from HBM. Let Q𝐺 be the ordered GPU ownership queue after current decisions and 𝐵𝐺capacity the GPU KV capacity. Define ∑︁ 𝐵𝐺peak = 𝑏b𝑟 , (4) 𝑟 ∈ Q𝐺

as its aggregate finish-peak footprint. If 𝐵𝐺peak > 𝐵𝐺capacity , the scheduler scans Q𝐺 from the eviction-priority tail, moving each scanned request’s 𝑏b𝑟 from the GPU peak into 𝐵𝐶migration until the remainder fits in HBM. This term represents GPUowned KV without guaranteed long-term HBM residency. The four terms connect the two schedulers. A Prefill-toCPU candidate increases 𝐵𝐶planned , whereas a Prefill-to-GPU candidate extends Q𝐺 and may increase 𝐵𝐶migration . Both destinations are admitted only when Equation 2 remains satisfied. The CPU scheduler, in turn, treats 𝐵𝐶migration as unavailable capacity while admitting new CPU decode requests. Based on the analysis above, the complete per-engine scheduling procedures are presented in Appendix A. 4.2

Stage-Boundary Cooperation

A naive dual-engine design faces a fundamental tradeoff. Executing every stage of a CPU-owned decode iteration locally preserves independent progress, but the post-attention dense operators and the output head become increasingly costly as the CPU batch grows. Conversely, transferring the requests back to the GPU engine for dense computation would reintroduce dependence on GPU-side admission. DualPace avoids both: the CPU engine retains request ownership and

YuXuan Lin, KeQiang Wu, and Kingsum Chow Stage boundary: GPU Worker check ready DenseAssist Stage boundary: GPU Worker check ready DenseAssist After GPU Enginge Schedule And Start Batch Transformer Layer x (N -1)

GPU Worker

Prelayer

no

Pre-Attention Stage

yes

Dense Assist

Communication CPU Worker

GPU Attention

Post-Attention Stage

Pre-Attention Stage

yes

Dense Assist

no

CPU Attention

Transformer Layer x N After CPU Enginge Schedule And Start Batch

Pre-Attention Stage

GPU Attention

yes

Post-Attention Stage

no

Post-Attention Stage

yes

Dispatch Dense Assist to the GPU Worker through Shared Memory

Return the calculated result Prelayer

no

Stage boundary: GPU Worker check ready DenseAssist

no yes

Post-layer

Dense Assist

Return the calculated result

Post-layer

Post-attention routing strategy: Determines whether the post-attention stage and post-layer are executed locally on the CPU

Figure 6. DualPace’s decoupled GPU–CPU pipeline with stage-boundary cooperation. scheduling authority, while the GPU worker executes only selected dense stages at explicit model-stage boundaries. We realize this design through GPU Dense Assist. A Dense Assist is an operator-level request issued after the CPU worker has independently advanced a batch to an assistance boundary. The CPU worker sends only the tensors required at that boundary; the GPU worker executes the requested post-attention stage or model post-layer and returns the result to the same CPU-owned batch. These requests never enter the GPU scheduler’s queues, retain their KV cache in host memory, and acquire no long-lived GPU KV blocks. Thus, Dense Assist accelerates selected dense computation without turning GPU assistance into a change in request residency, or scheduling control. CPU-owned decoding. As illustrated by the lower lane in Figure 6, DualPace assigns the model-level pre-layer and, in every transformer layer, the pre-attention stage and attention to the CPU execution path. This local prefix is essential to scheduling decoupling: it allows the CPU engine to admit and launch a batch without waiting for the GPU scheduler. The subsequent post-attention stage—and, after the final transformer layer, the model post-layer—does not access the request’s KV cache. The CPU worker can therefore execute these stages either locally or through Dense Assist without changing request ownership or KV residency. Since their relative CPU and GPU costs vary with the CPU batch size, the CPU worker selects their placement using two batch-size thresholds: 𝑃 and 𝐹 , the largest CPU-local batch sizes for post-attention and the post-layer, respectively. For an active CPU decode batch of size 𝑏, we denote the two-threshold routing policy by 𝜋𝑃,𝐹 (𝑏):   Full-CPU, 𝑏 ≤ 𝐹,    𝜋𝑃,𝐹 (𝑏) = Split-Final, 𝐹 < 𝑏 ≤ 𝑃,    Fused-Assist, 𝑏 > 𝑃 . 

(5)

Legal settings are (𝑃, 𝐹 ) = (0, 0) or 0 ≤ 𝐹 < 𝑃; Full-CPU executes every post-attention stage and the post-layer on the CPU; Split-Final keeps all transformer layers on the CPU and uses one GPU assist for the post-layer; Fused-Assist

delegates each post-attention stage to the GPU, with the final assist also executing the post-layer. Non-preemptive stage-boundary priority. The upper lane of Figure 6 shows GPU-main execution. Rather than launching a complete forward pass, the GPU worker represents GPU-main execution as FIFO stage tasks. DualPace never preempts a running CUDA kernel or cancels submitted work. Instead, it exposes safe boundaries after the pre-layer, after each transformer layer, and before the post-layer. At each boundary, the GPU worker services a ready Dense Assist request before dequeuing the next main stage. This limits assist response to the completion of the current stage while preserving deterministic GPU execution. We quantify this mechanism’s impact on GPU-main execution in the GPUside cost analysis of Section 6.4 (Figure 12). Equation 5 also identifies when this stage-boundary coordination is unnecessary. Let 𝐶 be the largest CPU decode batch that the CPU engine can form. If 𝐶 ≤ 𝐹 (and hence 𝐶 ≤ 𝑃), every reachable batch follows Full-CPU, so Dense Assist is statically unreachable. DualPace then activates its whole-forward GPU-main path, eliminating layer-wise assist checks and allowing the GPU and CPU engines to execute independently. 4.3

Profile-Guided Hardware Configuration

CPU–GPU crossover points and throughput saturation across model, devices, and NUMA placements [10, 17]. DualPace therefore profiles each deployment offline and derives three controls: post-attention routing thresholds, a GPU batchedtoken limit, and a CPU decode batch limit. These controls adapt cooperation to the hardware without placing configuration search on the serving critical path. Post-attention routing. The best Dense Assist mode depends on both CPU batch size and hardware characteristics. Full-CPU avoids GPU interference but can lengthen CPUowned execution; Fused-Assist accelerates the CPU batch by invoking the GPU at every layer but consumes GPU-main service time; Split-Final provides an intermediate trade-off. The purpose of post-attention profiling is to quantify this trade-off and recommend the thresholds (𝑃, 𝐹 ) that select the best mode for each CPU decode batch regime.

DualPace: Decoupled Scheduling and Cooperative GPU–CPU Decoding for LLM Serving

Algorithm 1 Profile-guided post-attention routing Require: profiled batch sizes B and route scores 𝑆𝑟 (𝑏) Require: transformer-layer count 𝑁 and regret tolerance 𝜖 Ensure: post-attention thresholds (𝑃, 𝐹 ) 1: Θ ← {(0, 0)} ∪ {(𝑃, 𝐹 ) | 𝑃 ∈ B, 𝐹 ∈ B ∪ {0}, 𝐹 < 𝑃 } 2: Calls(Full-CPU) ← 0 3: Calls(Split-Final) ← 1 4: Calls(Fused-Assist) ← 𝑁 5: for all (𝑃, 𝐹 ) ∈ Θ do 6: ThresholdScore(𝑃, 𝐹 ) ← mean𝑏 ∈ B 𝑆 𝜋𝑃,𝐹 (𝑏 ) (𝑏) 7: AvgAssistCalls(𝑃, 𝐹 ) ← mean𝑏 ∈ B Calls(𝜋𝑃,𝐹 (𝑏)) 8: end for 9: BestScore ← min (𝑃,𝐹 ) ∈Θ ThresholdScore(𝑃, 𝐹 ) 10: Θ𝜖 ← {(𝑃, 𝐹 ) ∈ Θ : ThresholdScore(𝑃, 𝐹 ) ≤ (1 + 𝜖) BestScore} 11: return the candidate in Θ𝜖 with the fewest AvgAssistCalls, then the lowest score, 𝑃, and 𝐹

Routing score (ms)

750

Throughput (K tokens/s)

Full-CPU Fused-Assist Split-Final

1000

500 250

15 10 Context length 64 256 128 512

5 0

0

100

200

300

400

500

64 128

Number of tokens

512

300

300

Sequence length 32 256 64 512 128

200

100 14 8

16

32

Batch size

64

Throughput (K tokens/s)

Throughput (K tokens/s)

256

Batch size (b) CPU Single-Layer Decode Throughput

(a) CPU Post-Attention Routing Score

different data paths; 𝑡¯𝑙𝑆 and 𝑡¯𝑙𝐴 denote their respective waitpost-layer latencies. The CPU-visible route costs are CPU 𝑇full (𝑏) = 𝑁 𝑡𝑝 (𝑏) + 𝑡𝑙 (𝑏), CPU 𝑇split (𝑏) = 𝑁 𝑡𝑝 (𝑏) + 𝑡¯𝑙𝑆 (𝑏),

(7)

CPU 𝑇fused (𝑏) = (𝑁 − 1)𝑡¯𝑝 (𝑏) + 𝑡¯𝑙𝐴 (𝑏).

The wait terms include the required Assist Buffer movement and the CPU-visible wait for GPU completion. Let 𝑑𝑝 denote the GPU disturbance of an assisted post-attn stage, and let 𝑑𝑙𝑆 and 𝑑𝑙𝐴 denote the route-specific post-layer disturbance. Then GPU 𝐷 full (𝑏) = 0, GPU 𝐷 split (𝑏) = 𝑑𝑙𝑆 (𝑏),

(8)

GPU 𝐷 fused (𝑏) = (𝑁 − 1)𝑑𝑝 (𝑏) + 𝑑𝑙𝐴 (𝑏).

Each disturbance term includes Dense Assist CUDA execution and the GPU worker’s launch/finalization overhead. Equations 7 and 8 explicitly account for the amplification of per-layer assistance across an 𝑁 -layer model. Appendix B details how the profiler obtains these component functions. The profiler then fits the two-threshold policy in Equation 5. For every legal threshold pair, it evaluates 1 ∑︁ 𝑆 𝜋𝑃,𝐹 (𝑏 ) (𝑏), (9) 𝐽 (𝑃, 𝐹 ) = |B| 𝑏∈B

200 Mean Smoothed

100 26

28

210

212

214

Number of tokens

(c) GPU Single-Layer Prefill Throughput by Batch (d) GPU Single-Layer Prefill Throughput

Figure 7. Profile-guided configuration decision visualization chart for Llama-3.1-8B on a 12-core Intel Xeon Silver 4510 CPU and an NVIDIA RTX 4090 GPU platform.

For each profiled batch size 𝑏 and route 𝑟 , the profiler constructs 𝑆𝑟 (𝑏) = 𝑇𝑟CPU (𝑏) + 𝜆GPU 𝐷𝑟GPU (𝑏),

(6)

where 𝑟 is Full-CPU, Split-Final, or Fused-Assist. 𝑇𝑟CPU is the CPU-visible time to advance the CPU-owned batch, and 𝐷𝑟GPU is the GPU-main service time displaced by Dense Assist. We use 𝜆GPU = 1 in our evaluation. Importantly, the profiler does not simply select the route that minimizes CPU execution time; it jointly accounts for CPU-side acceleration and the GPU service capacity consumed to achieve it. This allows DualPace to accelerate CPU-resident decoding without unnecessarily degrading GPU-main throughput. Let 𝑁 be the number of transformer layers. To keep the cost model readable, 𝑡𝑝 and 𝑡𝑙 denote CPU-local post-attn and post-layer latency, while 𝑡¯𝑝 denotes wait-post-attn latency. Split-Final and Fused-Assist enter the post-layer through

over the profiled batch sizes B. Algorithm 1 enumerates every legal (𝑃, 𝐹 ) pair. Among candidates within a fractional tolerance 𝜖 of the minimum, the algorithm first minimizes the average number of Dense Assist calls, then the score, and finally the two thresholds. This near-optimal tie break preserves CPU progress while reducing stage-boundary coordination. Figure 7(a) illustrates the resulting decision for our RTX 4090 configuration. The exact score minimum is (𝑃, 𝐹 ) = (32, 0). With 𝜖 = 0.25%, the profiler selects (32, 16) at only 0.070% higher cost, assigning Full-CPU to batches up to 16, Split-Final to batches 17–32, and Fused-Assist to larger batches. Thus, a compact threshold pair reproduces the hardware specific route crossover without an online search. GPU batched-token limit. Figure 7(c) shows that different (𝐵, 𝐿) shapes rapidly converge a similar throughput plateau. This indicates that, once GPU throughput reaches this plateau, admitting additional optional prefill or swap-in tokens only lengthens the iteration without increasing useful throughput. Inspired by chunked prefill [1, 2], we profile one decoder layer across batch sizes 𝐵 and sequence lengths 𝐿, then group the measurements by total batched tokens 𝑛 = 𝐵𝐿 and average them to obtain Figure 7(d). We apply a nondecreasing least-squares fit to the mean throughput at each 𝑛 [7], estimate the plateau from the last 𝐾 fitted points, and select the earliest 𝑛 within 𝛿 of that plateau. Figure 7(d)

YuXuan Lin, KeQiang Wu, and Kingsum Chow

Prompt length (mean: 2348) Output length (mean: 22)

400

Count

Count

300

200

100

Prompt length (mean: 1599) Output length (mean: 114)

300 200 100

0 0 10

101

102

103

104

0 0 10

101

102

103

Number of tokens

Number of tokens

(a) Azure-Code

(b) Azure-Conv

104

Figure 8. Prompt and output length distributions of the Code and Conversation workloads in the Azure LLM Inference Trace [23, 40]: (a) Code and (b) Conversation. uses 𝐾 = 5 and 𝛿 = 1%, selecting a GPU batched-token limit of 1,024. CPU decode batch limit. Host KV capacity can make many requests resident, but CPU decode throughput stops scaling once AMX execution and memory bandwidth saturate. DualPace profiles one checkpoint-backed decoder layer across CPU batch sizes and KV-context lengths and selects the throughput-maximizing operating region. As shown in Figure 7 (b), CPU decode throughput peaks at a batch size of 256 requests. We therefore set 256 as the CPU decode batch limit for throughput-oriented runs. This profile is advisory rather than mandatory, latency-oriented deployments may choose a smaller batch.

5

Implementation

DualPace is implemented in Python and PyTorch [33]. Its FastAPI/Uvicorn serving interface [9] follows a DistServestyle asynchronous engine organization [49]. Asynchronous runtime. The LLMEngine owns global request state and launches three asyncio tasks: GPU engine, CPU engine, and completion loops. Each engine follows a strict local step—schedule one batch, submit it, await its result, and apply state before scheduling again—but the two loops share no iteration barrier. GPU and CPU workers are Ray actors, providing process isolation, lifecycle management, and resource placement [25]. Ray RPCs are used for setup and teardown, not per-stage serving. Shared-memory control. Fixed-layout shared-memory blocks carry GPU-main, CPU-decode, Dense Assist, and tensor-parallel control state. Separate shared pools hold CPU KV cache and assist activations. GPU workers attach and CUDA-register these host regions, enabling explicit H2D/ D2H copies without Ray object-store serialization. Dense Assist follows a CPU-Ready → GPU-Running → GPU-Done handshake, while Prefill-to-CPU KV copies use a dedicated CUDA stream and complete layer by layer. Stage-boundary cooperation. The GPU worker expands each GPU batch into FIFO tasks for the pre-layer, transformer layers, and post-layer. A CUDA event tracks the active task. The worker polls event completion and cooperatively

yields to other asynchronous work. Once the current task completes, it services a ready Dense Assist request before dequeuing the next GPU-main task. This implements nonpreemptive stage-boundary priority without device-wide synchronization or interruption of an active kernel. Efficient kernels and parallelism. GPU execution uses checkpoint-backed BF16 Llama kernels and FlashInfer ragged-prefill and HND paged-decode attention [46]; native CUDA extensions implement KV append and migration. Following vLLM’s packed CPU-linear organization, CPU dense operators use custom C++ oneDNN BF16 handlers for QKV, output, MLP, and LM-head projections [17, 42, 44]; CPU attention uses our AVX512-BF16/OpenMP PagedAttention kernel directly over the shared HND KV pool. Both paths support tensor parallelism through PyTorch Distributed: GPU ranks use NCCL, CPU ranks use Gloo, and CPU KV/Assist buffers are rank-sharded and NUMA-local [29, 33].

6

Evaluation

We evaluate DualPace by answering four questions: First, does leveraging host compute and memory increase the sustainable request rate under load? Second, does independent CPU scheduling preserve the TTFT benefit of CPU offloading while improving tail token-generation latency? Third, how do CPU tensor parallelism and host KV-cache capacity affect these two benefits? Finally, what overhead does GPU Dense Assist introduce? 6.1

Experimental Setup

Model and server configurations. We evaluate three model–server configurations. Llama-3.1-8B [22] runs on one NVIDIA RTX 4090 with 12 physical cores from an Intel Xeon Silver 4510 CPU. Qwen3-14B [37] runs on two NVIDIA H20 GPUs with 24 physical cores from an Intel Xeon Platinum 8469C (Sapphire Rapids) CPU, while Qwen3-32B runs on four H20 GPUs with 48 physical cores from the same CPU model. Due to resource constraints, each CPU allocation is confined to a single NUMA node. Workloads. We use the Conversation and Code workloads from the Azure LLM Inference Trace [23, 40], collected from Microsoft Azure’s production environment. For each workload, we select the first 2,000 requests. Figure 8 shows their prompt- and output-length distributions. Requests arrivals follow fixed-seed exponential inter-arrival gaps normalized to the target offered request rate. Baselines. We compare DualPace with vLLM [18], a widely used GPU serving engine, and NEO [15], a state-of-the-art open-source CPU-offloading system. We configure vLLM 0.22.1 without host KV-cache offloading and enable chunked prefill. Prefix caching is disabled because the length-preserving synthetic prompts reuse token IDs and would otherwise create artificial cross-request cache hits.

DualPace: Decoupled Scheduling and Cooperative GPU–CPU Decoding for LLM Serving

Normalized Latency (s/token)

DualPace (ours)

NEO

vLLM

2.0

2.0

2.0

1.5

1.5

1.5

1.0

1.0

1.0

0.5

0.5

0.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

0.5

0.0

2.0

2.5

Request Rate (req/s)

3.0

3.5

4.0

4.5

5.0

0.0

(a) Llama-3.1-8B, RTX 4090, 12 CPU cores, Azure-Conv

(b) Qwen3-14B, 2×H20, 24 CPU cores, Azure-Conv

2.0

1.5

1.5

1.5

1.0

1.0

1.0

0.5

0.5 2.5

3.0

3.5

2.50

4.0

4.5

0.0

2.75

3.00

3.25

3.50

3.75

4.00

(c) Qwen3-32B, 4×H20, 48 CPU cores, Azure-Conv

2.0

2.0

2.25

Request Rate (req/s)

2.0

0.0

2.00

Request Rate (req/s)

0.5 2.00

2.25

2.50

2.75

3.00

3.25

3.50

3.75

4.00

0.0

2.0

2.2

2.4

2.6

2.8

3.0

Request Rate (req/s)

Request Rate (req/s)

Request Rate (req/s)

(d) Llama-3.1-8B, RTX 4090, 12 CPU cores, Azure-Code

(e) Qwen3-14B, 2×H20, 24 CPU cores, Azure-Code

(f) Qwen3-32B, 4×H20, 48 CPU cores, Azure-Code

Figure 9. Normalized latency under increasing offered load on the Azure-Conversation and Azure-Code workloads. Upstream NEO is Llama-specific. For the Qwen3 experiments, we apply protocol-compatibility and dense-Qwen support patches that preserve NEO’s original offloading scheduler, asymmetric pipeline, and execution policy. Metrics and sustainable request rate. Following Orca and vLLM [18, 47], we use normalized latency, the mean of every request’s end-to-end latency divided by its output length, to characterize serving throughput under increasing offered load. A high-throughput system retains low normalized latency as the offered request rate increases. We additionally report P99 client-observed TTFT and ITL to characterize tail responsiveness during request admission and token generation, respectively. P99 TPOT results are reported separately in Appendix C. For each metric, we define the sustainable request rate at a fixed target as the intersection between the target and the line segment joining adjacent measured request-rate points. We use linear interpolation and never extrapolate beyond the measured sweep. The main figures use targets of 1 s/token normalized latency, 5 s P99 TTFT, and 1.5 s P99 ITL. 6.2

CPU-resident requests and advances them with host PagedAttention and AMX-accelerated dense operators. DualPace therefore benefits from both sides of host offloading: more KV residency creates larger effective batches, and independent host execution turns those batches into additional service capacity. Results on Azure-Code identify the workload boundary of decode-tier expansion. Because all three systems execute prefills on the GPU, Azure-Code’s long prompts and 22-token average outputs keep GPU prefill on the dominant path. CPUresident decoding contributes relatively little work, making it difficult to amortize movement of a long prompt’s KV state. The GPU-only baseline is therefore favored in the Qwen3 configurations, while DualPace remains ahead of NEO by converting a larger fraction of host compute into useful progress. These results show that DualPace is most effective when GPU KV pressure coincides with sustained decode work, allowing host memory and CPU compute to contribute meaningful additional service capacity. Based on this workload characterization and space constraints, the remaining main-text experiments focus on Azure-Conversation; Appendix C reports the corresponding Azure-Code tail-latency results.

Sustainable Request Rate under Load

Figure 9 plots normalized latency as the offered request rate increases. Each system first exhibits gradual latency growth and then a sharp rise once arrivals exceed its service capacity and requests accumulate. On Azure-Conversation, DualPace shifts this saturation knee to the right in all three configurations. At the 1 s/token normalized-latency target, it sustains 43.2%, 6.9%, and 18.1% higher request rates than NEO and 90.6%, 11.5%, and 23.8% higher rates than vLLM, for Llama-3.1-8B, Qwen3-14B, and Qwen3-32B, respectively. These gains come from converting host resources into an additional decode tier. Host memory admits KV states that would otherwise consume HBM or wait for GPU KV blocks, allowing the GPU to continue admitting prefills and GPUresident decodes. Independently, the CPU engine batches

6.3

Tail TTFT and ITL

Figure 10 (top) shows that CPU offloading moves the P99 TTFT saturation point to higher loads. Both DualPace and NEO use Prefill-to-CPU: when the GPU main KV pool cannot retain a prompt’s full KV state, the system executes the prefill on the GPU, allocates persistent KV blocks in host memory, and asynchronously transfers the generated K/V states to the CPU layer by layer. Avoiding long-lived GPU KV residency allows the GPU to admit subsequent prefills under KV pressure, reducing the queueing component of TTFT. Across the three configurations, DualPace remains

P99 Inter-Token Latency (s)

P99 Time to First Token (s)

YuXuan Lin, KeQiang Wu, and Kingsum Chow

DualPace (ours)

NEO

vLLM

10

10

10

8

8

8

6

6

6

4

4

4

2

2

2

0

0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

2.0

2.5

Request Rate (req/s)

3.0

3.5

4.0

4.5

5.0

0

3.0

3.0

2.5

2.5

2.5

2.0

2.0

2.0

1.5

1.5

1.5

1.0

1.0

1.0

0.5

0.5

0.0

0.0

2.0

2.5

3.0

3.5

4.0

2.25

4.5

5.0

2.50

2.75

3.00

3.25

3.50

3.75

4.00

3.75

4.00

Request Rate (req/s)

3.0

1.5

2.00

Request Rate (req/s)

0.5 2.0

2.5

3.0

3.5

4.0

4.5

5.0

0.0

2.00

2.25

2.50

2.75

3.00

3.25

3.50

Request Rate (req/s)

Request Rate (req/s)

Request Rate (req/s)

(a) Llama-3.1-8B, RTX 4090, 12 CPU cores

(b) Qwen3-14B, 2×H20, 24 CPU cores

(c) Qwen3-32B, 4×H20, 48 CPU cores

P99 Inter-Token Latency P99 Time to First Token Normalized Latency (s) (s) (s/token)

Figure 10. Tail latency under increasing offered load on the Azure-Conversation workload.

CPU TP1

30 GiB 60 GiB

CPU TP2

120 GiB 240 GiB

vLLM achieves lower P99 ITL because its decode requests remain GPU-resident, where per-token execution is faster than on the CPU. DualPace and NEO use CPU computation to serve host-resident decode requests, so their P99 ITL also includes the intrinsic CPU–GPU execution-speed gap. The key comparison among CPU-offloading systems is therefore whether host-resident requests make continuous progress. DualPace substantially reduces the scheduling-induced tail latency observed in NEO while preserving the TTFT benefit of Prefill-to-CPU. Appendix C reports P99 TPOT and the corresponding P99 TTFT and ITL results on Azure-Code.

2.0 1.5 1.0 0.5 0.0 10 8 6 4 2 0 2.0 1.5 1.0

6.4

0.5 0.0

2.0

2.5

3.0

3.5

4.0

4.5

5.0

2.0

2.5

3.0

3.5

4.0

4.5

Request Rate (req/s)

Request Rate (req/s)

(a) Impact of CPU tensor parallelism

(b) Impact of Host kv cache capacity

Figure 11. Sensitivity to CPU tensor parallelism and KV capacity.

close to NEO at the 5 s P99 TTFT target, showing that independent CPU scheduling preserves the first-token benefit of Prefill-to-CPU. Although DualPace and NEO provide comparable P99 TTFT, their tail behavior diverges during token generation. At the 1.5 s P99 ITL target in Figure 10 (bottom), DualPace sustains 50.5%, 15.2%, and 7.3% higher request rates than NEO from left to right. NEO advances CPU-resident requests only when its GPU pipeline exposes a suitable overlap window; requests can therefore accumulate in host memory without receiving regular decode service. DualPace instead gives CPU-resident requests an independent CPU scheduling clock and execution budget. This sustained CPU-side progress substantially removes the scheduler-induced component of tail ITL, consistent with Figure 1.

Sensitivity Studies

CPU tensor parallelism. Figure 11 (left) compares CPU TP = 1 and CPU TP = 2 for Llama-3.1-8B on one RTX 4090 under Azure-Conversation. Each CPU TP rank uses 12 physical cores and is pinned to one NUMA node, while both configurations use the same aggregate host KV-cache capacity. CPU TP = 2 raises the sustainable request rate by 15.0%, 14.3%, and 29.7% under the normalized-latency, P99 TTFT, and P99 ITL targets, respectively. Adding an independently bound rank exposes more parallel AMX work and distributes host PagedAttention across NUMA memory domains. The resulting compute parallelism and aggregate memory bandwidth accelerate both dense operators and KV reads, improving throughput and both tail metrics. Host KV-cache capacity. Figure 11 (right) varies host KV capacity from 30 to 240 GiB. At the normalized-latency target, 120 GiB sustains 28.7% and 18.5% higher request rates than 30 and 60 GiB, respectively. At the P99 TTFT target, the corresponding improvements are 28.3% and 14.6%. More host KV capacity admits additional Prefill-to-CPU and migrated requests. This both releases GPU KV blocks sooner and supplies the CPU engine with enough resident work to form efficient batches.

DualPace: Decoupled Scheduling and Cooperative GPU–CPU Decoding for LLM Serving

GPU-side cost of Dense Assist. We quantify the effect of stage-boundary cooperation on GPU-main execution. When no Dense Assist is ready at a layer boundary, polling the assist queue adds only 2–5 𝜇s per layer relative to the wholeforward GPU path, or 𝑂 (0.1 ms) for a 32–64-layer batch. Figure 12 profiles the GPU time attributable to Dense Assist under Azure-Conversation for Qwen3-32B on four H20 GPUs and Llama-3.1-8B on one RTX 4090. CUDA execution accounts for 60.4% and 70.0% of the measured component time, respectively. Within CUDA execution, the postattention stage—output projection and FFN—accounts for 79.8% and 79.2%, while H2D transfer accounts for 15.3% and 12.6%. Thus, most device time is spent on the dense computation that Dense Assist is intended to accelerate. Host launch and finalization accounts for the remaining 39.6% and 30.0%. This component denotes host orchestration performed by the GPU worker along its serialized service path. Launch includes Dense Assist admission, payload preparation, and CUDA-work submission; finalization covers host-side result handling after CUDA completion, with launch accounting for most of this component. Our experiments set the CPU decode batch limit to 256. Because each active request contributes one token per decode iteration, an assisted post-attention stage processes at most 256 decode tokens and its CUDA segment completes in only 0.6–0.8 ms; the fixed host-submission cost therefore becomes visible at this granularity. CUDA submission is asynchronous, so host launch and CUDA execution can partially overlap. The stacked bars in Figure 12 (left) therefore represent component work rather than strictly serialized end-to-end latency. Moreover, when the host provides sufficient compute capacity, DualPace can keep more dense stages CPU-local, reducing the frequency and aggregate impact of this fixed overhead.

CUDA execution Host launch/finalize

H2D Post-Attention Stage

80

100 39.6%

30.0%

60 40 60.4%

70.0%

20 0

GPU Dense Assist CUDA-time breakdown (%)

100

GPU Dense Assist occupied-time breakdown (%)

The normalized-latency and P99 TTFT results plateau between 120 and 240 GiB. At 120 GiB, the host KV pool is already large enough to keep the CPU worker processing its largest effective decode batch while accommodating most KV states offloaded by the GPU. Additional capacity allows more requests to reside in host memory, but it does not make the CPU worker advance each decode batch faster. CPU computation and memory bandwidth, rather than host KV capacity, therefore become the primary bottlenecks. P99 ITL exposes the complementary placement trade-off. The 30 and 60 GiB configurations keep a larger fraction of decodes on the GPU and therefore attain lower P99 ITL, whereas 120 and 240 GiB prioritize host admission capacity and are nearly identical. Host KV capacity is therefore an admission-control knob rather than a monotonic ITL knob: increasing it improves sustainable throughput and TTFT until the CPU execution tier saturates, while smaller capacities favor GPU-resident per-token latency. P99 TPOT sensitivity results appear in Appendix C.

Qwen3-32B Llama-3.1-8B 4×H20, 48 CPU cores RTX 4090, 12 CPU cores

D2H Sampling 5.9%

80 60

79.8%

79.2%

40 20 15.3%

12.6%

0

Qwen3-32B Llama-3.1-8B 4×H20, 48 CPU cores RTX 4090, 12 CPU cores

Figure 12. GPU Dense Assist component-time breakdown. Left: normalized host-orchestration and CUDA component times. Right: CUDA-time decomposition.

The post-attention routing thresholds (𝑃, 𝐹 ), defined in Equation 5 and selected by the offline profiler in Section 4.3, provide a direct criterion for deciding when to avoid Dense Assist: a large recommended 𝑃 means that, after accounting for GPU launch/finalization and CUDA disturbance, CPUlocal post-attention remains preferable up to a large CPU batch. In this scenario, if the desired maximum CPU decode batch size is similar to the 𝑃 value, we can configure the system to use the whole-forward GPU-main path. The GPU then submits each GPU-main batch as a whole, bypassing layer-wise assist checks and performing no Dense Assist for CPU-owned batches. The two engines consequently advance their respective batches independently.

7

Related Work

GPU-centric LLM serving. Orca introduces iterationlevel scheduling, whereas vLLM and vAttention improve KV-cache management [18, 36, 47]. Sarathi-Serve and PODAttention coordinate prefill and decode within a GPU, while Splitwise and DistServe separate the two phases across GPU replicas [1, 16, 34, 49]. These systems improve batching, memory management, or kernel overlap for GPU-resident work. DualPace extends the serving tier beyond the GPU by combining host-resident KV capacity with an independently scheduled CPU decode engine. Heterogeneous LLM inference. FlexGen exploits GPU, CPU, and storage capacity for throughput-oriented inference [38]. FastDecode, NEO, APEX, and OmniServe execute attention near host-resident KV state and coordinate it with GPU dense computation [8, 10, 15, 24]. LIA further shows that AMX-capable CPUs can execute dense LLM operators [17]. DualPace differs in where scheduling authority resides: CPU-resident requests remain owned and scheduled by an independent CPU engine, while profile-guided GPU Dense Assist selectively accelerates dense stages without transferring request ownership back to the GPU scheduler.

YuXuan Lin, KeQiang Wu, and Kingsum Chow

8

Conclusion

DualPace turns host memory and AMX compute into an independently scheduled LLM decode tier, with profile-guided GPU Dense Assist for selective cross-engine cooperation. In decode-oriented scenarios, DualPace improves serving throughput by 6.9–43.2% over NEO and by 11.5–90.6% over vLLM, while maintaining the same level of latency. DualPace also preserves NEO’s P99 TTFT advantage over vLLM while sustaining 7.3–50.5% higher request rates than NEO at the same level of P99 ITL. These results show that decoupling scheduling ownership from execution placement allows CPU-resident requests to make sustained decode progress, turning host KV capacity and CPU compute into effective serving capacity.

Acknowledgments Codex was used for selected system-development and language editing. The authors directed and verified all paper content, code, and experimental results.

References [1] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 117–134. https://www.usenix.org/conference/osdi24/ presentation/agrawal [2] Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, and Ramachandran Ramjee. 2023. SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills. arXiv:2308.16369 [cs.LG] https://arxiv.org/abs/2308.16369 [3] Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, and Yuxiong He. 2022. DeepSpeed-Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, Dallas, TX, USA, 1–15. doi:10.1109/SC41404.2022.00051 [4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are FewShot Learners. In Advances in Neural Information Processing Systems, Vol. 33. Curran Associates, Inc., Vancouver, Canada, 1877– 1901. https://proceedings.neurips.cc/paper_files/paper/2020/file/ 1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf [5] Tri Dao. 2024. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. In International Conference on Learning Representations. https://proceedings.iclr.cc/paper_files/paper/2024/ file/98ed250b203d1ac6b24bbcf263e3d4a7-Paper-Conference.pdf [6] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh

(Eds.), Vol. 35. Curran Associates, Inc., New Orleans, LA, USA, 16344– 16359. doi:10.52202/068431-1189 [7] Jan de Leeuw, Kurt Hornik, and Patrick Mair. 2009. Isotone Optimization in R: Pool-Adjacent-Violators Algorithm (PAVA) and Active Set Methods. Journal of Statistical Software 32, 5 (2009), 1–24. doi:10.18637/jss.v032.i05 [8] Jiakun Fan, Yanglin Zhang, Xiangchen Li, and Dimitrios S. Nikolopoulos. 2025. APEX: Asynchronous Parallel CPU-GPU Execution for Online LLM Inference on Constrained GPUs. arXiv:2506.03296 [cs.DC] https://arxiv.org/abs/2506.03296 [9] FastAPI Contributors. 2026. FastAPI. Retrieved August 19, 2026 from https://github.com/fastapi/fastapi [10] Jiaao He and Jidong Zhai. 2024. FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines. arXiv:2403.11421 [cs.DC] https://arxiv.org/abs/2403.11421 [11] Hugging Face. 2022. Large Language Model Text Generation Inference. Retrieved August 19, 2026 from https://github.com/huggingface/textgeneration-inference [12] Intel Corporation. 2023. 5th Gen Intel Xeon Processors. Retrieved August 19, 2026 from https://www.intel.com/content/www/ us/en/products/details/processors/xeon/5th-gen-xeon-scalableprocessors.html [13] Intel Corporation. 2026. Intel 64 and IA-32 Architectures Software Developer Manuals. Retrieved August 19, 2026 from https://www.intel.com/ content/www/us/en/developer/articles/technical/intel-sdm.html [14] Chaoyi Jiang, Lei Gao, Hossein Entezari Zarch, and Murali Annavaram. 2025. KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputation. In Findings of the Association for Computational Linguistics: ACL 2025. Association for Computational Linguistics, Vienna, Austria, 19474–19488. doi:10.18653/v1/2025.findings-acl.997 [15] Xuanlin Jiang, Yang Zhou, Shiyi Cao, Ion Stoica, and Minlan Yu. 2025. NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference. In Proceedings of Machine Learning and Systems, M. Zaharia, G. Joshi, and Y. Lin (Eds.), Vol. 7. MLSys, Santa Clara, CA, USA. https://proceedings.mlsys.org/paper_files/paper/2025/file/ 66a026c0d17040889b50f0dfa650e5e0-Paper-Conference.pdf [16] Aditya K. Kamath, Ramya Prabhu, Jayashree Mohan, Simon Peter, Ramachandran Ramjee, and Ashish Panwar. 2025. POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. ACM, Rotterdam, Netherlands, 897–912. doi:10.1145/3676641.3715996 [17] Hyungyo Kim, Nachuan Wang, Qirong Xia, Jinghan Huang, Amir Yazdanbakhsh, and Nam Sung Kim. 2025. LIA: A Single-GPU LLM Inference Acceleration with Cooperative AMX-Enabled CPU-GPU Computation and CXL Offloading. In Proceedings of the 52nd Annual International Symposium on Computer Architecture. ACM, Tokyo, Japan, 544–558. doi:10.1145/3695053.3731092 [18] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles. ACM, Koblenz, Germany, 611–626. doi:10.1145/3600006.3613165 [19] LangChain. 2023. LangChain. Retrieved August 19, 2026 from https: //github.com/langchain-ai/langchain [20] Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. 2024. InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 155–172. https://www.usenix.org/conference/ osdi24/presentation/lee

DualPace: Decoupled Scheduling and Cooperative GPU–CPU Decoding for LLM Serving

[21] Bin Lin, Chen Zhang, Tao Peng, Hanyu Zhao, Wencong Xiao, Minmin Sun, Anmin Liu, Zhipeng Zhang, Lanbo Li, Xiafei Qiu, Shen Li, Zhigang Ji, Tao Xie, Yong Li, and Wei Lin. 2024. Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache. arXiv:2401.02669 [cs.DC] https://arxiv.org/abs/2401.02669 [22] Meta AI. 2024. Introducing Llama 3.1: Our Most Capable Models to Date. Retrieved August 24, 2026 from https://ai.meta.com/blog/metallama-3-1/ [23] Microsoft Azure. 2024. Azure LLM Inference Trace 2024. Collected May 10–19, 2024. Retrieved August 19, 2026 from https://github.com/Azure/AzurePublicDataset/blob/master/ AzureLLMInferenceDataset2024.md [24] Zizhao Mo, Junlin Chen, Huanle Xu, and ChengZhong Xu. 2026. Serving Hybrid LLM Loads with SLO Guarantees Using CPU-GPU Attention Piggybacking. Proceedings of the ACM on Management of Data 4, 3 (2026), 1–26. doi:10.1145/3802107 [25] Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, and Ion Stoica. 2018. Ray: A Distributed Framework for Emerging AI Applications. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). USENIX Association, Carlsbad, CA, 561–577. https://www.usenix.org/conference/osdi18/ presentation/moritz [26] Sharan Narang and Aakanksha Chowdhery. 2022. Pathways Language Model (PaLM): Scaling to 540 Billion Parameters for Breakthrough Performance. Retrieved August 24, 2026 from https://research.google/blog/pathways-language-model-palmscaling-to-540-billion-parameters-for-breakthrough-performance/ [27] NVIDIA. 2019. FasterTransformer. Retrieved August 19, 2026 from https://github.com/NVIDIA/FasterTransformer [28] NVIDIA. 2022. GeForce RTX 4090 Graphics Cards for Gaming. Retrieved August 19, 2026 from https://www.nvidia.com/en-us/geforce/ graphics-cards/40-series/rtx-4090/ [29] NVIDIA. 2023. NVIDIA Collective Communications Library (NCCL). Retrieved August 19, 2026 from https://developer.nvidia.com/nccl [30] NVIDIA. 2024. TensorRT-LLM. Retrieved August 19, 2026 from https://github.com/NVIDIA/TensorRT-LLM [31] OpenAI. 2022. Introducing ChatGPT. Retrieved August 19, 2026 from https://openai.com/index/chatgpt/ [32] OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] https://arxiv.org/abs/2303.08774 [33] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems, Vol. 32. Curran Associates, Inc., Vancouver, Canada, 8024– 8035. https://proceedings.neurips.cc/paper_files/paper/2019/file/ bdbca288fee7f92f2bfa9f7012727740-Paper.pdf [34] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, Buenos Aires, Argentina, 118–132. doi:10.1109/ISCA59077.2024. 00019 [35] Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2022. Efficiently Scaling Transformer Inference. arXiv:2211.05102 [cs.LG] https://arxiv.org/abs/2211.05102 [36] Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ramjee, and Ashish Panwar. 2025. vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention. In Proceedings of the 30th

ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1. ACM, Rotterdam, Netherlands, 1133–1150. doi:10.1145/3669940.3707256 [37] Qwen Team. 2025. Qwen3: Think Deeper, Act Faster. Retrieved August 24, 2026 from https://qwenlm.github.io/blog/qwen3/ [38] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU. In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202), Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (Eds.). PMLR, 31094–31116. https://proceedings.mlr.press/ v202/sheng23a.html [39] Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. 2024. PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles. ACM, Austin, TX, USA, 590–606. doi:10.1145/3694715. 3695964 [40] Jovan Stojkovic, Chaojie Zhang, Íñigo Goiri, Josep Torrellas, and Esha Choukse. 2025. DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency. In 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, Las Vegas, NV, USA, 1348–1362. doi:10.1109/HPCA61900.2025.00102 [41] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, MarieAnne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971 [cs.CL] https://arxiv. org/abs/2302.13971 [42] UXL Foundation. 2024. oneAPI Deep Neural Network Library (oneDNN). Retrieved August 19, 2026 from https://github.com/ uxlfoundation/oneDNN [43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems, Vol. 30. Curran Associates, Inc., Long Beach, CA, USA, 5998–6008. https://proceedings.neurips.cc/paper/2017/hash/ 3f5ee243547dee91fbd053c1c4a845aa-Abstract.html [44] vLLM Project. 2026. vLLM. Retrieved August 19, 2026 from https: //github.com/vllm-project/vllm [45] Samuel Williams, Andrew Waterman, and David Patterson. 2009. Roofline: An Insightful Visual Performance Model for Multicore Architectures. Commun. ACM 52, 4 (2009), 65–76. doi:10.1145/1498765. 1498785 [46] Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, and Luis Ceze. 2025. FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving. In Proceedings of Machine Learning and Systems, M. Zaharia, G. Joshi, and Y. Lin (Eds.), Vol. 7. MLSys, Santa Clara, CA, USA. https://proceedings.mlsys.org/paper_files/paper/2025/file/ dbf02b21d77409a2db30e56866a8ab3a-Paper-Conference.pdf [47] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, Carlsbad, CA, 521–538. https://www.usenix.org/conference/ osdi22/presentation/yu [48] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Structured Language Model Programs. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey,

YuXuan Lin, KeQiang Wu, and Kingsum Chow

D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37. Curran Associates, Inc., Vancouver, Canada, 62557–62583. doi:10.52202/079017-2000 [49] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 193– 210. https://www.usenix.org/conference/osdi24/presentation/zhongyinmin [50] Kan Zhu, Yufei Gao, Yilong Zhao, Liangyu Zhao, Gefei Zuo, Yile Gu, Dedong Xie, Zihao Ye, Keisuke Kamahori, Chien-Yu Lin, Ziren Wang, Stephanie Wang, Arvind Krishnamurthy, and Baris Kasikci. 2025. NanoFlow: Towards Optimal Large Language Model Serving Throughput. In 19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25). USENIX Association, Boston, MA, 749– 765. https://www.usenix.org/conference/osdi25/presentation/zhukan

A

Per Engine Scheduling Procedures

DualPace realizes scheduling independence with two local event loops rather than one synchronized heterogeneous batch. In each round, an engine constructs a logical schedule, commits its KV-block and ownership changes, executes at most one worker batch, and applies the result before beginning its next round. GPU and CPU rounds need not start or finish together. The engines share only request ownership, transfer readiness, future migration intent, and the host KV obligations in Equation 2. A.1

GPU Engine Scheduling Round

Each GPU engine round proceeds as follows. 1. Establish a transfer-safe scheduling frontier. The GPU engine first retires completed Prefill-to-CPU transfers. A transferred request becomes eligible for CPU decoding only after all layer-wise KV copies have completed. Before constructing a new GPU schedule, the engine also waits for the preceding Prefill-to-CPU copy group to drain, so temporary staging blocks are never reused while a copy is in flight. 2. Restore next-token HBM feasibility. Existing GPUresident decodes have priority because their KV state already occupies HBM. The scheduler first accounts for the blocks required for every resident request to append its next token. If this footprint exceeds HBM capacity, it searches the GPU decode queue from the eviction-priority tail for requests whose current KV state can be accepted by the host. Host pressure does not force the entire GPU engine to stop. A tail request that cannot fit in host memory may be skipped in favor of an earlier feasible victim. If no eviction is feasible, the round still executes the subset of GPU decodes whose next-token growth fits the currently available HBM. If the selected evictions free only part of the required capacity, the engine

commits a migration-only round and re-evaluates the remaining pressure after those transfers complete. 3. Reserve host space for future GPU evictions. For the remaining GPU ownership queue, the scheduler computes the finish-peak KV footprint of each request. When their aggregate peak exceeds HBM capacity, a reverse queue scan identifies the suffix that may later require eviction. Its host footprint is published as 𝐵𝐶migration . The CPU scheduler treats this reserve as unavailable, preventing CPU admission from consuming the space required to preserve future GPU progress. 4. Admit GPU decodes and relieve a saturated CPU tier. All remaining GPU-resident decodes enter the candidate batch first. If the CPU active batch is full, its waiting queue is nonempty, and the current GPU round performs no swap-out, the GPU then admits a ready FIFO prefix of CPU-waiting decodes before considering new prefills. This early swap-in gives an already backlogged CPU tier access to residual HBM before prompt admission consumes it. Each admitted request must fit both the next-token HBM budget and its finish-peak footprint, so it can remain GPUresident through completion. 5. Admit prefills and choose their decode residency. Waiting prompts are considered in FIFO order until the GPU row limit, profiled batched-token limit, hard token limit, or temporary staging capacity is reached. A prompt follows Prefill-to-GPU when its persistent KV state fits in HBM; adding it to the future GPU population also updates 𝐵𝐶migration . Otherwise, it follows Prefill-to-CPU: the request receives persistent host KV blocks and temporary single-layer GPU staging blocks for layer-wise transfer. Once the FIFO scan selects its first Prefill-to-CPU request, later prompts admitted in the same round use the same destination. This preserves queue order instead of allowing a smaller later prompt to bypass the first host-bound request. A Prefill-to-CPU candidate is admitted only while the host decode-admission limit and the global KV envelope remain satisfied, including materialized use, planned destinations, finish growth of CPU-running requests, and future migration reserve. 6. Use residual HBM for additional swap-in. After prefill admission, a round with no swap-out again considers a ready FIFO prefix of the CPU waiting queue. This second pass uses HBM left by the final prefill decision. If the CPU waiting queue was empty before the early-swap-in pass, the scheduler may instead mark eligible CPU-running requests for a future swap-in. A mark reserves future GPU rows, tokens, and finishpeak HBM but does not change ownership. The CPU scheduler yields the marked requests at its next batch

DualPace: Decoupled Scheduling and Cooperative GPU–CPU Decoding for LLM Serving

boundary, and a later GPU round performs the actual transfer. Restricting deferred swap-in to an initially empty waiting queue prevents GPU reclamation from bypassing an existing CPU backlog. 7. Commit migrations and execute one GPU batch. The scheduler materializes a decision only after all budgets pass. Swap-out captures the source GPU block table and allocates the host destination before releasing GPU ownership. Swap-in allocates the GPU destination before releasing host ownership. These ownership migrations complete before GPU forward execution; a round containing only migrations remains valid. Prefill-to-GPU allocates persistent HBM blocks. Prefillto-CPU allocates persistent host blocks and GPU staging blocks, places the request at the CPU waiting frontier, and keeps it unready until its transfer-completion event arrives. The GPU engine then submits at most one GPU-main batch and applies all generated-token, readiness, and ownership updates before its next scheduling round. A.2

waiting queue in FIFO order. It removes finished entries but stops at the first unready request, including a Prefill-to-CPU request whose layer-wise copies are incomplete or a swapped-out request whose migration has not yet become visible. Ready requests are admitted while the row, token, immediate KV, finish-growth, and migration reserve constraints remain satisfied. 4. Wait on readiness rather than bypassing ownership. If no request is selected and the queue head is unready, the CPU engine waits on that request’s readiness event. It does not skip the head to assemble a younger batch. If the round is blocked only by the migration reserve, or if no host work exists, the engine yields to the other asynchronous runtime tasks and retries later. 5. Commit and execute one CPU decode batch. For each selected request, the scheduler materializes only the blocks needed for the next decode token and constructs the physical block table consumed by the CPU worker. The finish-peak quantity remains an admission reservation rather than an eager allocation. The CPU engine submits one batch, applies its sampled tokens, releases the KV state of finished requests, and leaves unfinished requests CPU-owned and ready for the next local round.

CPU Engine Scheduling Round

The CPU engine advances host-resident requests on its own scheduling clock using the latest 𝐵𝐶migration published by the GPU scheduler. Each round proceeds as follows. 1. Initialize local admission budgets. The CPU scheduler snapshots materialized host KV use, the profiled active-batch limit, the hard token limit, and the current migration reserve. It initializes an empty candidate batch together with the cumulative next-token allocation and finish-growth obligations of requests selected for this round. 2. Preserve the running frontier. Existing CPU running requests are examined first in FIFO order. A request marked for future GPU swap-in is yielded to the front of the CPU waiting queue; its host KV remains allocated until the later GPU round commits the swap-in, preserving single ownership throughout the handoff. Every other running request is retained while the row and token limits, immediate next-token allocation, finish-growth obligation, and 𝐵𝐶migration satisfy the host KV envelope. When a boundary is reached, the unscheduled running suffix moves to the front of the waiting queue. If any running request is moved or yielded, the scheduler does not admit older waiting work in the same round. This rule preserves the priority of requests that had already entered the active CPU batch. 3. Admit the ready waiting prefix. When the running frontier remains intact, the scheduler scans the CPU

These procedures enforce two properties simultaneously. First, each request has exactly one scheduling owner at every point, and transfer readiness separates destination allocation from destination execution. Second, resource backpressure remains local: host pressure cannot prevent a runnable GPU decode subset from advancing, while the migration reserve prevents CPU admission from eliminating the GPU’s future eviction path. Neither property requires a shared GPU–CPU batch or a cross-engine iteration barrier.

B

Construction of Post-Attention Routing Scores

Section 4.3 defines the routing score 𝑆𝑟 (𝑏), the CPU-visible route costs in Equation 7, and the corresponding GPU disturbance in Equation 8. This appendix specifies how DualPace obtains the latency functions used by those equations. Threshold fitting then follows Algorithm 1; no additional routing objective is introduced here. Serving-shaped primitive profiles. For each CPU decode batch size 𝑏, DualPace profiles the same CPU kernels, GPU execution paths, shared Assist Buffers, tensor-parallel topology, CPU affinity, and NUMA placement used during serving. After 𝑊 warm-up iterations, each of 𝑅 repeats measures 𝐼 timed invocations. Let Q𝑥 be the worker ranks participating in primitive 𝑥. The critical-path sample for repeat 𝑗 and its profiled latency are

YuXuan Lin, KeQiang Wu, and Kingsum Chow

ℓ𝑥,𝑗 (𝑏) = max

end (𝑏) − 𝜏 start (𝑏) 𝜏𝑥,𝑗,𝑞 𝑥,𝑗,𝑞

,

𝑞 ∈ Q𝑥

𝐼 (10)

and host-side coordination. 𝑂 𝑥,𝑢 captures orchestration by the GPU worker on the serialized GPU-main service path. DualPace then averages over the row groups rather than over all calls:

𝑅

𝑀𝑥 (𝑏) =

1 ∑︁ ℓ𝑥,𝑗 (𝑏). 𝑅 𝑗=1

CPU primitives use wall-clock latency, while GPU primitives use the rank-critical CUDA interval reported by the GPU worker. The maximum over ranks captures the tensorparallel critical path and disappears for single-rank execution. For the RTX 4090 profile in Figure 7(a), we use 𝑊 = 5, 𝐼 = 100, and 𝑅 = 3. The CPU profiles directly instantiate the main-text latency functions: 𝑡𝑝 (𝑏) = 𝑀𝑝CPU (𝑏), 𝑎 𝑤 (𝑏) = 𝑀𝑤CPU (𝑏), ℎ 𝑤 (𝑏) = 𝑀ℎCPU (𝑏).

𝑢 ∈ U𝑥

(13)

Mapping measurements to route costs. The profiler combines the row-independent calibration constants with the batch-dependent GPU measurements as 𝑤 𝑥 (𝑏) = 𝑞𝑥 + 𝑔𝑥 (𝑏), 𝑑𝑥 (𝑏) = 𝑜 𝑥 + 𝑔𝑥 (𝑏),

𝑥 ∈ {𝑝, 𝑙 𝐴 , 𝑙 𝑆 }.

(14)

(11)

Here, 𝑀𝑝CPU measures one CPU-local post-attn stage, and CPU 𝑀𝑙 measures the post-layer stage comprising final normalization, LM head, and sampling. The remaining primitives measure the data paths surrounding Dense Assist: 𝑀𝑤CPU writes the residual and attention output to the Assist Buffer, 𝑀𝑟CPU reads the GPU-produced hidden state, and 𝑀ℎCPU writes the CPU-produced final hidden state used by Split-Final. Microbenchmarks on the GPU worker produce three batchdependent CUDA latencies. We denote them by 𝑔𝑝 (𝑏) for a non-final post-attn assist, 𝑔𝑙𝐴 (𝑏) for the final Fused-Assist call, and 𝑔𝑙𝑆 (𝑏) for the Split-Final post-layer call. Each measurement covers the complete CUDA path required by that assist, including its data movement and kernels. End-to-end calibration. We run microbenchmarks to capture batch-dependent execution latency, but a Dense Assist call also incurs control-path costs in the complete serving runtime. DualPace obtains these costs from end-to-end calibration runs executed under the same model, worker topology, affinity, and NUMA placement. For assist path 𝑥 ∈ {𝑝, 𝑙 𝐴 , 𝑙 𝑆 }, calibration calls are grouped by their observed batch-size row 𝑢. Let I𝑥,𝑢 be the calls in one such group. For call 𝑖, let 𝑊𝑥,𝑢,𝑖 be the CPU-visible assist wait, 𝐺𝑥,𝑢,𝑖 the measured CUDA execution interval, and 𝐿𝑥,𝑢,𝑖 and 𝐹𝑥,𝑢,𝑖 the host-side launch and finalization intervals recorded by the GPU worker. The profiler first computes   𝑄 𝑥,𝑢 = mean 𝑊𝑥,𝑢,𝑖 − 𝐺𝑥,𝑢,𝑖 + , 𝑖 ∈ I𝑥,𝑢
𝑂 𝑥,𝑢 = mean 𝐿𝑥,𝑢,𝑖 + 𝐹𝑥,𝑢,𝑖 ,

𝑜 𝑥 = mean 𝑂 𝑥,𝑢 ,

𝑢 ∈ U𝑥

where U𝑥 is the set of calibration groups for path 𝑥. Equal group weighting prevents the request-frequency distribution of a calibration workload from becoming an implicit routingpolicy prior.

𝑡𝑙 (𝑏) = 𝑀𝑙CPU (𝑏), 𝑎𝑟 (𝑏) = 𝑀𝑟CPU (𝑏),

𝑞𝑥 = mean 𝑄 𝑥,𝑢 ,

(12)

𝑖 ∈ I𝑥,𝑢

where [𝑧] + = max{𝑧, 0}. 𝑄 𝑥,𝑢 isolates the CPU-visible non-CUDA wait, including control publication, visibility,

Here, 𝑤 𝑥 (𝑏) is the CPU-visible wait for one assisted stage, while 𝑑𝑥 (𝑏) is the GPU-main service displaced by that stage. These primitives instantiate the wait terms in the main text: 𝑡¯𝑝 (𝑏) = 𝑎 𝑤 (𝑏) + 𝑤 𝑝 (𝑏) + 𝑎𝑟 (𝑏), 𝑡¯𝑙𝑆 (𝑏) = ℎ 𝑤 (𝑏) + 𝑤𝑙 𝑆 (𝑏),

(15)

𝑡¯𝑙𝐴 (𝑏) = 𝑎 𝑤 (𝑏) + 𝑤𝑙 𝐴 (𝑏). Substituting Equations 11, 14, and 15 into Equations 7 and 8 constructs one score row 𝑆𝑟 (𝑏) for each route and profiled batch size. CUDA execution appears in both 𝑤 𝑥 (𝑏) and 𝑑𝑥 (𝑏) because it has two effects: it delays completion of the CPU-owned batch and consumes GPU service that could otherwise advance GPU-main work.

C

Additional Tail-Latency Results

The main evaluation uses P99 TTFT and ITL on AzureConversation to separate admission latency from tokengeneration progress. This appendix reports the corresponding Azure-Code tail results and adds P99 time per output token (TPOT) on both traces. For a completed request with 𝑛 out > 1, client-observed TPOT is (𝑇e2e − 𝑇TTFT )/(𝑛 out − 1); we report the 99th percentile across requests. Unlike ITL, which captures individual token-arrival gaps, TPOT measures each request’s average generation cadence after its first token. P99 TTFT on Azure-Code. Figure 13 (top) shows that DualPace extends the TTFT-compliant load range most clearly for Llama-3.1-8B on the RTX 4090. The smaller HBM pool reaches prompt-KV pressure earlier, so Prefill-to-CPU’s release of persistent GPU KV capacity directly increases prompt admission. On the two multi-H20 configurations, DualPace closely tracks vLLM at the TTFT target while sustaining a higher request rate than NEO. These results show that combining Prefill-to-CPU with an independent decode tier

P99 Inter-Token Latency (s)

P99 Time to First Token (s)

DualPace: Decoupled Scheduling and Cooperative GPU–CPU Decoding for LLM Serving

DualPace (ours)

NEO

vLLM

10

10

12

8

8

10

6

6

4

4

2

2

0

0

2.0

2.5

3.0

3.5

4.0

4.5

8 6 4 2 2.00

2.25

2.50

Request Rate (req/s)

2.75

3.00

3.25

3.50

3.75

4.00

0

2.0

2.2

2.4

Request Rate (req/s)

2.8

3.0

2.8

3.0

2.0

1.5

0.8

2.6

Request Rate (req/s)

1.5 1.0

0.6 2.0

2.5

3.0

3.5

4.0

4.5

2.00

2.25

2.50

2.75

3.00

3.25

3.50

3.75

4.00

2.0

2.2

2.4

2.6

Request Rate (req/s)

Request Rate (req/s)

Request Rate (req/s)

(a) Llama-3.1-8B, RTX 4090, 12 CPU cores

(b) Qwen3-14B, 2×H20, 24 CPU cores

(c) Qwen3-32B, 4×H20, 48 CPU cores

Figure 13. P99 client-observed TTFT and ITL under increasing offered load on the Azure-Code workload.

P99 Time Per Output Token (s)

DualPace (ours) 3.0 2.5 2.0 1.5 1.0 0.5 0.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

3.0 2.5 2.0 1.5 1.0 0.5 0.0

2.0

NEO

2.5

3.0

3.5

vLLM

4.0

4.5

5.0

3.0 2.5 2.0 1.5 1.0 0.5 0.0

2.00

2.25

2.50

2.75

3.00

3.25

3.50

3.75

4.00

Request Rate (req/s)

Request Rate (req/s)

Request Rate (req/s)

(a) Llama-3.1-8B, RTX 4090, 12 CPU cores, Azure-Conv

(b) Qwen3-14B, 2×H20, 24 CPU cores, Azure-Conv

(c) Qwen3-32B, 4×H20, 48 CPU cores, Azure-Conv

0.8 2.0 1.5 0.6

1.5 1.0 2.0

2.5

3.0

3.5

4.0

4.5

2.00

2.25

2.50

2.75

3.00

3.25

3.50

3.75

4.00

2.0

2.2

2.4

2.6

2.8

3.0

Request Rate (req/s)

Request Rate (req/s)

Request Rate (req/s)

(d) Llama-3.1-8B, RTX 4090, 12 CPU cores, Azure-Code

(e) Qwen3-14B, 2×H20, 24 CPU cores, Azure-Code

(f) Qwen3-32B, 4×H20, 48 CPU cores, Azure-Code

Figure 14. P99 client-observed TPOT under increasing offered load on the Azure-Conversation and Azure-Code workloads. preserves the first-token benefit of host KV residency across both single- and multi-GPU servers. P99 ITL on Azure-Code. Figure 13 (bottom) shows a complementary effect during generation: DualPace provides the lowest P99 ITL over most evaluated request rates. AzureCode’s long GPU prefills contend with ongoing GPU-resident decoding, while its short outputs bound the amount of work assigned to the CPU tier. DualPace can therefore advance host-resident decodes on an independent clock while the GPU serves prompts, avoiding the GPU-gated progress gaps of NEO and, over much of the measured range, offsetting the CPU’s lower per-token execution speed. This result identifies prefill-heavy contention as a second regime in which independent decode progress improves the tail. P99 TPOT on Azure-Conversation. Figure 14 (top) corroborates the main P99 ITL result with a request-level metric. At the 1.5 s P99 TPOT target, DualPace sustains 61%, 15%, and 3% higher request rates than NEO from left to right. Independent CPU scheduling thus improves not only isolated

inter-token gaps but also the tail request’s average generation cadence. vLLM maintains a lower TPOT because its requests decode entirely on the GPU; among CPU-offloading systems, DualPace converts host residency into substantially more regular progress.

P99 TPOT on Azure-Code. Figure 14 (bottom) shows that DualPace generally outperforms NEO and frequently matches or outperforms vLLM across the measured rates. This operating point differs from Azure-Conversation: long incoming prefills dominate GPU work, whereas each admitted request has only a short decode phase. GPU-only decoding must share the device with those prefills, and GPU-gated CPU offloading must wait for suitable GPU-side opportunities. DualPace instead uses the host as an independently progressing decode tier, allowing token generation to continue while the GPU processes prompt work. The TPOT results therefore complement the normalized-latency boundary in the main text: CPU offloading leaves the GPU-dominated

P99 Time Per Output Token (s)

YuXuan Lin, KeQiang Wu, and Kingsum Chow

CPU TP1

30 GiB 60 GiB

CPU TP2

120 GiB 240 GiB

2.0 1.5 1.0 0.5 0.0

2.0

2.5

3.0

3.5

4.0

4.5

5.0

2.0

2.5

3.0

3.5

4.0

4.5

Request Rate (req/s)

Request Rate (req/s)

(a) Impact of CPU tensor parallelism

(b) Impact of Host kv cache capacity

Figure 15. P99 client-observed TPOT sensitivity on Azure-Conversation for Llama-3.1-8B on one RTX 4090. Left: CPU tensor parallelism, with 12 physical cores per rank and the same aggregate host KV-cache capacity. Right: host KV-cache capacity. The horizontal dashed line marks the 1 s P99 TPOT target; vertical dashed lines mark interpolated sustainable request rates when the target is crossed within the measured sweep. prefill path unchanged, while independent host progress isolates the generation cadence that follows admission from that path. CPU tensor parallelism. Figure 15 (left) complements the normalized-latency, TTFT, and ITL results in Figure 11.

At the 1 s P99 TPOT target, CPU TP = 2 sustains a 31% higher request rate than CPU TP = 1. Each rank uses 12 physical cores pinned to one NUMA node, and both configurations use the same aggregate host KV-cache capacity. The additional rank exposes more parallel AMX execution and aggregate memory bandwidth, accelerating both dense operators and KV-cache reads. CPU tensor parallelism therefore improves the generation cadence of tail requests in addition to increasing sustainable throughput. Host KV-cache capacity. Figure 15 (right) exhibits the same placement trend as the P99 ITL result in the main text. The 120 and 240 GiB configurations attain nearly identical P99 TPOT, while the 30 and 60 GiB configurations provide lower P99 TPOT over the measured request-rate range. A smaller host KV pool keeps a larger fraction of requests GPUresident, favoring per-token execution latency. Increasing host KV capacity instead admits more Prefill-to-CPU and migrated requests, expanding the host-resident decode tier and improving the sustainable load and TTFT results reported in the main text. Once 120 GiB can retain most KV states selected for offloading, increasing the pool to 240 GiB no longer changes the active CPU execution regime, producing nearly overlapping TPOT curves.

Related documents

• DualPace: Decoupled Scheduling and Cooperative GPU--CPU Decoding for LLM Serving#1455982
• The Token-Energy Law of LLM Decode: Active Bytes and Realized Time Account for Joules per Token#1412633
• The Token-Energy Law of LLM Decode: Active Bytes and Realized Time Account for Joules per Token#1394205
• Resource-Efficient Speculative Decoding for Long-Context LLM Serving#1117896
• Fast Recovery for LLM Serving via Decoupled Device Memory Lifetime in Dynamo#1061091