PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers
Hongbin Zhang 1 Taosheng Wei 1 Jiazhi Jiang 1 Hui Yan 1 Jiangsu Du 1 Zhiguang Chen 1
arXiv:2605.02189v1 [cs.DC] 4 May 2026
Abstract
as RTX 5090 are approximately 3× more cost-effective under the same computational capability (Feng et al., 2023) and constitute a large fraction of the deployed GPU infrastructure (Du et al., 2025). Second, high-end GPUs with high-bandwidth interconnects (e.g., NVLink), such as H100 and B200, are in practice preferentially reserved for latencycritical LLM serving. However, efficiently exploiting commodity GPU servers for high-throughput LLM inference remains challenging.
Offline LLM inference seeks to maximize request processing under fixed budgets, making commodity GPU servers a promising choice. However, prior work typically considers offloading and parallelism in isolation, resulting in suboptimal performance. In this paper, we propose PipeMax, a high-throughput LLM inference system that integrates pipeline parallelism with offloading to overcome interconnect and memory constraints on GPU servers. Particularly, pipeline parallelism naturally incurs low communication overhead and keeps only one batch active on each GPU at a time, which enables offloading the KV cache of inactive batches. By coordinating computation with offloading data movement, PipeMax effectively expands GPU memory capacity and sustains large-batch execution. Experiments show that PipeMax achieves up to 2.51× higher throughput than vLLM, and up to 1.42× and 1.38× higher throughput than state-of-the-art high-throughput LLM systems, respectively, on an 8-GPU node.
LLM inference imposes heavy GPU memory demands due to large model weights and extensive KV cache, which often exceeds the footprint of the weights. Existing systems (Sheng et al., 2023; Zhang et al., 2025) attempt to expand memory capacity via either offloading or parallelization. Purely offloading-based approaches are fundamentally constrained by limited CPU–GPU bandwidth that transferring model weights and KV cache dominates execution time, leaving GPUs largely underutilized. Parallelism-based approaches include tensor parallelism and pipeline parallelism. Among them, tensor parallelism is communication-bound due to frequent all-reduce operations, rendering it impractical on bandwidth-limited nodes. Consequently, pipeline parallelism emerges as the most promising option for commodity GPU servers.
1. Introduction
However, standalone pipeline parallelism falls short of realizing its full memory efficiency. During the decode stage under pipeline parallelism, only the KV cache of a single batch is active for each GPU at any given time, leaving most GPU memory occupied by inactive batches and limiting effective memory expansion. To address this limitation, we propose PipeMax, a high-throughput LLM inference system tailored for commodity GPU servers. PipeMax enhances pipeline parallelism with offloading by storing inactive batches in CPU memory, allowing pipeline parallelism to fully exploit its memory efficiency. In summary, we make the following contributions:
Large language models (LLMs) have been widely adopted across many domains (GitHub, 2023; Nazi & Peng, 2024; Rane et al., 2023). Due to their massive parameter scales, LLM inference requires numerous high-cost GPUs, making it extremely expensive to deploy. Beyond interactive applications such as chatbots, LLMs are increasingly used in offline workloads, including database processing (Liu et al., 2025) and information extraction (Xu et al., 2024). Unlike online LLM serving (Zhong et al., 2024; Agrawal et al., 2024; Du et al., 2025), which prioritizes latency SLOs, offline scenarios primarily target high-throughput execution.
• We identify a decode-phase inefficiency in pipeline parallelism: KV cache remains idle across inactive batches, limiting effective GPU memory utilization.
Providing high-throughput LLM inference on commodity GPU servers without high-bandwidth interconnects is increasingly important. First, commodity GPU servers such
• We propose PipeMax, a high-throughput LLM inference system that strategically integrates pipeline parallelism with offloading to expand effective GPU memory by evicting inactive KV cache.
1
School of Computer Science and Engineering, Sun Yat-sen University.
1
PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers prompt
• We evaluate PipeMax and demonstrate it has substantial throughput improvements over state-of-the-art systems across a range of model sizes and workloads.
LLM
is
all
need
you
.
<eos>
Transformer Model
KV Cache need
.
<eos>
prefill
2. Background
decode
Figure 1. Autoregressive generation in LLM inference.
2.1. LLM Processing Phases
Multi-batch Computation Layer-wise
Recent large language models (LLMs) adopt an autoregressive generation paradigm. As shown in Fig. 1, the model iteratively predicts tokens until the end-of-sequence (EoS), storing intermediate states as a KV cache to avoid redundant computation. Accordingly, LLM inference consists of two phases: prefill for processing the input sequence, and decode for sequential token generation using the KV cache.
CPU to GPU
𝑳𝒊
𝑩𝟏
GPU to CPU
𝑩𝟏
𝑩𝟐 𝑩𝟑 𝑩𝟒
𝑩𝟐
𝑩𝟑
𝑩𝟒
𝑳𝒊&𝟏
𝑩𝟏
𝑩𝟐
𝑩𝟑
𝑩𝟒
Load Weight
Offload KV
𝑩𝟏 𝑩𝟏
𝑩𝟐 𝑩𝟑 𝑩𝟒
𝑩𝟐
𝑩𝟑
𝑩𝟒
𝑩𝟏
𝑩𝟐
𝑩𝟑
𝑩𝟒 Time
Load KV
Computation
Figure 2. Existing offload-based LLM inference method.
parallelism assigns different layer groups to different GPUs and transfers intermediate activations between stages. Prior work (Zhang et al., 2025; Su et al., 2025) shows that tensor parallelism incurs high communication overhead on GPU servers without high-speed interconnects due to frequent all-reduce operations, thereby making pipeline parallelism a natural alternative. Motivated by this, TDPipe (Zhang et al., 2025) adopts temporally disaggregated pipeline parallelism with inter-batch work stealing to mitigate pipeline bubbles and improve decode-phase arithmetic intensity. Meanwhile, Seesaw (Su et al., 2025) re-shards the model across prefill and decode, using pipeline parallelism for prefill and tensor parallelism for decode. However, both approaches have limitations. TD-Pipe retains multiple decode batches in GPU memory, leaving much KV cache inactive (Section 2.3.2), while Seesaw assumes negligible decode-phase communication overhead, which is often nonnegligible on bandwidth-constrained commodity servers, and becomes increasingly restrictive as the system scales to more GPUs.
Prior work (Du et al., 2025; Jiang et al., 2025) shows that prefill is compute-intensive and efficient even with small batches, whereas decode is memory-bound and requires large batches for peak utilization. However, achieving large decode batches requires substantial GPU memory for KV cache. For instance, a batch size of 512 with a sequence length of 1024 in Qwen3-32B requires about 133 GB of KV cache, far beyond a single GPU’s capacity. Thus, decode throughput is fundamentally limited by GPU memory. 2.2. Existing High-Throughput LLM Inference To address above GPU memory issue, existing systems (Zhang et al., 2025; Kwon et al., 2023; Su et al., 2025; Sheng et al., 2023) commonly use offloading or model parallelism to expand GPU memory. 2.2.1. O FFLOADING A PPROACH Systems such as FlexGen (Sheng et al., 2023) and TightLLM (Hu et al., 2025) enable single-GPU highthroughput LLM inference by offloading model weights and/or KV cache to CPU memory. As illustrated in Fig. 2, they rely on two core techniques: layer-wise offloading, which keeps only a subset of layers on the GPU, and multibatch inference, which partitions requests into multiple batches to reduce the KV cache footprint. Both techniques use prefetching to overlap computation and data transfer.
2.3. Analysis of Pipeline Parallel for LLM Inference 2.3.1. P REFILL PHASE Fig. 4 shows that under continuous pipeline-parallel execution, prefill latency is dominated by the first pipeline stage for large request counts. The total execution time equals the first-stage time plus (n − 1) times the per-stage prefill time of the longest request, where n is the number of GPUs (Appendix A.1). As prefill is compute-intensive and free of inter-request dependencies, pipeline parallelism achieves high GPU utilization with minimal overhead.
However, offloading-only approaches remain bandwidthbound on commodity GPUs: limited PCIe bandwidth causes weight transfers to dominate execution, and on multi-GPU nodes, repeatedly transferring weights to each GPU leads to redundant communication and wasted bandwidth.
2.3.2. D ECODE PHASE In contrast, decode exhibits fundamentally different behavior under pipeline parallelism. Prior work (Zhang et al.,
The two mainstream model parallelism techniques are tensor parallelism and pipeline parallelism, both of which distribute model parameters across multiple GPUs to expand total memory capacity. As shown in Fig. 3, tensor parallelism partitions computation within each layer and requires all-reduce synchronization at every layer, whereas pipeline
(a) Tensor Parallel
GPU1 Send/Recv
All Reduce
All Reduce
All Reduce
GPU2
All Reduce
GPU1
All Reduce
GPU0 GPU0
GPU2
Send/Recv
2.2.2. M ODEL PARALLELISM A PPROACH
(b) Pipeline Parallel
Figure 3. Tensor parallelism and pipeline parallelism.
2
PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers Stage 1 Stage 2 Stage 3 Stage 4 𝑛=4
𝒕𝟏
𝒕𝟐
𝒕𝟑
𝒕𝟒
𝒕&
𝑛 − 1 × max{ 𝑡% }
Time
{ 𝑡! } Total Time = ! 𝑡! + 𝑛 − 1 × max ! !
Stage 1 Stage 2 Stage 3 Stage 4
%
Inter-decode-step Data Dependency
Incoming prefill batches
Prefill
Figure 4. Prefill phase under pipeline parallelism. The total execution time equals the first stage plus (n − 1) times the longest stage, where n is the number of GPUs.
C1: Uncertain Compute–Prefetch Overlap. Achieving effective KV cache prefetching requires overlapping prefetch with decode computation, so that GPU memory only needs to hold the KV cache of the current and upcoming batches. However, this ideal overlap is not always attainable in practice. Limited CPU–GPU bandwidth may prevent fully prefetching a batch’s KV cache within a single decode iteration, delaying subsequent iterations and causing GPU idle time. Moreover, decode execution time varies across iterations due to irregular request completion and dynamic batch composition, further complicating compute–communication overlap. C2: Prefetch-Aware Decode Batch Balancing. As discussed in Section 2.3.2, efficient decode execution requires balanced execution times across batches. TD-Pipe achieves this via inter-batch work stealing under the assumption of a closed set of GPU-resident requests. With prefetching, however, decode batches are formed from a broader pool of CPU-resident requests, invalidating this assumption. Consequently, batch balancing must jointly consider execution time and prefetch feasibility, making it significantly harder to maintain stable execution across iterations. C3: KV Cache Transfer Inefficiency under PagedAttention. Modern LLM inference frameworks adopt PagedAttention (Kwon et al., 2023) to improve GPU memory utilization, yet mainstream implementations are not transferfriendly. In systems such as vLLM (Kwon et al., 2023) and sglang (Zheng et al., 2024a), KV cache is organized in page-sized blocks and further separated along the layer dimension, fragmenting each request’s KV cache along two dimensions and leading to inefficient CPU–GPU transfers.
Despite such balancing, decode throughput under pipeline parallelism remains constrained by the memory footprint of the active batch. At any time, each GPU executes only one active decode batch, leaving the KV cache of other batches idle and resulting in low effective GPU memory utilization. The average token budget per batch can be expressed as (1)
where M denotes the per-GPU memory capacity, W the total model size, n the pipeline degree, and T the KV cache size per token; a detailed derivation is provided in Appendix A.2. As a result, although pipeline parallelism increases the aggregate KV cache capacity across devices, the KV cache available to the active batch remains fundamentally constrained. Importantly, multi-batch execution exhibits natural anti-locality: a batch’s KV cache remains unused until all other batches finish their decode iterations. 2.3.3. P REFILL -D ECODE IMBALANCE Frequent transitions between prefill and decode introduce pipeline bubbles due to execution-time mismatch, as illustrated in Fig. 6. TD-Pipe (Zhang et al., 2025) mitigates this overhead through temporal separation, which PipeMax adopts as its baseline execution model.
3. The Design of PipeMax In this section, we propose PipeMax, a high-throughput LLM inference system that leverages pipeline parallelism with offloading. Specifically, PipeMax leverages pipeline parallelism to partition model weights across GPUs, while offloading only KV cache to CPU memory.
2.4. Opportunities and Challenges The anti-locality in Section 2.3.2, together with pipeline parallelism’s multi-batch structure, naturally motivates offloading inactive KV cache to CPU memory and prefetching it on demand, thereby expanding the effective GPU memory for the active batch. Fig. 7 illustrates this mechanism. However, realizing this opportunity introduces three challenges: Inter-decode-step Data Dependency Stage 1 Stage 2 Stage 3 Stage 4
3.1. PipeMax Workflow As illustrated in Fig. 8, PipeMax temporally decouples prefill and decode, establishing a producer–consumer pipeline between the two stages.
Decode Batch Stage 1 Stage 2 Stage 3 Stage 4
(a) Inter-Batch Imbalance
Time
Time Bubble
Figure 6. Prefill-decode imbalance.
2025; Zhong et al., 2024) partitions GPU-resident requests into multiple autoregressive batches to fill the pipeline, but inter-step dependencies cause execution-time variations to amplify into inter-batch imbalance, leading to pipeline stalls (Fig. 5(a)). TD-Pipe (Zhang et al., 2025) mitigates this issue via inter-batch work stealing (Fig. 5(b)).
M − W/n , T
Decode Batch
3.1.1. P REFILL S TAGE As discussed in Section 2.3, PipeMax continuously performs prefill to maximize pipeline utilization. The generated KV
Time
(b) With Inter-batch Work Stealing
Figure 5. Decode phase under pipeline parallelism.
3
PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers KV Cache Composition
Stage 1 Stage 2 Stage 3 Stage 4
Batch 0
……
Prefetch
Figure 9. KV cache composition per batch in PipeMax under bandwidth constraints.
Data Dependency
Figure 7. Decode phase with KV cache prefetching.
A centralized engine serves as the control plane, while a distributed runtime constitutes the execution plane, jointly supporting the execution workflow described above. We next describe the key mechanisms of both components.
cache is asynchronously offloaded to CPU memory, allowing GPU memory to be reused by overwriting completed requests without blocking. This decouples prefill from GPU memory constraints and accumulates decode requests.
3.3. Centralized Engine
3.1.2. D ECODE S TAGE
The centralized engine integrates a decode execution-time estimator and a prefetch-aware scheduler to coordinate decode execution and KV cache prefetching.
Once sufficient requests are buffered, PipeMax enters the decode stage and addresses the challenges identified above. C1. PipeMax adopts a best-effort KV cache prefetching strategy. Before each iteration, it estimates decode execution time to determine the prefetch budget and incrementally prefetches KV cache by overwriting inactive batches. When bandwidth is insufficient, prefetching is amortized across iterations, resulting in a hybrid GPU-resident and prefetched KV cache per batch (Fig. 9).
3.3.1. D ECODE E XECUTION T IME E STIMATOR To overlap decode computation with KV cache prefetching, PipeMax estimates decode execution time for each batch. A decode iteration consists of two components: (1) linear operations (e.g., QKV projection and feed-forward networks) with cost O(b × h2 ) for batch size b and hidden size h; and (2) attention over the prefix KV cache with cost O(L), Pb where L = i=1 Li denotes the total prefix length of the batch, and Li is the prefix length of request i.
C2. PipeMax employs a prefetch-aware scheduler that jointly considers execution time and prefetch feasibility, aligning batch execution times to mitigate inter-batch imbalance.
Accordingly, PipeMax models the decode execution time of a batch as: α · b + β · L + δ, (2)
C3. PipeMax further introduces a transfer-efficient KV cache engine that maximizes CPU–GPU bandwidth utilization while remaining compatible with PagedAttention.
where α, β, and δ are parameters obtained via offline profiling, capturing the per-request linear cost, per-token attention cost, and constant overheads, respectively.
3.1.3. P REFILL –D ECODE S WITCHING P OLICY Unlike TD-Pipe, which frequently switches between prefill and decode to sustain compute intensity under GPU memory constraints, PipeMax buffers a large pool of decodeready requests in CPU memory and adopts a memory-driven switching policy.
3.3.2. P REFETCH - AWARE DECODE SCHEDULER To address C2 in Section 2.4, PipeMax introduces a prefetchaware scheduler for the decode stage that dynamically determines how many requests to prefetch and which ones to select. The scheduler aims to fully overlap computation with KV cache prefetching, while expanding effective memory capacity and maintaining balance across batches.
Prefill proceeds until CPU-resident KV cache approaches capacity, reserving headroom for decode, and resumes only when available KV cache falls below GPU capacity. As a result, prefill–decode switching is infrequent, and the imbalance in Fig. 6 can be safely ignored.
As discussed in Section 3.1.2, limited CPU–GPU bandwidth may leave portions of KV cache from multiple batches resident in GPU memory. To efficiently utilize GPU mem-
3.2. System Overview
request queue
As shown in Fig. 10, PipeMax adopts a hierarchical controller architecture that separates control from execution.
…
req. 1
req. 0
Centralized Engine(§3.3)
Timeline Prefill Stage
Prefetch-aware Scheduler (§3.3.2)
Decode Time Estimator (§3.3.1)
Distributed Runtime (§3.4)
Decode Stage Stage Transition
Continuously Executed Pipeline Parallelism Producer Offloading
Autoregressive Batch 3
GPU-Resident Prefetched
Time
Decode Execution
Batch 1 Batch 2
B! B" B# B$
B! B" B# B$
B! B" B# B$
B! B" B# B$
GPU 0
GPU 1
GPU 2
GPU 3
Pipeline Parallelism With Prefetch Consumer Prefetching
KV Cache Engine (§3.4.3) CPU Memory
CPU-Resident KV Cache Replicas
Activation Transfer
Figure 8. The workflow of PipeMax.
Offloading
Demand-Priority Transfer Stream
Prefetching
Metadata
Figure 10. Overview of PipeMax system.
4
PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers
While the above formulation specifies Pt per iteration, the scheduling policy evolves across iterations, and the decode execution is divided into warm-up and steady phases. In warm-up phase, PipeMax enlarges the prefetch budget across iterations by extending decode iterations, thereby expanding effective GPU memory. According to Eq. 2, when the total KV cache length of the current batch L is close to len(Djres ) + Bt , execution time is dominated by the t batch size b. PipeMax therefore prioritizes short requests to pack more requests into each batch, maximizing the batch size b under a fixed KV cache budget. This increases the prefetch budget in subsequent iterations via the time-based update, forming a positive feedback loop. However, as longer requests are admitted, KV cache growth under fixed GPU memory and CPU–GPU bandwidth imposes hard limits, causing execution time to fluctuate. In practice, after such temporary fluctuations, execution time converges to a bounded range, after which the system enters the steady phase. In steady phase, the primary objective shifts to maintaining execution-time balance across decode batches to mitigate inter-batch imbalance. To this end, PipeMax selects Pt such that the predicted execution time of the updated batch remains close to T̂t . Based on the execution-time model in Section 3.3.1, the retained set Djres contributes a detert res ministic execution time T̂t , computed using Eq. (2). The remaining execution-time gap is
ory while preserving autoregressive semantics, PipeMax partitions decode requests into n batches, where n is the pipeline depth, and formulates decode scheduling as a stateful, prefetch-aware iterative batch update problem. PipeMax maintains a set of decode batches D = {D0 , D1 , . . . , Dn−1 },
(3)
which are executed autoregressively in a cyclic order. At iteration t, the executing, prefetched, and overwritten batches are indexed as it = t mod n, jt = (it + 1) mod n,
(4)
kt = (it − 1) mod n. During iteration t, PipeMax executes batch Dit while concurrently prefetching KV cache from CPU memory to update Djt for execution in iteration t + 1, overlapping computation with data movement. Initial Decode Batches. Let R denote the set of requests with KV cache resident in GPU memory after prefill. PipeMax constructs an initial partition D such that n−1 [
Dk = R,
Dk ∩ Dk′ = ∅ ∀k ̸= k ′ .
(5)
k=0
Each subset Dk forms an initial decode batch. The initial partition divides requests into batches of equal size, while attempting to balance total KV cache length across batches to approximate similar decode execution times, thereby reducing inter-batch imbalance across pipeline stages.
∆T̂t = T̂t − T̂tres .
Selecting Pt is formulated as a subset selection problem, over CPU-resident requests, where each request r contributes α+βLr according to Eq. (2). The cumulative prefix length is constrained to nearly saturate the prefetch budget Bt , so as to fully utilize the available CPU–GPU bandwidth and maximally extend the effective GPU memory capacity. The objective is to make the execution-time contribution of CPU-resident requests as close as possible to ∆T̂t . PipeMax provides a greedy algorithm to efficiently solve this problem; the complete scheduling procedure and the detailed algorithm for selecting Pt are presented in Appendix A.3.
Iterative Prefetch-Aware Scheduling. After initialization, decode execution proceeds iteratively in an autoregressive manner. At each iteration t, PipeMax updates the next decode batch Djt using a prefetch-aware scheduling policy. First, PipeMax retains requests whose KV cache remains resident in GPU memory: Djres = Djt ∩ Rt , t
(6)
where Rt denotes the set of requests whose KV cache resides in GPU memory at the beginning of iteration t.
During prefetching for Djt , PipeMax reuses GPU memory by overwriting KV cache blocks associated with the inactive batch Dkt . The updated decode batch is
Second, PipeMax determines additional requests to prefetch from CPU memory. It predicts the execution time Tˆt of the currently executing batch Dit using the estimator in Section 3.3.1, and derives a prefetch budget Bt = B · T̂t ,
(8)
Djt = Djres ∪ Pt . t
(9)
3.4. PipeMax Runtime
(7)
The PipeMax runtime consists of two components to support model execution, as follows.
where B denotes the effective CPU–GPU bandwidth, profiled using KV cache transfers in the PagedAttention format.
3.4.1. S CHEDULER - COORDINATED M ODEL E XECUTOR
Given Bt , PipeMax selects CPU-resident requests Pt whose total prefix length approaches Bt , maximizing bandwidth utilization and effective GPU memory expansion.
The PipeMax runtime includes a scheduler-coordinated model executor that receives execution metadata from the 5
PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers Prefetch
Timeline
Continuous Memory
Layer 1
Block 1
Block 2
Block 3
Layer 2
Block 1
Block 2
Block 3
Layer 3
Block 1
Block 2
Block 3
Layer 4
Block 1
Block 2
Block 3
Block 1 Layer 1
Layer 2 1Layer 3 Block
Layer 4
Block 2 Layer 1
Layer 2 1Layer 3 Block
Layer 4
Block 3 Layer 1
Layer 2 1Layer 3 Block
Layer 4
(a) Layer-first Layout
GPU Prefetch
compute stream transfer stream
High-Priority Low-Priority
(b) Block-first Layout
submit
Activation Prefetching
Computation Offloading
Figure 13. Priority-based transfer orchestration in PipeMax. Figure 11. Layer-first vs block-first KV cache layouts for prefetching. The block-first layout enables contiguous block-level prefetching, while the layer-first layout incurs fragmented transfers.
by asynchronously offloading per-layer KV cache to CPU memory immediately after QKV computation, overlapping data transfer with subsequent computation. In Fig. 12, the offloaded KV cache is then reorganized on the CPU side into a block-first layout for later prefetching, incurring negligible overhead. Since PipeMax adopts a consistent block-first KV cache layout in both CPU and GPU memory, prefetched KV cache blocks can be directly transferred from CPU to GPU without any further reorganization. Demand-Priority Transfer Orchestration The PipeMax runtime involves three types of data movement: inter-stage activation transfers, KV cache prefetching, and KV cache offloading. On commodity GPU servers, all transfers are carried out over PCIe, a full-duplex CPU–GPU interconnect, where prefetching and offloading proceed in opposite directions without interfering with each other. Activation transfers occupy both PCIe directions and lie on the critical path of the next pipeline stage, whereas prefetching and offloading use only the CPU-to-GPU and GPU-to-CPU directions, respectively. Although activation transfers involve much smaller data volumes and incur negligible impact on prefetching and offloading, they are latency-critical for pipeline execution and highly sensitive to interference from concurrent prefetching and offloading.
centralized engine and executes scheduled model computation, asynchronously transferring intermediate activations across pipeline stages via peer-to-peer communication. 3.4.2. T RANSFER -E FFICIENT KV CACHE E NGINE PipeMax designs a transfer-efficient KV cache engine to efficiently utilize available CPU–GPU bandwidth. Similar to existing systems such as vLLM (Kwon et al., 2023) and sglang (Zheng et al., 2024a), PipeMax stores KV cache on both GPU and CPU memory using the PagedAttention format, and further enhances KV cache transfers as follows: Block-First Layout for Prefetching C3 in Section 2.4 identifies that existing PagedAttention-based designs adopt layer-first KV cache layouts and further partition them into blocks, causing KV cache prefetching to be sliced along both the layer and block dimensions. As illustrated in Fig. 11(a), when prefetching a block of a target request, the corresponding KV cache is not stored contiguously in memory, leading to inefficient CPU–GPU data transfers. PipeMax instead adopts a block-first KV cache layout to accelerate prefetching. Fig. 11(b) shows that this layout colocates the KV cache of all layers within the same block into contiguous memory regions. Thus, KV cache prefetching is sliced only along the block dimension, enabling efficient transfers. Notably, this layout change only affects KV cache storage and transfer, and remains fully compatible with PagedAttention without any implementation changes.
Inspired by priority-aware load management in AptMoE (Wei et al., 2024), PipeMax employs priority-based transfer orchestration with multiple priority queues, prioritizing activation transfers without degrading KV cache throughput. Fig. 13 shows that PipeMax maintains priority queues for issuing data transfer requests. PipeMax enforces priorities by controlling the submission order of PCIe transfer requests: activation transfers are issued promptly, while KV cache prefetching and offloading are scheduled opportunistically.
Asynchronous CPU-Assisted KV Cache Offloading Both prefill and decode stages continuously generate KV cache that must be preserved in CPU-resident replicas, and ideally transferred in a way that can be overlapped with computation. While the block-first layout improves prefetch efficiency, KV cache is generated on a per-layer basis, making block-wise offloading infeasible at generation time.
4. Evaluation We implement PipeMax based on vLLM v0.7.3. To demonstrate its effectiveness, we evaluate PipeMax across diverse hardware configurations and workloads, and compare it with state-of-the-art approaches. In addition, we conduct ablation studies to quantify the impact of individual components.
Fortunately, right after QKV projection, each layer produces contiguous KV cache tensors before they are partitioned into PagedAttention blocks. PipeMax exploits this observation Layer 1 GPU GPU->CPU CPU
…
QKV Projection
Layer i
…
Attention
Layer N
4.1. Experimental Setup
FFN
4.1.1. N ODE T ESTBED .
Offload Layouting
We conduct experiments on three nodes: two commodity GPU servers with 8×RTX 5090 and 8×L20 GPUs (both without NVLink), and one data-center server with 8×H100
Timeline
Figure 12. Execution timeline of per-layer KV cache generation, offloading, and CPU-side layouting.
6
PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers Table 1. GPU platforms and deployed models.
Table 2. Length statistics of the evaluation datasets.
Node
Workload (Models)
Dataset
InAvg
InMed
OutAvg
OutMed
8 × RTX 5090 8 × L20 8 × H100
LLaMA 2 70B (70B), Mixtral-8×7B(8×7B) LLaMA 2 70B (70B), Mixtral-8×7B(8×7B) Qwen3 235B-A22B (235B-A22B)
ShareGPT (ShareGPT, 2025) LongBench (Bai et al., 2024)
343.76 2686.89
148.00 2736.50
237.20 101.78
152.00 19.00
4.2. Overall Throughput
1
GPUs interconnected via NVLink . All GPUs connect to independent PCIe root complexes: RTX 5090 and H100 use PCIe 5.0 (64 GB/s), while L20 uses PCIe 4.0 (32 GB/s).
We compare PipeMax with all baselines in throughput, measured in tokens per second, across diverse hardware configurations and workloads (Section 4.1.2). Fig. 14 reports the normalized overall throughput results. PipeMax outperforms vLLM(TP), vLLM(PP), TD-Pipe, and Seesaw by up to 2.45×, 2.51×, 1.42×, and 1.38×, respectively.
4.1.2. M ODEL AND DATASET S ETUP. Table 1 summarizes the deployed dense and MoE models, each scaled to the limits of its GPU platform.
PipeMax leverages pipeline parallelism to reduce inter-GPU communication overhead compared to vLLM(TP), while eliminating the decode-phase anti-locality in prior pipelinebased designs such as TD-Pipe. Although vLLM (PP) also adopts pipeline parallelism, its decode phase suffers from pronounced inter-batch imbalance(Fig. 5) due to lack of load balancing, which limits overall performance. Seesaw shows lower-than-expected performance in our setting, as its decode phase relies on all-reduce communication, constrained by bandwidth in large-scale GPU configurations. Moreover, even on NVLink-equipped H100 servers, the extreme compute capability of H100 GPUs makes inter-GPU communication under tensor parallelism non-trivial. Consequently, PipeMax has the potential to deliver benefits on data-center GPU servers as well.
We select two representative datasets with distinct sequencelength characteristics. ShareGPT features balanced input and output lengths, while LongBench targets long-form inputs. Table 2 reports their input and output length statistics. 4.1.3. BASELINE SETUP. We compare PipeMax with the following baselines. Offloading-based systems such as FlexGen and TightLLM primarily target single-GPU inference and are not designed for multi-GPU execution, and thus are not included. vLLM (Kwon et al., 2023) is a widely adopted LLM inference engine supporting multiple parallelism strategies. TD-Pipe (Zhang et al., 2025) extends vLLM by addressing fundamental inefficiencies in pipeline parallelism, temporally decoupling prefill and decode to mitigate prefill–decode imbalance (Fig. 6) and employing inter-batch work stealing to alleviate inter-batch imbalance (Fig. 5).
4.3. Ablation Study This section studies the impact of design strategies in PipeMax via ablation experiments on RTX 5090 and L20 GPU servers with the 70B model and ShareGPT dataset.
Seesaw (Su et al., 2025) builds upon vLLM by adopting pipeline parallelism during the prefill stage and prioritizing tensor parallelism during the decode stage.
4.3.1. C ENTRALIZED E NGINE Decode Execution-Time Estimator To evaluate the accuracy of the decode execution-time estimator, we sample 100 consecutive decode steps during runtime and measure the ratio between the actual batch execution time and the predicted execution time at each step. Figure 15 reports these ratios for two representative workloads. Across both workloads, predictions closely match actual execution times, with over 90% of samples within 5% error and worst-case deviation below 8%. This confirms the estimator’s accuracy and suitability for prefetch-aware scheduling.
We evaluate vLLM under both tensor parallelism(TP) and pipeline parallelism(PP). TD-Pipe uses PP, while Seesaw follows its design, using PP for prefill and TP for decode. All baselines are implemented on top of vLLM v0.7.3 to ensure a fair comparison. Since the designs of Seesaw, TD-Pipe, and PipeMax are orthogonal to vLLM’s ongoing evolution, the relative performance trends reported in this paper are expected to remain valid on newer vLLM versions. 1 H100 is included solely as a high-end reference platform, while our primary focus is on commodity GPU servers.
vLLM(PP)
TD-Pipe
(a) RTX 5090 2
2
1
1
0
70B
8×7B
0
2
2
1
1
0
70B
8×7B
Seesaw
(b) L20
0
PipeMax (c) H100
±5% band
1
70B
8×7B
0
Actual / Predicted
Longbench ShareGPT
vLLM(TP)
Prefetch-aware Decode Scheduler To evaluate the effectiveness of the prefetch-aware decode scheduler, we compare it against static prefetching baselines. These baselines
235B-A22B
2 1
70B
8×7B
0
235B-A22B
Figure 14. Normalized overall throughput (tokens/s) across workloads and GPU servers, where vLLM (TP) is normalized to 1.
(a) 5090 + 70B
1.10 1.05 1.00 0.95 0.90 0
25
50 Step
75
±10% band
(b) L20 + 70B
1.10 1.05 1.00 0.95 0.90 100
0
25
50 Step
75
100
Figure 15. Ratio of actual to predicted decode execution time over 100 consecutive steps.
7
(b) L20 + 70B
1.0
0.5
0.5
0.0
5
10
15
20
25
Prefetch Ratio (%)
PipeMax
0.0
5
10
15
20
25
Prefetch Ratio (%)
(b) KV Cache Usage
(a) Decode Execution Time Trend 1.00 0.75 0.50
Steady Phase
0.25 0.00
PipeMax
KV Cache Usage (Normalized)
(a) RTX 5090 + 70B
1.0
Execution Time (Normalized)
Normalized Throughput
PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers
0
2500
5000 Step
7500
10000
2.0
GPU Capacity
Total KV Cache
1.5 1.0
Prefetched KV Cache
0.5 0.0 0
GPU-Resident
1000 2000 3000 4000 5000 Step
Figure 18. Decode runtime dynamics of PipeMax.
Figure 16. Normalized throughput of PipeMax vs. static prefetching with fixed prefetch ratios.
4.4. Runtime Dynamics during Decode In this section, we show the runtime behavior of PipeMax on RTX 5090 with the 70B model using the ShareGPT dataset.
are implemented by replacing the original decode scheduler with static prefetching policies that prefetch a fixed fraction of available GPU memory, ranging from 5% to 25%, while keeping all other system components unchanged. As shown in Fig. 16, PipeMax consistently outperforms the static prefetching baselines. This result indicates that PipeMax can dynamically overlap KV cache prefetching with model execution, whereas static prefetching fails to fully utilize available overlap opportunities due to mismatches between prefetching decisions and actual execution progress, resulting in either insufficient prefetching or overly long prefetch operations that interrupt model execution.
Fig. 18(a) shows that decode execution time increases rapidly after the prefill-to-decode transition due to shortrequest prefetching, then converges as GPU memory and PCIe bandwidth become limiting factors (Section 3.3.2). In the steady phase, PipeMax balances batch workloads to prevent inter-batch imbalance. Fig. 18(b) shows the KV cache footprint per batch in this phase. With total GPU memory normalized to 1.0, prefetched KV cache occupies a substantial fraction of GPU memory, indicating PipeMax effectively expands usable KV cache capacity via prefetching.
4.3.2. P IPE M AX RUNTIME
5. Related Work
Block-First Layout We replace PipeMax’s block-first KV cache layout with a layer-first layout to evaluate the effectiveness of the block-first design. As shown in Fig. 17(a), the block-first layout consistently outperforms the layerfirst layout, as it enables contiguous memory allocation that improves prefetch efficiency and allows larger KV cache prefetching, leading to greater effective memory capacity.
LLM Inference. LLM inference has attracted growing attention, motivating extensive system-level optimizations. Orca (Yu et al., 2022) introduces continuous batching, while vLLM (Kwon et al., 2023) proposes PagedAttention for efficient KV cache management. Subsequent work optimizes LLM inference for both online and offline scenarios. For online serving, systems such as DistServe (Zhong et al., 2024), Sarathi-Serve (Agrawal et al., 2023), EcoServe (Du et al., 2025), and Bullet (Lin et al., 2025) mitigate prefill– decode interference to improve service quality. For offline inference, beyond the parallelism and offloading techniques studied here, BatchLLM (Zheng et al., 2024b) and BlendServe (Zhao et al., 2024) improve throughput via prefix sharing. Prefix sharing is orthogonal to PipeMax and can be seamlessly combined with our approach.
We further measure the PCIe bandwidth utilization of both layouts. The results show that under the block-first layout, with the block size set to the vLLM default of 16, KV cache prefetching can saturate nearly 90% of the available PCIe bandwidth. In contrast, the layer-first layout achieves only about 30% bandwidth utilization. Asynchronous Offloading To validate that asynchronous offloading in Section 3.4.2 can be hidden by computation, we measure the execution-time breakdown of computation and KV cache offloading.
Pipeline Parallelism Enhanced by Offloading. Pipeline parallelism and offloading both improve resource utilization via concurrent batch execution, making their integration natural. Prior work explored this combination for training, including Mobius (Feng et al., 2023), APT-MoE (Wei et al., 2024), and PipeOffload (Wan et al., 2025), which overlap pipeline execution with data transfers via stage, expert, or activation offloading.
(a) Layer-First vs PipeMax Layer-First
1.0
PipeMax
0.5
0.0
5090 + 70B
L20 + 70B
6. Conclusion This paper presents PipeMax, a high-throughput LLM inference system for commodity GPU servers. PipeMax boosts pipeline-parallel inference by offloading inactive KV cache and dynamically scheduling computation and KV cache movement to maximize compute–data overlap. Experiments show that PipeMax outperforms state-of-the-art LLM systems by up to 2.51×, 1.42×, and 1.38× on 8 GPUs.
(b) Asynchronous Offload Overlap Normalized Execution Time
Normalized Throughput
Fig. 17(b) reports two representative prefill cases with input lengths of 1 and 256 tokens on RTX 5090 with the 70B model. In both cases, KV cache offloading and CPU-side processing are fully overlapped with attention and FFN computation; intermediate input lengths show similar behavior but are omitted for brevity. During decode, different batch sizes (reflecting decode lengths) exhibit the same trend, with larger attention cost further masking offloading latency. 1.0
Attn
FFN
Offload
CPU
0.5
0.0
Length = 1
Length = 256
Figure 17. Runtime ablation of PipeMax: KV cache layout and offloading.
8
PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers
Impact Statement
Lin, Z., Xu, H., Chen, G., Zhang, X., and Lu, Y. Bullet: Boosting gpu utilization for llm serving via dynamic spatial-temporal orchestration. arXiv preprint arXiv:2504.19516, 2025.
This paper presents work whose goal is to advance the field of Large Language Model Inference Systems. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Liu, S., Biswal, A., Kamsetty, A., Cheng, A., Schroeder, L. G., Patel, L., Cao, S., Mo, X., Stoica, I., Gonzalez, J. E., et al. Optimizing llm queries in relational data analytics workloads. Proceedings of Machine Learning and Systems, 7, 2025.
References Agrawal, A., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., and Ramjee, R. Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills. arXiv preprint arXiv:2308.16369, 2023.
Nazi, Z. A. and Peng, W. Large language models in healthcare and medical domain: A review, 2024. URL https://arxiv.org/abs/2401.06775.
Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B., Tumanov, A., and Ramjee, R. Taming {Throughput-Latency} tradeoff in {LLM} inference with {Sarathi-Serve}. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 117–134, 2024.
Rane, N. L., Tawde, A., Choudhary, S. P., and Rane, J. Contribution and performance of chatgpt and other large language models (llm) for scientific and research advancements: a double-edged sword. International Research Journal of Modernization in Engineering Technology and Science, 5(10):875–899, 2023.
Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., et al. Longbench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), pp. 3119–3137, 2024.
ShareGPT. ShareGPT Datasets. https: //huggingface.co/datasets/ anon8231489123/ShareGPT_Vicuna_ unfiltered, 2025. Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Chen, B., Liang, P., Ré, C., Stoica, I., and Zhang, C. Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning, pp. 31094–31116. PMLR, 2023.
Du, J., Zhang, H., Wei, T., Zheng, Z., Wu, K., Chen, Z., and Lu, Y. Ecoserve: Enabling cost-effective llm serving with proactive intra-and inter-instance orchestration. arXiv preprint arXiv:2504.18154, 2025. Feng, Y., Xie, M., Tian, Z., Wang, S., Lu, Y., and Shu, J. Mobius: Fine tuning large-scale models on commodity gpu servers. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, pp. 489– 501, 2023.
Su, Q., Zhao, W., Li, X., Andoorveedu, M., Jiang, C., Zhu, Z., Song, K., Giannoula, C., and Pekhimenko, G. Seesaw: High-throughput llm inference via model re-sharding. arXiv preprint arXiv:2503.06433, 2025. Wan, X., Qi, P., Huang, G., Lin, M., and Li, J. Pipeoffload: Improving scalability of pipeline parallelism with memory optimization. arXiv preprint arXiv:2503.01328, 2025.
GitHub. Github copilot: Your ai pair programmer. https: //github.com/features/copilot, 2023. Hu, Y., Liu, X., Yang, G., Li, L., Zeng, K., Zhao, Z., Chen, S., Zhao, L., Li, W., and Li, K. Tightllm: Maximizing throughput for llm inference via adaptive offloading policy. IEEE Transactions on Computers, 2025.
Wei, Y., Du, J., Jiang, J., Shi, X., Zhang, X., Huang, D., Xiao, N., and Lu, Y. Aptmoe: Affinity-aware pipeline tuning for moe models on bandwidth-constrained gpu nodes. In SC24: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1–14. IEEE, 2024.
Jiang, J., Chen, Y., Zhang, Z., He, B., Luo, P., Lu, M., Chen, Y., Zhang, H., Du, J., Huang, D., et al. Efficient kv cache spillover management on memory-constrained gpu for llm inference. IEEE Transactions on Parallel and Distributed Systems, 37(1):90–105, 2025.
Xu, D., Chen, W., Peng, W., Zhang, C., Xu, T., Zhao, X., Wu, X., Zheng, Y., Wang, Y., and Chen, E. Large language models for generative information extraction: A survey. Frontiers of Computer Science, 18(6):186357, 2024.
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pp. 611–626, 2023. 9
PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers
Yu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S., and Chun, B.G. Orca: A distributed serving system for TransformerBased generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp. 521–538, Carlsbad, CA, July 2022. USENIX Association. ISBN 978-1-939133-28-1. URL https://www.usenix.org/conference/ osdi22/presentation/yu.
Zheng, L., Yin, L., Xie, Z., Sun, C. L., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., et al. Sglang: Efficient execution of structured language model programs. Advances in neural information processing systems, 37:62557–62583, 2024a. Zheng, Z., Ji, X., Fang, T., Zhou, F., Liu, C., and Peng, G. Batchllm: Optimizing large batched llm inference with global prefix sharing and throughput-oriented token batching. arXiv preprint arXiv:2412.03594, 2024b.
Zhang, H., Wei, T., Zheng, Z., Du, J., Chen, Z., and Lu, Y. Td-pipe: Temporally-disaggregated pipeline parallelism architecture for high-throughput llm inference. In Proceedings of the 54th International Conference on Parallel Processing, pp. 689–698, 2025.
Zhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X., and Zhang, H. {DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 193–210, 2024.
Zhao, Y., Yang, S., Zhu, K., Zheng, L., Kasikci, B., Zhou, Y., Xing, J., and Stoica, I. Blendserve: Optimizing offline inference for auto-regressive large models with resourceaware batching. arXiv preprint arXiv:2411.16102, 2024.
10
PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers
A. Appendix A.1. Pipeline Parallelism for the Prefill Stage In this section, we present a proof for the total execution-time formula of the pure prefill pipeline shown in Fig. 4 (also Stated in Section 2.3.1). Setup. Consider m independent prefill requests executed on an n-stage pipeline (i.e., n workers/GPUs). Let ti denote the per-stage execution time of request i during prefill, assuming a homogeneous pipeline where all stages have identical execution time for a given request. 2 Let T (m, n) denote the total makespan to process all m requests on the n-stage pipeline. Assumption (No bubble at Stage 1). We assume the first stage is fully utilized (i.e., no idle bubble at Stage 1). This can be Ensured by continuously dispatching ready requests whose dependencies (if any) have been resolved. Theorem A.1 (Prefill pipeline execution time). Under the above assumption, the total execution time satisfies T (m, n) =
m X
ti + (n − 1) · max ti .
(10)
ti + (n − 1) · max ti .
(11)
1≤i≤m
i=1
Proof. Define f (m, n) ≜
m X
1≤i≤m
i=1
Our goal is to show that the total execution time satisfies T (m, n) = f (m, n),
(12)
for all m > 0 and n > 0. We prove the theorem by induction. Base cases.
When n = 1, the pipeline degenerates to sequential execution: T (m, 1) =
m X
ti = f (m, 1).
(13)
T (1, n) = n · t1 = f (1, n).
(14)
i=1
When m = 1, the single request traverses all n stages:
Inductive step.
Given the base cases, we assume that the equality T (i, j) = f (i, j),
∀ 1 ≤ i ≤ m, 1 ≤ j ≤ n.
(15)
It suffices to show that the equality is preserved when extending the pipeline by one stage or one request, i.e., T (m, n + 1) = f (m, n + 1)
and
T (m + 1, n) = f (m + 1, n).
(16)
We first prove that T (m, n + 1) = f (m, n + 1). Fix the number of pipeline stages to (n + 1) and consider increasing the number of requests. The equality holds for a single request, since T (1, n + 1) = (n + 1) · t1 = f (1, n + 1). (17) 2
Prefill has no cross-request data dependency; thus, the first stage can be kept continuously busy by scheduling ready requests.
11
PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers
It therefore suffices to show that, for any k ∈ [1, m − 1], T (k, n + 1) = f (k, n + 1) =⇒ T (k + 1, n + 1) = f (k + 1, n + 1).
(18)
Once this implication is established, T (m, n + 1) = f (m, n + 1) follows directly. Consider the pipeline dependency for processing the (k + 1)-th request on an (n + 1)-stage pipeline. Let T (k, n + 1) denote the time when the k-th request finishes stage (n + 1), and T (k + 1, n) denote the time when request (k + 1) finishes stage n. Due to pipeline precedence constraints, the execution of request (k+1) at stage (n+1) is governed by two dependencies, as illustrated in Fig. 19(a) and (b). …… …… …… …… Stage 𝒏 …… Stage 𝒏 + 𝟏 ……
…… …… …… …… Stage 𝒏 …… Stage 𝒏 + 𝟏 ……
𝒌+𝟏 𝒌+𝟏 𝒌+𝟏 𝒌+𝟏 𝑻(𝒌, 𝒏 + 𝟏)
Time
𝒌+𝟏 𝒌+𝟏 𝒌+𝟏 𝒌+𝟏
𝑻 𝒌, 𝒏 + 𝟏 + 𝒕𝒌"𝟏
𝑻(𝒌 + 𝟏, 𝒏)
(a) Case 1: request-order dependency.
Time
𝑻(𝒌 + 𝟏, 𝒏) + 𝒕𝒌"𝟏
(b) Case 2: stage-order dependency.
Figure 19. Precedence constraints when extending the pipeline by one additional request, which determine the start time of request (k + 1) at stage (n + 1).
Combining the above two cases, the completion time of request (k+1) at stage (n+1) is determined by the later of the two dependencies, and thus satisfies T (k+1, n+1) = max T (k, n+1), T (k+1, n) + tk+1 . (19) Define Mk ≜ max1≤i≤k ti . Case 1: tk+1 ≤ Mk . In this case, Mk+1 = Mk . Using the induction hypothesis, T (k, n + 1) =
k X
ti + n · Mk ,
(20)
i=1
T (k + 1, n) =
k+1 X
ti + (n − 1) · Mk+1 =
i=1
k+1 X
ti + (n − 1) · Mk .
(21)
i=1
Thus, T (k, n + 1) − T (k + 1, n) = nMk − tk+1 + (n − 1)Mk = Mk − tk+1
(22)
≥ 0. so max{T (k, n + 1), T (k + 1, n)} = T (k, n + 1). Plugging into (19), T (k + 1, n + 1) = T (k, n + 1) + tk+1 =
k+1 X
ti + n · Mk+1
i=1
= f (k + 1, n + 1). Case 2: tk+1 > Mk . 12
(23)
PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers
In this case, Mk+1 = tk+1 . Using the induction hypothesis, T (k, n + 1) =
k X
ti + n · Mk ,
(24)
i=1
T (k + 1, n) =
k+1 X
ti + (n − 1) · Mk+1 =
i=1
k+1 X
ti + (n − 1) · tk+1 .
(25)
i=1
Thus, T (k + 1, n) − T (k, n + 1) =
k+1 X
k X ti + (n − 1)tk+1 − ti + nMk
i=1
i=1
(26)
= ntk+1 − nMk = n(tk+1 − Mk ) > 0. Therefore, max{T (k, n + 1), T (k + 1, n)} = T (k + 1, n). Plugging into (19), T (k + 1, n + 1) = T (k + 1, n) + tk+1 =
k+1 X
ti + n · tk+1
i=1
=
k+1 X
ti + n · Mk+1
i=1
= f (k + 1, n + 1).
(27)
In both cases, the implication in (18) holds, and therefore T (m, n + 1) = f (m, n + 1) follows. We next prove that T (m + 1, n) = f (m + 1, n). Similar to the previous case, we fix the number of requests to (m + 1) and consider increasing the number of pipeline stages. The equality holds for a single stage, since T (m + 1, 1) =
m+1 X
ti = f (m + 1, 1).
(28)
i=1
It therefore suffices to show that, for any k ∈ [1, n − 1], T (m + 1, k) = f (m + 1, k) =⇒ T (m + 1, k + 1) = f (m + 1, k + 1).
(29)
Consider processing request (m + 1) on a (k + 1)-stage pipeline. Let T (m, k + 1) denote the time when request m finishes stage (k + 1), and let T (m + 1, k) denote the time when request (m + 1) finishes stage k. By pipeline precedence constraints (illustrated in Fig. 20), request (m + 1) can start stage (k + 1) only after both dependencies are resolved. Therefore, its completion time satisfies T (m + 1, k + 1) = max T (m, k + 1), T (m + 1, k) + tm+1 . (30)
…… Stage 𝒌 Stage k+𝟏
…… …… …… …… ……
𝒎+𝟏
……
𝒎+𝟏
Stage 𝒌
𝒎+𝟏 𝒎+𝟏 𝑻(𝒎, 𝒌 + 𝟏)
Stage 𝒌 + 𝟏
Time
𝑻 𝒎, 𝒌 + 𝟏 + 𝒕𝒎"𝟏
…… …… …… …… ……
𝒎+𝟏 𝒎+𝟏 𝒎+𝟏 𝒎+𝟏 𝑻(𝒎 + 𝟏, 𝒌)
(a) Case 1: request-order dependency.
Time
𝑻(𝒎 + 𝟏, 𝒌) + 𝒕𝒎"𝟏
(b) Case 2: stage-order dependency.
Figure 20. Precedence constraints for request (m + 1) when extending the pipeline by one additional stage, which determine its start time at stage (k + 1).
13
PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers
The dependency constraints in (30) are structurally identical to those in (19), with the roles of requests and pipeline stages exchanged. Accordingly, the same case analysis applies here. By distinguishing whether tm+1 is no larger than or exceeds max1≤i≤m ti , we can show that T (m + 1, k + 1) = f (m + 1, k + 1).
(31)
This establishes the implication in (29), and therefore T (m + 1, n) = f (m + 1, n) holds. Conclusion Since the equality holds for the base cases and is preserved when extending either the number of requests or the number of pipeline stages, we conclude that T (m, n) = f (m, n) holds for all m > 0 and n > 0.
Implication. The theorem implies that, once the first stage is kept bubble-free, the overall prefill makespan is dominated by (i) the cumulative work injected into Stage 1 and (ii) a fixed drain cost of (n − 1) maxi ti . When m ≫ n, the drain term becomes amortized, and the total time is effectively governed by Stage 1 throughput. Therefore, optimizing prefill reduces to keeping Stage 1 continuously saturated, which naturally connects to our batch construction and KV-cache memory management design. A.2. Pipeline Parallelism for the Decode Stage In this section, we derive the average token budget per decode batch under pipeline parallelism. Let M denote the per-GPU memory capacity, W the total model weight size, n the pipeline degree, and T the KV Cache size per token. Since the model weights are evenly partitioned across pipeline stages, the per-GPU weight footprint is Wgpu =
W . n
(32)
Accordingly, the memory available for KV Cache on each GPU is Mkv = M − Wgpu .
(33)
Since the KV Cache of each token is sharded across all n GPUs, the per-token KV Cache footprint on each GPU is Tgpu =
T . n
(34)
Maintaining full pipeline utilization during decode requires n concurrent batches to remain resident in GPU memory. Let max Ntoken denote the system-wide maximum number of storable tokens, which is given by Mkv Tgpu M − W/n = T /n nM − W = . T
max Ntoken =
(35)
Dividing this system-wide token capacity evenly across the n resident decode batches yields the average token budget per batch: N max Navg = token n (36) M − W/n = . T 14
PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers
A.3. Scheduler Algorithm Here we present the detailed scheduling procedure and the greedy algorithm for selecting Pt . A.3.1. S CHEDULING P ROCEDURE The scheduling procedure described in Section 3.3.2 is summarized in Algorithm 1. Algorithm 1 Prefetch-Aware Decode Scheduler Require: Decode request set R, pipeline depth n Ensure: Iterative decode batches D = {D0 , . . . , Dn−1 } 1: Initialization: 2: Partition R into n initial decode batches D 3: Initialize iteration counter t ← 0 4: Initialize steady-phase flag steady ← false 5: Iterative Scheduling: 6: while decode not finished do 7: it ← t mod n 8: jt ← (it + 1) mod n 9: kt ← (it − 1) mod n 10: Predict the execution time T̂t of batch Dit 11: Derive the prefetch budget Bt ← B · T̂t 12: Update steady-phase flag steady by monitoring whether Bt has stabilized 13: Retain GPU-resident requests in the next batch: 14: Djres ← Djt ∩ Rt t 15: if not steady then 16: // Ramp-up phase: prioritize short requests to fill the budget 17: Pt ← S HORT F IRST F ILL(Rcpu t , Bt ) 18: else 19: // Steady phase: match execution-time gap 20: Pt ← G REEDY S ELECT(Rcpu t , Bt , ∆T̂t ) 21: end if 22: Prefetch the KV cache of requests in Pt from CPU to GPU 23: Reclaim GPU memory by overwriting KV cache blocks of the inactive batch Dkt 24: Update the next decode batch: 25: Djt ← Djres ∪ Pt t 26: Submit batch Dit for execution and batch Djt for prefetching to the PipeMax runtime 27: t←t+1 28: end while
The scheduler operates in an iterative manner and adapts its behavior across iterations based on the evolution of the prefetch budget Bt . At each iteration, PipeMax first predicts the execution time of the currently executing batch and derives the corresponding prefetch budget. Requests whose KV cache already resides in GPU memory are retained in the next batch, while additional CPU-resident requests are selected for prefetching to fully utilize the available budget. PipeMax distinguishes the warm-up phase from the steady phase by tracking the stabilization of the prefetch budget. During the warm-up phase, the scheduler prioritizes short requests with smaller prefix lengths to best-effort utilize the limited prefetch budget and rapidly increase decode execution time. As longer requests are gradually admitted, execution time and the prefetch budget may exhibit temporary fluctuations due to GPU memory and CPU–GPU bandwidth constraints. Eventually, the system converges to a bounded execution regime, after which PipeMax enters the steady phase. Once the system reaches a steady phase, the scheduler switches to a prefetch-aware selection policy that matches the execution-time contribution of prefetched requests to the remaining time gap, thereby balancing decode batches and avoiding 15
PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers
inter-batch imbalance. In practice, PipeMax detects the onset of the steady phase by tracking the evolution of the prefetch budget Bt over a sliding window of recent iterations. At the beginning of decoding, Bt typically grows rapidly as execution time ramps up. As the system transitions into the steady phase, Bt stabilizes and fluctuates within a narrow range. Specifically, PipeMax maintains a sliding window of the most recent w iterations and considers the system to have entered the steady phase when the relative variation of Bt within the window falls below a predefined threshold. A.3.2. G REEDY A LGORITHM FOR S ELECTING Pt After PipeMax reaches a steady phase, it strives to maintain stable execution times across iterations to reduce inter-batch imbalance.
Problem Formulation. At iteration t, PipeMax selects a set of CPU-resident requests Pt to augment the next decode batch Djt . Each request r is associated with a prefix length Lr and contributes an execution-time cost α + βLr according to Eq. (2). Given the prefetch budget Bt = B · T̂t and the remaining execution-time gap ∆T̂t defined in Eq. (8), the goal is to select a subset Pt such that X
Lr ≈ Bt ,
r∈Pt
i.e., the selected requests aim to nearly saturate the prefetch budget without exceeding it, while making the total executiontime contribution X
(α + βLr )
r∈Pt
as close as possible to ∆T̂t .
Greedy Algorithm To select the set of CPU-resident requests Pt , PipeMax adopts a two-stage heuristic that combines greedy selection with exchange-based local refinement. The greedy stage prioritizes length utilization by selecting requests in a length-first manner, rapidly saturating the prefetch budget P to obtain a near-feasible initial solution. Under the execution-time model, this effectively drives the length-dependent term β r∈Pt Lr toward its budget-limited maximum. Building on this initial solution, the refinement stage performs limited exchange operations that replace a small number of selected requests with unselected ones, adjusting P the batch cardinality while preserving the budget constraint. Under the length budget, the cumulative length term β r∈Pt Lr remains largely fixed after greedy initialization. As a result, the refinement primarily adjusts the constant per-request component α|Pt | of the execution-time model. These local exchanges are guided by the execution-time model and aim to minimize the mismatch between the modeled batch execution time α|Pt | + β
X
Lr
r∈Pt
and the remaining execution-time gap ∆T̂t . Concretely, when the modeled execution time falls short of the target, the refinement replaces longer requests with multiple shorter ones of comparable total length, increasing the α|Pt | term. Conversely, when the modeled execution time exceeds the target, multiple shorter requests are replaced by a longer one to reduce the α|Pt | contribution, while keeping the total prefix length approximately unchanged. This exchange process continues until the mismatch falls below a predefined threshold or a maximum number of refinement steps is reached. 16
PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers
Algorithm 2 Greedy Selection with Local Refinement Require: CPU-resident requests Rcpu with prefix length Lr t Require: Prefetch budget Bt , target gap ∆T̂t Ensure: Selected subset Pt 1: Greedy Initialization: cpu 2: Sort Rt in descending order of Lr 3: Pt ← ∅, S ← 0 cpu 4: for each request r ∈ Rt do 5: if S + Lr ≤ Bt then 6: Pt ← Pt ∪ {r} 7: S ← S + Lr 8: end if 9: end for 10: Local Refinement: 11: for a fixed number of refinement steps do 12: T ← α|Pt | + βS 13: if |T − ∆T̂t | is below a threshold then 14: break 15: end if 16: if T < ∆T̂t then 17: Replace one long selected request with multiple shorter unselected ones 18: to increase |Pt | while keeping S ≈ Bt 19: else if T > ∆T̂t then 20: Replace multiple short selected requests with one longer unselected one 21: to decrease |Pt | while keeping S ≈ Bt 22: end if 23: end for 24: return Pt
17
▷ S: cumulative prefix length