P-PAS: Prefill-Pressure Adaptive Scheduling for Long-Context LLM Serving

P-PAS: Prefill-Pressure Adaptive Scheduling for Long-Context LLM Serving Timo Sämann Independent Researcher

arXiv:2608.15171v1 [cs.DC] 15 Aug 2026

Abstract Long-context LLM applications such as retrieval-augmented generation (RAG) and agentic systems often process tens of thousands of input tokens to produce short outputs, making end-to-end request latency an important serving objective. We show that the maximum number of batched tokens (MBT), which controls the token scheduling budget in vLLM, has a scheduling-pressure-dependent effect on latency. Larger token budgets can reduce latency under low scheduling pressure, while smaller budgets become preferable under higher pressure. Consequently, no single static MBT performs best across load regimes. We introduce Prefill-Pressure Adaptive Scheduling (P-PAS), a lightweight policy that dynamically adapts the scheduling budget based on concurrent prefill and decode state. PPAS retains a large token budget under low pressure and constrains prefill work as pressure increases. Across models, workloads, and GPUs, P-PAS maintains low end-to-end latency across changing load regimes, avoiding the limitations of a fixed MBT. Kernel-level profiling shows that large prefill chunks can improve execution efficiency under low scheduling pressure, but that this advantage varies across model–hardware configurations. As scheduling pressure increases, smaller chunks can instead reduce interference with active decoding, explaining the observed load-dependent MBT sensitivity. Code and artifacts for reproducing our results are available at https://github.com/TimoSaemann/ppas-vllm.

1

Introduction

Large language model (LLM) serving increasingly extends beyond interactive chat toward applications that consume large amounts of context while producing comparatively short outputs. Retrieval-augmented generation (RAG) [1], document analysis, and agentic systems can provide an LLM with retrieved documents, tool outputs, execution traces, or long conversation histories while requiring only a short answer or action in return [2]. We focus on such long-context, short-output workloads under bursty request arrivals, as illustrated in Figure 1(a). In agentic workflows in particular, downstream actions often depend on the completed LLM response [3], motivating our focus on end-to-end request latency. Serving engines for autoregressive LLMs interleave prefill and decode work on the GPU, making their interaction important for end-to-end latency [4]. vLLM [5], a widely used LLM serving engine, uses continuous batching and chunked prefill to manage concurrent requests. A key scheduling parameter is the maximum number of batched tokens (MBT), which limits the token budget available within one scheduler iteration. Larger MBTs allow more prefill tokens to be processed together, whereas smaller MBTs divide long prefills into smaller chunks. We show that the latency-optimal MBT depends on scheduling pressure. Under low pressure, larger token budgets can reduce end-to-end latency through more efficient prefill execution, while under high pressure, smaller budgets become preferable by limiting interference with active decoding. Consequently, no single static MBT performs best across load regimes. Figure 1(b) illustrates this behavior and motivates adaptive control of the scheduling budget. 1

Figure 1: Motivation and overview of P-PAS. (a) We target bursty, long-context workloads with 20k–30k-token prompts and short 16–64-token outputs, evaluated across multiple model architectures and sizes, as well as consumer- and datacenter-class GPUs. (b) The latency-optimal maximum batched tokens (MBT) depends on scheduling pressure: large budgets can perform better under low pressure, while smaller budgets become preferable as pressure increases. (c) P-PAS adapts the scheduling budget based on prefill pressure and active decode state; the figure shows the evaluated configuration with a prefill-pressure threshold of two and budgets of 16k and 2k. To adapt to this load-dependent behavior, we introduce Prefill-Pressure Adaptive Scheduling (P-PAS), a lightweight extension to the vLLM scheduler. As illustrated in Figure 1(c), P-PAS dynamically adapts the scheduling budget based on prefill pressure and decode state. It retains a large budget under low pressure and constrains the budget when concurrent prefills compete with active decoding. P-PAS operates directly on scheduler state and requires neither offline workload prediction nor a predefined request-rate threshold. We evaluate P-PAS primarily using Qwen2.5-3B on an NVIDIA RTX 5090 with 25k-token prompts and 32-token outputs, and further assess its robustness across model architectures and scales, prompt lengths from 20k to 30k tokens, output lengths from 16 to 64 tokens, and consumer- and datacenter-class GPUs. Across these configurations, P-PAS maintains low endto-end latency as scheduling pressure changes, avoiding the need to select a single fixed MBT for different load regimes. Finally, we use GPU profiling to investigate the source of MBT sensitivity. For Qwen2.5, larger prefill chunks can reduce the cumulative execution time of dominant attention and MLP kernels compared with processing the same prompt through multiple smaller chunks. This prefillefficiency advantage explains why large MBTs can reduce latency under low pressure. As pressure increases, however, larger prefill chunks interfere more strongly with active decoding, shifting the latency-optimal configuration toward smaller budgets. We further find that the large-chunk efficiency advantage can diminish for some model–hardware configurations, resulting in weaker MBT sensitivity. In summary, this work makes the following contributions: • We characterize MBT sensitivity in long-context, short-output LLM serving and identify a load-dependent crossover in which the latency-optimal token budget changes with scheduling pressure. • We introduce P-PAS, a lightweight scheduling policy that dynamically adapts the token budget based on prefill pressure and active decode state, avoiding a single static MBT choice. • We evaluate P-PAS across model architectures and scales, workload configurations, and consumer- and datacenter-class GPUs, showing that P-PAS maintains low end-to2

end latency across changing load regimes. • We provide a kernel-level analysis of MBT sensitivity, showing how prefill execution efficiency changes with chunk size and why the resulting MBT sensitivity varies across model– hardware configurations.

2

Background and Motivation: MBT Sensitivity

Autoregressive LLM inference consists of two phases. During prefill, the model processes the input tokens and stores the resulting keys and values in the KV cache. During decode, output tokens are generated autoregressively. In vLLM, continuous batching allows prefill and decode work from different requests to be scheduled together within an iteration. For long prompts, chunked prefill divides the prefill into smaller chunks that can be spread across multiple iterations. In vLLM, the maximum number of batched tokens (MBT) limits the total number of tokens that can be scheduled in one iteration. This budget is shared by prefill and decode work. A large MBT allows more prefill tokens to be processed at once, reducing the number of chunks required for long prompts, whereas a small MBT divides the prefill across more scheduling iterations. This creates a trade-off for long-context workloads. Larger token budgets can process long prefills efficiently, whereas smaller budgets provide more frequent opportunities for active decode requests to make progress. The latency-optimal MBT therefore depends on the balance between prefill efficiency and decode interference. Figure 1(b) demonstrates this behavior for Qwen2.5-3B on an RTX 5090 with 25k-token prompts and 32-token outputs. At low arrival rates, MBT 16k achieves lower end-to-end latency than MBT 2k. As the arrival rate increases, the ordering reverses: MBT 2k becomes preferable while the latency of MBT 16k increases more rapidly. We refer to this dependence of request latency on the scheduling budget as MBT sensitivity. Bursty workloads can move between these regimes during a single serving run, making either static choice suboptimal for part of the workload. This observation motivates dynamically adapting the scheduling budget to the current prefill pressure, which we introduce next.

3

Prefill-Pressure Adaptive Scheduling (P-PAS)

The MBT sensitivity observed in Section 2 suggests that the scheduling budget should depend on the current execution state. P-PAS follows a simple principle: retain a large prefill token budget when pressure is low, but reduce the budget when multiple prefills compete with active decoding. This preserves the efficiency of large prefill chunks when possible while limiting interference with active decoding as pressure increases. P-PAS derives its decision directly from the scheduler state. At each scheduling iteration, it considers running and waiting prefills together with running decode requests. Let Np denote the number of running and waiting prefills and Nd the number of running decode requests. P-PAS detects prefill pressure when Np reaches a threshold Nth while at least one decode request is active. P-PAS uses two prefill token budgets: Bmax under low pressure and a smaller cap Bcap under pressure. The global vLLM token budget remains unchanged; P-PAS introduces an additional constraint on the aggregate prefill tokens scheduled within an iteration. In our evaluation, we use Nth = 2, Bmax = 16384, and Bcap = 2048. These values define the evaluated P-PAS 3

Algorithm 1 Prefill-Pressure Adaptive Scheduling (P-PAS) running

waiting + Nprefill 2: Nd ← number of running decode requests 3: if Np ≥ Nth ∧ Nd > 0 then 4: Bprefill ← Bcap 5: else 6: Bprefill ← Bmax 7: end if 8: Limit aggregate prefill tokens to Bprefill

1: Np ← Nprefill

Figure 2: Bursty arrival workload used in our experiments. The arrival rate alternates between steady-load phases (λ = 0.2 requests/s) and higher-rate burst phases. Requests are sampled from a Poisson process within each phase. The example shows a burst rate of λ = 4.0 requests/s; the y-axis is broken for readability. configuration rather than the scheduling policy itself. P-PAS uses only information already available to the scheduler and requires no additional model execution or profiling. Since the decision is reevaluated every iteration, P-PAS can return to the larger prefill budget as soon as pressure subsides. Algorithm 1 summarizes the policy.

4

Experimental Setup

4.1

Workload

We evaluate long-context, short-output serving workloads representative of RAG and agentic applications. Unless stated otherwise, requests contain 25k prompt tokens and generate 32 output tokens. To evaluate scheduler behavior under changing load, we use 50 s traces that alternate between steady-load and burst phases. Request arrivals within each phase follow a Poisson process with the corresponding arrival rate. As illustrated in Figure 2, each trace consists of 10 s phases, with a steady arrival rate of 0.2 requests/s and alternating burst phases whose arrival rate is varied to control load. All experiments use this same temporal structure.

4

4.2

Models and Hardware

Our primary experiments use Qwen2.5-3B on an NVIDIA RTX 5090 (32 GB) with 25k-token prompts and 32-token outputs. We additionally evaluate Qwen2.5-0.5B and SmolLM3-3B, vary prompt lengths from 20k to 30k tokens and output lengths from 16 to 64 tokens, and perform experiments on NVIDIA A100 SXM (80 GB) and H100 SXM (80 GB) GPUs. The upper prompt length of 30k was chosen close to the 32,768-token context limit of Qwen2.5-3B, while the 20k–30k range provides multiple operating points within the long-context regime. These variations test the robustness of MBT sensitivity and P-PAS across model architectures and scales, workload shapes, and consumer- and datacenter-class GPUs.

4.3

Baselines and P-PAS Configuration

We compare P-PAS against fixed MBT configurations of 2,048 and 16,384 tokens, representing the small- and large-budget regimes identified in Section 2. For P-PAS, we use Nth = 2, Bmax = 16k, and Bcap = 2k throughout the evaluation. These configurations are held fixed across models, workloads, and GPUs.

4.4

Metrics

Our primary metric is end-to-end request latency, measured from request arrival to completion of the generated response. We additionally report time to first token (TTFT) and time per output token (TPOT) to characterize prefill and decode behavior, respectively. Makespan measures the elapsed time from the first request arrival until completion of the final request. For request-level metrics, we report both the mean and the 95th percentile (P95), where applicable. Results are averaged over five random seeds.

5

Evaluation

5.1

P-PAS under Changing Load

We first evaluate P-PAS under changing load using our primary configuration: Qwen2.5-3B on the RTX 5090 with 25k-token prompts and 32-token outputs. We vary the burst arrival rate from 0.4 to 1.4 requests/s while keeping the steady arrival rate fixed at 0.2 requests/s. Figure 3 compares P-PAS with fixed MBT configurations across the full range of burst rates. Detailed per-burst results for all fixed MBT configurations and P-PAS are provided in Appendix Table 7. At low load, MBT 16k provides lower end-to-end latency than MBT 2k, consistent with the MBT sensitivity identified in Section 2. As load increases, the ordering reverses and MBT 2k becomes preferable. P-PAS adapts across these regimes: it closely matches MBT 16k at low load, outperforms both MBT 2k and MBT 16k across intermediate burst rates, and remains close to MBT 2k at the highest load. The additional static configurations (MBT 1k, 4k, and 8k) exhibit the same limitation: none provides the lowest latency consistently across load regimes. To summarize performance across all load regimes, we compare P-PAS with each fixed-MBT baseline separately at every burst rate. For each metric, we first compute the ratio between P-PAS and the baseline; a ratio below one indicates that P-PAS performs better. We then aggregate these ratios across the K burst rates using the geometric mean:

5

Figure 3: Average end-to-end latency across burst arrival rates for Qwen2.5-3B on an RTX 5090 with 25k-token prompts and 32-token outputs. Fixed MBT configurations favor different load regimes, whereas P-PAS adapts the prefill token budget and maintains low latency as load changes. The inset highlights the low-load regime. Table 1: P-PAS improvement over fixed MBT schedulers for Qwen2.5-3B on the RTX 5090. Values are geometric-mean improvements of normalized ratios across burst rates 0.4–1.4 requests/s. Positive values indicate lower values and better performance. Metric Avg. latency P95 latency Avg. TTFT Avg. TPOT Makespan

Rm =

vs. MBT 2k

vs. MBT 16k

8.5% 3.0% 11.8% 3.4% 1.0%

11.3% 10.0% −18.5% 36.3% −3.9%

K Y mP-PAS,k k=1

!1/K

mbaseline,k

,

(1)

where m denotes the evaluated metric. We express the resulting ratio as a percentage improvement: Im = (1 − Rm ) × 100%.

(2)

Thus, positive values indicate an improvement over the fixed-MBT baseline. Table 1 summarizes the results across all six burst rates. Across the full load range, P-PAS reduces average end-to-end latency by 8.5% relative to MBT 2k and by 11.3% relative to MBT 16k. The breakdown also reflects the underlying scheduling trade-off: relative to MBT 16k, P-PAS increases TTFT but substantially reduces TPOT, while relative to MBT 2k it improves both. Makespan improves slightly relative to MBT 2k and increases slightly relative to MBT 16k, consistent with the higher throughput generally achieved by larger token budgets.

6

Table 2: Average-latency improvement of P-PAS across models on the RTX 5090. Values are geometric-mean improvements across the indicated representative low- and high-pressure burst rates. Positive values indicate lower latency. Model Qwen2.5-0.5B Qwen2.5-3B SmolLM3-3B

5.2

Burst rates

vs. MBT 2k

vs. MBT 16k

1.0, 4.0 0.4, 1.0 0.4, 1.0

7.1% 12.2% 13.9%

14.2% 9.0% 9.0%

Robustness across Models, Workloads, and GPUs

We next test whether the observed behavior is specific to the primary Qwen2.5-3B configuration. We vary model architecture and scale, prompt and output length, and GPU platform while keeping the P-PAS policy and its parameters unchanged. To limit the number of experimental runs, the robustness experiments use representative lowand high-pressure burst rates rather than the full six-rate sweep of the primary evaluation. Aggregate improvements in this section are therefore computed over the burst rates reported for each experiment and can differ from the values obtained from the full sweep above. Model architecture and scale. We evaluate Qwen2.5-0.5B, Qwen2.5-3B, and SmolLM33B on the RTX 5090. Because Qwen2.5-0.5B processes requests substantially faster, higher burst rates are required to expose comparable low- and high-pressure regimes. Table 2 reports the geometric-mean improvement in average end-to-end latency over the selected burst rates; the corresponding absolute results are provided in Appendix Table 11. P-PAS reduces average latency relative to both fixed MBT baselines for all three models. The Qwen2.5-0.5B results further show that the benefit is not restricted to models at the 3B scale. Prompt and output length. We next vary the workload shape for Qwen2.5-3B. As shown in Table 3, P-PAS reduces average latency relative to both fixed MBT baselines across all evaluated prompt and output lengths. The magnitude of the improvement varies with workload shape, reaching 23.2% relative to MBT 16k for 16-token outputs. P95 latency follows the same general trend, with one exception: for 30k-token prompts, P-PAS is 2.7% worse than MBT 2k while remaining 7.1% better than MBT 16k. The results further show that the benefit of P-PAS is not restricted to the 25k-token prompt and 32-token output configuration used in our primary experiments. GPU platform. We next evaluate P-PAS on an NVIDIA A100 SXM to determine whether its benefits extend from the consumer-class RTX 5090 to datacenter hardware. Table 4 shows that P-PAS reduces average end-to-end latency relative to both fixed MBT baselines for both evaluated models. For Qwen2.5-3B, P-PAS improves average latency by 7.8% over MBT 2k and 6.6% over MBT 16k. For SmolLM3-3B, the corresponding improvements are 10.2% and 5.4%. These results show that the benefits observed on the Blackwell-based RTX 5090 persist on the earlier-generation Ampere-based A100 datacenter GPU. Absolute per-burst measurements for both models on the A100 SXM are provided in Appendix Table 9. Overall, these experiments show that the benefits of P-PAS persist across model architectures and scales, workload shapes, and both consumer- and datacenter-class GPUs. The magnitude of the improvement varies across configurations, consistent with MBT sensitivity depending on the interaction between model, workload, and hardware. We additionally verify that the 7

Table 3: Robustness of P-PAS across prompt and output lengths for Qwen2.5-3B. Values are geometric-mean improvements across burst rates 0.8 and 1.2. Positive values indicate lower latency. Experiment

Value

vs. MBT 2k

vs. MBT 16k

Avg. Lat.

P95

Avg. Lat.

P95

Prompt

20k 25k 30k

13.2% 7.0% 6.3%

8.4% 1.4% -2.7%

11.6% 13.9% 15.2%

11.5% 14.4% 7.1%

Output

16 32 64

5.9% 7.0% 8.4%

1.3% 1.4% 5.4%

23.2% 13.9% 1.8%

20.1% 14.4% 3.4%

Table 4: P-PAS performance on the NVIDIA A100 SXM. Values are geometric-mean improvements over the corresponding fixed-MBT baseline across representative low- and high-pressure burst rates 0.4 and 1.0. Positive values indicate lower values and better performance. Model

Metric

vs. MBT 2k

vs. MBT 16k

Qwen2.5-3B

Avg. latency P95 latency Avg. TTFT Avg. TPOT Makespan

7.8% 7.0% 7.8% 8.2% 0.4%

6.6% 14.8% −5.6% 13.6% −1.6%

SmolLM3-3B

Avg. latency P95 latency Avg. TTFT Avg. TPOT Makespan

10.2% 10.6% 10.6% 9.7% 0.7%

5.4% 7.3% −11.7% 20.8% −1.5%

underlying load-dependent MBT crossover occurs on an H100 SXM; detailed results are provided in Appendix Table 10.

6

Understanding MBT Sensitivity

We next investigate the mechanisms underlying MBT sensitivity using GPU profiling with NVIDIA Nsight Systems and Nsight Compute. We first analyze why large token budgets can improve prefill execution under low scheduling pressure and why this advantage varies across serving configurations (Section 6.1). We then examine why smaller token budgets become preferable as scheduling pressure increases (Section 6.2).

6.1

Large-Chunk Prefill Efficiency and Its Limits

Under low scheduling pressure, there is little concurrent decode work for large prefill chunks to interfere with. In this regime, the latency difference between MBT configurations is therefore strongly influenced by the efficiency with which the prefill itself is executed. Figure 4 illustrates this effect for one Qwen2.5-3B decoder block using kernel execution times measured with Nsight Systems. We compare processing the same number of prompt tokens 8

(a) Attention

(b) MLP

Figure 4: Kernel-level comparison of small- and large-chunk prefill execution for one Qwen2.53B decoder block. Processing the same 16,384-token prompt through eight 2k chunks incurs a higher cumulative execution time for dominant attention and MLP operations than processing it as a single 16k chunk. Table 5: Total prefill execution time for Qwen2.5 models on the RTX 5090, measured with Nsight Systems. The final column reports the reduction when using MBT 16k instead of MBT 2k. Model Qwen2.5-0.5B Qwen2.5-3B Qwen2.5-7B

MBT 2k [ms]

MBT 16k [ms]

Reduction

148.0 755.8 1373

132.4 668.6 1365

10.5% 11.6% 0.6%

using multiple 2k chunks with processing them as a single 16k chunk. The figure highlights FlashAttention and the fused gate-up projection, two computationally expensive operations during prefill. For FlashAttention, the eight 2k chunks require 7.21 ms in total, compared with 5.83 ms for a single 16k chunk. This cumulative cost includes the progressively increasing execution time of the individual 2k chunks as the KV sequence grows. The fused gate-up projection shows a smaller but similar effect: its cumulative execution time decreases from 7.16 ms for the 2k execution to 6.66 ms for 16k. Thus, larger prefill chunks can improve the execution efficiency of dominant prefill operations rather than merely reducing the number of scheduler iterations. We next examine whether this kernel-level advantage translates to the complete prefill and whether it persists across model sizes. Table 5 reports total prefill execution time for three Qwen2.5 models. For Qwen2.5-0.5B and Qwen2.5-3B, increasing the MBT from 2k to 16k reduces total prefill time by 10.5% and 11.6%, respectively. This provides a meaningful execution-efficiency advantage for

9

Table 6: Nsight Compute measurements for the dominant gate-up projection during prefill on the RTX 5090. Throughput values are reported as percentages of the corresponding sustained peak. Model

MBT

SM [%]

L2 Hit [%]

DRAM [%]

Qwen2.5-0.5B

2k 16k

40.64 45.76

86.95 90.03

12.37 11.91

Qwen2.5-3B

2k 16k

43.64 48.06

91.73 90.53

8.77 11.38

Qwen2.5-7B

2k 16k

47.92 48.40

96.04 88.95

7.69 22.14

the large MBT under low scheduling pressure. For Qwen2.5-7B, however, the reduction is only 0.6%. Consistent with this result, MBT 16k provides only a small latency advantage over MBT 2k at low load for Qwen2.5-7B, resulting in much weaker MBT sensitivity. P-PAS consequently provides little additional latency benefit for this configuration. To investigate why the large-chunk advantage diminishes for Qwen2.5-7B, we profile the dominant fused gate-up projection with Nsight Compute. Table 6 reports SM throughput, L2 hit rate, and DRAM throughput for MBT 2k and MBT 16k. Here, SM throughput reflects compute utilization relative to the GPU’s sustained peak, L2 hit rate indicates the fraction of L2 cache accesses served by the cache, and DRAM throughput reflects memory traffic relative to sustained DRAM bandwidth. For the two smaller models, increasing the prefill chunk size improves SM throughput. Qwen2.50.5B increases from 40.64% to 45.76%, while its L2 hit rate increases and DRAM throughput remains approximately unchanged. Qwen2.5-3B similarly increases SM throughput from 43.64% to 48.06%, with only modest changes in L2 hit rate and DRAM throughput. Qwen2.5-7B exhibits a different behavior. Increasing the MBT from 2k to 16k barely changes SM throughput, from 47.92% to 48.40%, while the L2 hit rate decreases from 96.04% to 88.95% and DRAM throughput increases from 7.69% to 22.14%. The larger prefill operation therefore places substantially more pressure on the memory hierarchy without translating this into higher SM throughput. This behavior coincides with the near disappearance of the whole-prefill advantage of MBT 16k in Table 5. These results show that the low-pressure advantage of a large MBT depends on the execution characteristics of the particular model–hardware configuration. When larger chunks improve prefill efficiency, as for Qwen2.5-0.5B and Qwen2.5-3B on the RTX 5090, there is a meaningful advantage for P-PAS to preserve under low pressure. When this advantage is small, as for Qwen2.5-7B, the opportunity for adaptive scheduling is correspondingly reduced.

6.2

Decode Interference under Scheduling Pressure

The preceding analysis explains why a large MBT can be advantageous under low scheduling pressure. It does not, however, explain why the ordering reverses as pressure increases. Under higher scheduling pressure, multiple requests can require prefill while other requests are already decoding. vLLM can schedule prefill and decode tokens together within the same iteration. With a large MBT, however, the decode work can be accompanied by a large amount of prefill work, increasing the execution time of the iteration. Although active decode requests make progress in that iteration, their next decode step cannot be scheduled until the iteration 10

Figure 5: Prefill–decode interference under scheduling pressure. Prefill and decode tokens share the MBT within each iteration. (a) MBT 16k processes the prefill in one large iteration, resulting in less frequent decode progress. (b) MBT 2k distributes approximately the same prefill work across eight iterations, enabling more frequent decode progress and lower TPOT. completes. Figure 5 illustrates this interaction conceptually. With MBT 16k, a large amount of prefill work can be scheduled alongside the active decode requests, resulting in fewer but longer iterations. With MBT 2k, approximately the same prefill work is distributed across multiple smaller iterations. Decode requests therefore make progress more frequently. This more frequent decode progress improves TPOT, but requires long prefills to be distributed across more iterations and can therefore increase TTFT. Under sufficient scheduling pressure, the TPOT improvement can outweigh this TTFT disadvantage, making MBT 2k preferable for end-to-end latency. MBT sensitivity therefore results from two competing effects. A large MBT can improve prefill execution efficiency, while a small MBT limits prefill–decode interference under scheduling pressure. A pronounced load-dependent crossover arises when both effects are sufficiently strong. P-PAS exploits this crossover by retaining the large prefill budget under low pressure and constraining it as prefill pressure increases. When the large-MBT prefill advantage is weak, as observed for Qwen2.5-7B on the RTX 5090, the opportunity for adaptive scheduling is correspondingly reduced.

7

Related Work

LLM serving systems dynamically batch requests at different stages of autoregressive inference to efficiently serve concurrent workloads. Orca [6] introduced iteration-level scheduling, allowing the composition of a batch to change between decoding iterations. vLLM [5] combines continuous batching with PagedAttention to efficiently manage the dynamically growing KV cache. Several systems specifically address the interaction between prefill and decode. Sarathi [7] introduced chunked prefill and decode-maximal batching, which combine prefill chunks with active decode requests to improve serving efficiency. DeepSpeed-FastGen [8] pursues a related approach with Dynamic SplitFuse, which splits and combines prompt and generation work into more uniformly sized forward passes. Sarathi-Serve [4] develops chunked-prefill scheduling into a 11

stall-free serving scheduler that prioritizes ongoing decodes and fills the remaining token budget with prefill work, limiting prefill-induced generation stalls. Chunked prefill has since been integrated into vLLM, where max_num_batched_tokens (MBT) limits the total token budget available to prefill and decode work within an iteration. Unlike Sarathi-Serve, which uses a pre-configured chunk size, P-PAS adapts the scheduling token budget online based on the current scheduler state. More recent systems explore alternative ways of controlling prefill execution. Liu et al. [9] regulate prefill concurrency and scheduling based on fairness and latency objectives. FlowPrefill [10] enables preemption at operator boundaries, allowing large prefill chunks to be interrupted to protect responsiveness, while Layered Prefill [11] partitions prefill along transformer layers to reduce the weight-loading overhead of token-level chunking for MoE models. These approaches address related prefill scheduling and execution trade-offs but differ from our focus on the loaddependent MBT crossover and online adaptation of the scheduling token budget. Alternatively, Splitwise [12] and DistServe [13] avoid colocated prefill–decode interference by executing the two phases on separate hardware resources. Beyond prefill–decode scheduling specifically, recent work has shown that inference-engine configuration effects can be strongly model- and workload-dependent. Zine et al. [14] systematically evaluate attention kernels, prefix caching, and chunked prefill in vLLM, finding that no evaluated configuration is universally optimal across models and workloads. In their setting, the observed effects are primarily driven by attention type and prefix caching, while enabling chunked prefill has limited impact. While their study characterizes configuration sensitivity across models and workloads, we focus on the load-dependent behavior of the chunked-prefill token budget within a serving configuration and adapt this budget online as scheduling pressure changes.

8

Conclusion and Future Work

We presented Prefill-Pressure Adaptive Scheduling (P-PAS), a lightweight extension to the vLLM scheduler for long-context, short-output workloads. Our results show that the latencyoptimal scheduling token budget depends on prefill pressure: Under low pressure, large budgets can improve prefill execution efficiency, while under high pressure, smaller budgets reduce interference with active decoding. P-PAS exploits this crossover by dynamically constraining the prefill budget based only on the current scheduler state. Across the evaluated models, workload configurations, and GPU platforms, this simple policy maintains low end-to-end latency across changing load regimes without requiring a single fixed MBT choice. Our profiling further shows that MBT sensitivity emerges from the interaction between prefill execution efficiency and prefill–decode interference, and that its magnitude depends on the model–hardware configuration. Future work should extend the evaluation to a broader workload range, including shorter and longer prompts and more diverse output lengths, and investigate whether MBT sensitivity persists for substantially larger models. Another direction is to make the P-PAS configuration itself adaptive rather than using fixed pressure thresholds and budget levels. More broadly, our results motivate a systematic characterization of how model architecture and size, workload characteristics, and hardware determine MBT sensitivity, which could help predict when adaptive scheduling is beneficial. Extending this analysis to other GPU platforms and vendors, including AMD GPUs, would further establish how hardware-specific these effects are. Finally, the observed differences in cache behavior, DRAM traffic, and kernel efficiency across configurations motivate deeper kernel-level analysis and may reveal additional optimization opportunities for large-chunk prefill execution.

12

References [1] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrievalaugmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33:9459–9474, 2020. [2] Jianshu She, Zonghang Li, Hongchao Du, Shangyu Wu, Wenhao Zheng, Eric Xing, Zhengzhong Liu, Huaxiu Yao, Jason Xue, and Qirong Ho. Laps: A length-aware-prefill llm serving system. arXiv preprint arXiv:2601.11589, 2026. [3] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations, 2023. [4] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming {Throughput-Latency} tradeoff in {LLM} inference with {Sarathi-Serve}. In 18th USENIX symposium on operating systems design and implementation (OSDI 24), pages 117–134, 2024. [5] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th symposium on operating systems principles, pages 611–626, 2023. [6] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for {Transformer-Based} generative models. In 16th USENIX symposium on operating systems design and implementation (OSDI 22), pages 521–538, 2022. [7] Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, and Ramachandran Ramjee. Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills. arXiv preprint arXiv:2308.16369, 2023. [8] Connor Holmes, Masahiro Tanaka, Michael Wyatt, Ammar Ahmad Awan, Jeff Rasley, Samyam Rajbhandari, Reza Yazdani Aminabadi, Heyang Qin, Arash Bakhtiari, Lev Kurilenko, et al. DeepSpeed-FastGen: High-throughput text generation for LLMs via MII and DeepSpeed-Inference. arXiv preprint arXiv:2401.08671, 2024. [9] Haoxin Liu, Jiayi Wang, Yueshen Xu, and Rui Li. Fairness-aware and latency-controllable scheduling for chunked-prefill llm serving. arXiv preprint arXiv:2606.09061, 2026. [10] Chia-chi Hsieh, Zan Zong, Xinyang Chen, Jianjiang Li, Jidong Zhai, and Lijie Wen. Flowprefill: Decoupling preemption from prefill scheduling granularity to mitigate head-of-line blocking in llm serving. arXiv preprint arXiv:2602.16603, 2026. [11] Gunjun Lee, Jiwon Kim, Jaiyoung Park, Younjoo Lee, and Jung H Ahn. From tokens to layers: Redefining stall-free scheduling for moe serving with layered prefill. Proceedings of Machine Learning and Systems, 8:398–421, 2026. [12] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative llm inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pages 118–132. IEEE, 2024.

13

[13] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 193–210. USENIX Association, 2024. [14] Nada Zine, Tristan Coignion, Vincenzo Stoico, Clément Quinton, Romain Rouvoy, and Patricia Lago. Attention to detail: Evaluating energy, performance, and accuracy tradeoffs across vllm configurations. arXiv preprint arXiv:2607.09172, 2026.

14

9

Appendix

Table 7: Detailed results for Qwen2.5-3B on the RTX 5090 with 25k-token prompts and 32-token outputs. Values are averages over five seeds. Lower is better for all metrics; the best result for each burst rate and metric is shown in bold. Burst Scheduler Avg. Lat. [s] P95 Lat. [s] TTFT [s] TPOT [s] Makespan [s]

0.4

MBT 1k MBT 2k MBT 4k MBT 8k MBT 16k P-PAS

3.503 3.348 2.993 3.285 2.761 2.735

5.393 5.168 4.980 6.165 4.502 4.672

2.379 2.079 1.850 1.985 1.695 1.678

0.0363 0.0409 0.0369 0.0419 0.0344 0.0341

39.476 39.226 39.089 39.036 39.000 38.970

0.6

MBT 1k MBT 2k MBT 4k MBT 8k MBT 16k P-PAS

4.506 4.399 4.252 4.005 3.678 3.764

7.505 7.057 7.767 7.396 6.854 6.659

3.098 2.551 2.208 2.142 2.039 2.020

0.0454 0.0596 0.0659 0.0601 0.0529 0.0563

40.181 39.661 39.241 39.040 38.982 39.022

0.8

MBT 1k MBT 2k MBT 4k MBT 8k MBT 16k P-PAS

5.280 5.214 5.463 5.340 4.900 4.705

8.774 8.438 9.630 10.405 9.909 8.387

3.741 3.033 2.556 2.493 2.303 2.579

0.0496 0.0703 0.0938 0.0918 0.0838 0.0686

44.540 43.889 43.429 43.225 43.180 43.473

1.0

MBT 1k MBT 2k MBT 4k MBT 8k MBT 16k P-PAS

8.110 7.451 8.041 8.625 8.420 7.037

13.891 12.155 12.390 14.029 14.668 11.796

6.300 4.715 3.933 3.591 3.466 4.288

0.0584 0.0882 0.1325 0.1624 0.1598 0.0887

49.369 46.926 45.846 45.297 45.110 46.326

1.2

MBT 1k MBT 2k MBT 4k MBT 8k MBT 16k P-PAS

12.643 10.687 11.206 13.016 13.264 10.251

21.440 17.237 16.478 18.148 19.493 16.869

10.599 7.382 6.001 5.403 5.226 6.856

0.0660 0.1066 0.1679 0.2456 0.2593 0.1095

56.444 49.907 47.301 46.275 45.670 48.755

1.4

MBT 1k MBT 2k MBT 4k MBT 8k MBT 16k P-PAS

14.614 12.560 13.050 15.156 16.649 12.895

24.395 19.881 18.978 20.466 23.004 20.537

12.567 9.180 7.217 6.319 6.066 9.416

0.0660 0.1091 0.1881 0.2851 0.3414 0.1122

60.081 53.204 49.297 47.831 47.217 53.688

Table 7 shows a small reversal at the highest burst rate of the main sweep (λ = 1.4), where MBT 2k outperforms P-PAS. To verify that this does not indicate increasing divergence at higher pressure, we additionally evaluate both configurations at λ = 1.6 requests/s. The results are shown in Table 8.

15

Table 8: High-load follow-up for Qwen2.5-3B on the RTX 5090 with 25k-token prompts and 32-token outputs at λ = 1.6 requests/s. Values are averages over five seeds. Configuration MBT 2k P-PAS

Avg. latency

P95 latency

Avg. TTFT

Avg. TPOT

Makespan

14.763 14.324

22.821 22.373

11.282 10.827

0.1123 0.1128

56.399 55.562

Table 9: Detailed A100 SXM results for Qwen2.5-3B and SmolLM3-3B with 25k-token prompts and 32-token outputs. Values are averages over five seeds. Lower is better for all metrics; the best result for each model, burst rate, and metric is shown in bold. Model

Burst Scheduler Avg. Lat. [s] P95 Lat. [s] TTFT [s] TPOT [s] Makespan [s] 0.4

MBT 2k MBT 16k P-PAS

3.179 2.750 2.706

5.037 4.882 4.233

2.049 2.024 1.818

0.0365 0.0234 0.0287

39.261 39.096 39.106

1.0

MBT 2k MBT 16k P-PAS

6.853 7.724 6.850

11.189 13.756 11.514

4.380 3.384 4.200

0.0798 0.1400 0.0855

46.417 44.791 46.190

0.4

MBT 2k MBT 16k P-PAS

3.368 2.764 2.798

5.504 4.301 4.400

2.140 1.845 1.841

0.0396 0.0296 0.0309

39.344 39.139 39.141

1.0

MBT 2k MBT 16k P-PAS

7.244 7.959 7.030

11.784 14.040 11.784

4.646 3.453 4.315

0.0838 0.1453 0.0876

46.790 45.005 46.350

Qwen2.5-3B

SmolLM3-3B

Table 10: MBT sensitivity on H100 SXM (Qwen2.5-3B, 25k prompt, 32 output tokens). Lower is better for all metrics; the best result for each burst rate and metric is shown in bold. Burst

Scheduler

Avg. Lat.

P95 Lat.

Avg. TTFT

Avg. TPOT

Makespan

1.0

MBT 2k MBT 16k

1.393 1.272

2.605 2.397

1.049 0.949

0.0111 0.0104

40.904 40.835

2.0

MBT 2k MBT 16k

2.285 2.448

3.957 5.016

1.632 1.333

0.0211 0.0360

43.276 43.248

16

Table 11: Performance across representative load regimes and model configurations on the RTX 5090. Results are averages over five seeds. Lower is better for all metrics; the best result for each burst rate and metric is shown in bold. Burst Scheduler Avg. Lat. P95 Lat. TTFT [s] [s] [s]

TPOT Makespan [s] [s]

Qwen2.5-0.5B 1.0 MBT 2k MBT 16k P-PAS 4.0 MBT 2k MBT 16k P-PAS

0.642 0.566 0.571 2.655 3.529 2.576

1.133 0.476 0.0053 0.955 0.424 0.0046 0.973 0.423 0.0048 4.656 1.936 0.0232 6.160 1.073 0.0792 4.628 1.849 0.0235

40.496 40.430 40.430 43.448 43.421 43.421

Qwen2.5-3B 0.4 MBT 2k MBT 16k P-PAS 1.0 MBT 2k MBT 16k P-PAS

3.348 2.761 2.735 7.451 8.420 7.037

5.168 2.079 0.0409 4.502 1.695 0.0344 4.672 1.678 0.0341 12.155 4.715 0.0882 14.668 3.466 0.1598 11.796 4.288 0.0887

39.226 39.000 38.970 46.926 45.110 46.326

SmolLM3-3B 0.4 MBT 2k MBT 16k P-PAS 1.0 MBT 2k MBT 16k P-PAS

3.658 2.958 2.974 8.415 9.318 7.680

5.638 2.194 0.0472 4.933 1.784 0.0379 4.953 1.775 0.0387 13.513 5.407 0.0970 15.007 3.757 0.1794 12.754 4.723 0.0954

39.410 39.062 39.064 48.150 45.657 47.080

17

Related documents

• P-PAS: Prefill-Pressure Adaptive Scheduling for Long-Context LLM Serving#612594
• P-PAS: Prefill-Pressure Adaptive Scheduling for Long-Context LLM Serving#611117
• Coordinated Scheduling for MoE LLM Serving#280179
• LEO-Aware DRL Meta-Scheduler for 5G Non-Terrestrial Network Slicing#423895
• Fairness-Aware and Latency-Controllable Scheduling for Chunked-Prefill LLM Serving#267618