Scaling LLM Inference Beyond Amdahl’s Limits via Eliminating Non-Scalable Overheads Cyril Y. He scitix
Wei Xu Tsinghua University
Abstract
4×(t=1)
Deployers of online LLM services usually seek to maximize cluster-wide performance given a fixed number of GPUs. Tensor parallelism (TP) is necessary to fit modern models but scales sub-linearly as the TP degree t grows, due to crossGPU communication and non-scalable runtime work, as predicted by Amdahl’s Law. Conversely, increasing t improves memory efficiency and alleviates KV-cache contention and swapping. We identify and validate an empirical optimal TP degree te that balances these effects. We present Albireo, a parallel inference system that raises the attainable te by shrinking the non-scalable portion via overlap of scheduling and I/O with compute and sequence-parallel sampling, without changing model architectures. Across models and benchmarks, Albireo achieves up to 1.9× higher throughput, 48% lower latency, 28% higher GPU utilization, and 54% lower energy than vLLM; in production it yields up to 2× higher throughput.
1
2×(t=2)
1×(t=4)
12
×103 Tokens/s
arXiv:2606.01927v1 [cs.DC] 1 Jun 2026
Alan Zhao scitix
9 6 3 0
Llama-2-13B Llama-2-13B Qwen-2.5-32B Qwen-2.5-32B with vLLM with Albireo with vLLM with Albireo
Figure 1: Throughput of different LLMs under various TP degrees (t), evaluated with vLLM and Albireo on the H100N testbed (detailed in §8.1). Each label x × (t=y) denotes total throughput of x inference instances, each with t=y. Batch size 128, evenly split (e.g., 64 per instance if 2 instances). on a single GPU is challenging. To mitigate these constraints, serving systems commonly employ tensor parallelism (TP)1 to distribute both weights and the KV cache across t (TP degree) GPUs [37, 47]. The main challenge of applying TP is that cross-GPU collectives and non-parallelizable portion preclude linear scaling: as the TP degree t grows, communication increasingly dominates and speedups become sub-linear, with eventual throughput degradation [4, 14, 38]. However, LLM inference is memory-bound; increasing t expands an instance’s effective memory budget, especially for the KV cache, which mitigates request preemption and GPU–CPU KV swapping [9, 17, 22]. The tension between these effects yields an empirical optimum te (per model/workload/hardware). We denote T (x) as the throughput when an instance is deployed with TP degree x and observe superlinear scaling from extensive experiments (§8) when t ≤ te : T (t) ≥ 2 × T ( 2t ) and n × T (te ) ≥ T (n × te ), n ≥ 1. This indicates, for a fixed GPU budget, configuring each instance with t = te maximizes per-GPU inference performance and thus cluster throughput. Concretely, in Figure 1 on a 4-GPU system running vLLM [17], we observe te = 2; two instances at t = 2 outperform four instances
Introduction
In data centers, large transformer-based language models (LLMs) [21, 26, 41, 46] require substantial GPU resources to deliver high-throughput, low-latency online inference. Deployments are typically constrained by a fixed GPU budget, and operators aim to maximize cluster-wide efficiency—high throughput subject to a latency SLO—for a given number of GPUs. A natural strategy is to run one instance per GPU, avoiding cross-GPU collectives and simplifying scheduling. This is usually infeasible because the combined memory footprint of the model weights and the runtime key–value (KV) cache [17, 29, 44] exceeds a single GPU’s capacity. Weight memory scales roughly linearly with model size—for example, a 32B-parameter model in FP16 uses about 64 GB for weights alone—while the KV cache grows with sequence length and batch size and can exceed the weight footprint [17,32,55]. Given that even high-end GPUs (e.g., H100) provide at most 80 GB per device, fitting 32B or larger models
1 Pipeline parallelism (PP) is suboptimal and reserved for ultra-large models, such as the 1TB Kimi K2 [40], which span multiple nodes. This paper focuses on intra-node parts of inference systems, so PP is beyond the scope.
1
at t = 1 and a single instance at t = 4. Equivalently, total throughput increases with t for t < te (memory wins) and decreases for t > te (communication wins). Consequently, for an N-GPU cluster, we have N N (1) × T (te ) ≥ × T (t), ∀t ∈ [1, N]. te t This implies two paths to maximizing the overall performance: (1) increasing T (t) for a fixed t, and (2) increasing te . However, both directions are challenging. First, increasing te is difficult because existing TP implementations [17,24,55] parallelize only the forward pass when executing LLMs. Ac1 cording to Amdahl’s Law [10] (Acceleration A = (1−P)+P/t , where P is the scalable portion, i.e., forward), the overall speedup is bounded by the non-scalable fraction (1 − P), which becomes more pronounced as t increases. The problem worsens for larger LLMs such as Llama-3.1-70B, where TP is typically set to t = 4 [46] or t = 8 [6]. Our measurements (§8) show that, under vLLM, throughput at t = 8 is 21% lower than 2× that at t = 4. Second, increasing T (t) for a fixed t is also non-trivial, especially when t is already near te . Most prior work on inference acceleration tries to increase T (t) by optimizing the forward pass. Examples include reducing communication overhead [17, 56], employing KV cache sparsity [2, 3, 19, 48, 54] or pre-fetching [9,18] to minimize cache swapping during forward, and utilizing multi-head latent attention [20] for faster forward computations. However, the non-scalable portion (1 − P) dominates execution time, so further optimizing the forward pass (increasing P) via prior techniques or faster GPUs yields diminishing returns (validated in §8.2). To overcome the speedup limits imposed by Amdahl’s Law, one must reduce the non-scalable portion by increasing the scalable portion P toward 100%. This paper introduces Albireo, which tackles inference scaling bottlenecks through overlap (§4, §5) and parallelization (§6) strategies, enabling faster inference and better resource utilization. Specifically, Albireo overlaps non-scalable scheduling and I/O with computation, and parallelizes sampling across GPUs—both largely under-optimized in prior systems—thereby improving scalability without changing model architectures. With Albireo, te shifts to a larger TP degree than that of other engines, and extensive experiments show that it can be estimated using a simple memory-budget rule of thumb (validated in §8): te = 4M /C (2) where C is the per-GPU memory capacity and M is the LLM weight size. As shown in Figure 1 (more example in §8.2), Albireo consistently outperforms vLLM for fixed t, and can sustain a higher te —e.g., for Qwen-2.5-32B, te increases from 2 to 4. Since throughput exhibits superlinear scaling when t ≤ te , i.e., T (t) ≥ 2 × T ( 2t ), this larger te directly translates into higher aggregate throughput under a fixed GPU budget. Our evaluation of Albireo shows substantial gains: 2×
higher inference throughput, 48% lower latency, 28% higher GPU utilization, and 54% lower energy consumption compared to vLLM [17], the state-of-the-art inference engine. In summary, our contributions are: (1) Performance analysis: We identify key scalability bottlenecks in LLM inference through detailed profiling. (2) Scalable parallelization: We propose resource-efficient techniques to boost inference scalability and GPU utilization. (3) System implementation: We develop Albireo, an LLM inference engine featuring asynchronous execution and parallel sampling. (4) Comprehensive evaluation: We evaluate Albireo on realistic workloads, demonstrating clear advantages over state-of-the-art systems such as vLLM [17] and SGLang [55].
2 2.1
Background LLM Inference Basics
(A) LLM Inference Workflow: Figure 2 overviews autoregressive generation. In LLM inference, an inference engine (e.g., vLLM [17]) invokes a trained model to produce an output from a user prompt. The prompt is tokenized into integer token IDs using standard tokenizers [8, 15, 34]. We call the concatenation of prompt and generated tokens the sequence. Inference proceeds through two stages—prefill and decode—to produce the final output. Prefill. In this stage, the model processes the prompt token IDs (e.g., seq0 {13, 4, 11} in iteration 0) in batches through one or more forward passes. The output is a probability distribution over possible next token IDs for each input position. Decode. Starting from the end of the prompt, the model iteratively generates one new token ID at a time. At each step, it uses the prompt and previously generated IDs (e.g., seq0 {13, 4, 11, 2} in iteration 1) to compute a probability distribution, from which the next token ID (referred to as the sampled token ID) is selected using a sampling strategy (e.g., greedy decoding). This ID is then appended to the sequence. The process repeats until a stopping condition is reached. Iteration. In LLM inference, an iteration refers to a cycle of token generation during either the prefill or decode stage. Regardless of the stage, an iteration in popular LLM inference frameworks, such as vLLM [17] and SGLang [55], includes five tasks. Figure 2 also illustrates these tasks. Tin denotes the execution of task Ti in the n-th iteration. (1) Scheduling (T1 ): Existing inference frameworks [1,17,51] use iteration batching for scheduling. At the start of each iteration, the scheduler selects a subset of unfinished sequences based on a priority strategy (e.g., seq0 and seq1 in iteration 0), allocates KV cache blocks for them, and generates scheduling outputs. At the end of the iteration, completed sequences are removed, and new ones are added, reducing waiting times for pending sequences. (2) Input processing (T2 ): This task computes the metadata required for model execution and converts the scheduling 2
Prompt:
outputs into model input, such as updating cache, transforming lists into tensors and transferring tensors to the GPU. When using TP, it also broadcasts the computed results to all GPUs. (3) Forward (T3 ): Once the model input is prepared, the model computes a probability distribution over possible output tokens, which is then passed to the next task as logits. (4) Sampling (T4 ): Sampling selects the next token ID based on the predicted probability distribution over the vocabulary, which is the set of tokens the model can understand and generate. The selected ID(s) are chosen using strategies like greedy search, top-k, or nucleus sampling, balancing accuracy and diversity. For example, {2} for seq0 in Iteration 0. (5) Output processing (T5 ): This task appends the sampled token ID to the sequence and converts it into a human-readable string through detokenization. It also checks for termination conditions and updates the sequence state. (B) Key-value Cache: In LLM inference, the key-value (KV) cache stores runtime states that are reused in computation, improving throughput and reducing latency. For example, in iteration 1 of seq1 in Figure 2, the KV cache prevents the engine from recomputing intermediate results for 13, 8, 4, 11, 2. However, the KV cache is memory-intensive (e.g., about 2.5MB per token ID in Qwen-2.5-32B). Given sequences with thousands of tokens [5] and large batch sizes (e.g., 256), the total GPU memory required can quickly reach hundreds of GB. This makes scaling GPU resources essential for efficient inference. (C) Tensor Parallelism in LLM Inference: Tensor parallelism (TP) [38] is a widely used technique to expand KV cache capacity in LLM inference [37,47]. It distributes model weights across multiple GPUs, with each GPU computing for only a portion of the model weights. This eliminates redundant weight loading and improves memory efficiency, enabling the model to support more concurrent requests without exceeding the memory capacity. In contrast, data parallelism, which is functionally equivalent to TP when t = 1, replicates the full model across GPUs and partitions the input data. For example, with two GPUs each equipped with 80 GB of memory, a TP degree of t = 1 leaves 2 × (80 − 64) = 32 GB available for KV cache when serving Qwen-2.5-32B. In contrast, with t = 2, the model weights are split across GPUs, leaving 2 × 80 − 64 = 96 GB available for KV cache.
2.2
How are you How old are you Can I help you tokenize
Prompt 13 4 11 13 8 4 11 13 1 7 11 Token IDs Autoregressive Generation Output 2 6 2 5 10 8 2 3 9 Token IDs detokenize Output: I’m great I’m eight years old I’m all set Autoregressive Generation Unifinished sequences Iteration 0 seq0: {13, 4, 11} seq0 seq1 seq1: {13, 8, 4, 11} 3 T! T%3 seq2: {13, 1, 7, 11} Unifinished sequences Iteration 1 seq0: {13, 4, 11, 2} seq1 seq2 seq1: {13, 8, 4, 11, 2} ! T! T%! seq2: {13, 1, 7, 11}
2 T'3
T&3
2
T$3
5 T'!
T&!
2
Vocabulary Table Token Token ID I 1 I’m 2 all 3 are 4 eight 5 great 6 help 7 old 8 set 9 years 10 you 11 can 12 how 13
T$!
Figure 2: Illustration of the autoregressive generation process. long-context scaling: DistServe and Sarathi-Serve separate prefill/decoding [1,56], ServerlessLLM trims checkpoint loading latency [7], and µserve partitions Transformer modules across devices [33]. KV-cache methods [3, 19, 48, 54] compress or share state, RadixAttention in SGLang [55], and multi-tier designs InfiniGen and CachedAttention [9, 18]. Newer techniques [12, 49, 52, 53] further shrink or hide movement/compute by compression, prefetch, and speculative early-exit. As a supplement, Albireo targets dependency elimination across I/O, prefill, decode, and sampling to make all tasks parallelizable or overlappable. Within this line of work, non-scalable overheads have also drawn increasing attention. Recently, several systems reduce CPU–GPU synchronization costs through engineering-level asynchrony—e.g., SGLang’s zero-overhead scheduler [35], NanoFlow’s asynchronous pipelines [58], and vLLM’s emerging asynchronous scheduler [45]. Although these efforts reduce certain partial idle-time bottlenecks in T1 , they do not achieve full asynchrony across T2 /T4 /T5 due to the absence of task-decoupling mechanisms, nor do they introduce new opportunities for parallel execution. Consequently, their performance gains are limited (1.8 ∼ 7.1% [45]). In contrast, Albireo eliminates all major sources of serialization and enables principled parallelism across tasks (T1 /T2 /T4 /T5 ).
3
Design Overview
Related Work 3.1
Researchers expand LLM inference parallelism by attacking serialization in kernels and schedulers: Llumnix and SGLang pursue asynchronous execution to curb GPU idle time [39,55]. Kernel/runtime advances push this further—POD-Attention fuses prefill and decode to enable full overlap [13], vAttention uses CUDA virtual memory for dynamic attention-state management [29], and FlashInfer supplies customizable highperformance attention/sampling kernels [50]. Complementary serving work improves throughput–latency trade-offs and
Motivation, Challenges and Solutions
Taking vLLM as an example, we investigate why it fails to achieve a larger te , as estimated by Equation 2; specifically, we profile Qwen-2.5-32B inference with t = 4. Figure 3 shows 4+4+6+0.5 non-scalable tasks account for 4+4+21+6+0.5 ≈ 41% of each iteration, forming a throughput bottleneck—a trend consistent across models (§8.2). According to Amdahl’s law, the inference speedup is bounded by P +P +P 1+P +P /t . On modern 1 2 4 3 5 GPUs like the H100, the non-scalable tasks dominate exe3
Time Sampling Metadata T!2 T%2 4 ms 4 ms
T'2 21 ms Model Input
T&2 T$2 T!2#! T%2#! 6 ms 0.5 ms Resource Demand per Sequence
scalable
non-scalable (n+1)-th iteration Data Dependency Resource Dependency
non-scalable
n-th iteration A=
! !"
"!#"$#"%#"## $
Challenge 2: Data dependencies in input/output processing. Cross-task data dependencies (solid arrows in Figure 3) serialize execution: the input processing of the (n + 1)-th iteration depends on the output of the n-th iteration’s output processing to generate the correct model input for forward, while that output processing, in turn, waits on non-overlappable sampling to produce token IDs. These dependencies prevent overlapping input/output processing with compute, thereby limiting parallelism and reducing overall efficiency. Optimization 2: Albireo introduces early-feedback backfill to enable asynchronous input and output processing. Early-feedback backfill (§5) establishes a fast path from sampling to input and output processing, enabling sampled token IDs to be forwarded immediately upon generation. This is based on the observation that data dependencies are minimal—typically just sampled token IDs. As a result, Albireo can initiate output processing without waiting for sampling to finish, and input processing can proceed concurrently. This overlapping of tasks improves overall inference efficiency. Challenge 3: Scaling communication on data dependencies of sampling. Original sampling process can be formulated as:
Sampled Token IDs
Figure 3: Sequential workflow of inference. Only T3n is parallelizable due to dependency constraints. The timing shown is per iteration, and Pi represents the average time proportion of Ti . Measurements are based on vLLM 0.11.2 (t = 4, batch size = 128) running Qwen-2.5-32B on the H100N testbed. cution time, so further P3 /t shrinking (via larger t or faster GPUs) yields little gain. Albireo addresses this by minimizing non-scalable overhead. It classifies such overhead as either overlappable (CPU T1 , T2 , T5 ) or non-overlappable (T4 ), overlapping the former with T3 and introducing a parallel mechanism for T4 . These efforts yield three key optimizations that tackle the core inference bottlenecks. Challenge 1: Resource dependencies in scheduling. To address the challenges from non-scalable overhead, we formalize scheduling and make its resource dependencies explicit (dashed arrows in Figure 3). A scheduler must respect three system budgets during inference: the per-iteration new tokens Bt , the number of concurrent sequences Bseq , and GPU memory. With PagedAttention [17], the KV cache is grouped into OS-like blocks of every Bc tokens. Therefore, the count of free blocks Bb serves as the effective memory budget and directly shapes scheduling. Let S be current sequences; each seq has length Lseq and needs Nseq new tokens. The framework chooses S′ ⊆ S to maximize its size subject to max |S′ | s.t. S′ ⊆ S, |S′ | ≤ Bseq , Lseq + Nseq N ≤ B , ≤ Bb . ∑ seq t ∑ ′ Bc seq∈S′ seq∈S
[Pr1v , Pr2v ] = DecodingForward(X) Pr = g(Pr1v , Pr2v )
(4)
Y = Sample(Pr), where X is the model input, Pr is the logits matrix representing the probabilities of v vocabulary tokens for s requests from forward, g signifies the gather operation, and Y is the sampling results. The superscript (·)v denotes partitioning by vocabulary. Sampling remains single-threaded in existing frameworks for two reasons. First, batch-parallel sampling ideally parallelizes along the request (batch) dimension, but the computation is over the vocabulary dimension, which TP already shards. Involving all TP workers requires expensive collectives, adding millisecond-scale overhead each iteration. Second, sampling requires per-sequence metadata (solid arrows in Figure 3), such as penalties [16, 25]. In our measurement, serializing and broadcasting this data—e.g., via Pickle [30]—costs more than 10 ms per iteration and causing up to 37% throughput loss on Qwen-2.5-32B in vLLM. Optimization 3: Albireo introduces sequence-parallel sampling. We propose sequence-parallel sampling (§6), which partitions sampling along the batch dimension so workers can sample independently. It is fully TP-compatible, requiring no model reloads or extra GPU memory. By overlapping scatter of sampling metadata and exchanging logits via all-to-all, Albireo removes the communication bottleneck that limits parallel sampling efficiency.
(3)
Asynchronous scheduling complicates deriving the correct set S because the previous iteration may still be in flight. Some sequences in S could finish during output processing of the previous iteration and change state. Since S affects the ′ scheduler’s selection of S , stale membership can violate the resource constraints in Equation 3. Additionally, misestimating Lseq and Nseq in asynchronous scheduling can likewise risks breaching these resource budgets. Optimization 1: Albireo introduces optimistic singleiteration prediction to enable asynchronous scheduling. Albireo adopts an optimistic prediction strategy (§4) to break the resource dependencies between adjacent tasks. In a nutshell, it allocates memory under the assumption that requests never encounter stop conditions and computes Lseq and Nseq based on this assumption. This enables the scheduling of the next iteration before the current one completes.
3.2
Overview
Architecture: Albireo makes minimal changes to existing inference stacks by adding four components: an asynchronous scheduler, input processor, output processor, and parallel 4
2
Scheduler
Scheduling 1 Outputs
Predictor
T!3
Input Processor
3 X X Async Input Processor
Async Scheduler Backfill 5
4 Pr
Output Processor Y Y Async Output Processor
6 Sampler Sampler Sampler
T%!
Time <80μs T!% T'3
21ms
T&3 3 1.5ms T$
T%%
T'!
T&! T$!
Figure 5: Albireo’s execution pipeline. Conducted on Qwen2.5-32B with 4×80GB H100 (t=4, batch size=128).
X
Model
7 Sequences’ Resource Demand
T%3
$ A= "&#"%
T!!
Sampling Metadata
4
Parallel Sampler
Optimistic Asynchronous Scheduling
Figure 4: Architecture of Albireo. Hollow components represent the baseline workflow described in §2.1; solid components enable asynchronous and parallel execution.
In iteration batching scheduling, the (n + 1)-th iteration depends on the completion of the n-th iteration. This synchronous scheduling enforces strict sequential execution, preventing asynchronous input/output processing.
sampler (Figure 4). The asynchronous scheduler incorporates a predictor that estimates resource needs from current scheduling outputs, allowing the next round to start before the current iteration finishes. The input processor asynchronously queues model inputs (X) and consumes them in order. The parallel sampler launches multiple sampler instances to partition the sampling task; their results (Y ) are enqueued to the ordered queue of output processor for detokenization. Workflow: In this section, we illustrate the workflow of Albireo through an example in which the n-th iteration of forward commences on the GPU while the scheduling for the (n + 1)-th iteration simultaneously initiates on the CPU. On the CPU, the scheduler inspects sequence states and, under a policy, e.g., First-Come-First-Serve (FCFS), emits scheduling outputs for n+1, forwarding them to the predic1 tor and input processor (⃝). The predictor estimates the KV blocks needed in the next iteration per scheduled sequence 2 and pre-updates their states (⃝). In parallel, the input processor converts these outputs into model inputs and sampling metadata in a queue. On the GPU, Albireo dequeues iteration-n inputs and sampling metadata and dispatches them concurrently to the model 3 and the parallel sampler (⃝). After forward, the sampler 4 shards the model logits Pr to sampler instances (⃝), which perform parallel sampling using the metadata. As each token ID is produced, the sampler backfills it to the 5 input processor to complete iteration-n+1 inputs (⃝), resolving placeholders inserted when a sequence spans iterations and its last token was unknown at scheduling. When sampling 6 finishes, the sampler sends results to the output processor (⃝). Simultaneously, the output processor continuously extracts sampling results from the queue, updates the states of unfin7 ished sequences, and removes completed ones (⃝). Optimized inference pipeline. Albireo ’s pipeline is primarily dominated by the model forward. Figure 5 shows the optimized design: relative to the sequential workflow (Figure 3), Optimizations 1–2 shrink CPU time from 8.5 ms to < 80µs, and Optimization 3 cuts sampling on four H100s from 6 ms to 1.5 ms. These remove > 89% of non-scalable overhead, yielding an ∼ 1.7× speedup over vLLM.
Upon re-examining Equation 3, asynchronous scheduling still appears feasible. Before a sequence seq is finished, we n+1 ← Ln + N n , where the initial can easily deduce that Lseq seq seq 0 n can be inferred based on the length of Lseq is 0, and Nseq sequence’s stage (prefill or decode), which is determined by n . The remaining challenge lies in predicting whether a Lseq sequence is finished in asynchronous scheduling, ensuring the accurate identification of the sequence set S. Iteration-dependent sequence management. We first present a revised sequence management mechanism tailored for asynchronous scheduling. As Figure 5 shows, during T1n+1 , a sequence seq involving in the (n − 1)-th iteration n−1 prior to T n−1 and may exhibit two uncertain length: Lseq 5 n−1 + N n−1 after T n−1 . This dual-state condition creates poLseq seq 5 tential ambiguities for the scheduler. Enforcing the sequential execution of T1n+1 and T5n−1 in any order undermines the advantages of asynchrony and introduces unnecessary event management overhead, reducing inference performance. To resolve sequence state inconsistencies, Albireo introduces iteration-dependent sequence management, which assigns a virtual state to each token ID in every sequence. The asynchronous scheduler tracks three states for each sequence during the n-th iteration: (1) Expected length (EL): The sequence length at the start of the n-th iteration; (2) Current length (CL): The sequence length at the end of the n-th iteration; (3) Number of new token IDs (NNT): The number of tokens expected to be generated by the end of the n-th iteration. This scheme lets the scheduler query future states based on how many iterations a sequence has been scheduled, ensuring consistent state tracking during asynchronous execution of T1n+1 . Optimistic prediction. The essence of asynchronous scheduling involves two key aspects: (A1) determining the number of KV cache blocks required by a sequence in each scheduling iteration, and (A2) predicting whether the sequence should continue generating the next token ID (i.e., whether it needs further processing). The core of A1 is to compute the number of KV cache blocks Cn needed for a sequence, given its scheduled iteration n. Cn can be calculated interactively as 5
XT
Metadata s’
(n+1)-th Iteration Input Processing Model Input (X) XM X-XM-XT
Early-feedback Backfill
n-th Iteration Output Processing
seq2 seq3 seq0 seq1
seq2 seq3 seq0 seq1
seq2 seq3 seq4 seq5
n-th Iteration Decoding Forward
n-th Iteration Sampling
(n+1)-th Iteration Decoding Forward
Sequence Parallel
Scheduling Outputs
seq2 seq3 seq4 seq5
Scatter ① (s’/t)
Ln Cn = , Bc
Gather ④
Sampling Output
v 0 1 … 𝑠
Logits
s
0 … 𝑠/𝑡
Remap GPU 0 Logits 0 s/t
0 … 𝑠/𝑡
GPU t Logits t
… s/t
s=de-padding(s’)
Figure 7: Sequence-parallel sampling workflow.
(5)
To further decouple input processing and output processing from the sequential execution workflow and eliminate the dependency of input processing on output processing, Albireo introduces an early-feedback backfill mechanism. This mechanism establishes a fast path from the sampler to both processors. During sampling, each newly generated token ID is immediately forwarded to these processors, which aligns model inputs with those in the sequential execution workflow and corrects potential inference errors.
Here, L0 = 0, N p represents the number of prompt token IDs in the sequence, and Nc denotes the chunk size of token IDs processed in prefill. During the prefill stage (when Ln−1 < N p ), Albireo updates Ln based on the preset Nc to enhance the efficiency of the prefill process. In contrast, during the decode stage, only one token ID is generated per iteration, so it is sufficient to increase the current length by only one. For A2, Albireo optimistically predicts that each sequence requires further inference. This strategy is highly effective in practice because, for each prompt generating N tokens, the prediction succeeds N − 1 times and fails only once. Albireo uses single-iteration asynchronous scheduling, where only the (n + 1)-th iteration is scheduled during the n-th iteration. This approach offers two benefits: first, we can hide the CPU computation by fully overlapping it with the forward, only adding a negligible 80 µs overhead to dequeue model input and enqueue sampling outputs (Figure 5). Second, in online serving, when a new request arrives, Albireo must recompute the scheduling for prefill tasks to ensure consistent latency for the generation of the first token. Consequently, the more iterations advanced in asynchronous scheduling, the higher the likelihood of scheduling invalidation. Singleiteration asynchronous scheduling strikes an effective balance between computational overhead and performance gains.
5
(s’/t)×v (s’/t)
Driver
All-to-all ②
s’
follows:
s’=padding(s)
s’×(v/t)
Sampling ③
Figure 6: Illustration of asynchronous input processing.
( Ln−1 + 1, Ln−1 ≥ N p , Ln = Ln−1 + Nc , Ln−1 < N p .
Logits
Input processing. The input processing of the (n + 1)-th iteration relies on both the scheduling output of the (n + 1)-th iteration and the sampling output of the n-th iteration. These dependencies are the reason why existing inference frameworks cannot execute input processing asynchronously. In Albireo, the input processor can directly access the output from the asynchronous scheduling, reducing the data dependency to only the sampling output from the n-th iteration. As Figure 6 shows, a model input X comprises metadata (XM ) and tensors (X − XM ). The input processor’s dependency on the sampler affects only one tensor in X − XM —the last sampled token IDs (XT ). To prevent input processing from stalling due to unavailable sampling output from the previous iteration, Albireo divides input processing into three steps. First, it computes XM , such as the metadata used for KV cache. Second, it allocates every tensor in X − XM , including input positions and the last sampled token IDs. Third, it transfers X − XM − XT to GPUs. Although the content of XT depends on sampling outputs, its shape is determined exclusively by scheduling outputs alone. This allows the input processor to execute the three processing steps as soon as it receives scheduling outputs, without waiting for sampling. Albireo corrects XT in two scenarios. If a sequence was not scheduled in the previous iteration, (e.g., seq4 and seq5 in Figure 6), its last token ID is used as the sampled token ID. Otherwise, the sampler backfills token IDs for these sequences (e.g., seq2 and seq3 ) after the sampling process completes.
Asynchronous Input/Output Processing
By decoupling scheduling from the sequential execution workflow as an asynchronous operation, input processing can also be performed asynchronously in advance. However, since the input processor also depends on the outputs from the output processor to generate model input, and the output processor in turn relies on the sampling, challenges remain for achieving asynchronous input/output processing fully. Early-feedback backfill. When generating the (n + 1)-th token ID in the (n + 1)-th iteration, the model requires the n-th token ID of the sequence as part of inputs. Without this n-th token ID, the model will produce incorrect token IDs, resulting in inference errors.
Output processing. Although Albireo overlaps GPU and CPU tasks extensively, it still requires rapid output processing because the forward time may be insufficient to hide the over6
head of all CPU tasks 2 . Consequently, Albireo preserves the existing output processing logic while incorporating several engineering optimizations (detailed in Appendix A of the supplementary material), including utilizing two lookup tables, replacing de-tokenizer calls with fast table lookups, and parallelizing sequence processing and stop checking, reducing output processing time to under 2 ms and thus effectively hiding CPU time in the execution timeline.
6
the substantial sampling metadata is only needed after the forward computation completes. To leverage this, Albireo delays the scatter operation until the forward begins, effectively concealing the communication overhead within the inevitable GPU computation time. We define the ratio of sampling metadata scattering time to forward pass time as Rs . In most cases, Rs is lower than 20% even on H100 GPUs because the metadata size is small (∼1.5 KB per request for a sequence length of 1000). This full overlap effectively eliminates the communication overhead associated with scattering. Efficient logit exchange. Albireo adopts a more efficient approach to exchanging logits. In conventional setups, the driver worker gathers probabilities along the vocabulary dimension from all workers for sampling. However, once the sampling workload is distributed across all workers, each worker must access these probabilities. Therefore, Albireo replaces the traditional gather with an all-to-all operation, which gathers probabilities along the vocabulary dimension and partitions them according to batch size simultaneously. This approach eliminates the need for workers to individually partition the outputs of an all-gather operation. As a result, each worker transfers the same amount of data as in a single gather operation, ensuring that parallel sampling does not introduce additional communication overhead at this step. The driver worker collects results only after the sampling task is completed. Each result consists of a single token ID and sequence ID per sequence. As a result, the communication data remains minimal, even for large batch sizes. We only observe 200 µs delay on average for 256 requests. Deterministic random number generation. Independent RNG states make per-worker draws diverge from singleworker outcomes. To retain determinism under a fixed seed, Albireo pre-generates all k random numbers on all t GPUs and lets each worker consume its slice. This approach trades additional GPU memory (approximately 128 KB per request) for lower communication overhead (saving about 1 ms to retrieve them from remote GPUs). Batch padding. Collectives often require divisibility by the number of peers. Albireo applies batch padding to make s′ mod t = 0 with synthetic metadata and extra logit rows; the driver then discards padded outputs before early-feedback backfill. Although this introduces some additional communication padding overhead (less than 10 KB for metadata padding), the advantages of parallel sampling far outweigh the costs, leading to an overall speedup in inference.
Sequence-Parallel Sampling
6.1
Parallel Sampling Workflow
As highlighted in §3.1, TP focuses exclusively on the computationally intensive forward task, but the sampling task only runs on a single GPU. Consequently, the driver worker (the first process among all worker processes) alone handles sampling, leaving the GPUs of other workers idle. We notice that in non-parallel sampling, operations on the (s × v) logits (Figure 7) are independent along the sequence dimension. This independence allows the sampling workload to be partitioned across multiple GPUs, thereby improving scalability. In contrast to the original workflow described in Equation 4, the sequence-parallel sampling process can be formulated as [Pr1v , Pr2v ] = DecodingForward(X) [Pr1s , Pr2s ] = ĝ(Pr1v , Pr2v ) Y1s = Sample(Pr1s ) and Y2s = Sample(Pr2s )
(6)
Y = g([Y1s ,Y2s ]), where g and ĝ are gather and all-to-all operations, respectively. The Albireo sampling workflow includes four steps: (1) The driver worker uses scatter to distribute sampling metadata of different sequences to workers. (2) Each worker performs an all-to-all operation to exchange vocabulary probabilities after forward, splitting the data along the sequence dimension. (3) The core sampling computation remains unchanged, but each worker processes a smaller subset of requests independently. (4) The driver worker collects the sampled token IDs from all workers into an array and sends it via early-feedback backfill.
6.2
Key Optimizations
The parallel sampling workflow poses two primary challenges: (1) the additional communication overhead, which may degrade performance, and (2) the risk of altering the final output due to parallel computation. In this section, we present the key optimizations designed to address both issues and ensure consistent, high-performance results. Overlapping scattering with GPU computation. The forward preceding sampling is a GPU-bound operation, while
7
Implementation
The core functionality of Albireo is implemented as a framework-independent plugin, comprising approximately 8,500 lines of C++ code. This modular design allows Albireo to be easily incorporated into various frameworks with minimal modifications. For our evaluation, we integrated the plugin with vLLM, requiring only ∼200 lines of code changes.
2 The profiling results in Figure 3 show only the unhidden portion of
output processing, while the full process takes approximately 7 ms.
7
SGLang Albireo 12 9 6 3 0 B 2B 4B 2B Llama-2-13 Qwen-2.5-1 Qwen-2.5-3 QwQ-3 (b) t=2, Small LLMs (c) t=4, Moderate LLMs
×103 Tokens/s
12 9 6 3 0
×103 Tokens/s
B Llama-2-7B Qwen-2.5-7 (a) t=1, Tiny LLMs
×103 Tokens/s
×103 Tokens/s
vLLM
12 9 6 3 0
12 9 6 3 0 70B -72B Llama-3.1- Qwen-2.5 (d) t=8, Large LLMs
Figure 8: Inference throughput across different model sizes, measured using the default configuration on the H100N testbed.
8
Evaluation
vLLM
SGLang
8.1
×103 Tokens/s
We evaluate Albireo to answer the following key questions: (1) What performance improvements does Albireo deliver, and on which metrics? Is Albireo effective across different models, settings, and GPUs?(§8.2, §8.3) (2) How does Albireo perform in production? (§8.4) (3) Can Albireo make more efficient use of resources? (§8.5) (4) What are the primary sources of performance gain? Does Albireo make the correct design decisions? (§8.6)
Albireo
8
×103 Tokens/s
8 6
6
4
4
2
2
0
7B 3B 2B 0B a-2- a-2-1 -2.5-3a-3.1-7 Llam Llam QwenL lam (a) A100 (NVLink)
0
7B 3B 2B 0B a-2- a-2-1 -2.5-3a-3.1-7 Llam Llam QwenL lam (b) A100 (PCIe)
Figure 9: Throughput across different devices. presents comparisons under low-load conditions. Workload. We randomly sample prompts from the Databricks dataset [5] as user inputs. For production deployment, we adopt bentoML [57] to launch servers and clients. All sampling features are enabled, including top-p, top-k, min-p, temperature, and repetition, presence, and frequency penalties. Metrics. We evaluate inference performance using four key metrics: Throughput — the number of tokens generated per second; Latency — the average time to generate the next token; GPU Utilization — the efficiency of GPU resource usage; Power Usage — the real-time GPU power consumption.
Experimental Setup
Testbed. We evaluate Albireo on three testbeds: H100N , A100N and A100P . Each has 8 GPUs (80 GB memory): H100N uses NVLink-connected H100 GPUs, A100N uses NVLink-connected A100 GPUs, and A100P uses PCIeconnected A100 GPUs. All testbeds feature 2 TB RAM, an Intel® Xeon® Platinum 8468 CPU with 192 logical cores. Baseline. We evaluate Albireo against vLLM (v0.11.2) [17] and SGLang (v0.5.5) [55], using their latest stable releases. All frameworks are evaluated with their default configurations, ensuring that all built-in performance optimizations are enabled. 3 Models. We evaluate eight LLMs with FP16 precision spanning a range of commonly used sizes, grouped into four categories: (1) Tiny — Llama-2-7B [23], Qwen-2.5-7B [41]; (2) Small — Llama-2-13B [43], Qwen-2.5-14B; (3) Moderate — Qwen-2.5-32B [41], QwQ-32B [42]; and (4) Large — Llama3.1-70B [27], Qwen-2.5-72B [41]. Larger models such as DeepSeek V3 [21] are not included, as they require PP across multiple nodes, which is beyond our scope. Configuration. In all tests, PP is disabled, and DP is expressed as a TP degree of t = 1. By default, each model category adopts the te derived from Equation 2, that maximizes overall performance within a node: 1 for Tiny, 2 for Small, 4 for Moderate, and 8 for Large. We also vary t in §8.2 to validate performance across different TP degrees. Following prior work [36], we adopt a default per-GPU batch size of 32 (e.g., a total batch size of 128 when t = 4) to better utilize resources under high-load datacenter settings; §8.4 further
8.2
End-to-End Throughput Improvement
Throughput improvements for LLMs of all sizes. We first evaluate the performance of each LLM on the H100N testbed using default configurations, as shown in Figure 8. We observe that Albireo yields improvements of about 1.3×, 1.5×, 1.7× and 1.9× for Tiny, Small, Moderate, and Large LLMs, respectively. These differences arise from the varying te across model sizes. For a given model, Albireo reduces overhead by about T1 + T2 + (1 − 1t ) × T4 + T5 in each iteration. For Tiny models with t = 1, parallel sampling and early feedback in asynchronous input processing (T2 ) are disabled, leading to only partial benefits from asynchronous execution. In contrast, larger models (with t > 1) fully exploit these mechanisms because parallel sampling removes roughly (1 − 1t ) × T4 of the sampling overhead, leading to greater speedups. Models within the same category show similar performance gains, as they use the same TP degree. Cross-platform validation of Albireo. To assess the generality of Albireo, we then evaluate its performance across diverse GPU architectures, conducting experiments on the A100N and A100P testbeds, as shown in Figures 9(a) and 9(b).
3 Sections 8.4 to 8.6 compare only against vLLM, since Albireo is implemented as a vLLM plugin, highlighting the incremental impact of its optimizations.
8
vLLM
×103 Tokens/s
18 15 12 9 6 3 0
1 2 4 TP degree (t) (a) Llama-2-13B
18 15 12 9 6 3 0
1 2 4 TP degree (t) (b) Qwen-2.5-14B
18 15 12 9 6 3 0
SGLang
2 4 8 TP degree (t) (c) Qwen-2.5-32B
18 15 12 9 6 3 0
Albireo
2 4 8 TP degree (t) (d) QwQ-32B
18 15 12 9 6 3 0
2 4 8 TP degree (t) (e) Llama-3.1-70B
18 15 12 9 6 3 0
2 4 8 TP degree (t) (f) Qwen-2.5-72B
Figure 10: Impact of TP degree on throughput. The dashed line indicates the t e . To the left of the dashed line, Albireo not only outperforms others but also achieves superlinear scalability, with throughput at t = 2n exceeding 2× that at t = n. Since models of similar size yield comparable speedups, we report results for one representative model per size category, all measured with the default configuration. Albireo delivers up to 1.7× and 1.4× performance speedups over vLLM on A100N and A100P , respectively, demonstrating its effectiveness across GPU types. Speedups are higher on A100N due to faster inter-GPU communication during forward passes when t > 1, which reduces T3 and improves overall performance. Compared to H100, A100 shows lower overall gains—an expected result, as H100’s 3× higher FLOPs makes T3 less dominant. This trend highlights the increasing impact of nonscalable overhead as computational power grows, reinforcing the importance of Albireo’s optimizations. Higher aggregate throughput from superlinear scaling and larger te . When memory permits, models can be deployed with different TP degrees. For example, on the H100N testbed, a node supports 8t inference engine instances with TP degree t. This section investigates whether Albireo can support a larger te and achieve higher aggregate throughput under the same GPU budget. Figure 10 shows the throughput of a single inference engine under different TP degrees, denoted as E t . Results for Moderate and Large LLMs at t = 1 are omitted: the former exhibits poor performance, while the latter runs out of memory. In both vLLM and SGLang, E te < 2 × E te /2 , indicating that the actual effective TP degree te′ is lower than te . This trend is more pronounced for Large LLMs, where communication and coordination overheads increase with higher TP degrees. Such behavior aligns with Amdahl’s Law, which predicts diminishing returns as parallelism grows due to the impact of communication and other non-scalable tasks. In contrast, Albireo increases the effective te from 2 to 4 for Moderate LLMs and from 4 to 8 for Large LLMs. At the same TP degree t, Albireo consistently outperforms the baselines and demonstrates superlinear scaling across all model sizes for t ≤ te , i.e., E te ≥ 2 × E te /2 , which leads to higher aggregate throughput, i.e., t8e × E te . This is primarily because Albireo significantly reduces non-scalable bottlenecks and efficiently shares model weights among GPUs at higher TP degrees, thus freeing redundant memory for the KV cache.
TPOT (milliseconds)
vLLM
SGLang
Albireo
60 40 20 0
B 13B B B 32B 32B B 72B -14 -70 .5-7 -2-7 -3.1 en-2.5 ma wen-2 ama-2 en-2.5 en-2.5 QwQ a a l l m L w a L Q l Qw Qw Q L
Figure 11: TPOT across different models and engines. For memory-bound inference workloads, this enhances cache hit rates and increases resource utilization on GPU compute (validated in §8.5), driving superlinear throughput gains.
8.3
Reduced End-to-End Latency
Time-to-First-Token (TTFT). TTFT is dominated by the prefill phase. In this phase, each batch typically contains thousands of tokens (e.g., 8192), and iterations are largely spent on GPU forward passes; moreover, prefill does not involve sampling. As a result, Albireo’s optimizations have minimal effect, and TTFT is nearly identical to vLLM; detailed results are omitted for brevity. Time-per-Output-Token (TPOT). TPOT measures the average time to generate each output token ID after the previous one. We evaluate TPOT for each model on the H100N testbed using default configurations, as shown in Figure 11. Albireo consistently achieves a 22% to 48% reduction in average TPOT across different LLMs, with the largest improvement observed on Qwen-2.5-72B. By reducing non-scalable overhead (1 − P), Albireo shortens iteration time and lowers token generation latency. Our results show consistent performance gains from Albireo for all scenarios, especially for Moderate and Large LLMs.
8.4
Online Inference in Production
Albireo is integrated into our cloud-based Model-as-aService (MaaS) platform, enabling users to deploy their online inference tasks like question answering using user-provided models. We evaluate its performance below. 9
21 22 23 24 25 26 27 28 29 Concurrent Requests (a) QwQ-32B
vLLM
21 22 23 24 25 26 27 28 29 Concurrent Requests (b) Llama-3.1-70B
Figure 12: Normalized inference throughput under varying loads on an H100 node, for TP degrees t=4 (left) and t=8 (right). Each value is normalized to vLLM under the same load (baseline = 1).
0
100 200 300 TPOT (ms) (a) QwQ-32B
400
1.0 0.8 0.6 0.4 0.2 0
0
0 2 4 6 8 10 12 14 16 (a) Time (s)
Albireo
500 400 300 200 100 0
0 5 10 15 20 25 30 35 (b) Time (s)
Figure 14: GPU utilization and power usage on Qwen-2.532B (t = 4) on the H100N testbed. vLLM
Asynchronous Execution
Parallel Sampling
Qwen-2.5-72B (t = 8) Llama-3.1-70B (t = 8) QwQ-32B (t = 4) Qwen-2.5-32B (t = 4) Qwen-2.5-14B (t = 2) Llama-2-13B (t = 2) Qwen-2.5-7B (t = 1) Llama-2-7B (t = 1)
Albireo CDF
CDF
vLLM 1.0 0.8 0.6 0.4 0.2 0
100 80 60 40 20 0
Power Usage (W)
Albireo
2 1.5 1 0.5 0
GPU Util (%)
Normalized Throughput
vLLM
2 1.5 1 0.5 0
0
100 200 300 400 TPOT (ms) (b) Llama-3.1-70B
2 4 6 8 10 Throughput (×103 Token/s)
12
Figure 15: Throughput ablation study of Albireo’s optimizations across various models on the H100N testbed.
Figure 13: Cumulative distribution of inference latency across models and inference engines, aggregated over one week.
ing inference engines, typically implemented in Python, rely on asyncio [31] to simulate multithreading for CPU tasks. However, the OS scheduling and event-loop overhead cause performance variability in asyncio tasks. Since existing engines do not overlap CPU and GPU tasks, this variability increases end-to-end latency. In contrast, Albireo ’s overlapping mechanism effectively hides CPU-side fluctuations, resulting in more stable tail latency.
Throughput under varying load conditions. Unlike the controlled benchmarks above, which maintain a large batch size to stress-test inference engines, the load in production environments fluctuates in real time—sometimes underloaded, sometimes overloaded. Figure 12 illustrates the performance comparison of two popular LLMs on our MaaS platform, which serves users, after transitioning from vLLM to Albireo. We observe that the performance benefits of Albireo grow with increasing load pressure and eventually stabilize (1.7× for QwQ-32B and 2× for Llama-3.1-70B). Through deep profiling, we find that before the system becomes overloaded (i.e., concurrent requests ≤ 256), increasing requests from n to 2n results in faster growth of non-scalable tasks (T1 /T2 /T4 /T5 ) than the forward computation (T3 ) because the system is memory-bound. With increasing batch size, non-scalable overheads grow disproportionately and begin to dominate the runtime, reducing the relative contribution of forward computation. This shift increasingly constrains overall performance under heavy load. Latency distribution. In online inference scenarios, tail latency—especially the 99% and 99.9% percentiles—is critical for ensuring user experience, in addition to average latency. Figure 13 presents the TPOT distribution for QwQ-32B and Llama3.1-70B over a one-week period in production, showing that Albireo substantially reduces tail latency. For example, QwQ-32B’s 99th and 99.9th percentile TPOT under vLLM are 271 ms and 298 ms, while Albireo reduces them to 75ms and 76ms. This improvement stems from two key factors: first, Albireo achieves shorter iteration latency through more efficient CPU-GPU parallelism; second, exist-
8.5
Resource Usage
Albireo optimizes GPU utilization of inference systems. A key design goal of Albireo is to improve GPU utilization. Our measurements show that by reducing GPU idle time in stages T1 , T2 , and T5 through asynchronous execution, and by distributing the sampling workload across all available GPUs, Albireo increases average GPU utilization by about 10% to 40%. Figure 14(a) presents the average GPU utilization measured every 20 ms, comparing Albireo with vLLM. As an example, Albireo improves average GPU utilization by 28% on Qwen-2.5-32B with the default configuration. Albireo completes inference tasks with less energy. While achieving higher GPU utilization, Albireo also enables more energy-efficient inference. Figure 14(b) shows the average GPU power consumption recorded every 20 ms during inference on 256 concurrent requests using different frameworks. Compared to vLLM, Albireo reduces average GPU power by 15%, shortens inference time by 47%, and achieves an overall energy saving of 54%. This 15% power reduction aligns with prior work [11], which shows that GPU energy efficiency (performance per 10
Table 1: Single-task time reduction for Qwen-2.5-32B at t = 4 on the H100N testbed, achieved by Albireo. T1 ∼ 4 ms ∼ 5 µs
T2 ∼ 4 ms ∼ 40 µs
T4 ∼ 6 ms ∼ 1.5 ms
Number of Blocks
Task vLLM Albireo
Used
T5 ∼ 0.5 ms ∼ 25 µs
Watt) drops when memory request rates exceed peak bandwidth. Without balancing memory-intensive tasks among GPUs, an inference system may consume more power while delivering lower computational throughput.
Allocated
18000 17000 16000 15000
0
0.2
0.4 0.6 Time (s)
1
Figure 16: Blocks allocated by Albireo compared to worstcase usage. 0.22
32
0.20
Ablation Studies
Batch Size
8.6
0.8
Performance impact of the asynchronous execution and sequence-parallel sampling. Figure 15 illustrates the sources of throughput improvements for each LLM under the default configuration. The bar vLLM represents the throughput of vanilla vLLM. Asynchronous Execution and Parallel Sampling represent the respective proportions of throughput contributed by the two mechanisms in the overall throughput. For tiny and small LLMs, the performance gains come almost entirely from Asynchronous Execution, as parallel sampling is either disabled or provides limited benefits due to insufficient parallelism. In contrast, for moderate and large LLMs, both mechanisms contribute comparably to the overall performance improvements. Performance impact of Albireo optimizations on inference tasks. Albireo demonstrates consistent performance across all models, significantly reducing non-scalable overhead in each iteration. Table 1 shows the performance improvements achieved by Albireo on Qwen-2.5-32B at each task of the inference workflow. Since sampling is a hybrid task involving both CPU and GPU computation, we introduce a GPU-CPU synchronization before sampling to isolate the influence of forward computation. We highlight the following key observations: (1) asynchronous execution in Albireo reduces the time that CPU tasks block the program from 8.5 ms to a negligible 70 µs, achieving a reduction of over 99%; (2) With 4 GPUs, parallel sampling saves 75% of the time in T4 . Combined, the two parallel mechanisms reduce the non-scalable overhead in each iteration by more than 89%. Optimistic single-iteration scheduling incurs minimal extra GPU memory usage in the worst case. As Albireo’s scheduler always assumes sequence continuation, a sequence that stops early can at worst waste one KV block. To assess this, we measure actual GPU memory usage and allocated memory per iteration (Figure 16 shows the results from the last second for Qwen-2.5-32B on the H100N testbed). Every Bc tokens (Bc =16 in Albireo and vLLM) trigger preallocation of one block; thereafter, used and allocated blocks match, and a new block is allocated only when the current one fills. If a sequence completes exactly when a block is preallocated,
64 0.18 128
0.16
256
0.14 500 1000 1500 2000 Sequence Output Length
Figure 17: Rs on Qwen-2.5-32B at t = 4 using H100N testbed. the surplus is reclaimed within one iteration. Forward computation overlaps extra scattering. To validate our claim that the forward pass can effectively hide the overhead of sampling metadata scattering, we measure the Rs (ratio of sampling metadata scattering time to forwarding pass time) as the batch size and sequence length increase in Figure 17, where warmer colors indicate higher Rs . Across all configurations on the H100N testbed, we observe that the Rs is significantly small, accounting for only 12% to 22%. Even with future devices offering improved performance, we estimate that the ratio of PCIe bandwidth to GPU computational power would need to exceed 4× the current level to challenge this assumption. Thus, our overlapping technique is expected to remain effective in most scenarios.
9
Limitations and Future Work
Looking ahead, we expect that rapidly increasing GPU computational capability—outpacing improvements in CPU technology—will continue to shrink the relative cost of forward passes within each iteration. For example, our evaluation shows that upgrading from A100 to H100 substantially boosts compute throughput while leaving non-scalable components largely unchanged, making these overheads a growing fraction of end-to-end latency. At the same time, steady growth in per-GPU memory capacity across generations is pushing model deployments increasingly toward single-node settings. Together, these trends underscore the rising importance of reducing non-scalable overheads and optimizing intra-node parallel execution as GPU compute becomes less of a bottleneck. Albireo focuses on efficient inference within a single node, leveraging parallelism and asynchronous execution to 11
reduce these non-scalable overheads. However, its current design does not directly address multi-node deployments for ultra-large models, which introduce fundamentally different scalability challenges. In such environments, TP all-reduce operations often dominate execution time, and hybrid TP–PP brings additional issues such as inter-stage communication bubbles and stage load imbalance (e.g., extra sampling work in the final stage). These challenges are largely orthogonal to the intra-node optimizations explored in this work. Future work will extend Albireo to hybrid TP–PP multi-node deployments, addressing inter-stage communication efficiency and improving load balance across pipeline stages.
Serverlessllm: Low-latency serverless inference for large language models. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 135–153. USENIX Association, 2024. [8] Philip Gage. A new algorithm for data compression. The C Users Journal, 12(2):23–38, 1994. [9] Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. Cost-efficient large language model serving for multi-turn conversations with cachedattention. In 2024 USENIX Annual Technical Conference (USENIX ATC 24), pages 111–126, 2024.
References
[10] John L Hennessy and David A Patterson. Computer architecture: a quantitative approach. Elsevier, 2011.
[1] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming throughputlatency tradeoff in llm inference with sarathi-serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 117–134, Santa Clara, CA, July 2024. USENIX Association.
[11] Sunpyo Hong and Hyesoon Kim. An integrated GPU power and performance model. In Proceedings of the 37th annual international symposium on Computer architecture, pages 280–289, 2010. [12] Dongwon Jo, Jiwon Song, Yulhwa Kim, and Jae-Joon Kim. Fastkv: Kv cache compression for fast longcontext processing with token-selective propagation. arXiv preprint arXiv:2502.01068, 2025.
[2] Adithya Bhaskar, Alexander Wettig, Tianyu Gao, Yihe Dong, and Danqi Chen. Cache me if you can: How many kvs do you need for effective long-context lms? arXiv preprint arXiv:2506.17121, 2025.
[13] Aditya K. Kamath, Ramya Prabhu, Jayashree Mohan, Simon Peter, Ramachandran Ramjee, and Ashish Panwar. Pod-attention: Unlocking full prefill-decode overlap for faster llm inference. In Proceedings of ASPLOS ’25, Rotterdam, Netherlands, 2025. ACM.
[3] Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, et al. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. arXiv preprint arXiv:2406.02069, 2024.
[14] Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023.
[4] Krishna Teja Chitty-Venkata, Siddhisanket Raskar, Bharat Kale, Farah Ferdaus, Aditya Tanikanti, Ken Raffenetti, Valerie Taylor, Murali Emani, and Venkatram Vishwanath. Llm-inference-bench: Inference benchmarking of large language models on ai accelerators. In SC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1362–1379. IEEE, 2024.
[15] Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Eduardo Blanco and Wei Lu, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium, November 2018. Association for Computational Linguistics.
[5] Databricks. Databricks-dolly-15k is an open source dataset of instruction-following records generated by thousands of Databricks employees in several of the behavioral categories outlined in the InstructGPT paper, 2024. https://huggingface.co/datasets/databr icks/databricks-dolly-15k. [6] Dell. Llama 2. https://infohub.delltechnologi es.com/ja-jp/l/llama-2-inferencing-on-a-s ingle-gpu/introduction-3976/, 2025.
[16] Ilya Kulikov, Alexander H Miller, Kyunghyun Cho, and Jason Weston. Importance of a search strategy in neural dialogue modelling. arXiv preprint arXiv:1811.00907, 2, 2018.
[7] Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yuvraj Patel, and Luo Mai.
[17] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, 12
Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626, 2023.
[28] Steven T Piantadosi. Zipf’s word frequency law in natural language: A critical review and future directions. Psychonomic bulletin & review, 21(5):1112–1130, 2014. [29] Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ramjee, and Ashish Panwar. vattention: Dynamic memory management for serving llms without pagedattention. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, pages 1133–1150, 2025.
[18] Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. Infinigen: Efficient generative inference of large language models with dynamic KV cache management. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 155–172, Santa Clara, CA, July 2024. USENIX Association. [19] Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: LLM knows what you are looking for before generation. arXiv preprint arXiv:2404.14469, 2024.
[30] Python. pickle — python object serialization, 2024. https://docs.python.org/3/library/pickle.h tml.
[20] Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024.
[32] Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: Trading more storage for less computation—a {KVCache-centric} architecture for serving {LLM} chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST 25), pages 155–170, 2025.
[31] Python. Asynchronous I/O. https://docs.python. org/3/library/asyncio.html, 2025.
[21] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024.
[33] Haoran Qiu, Weichao Mao, Archit Patke, Shengkun Cui, Saurabh Jha, Chen Wang, Hubertus Franke, Zbigniew Kalbarczyk, Tamer Başar, and Ravishankar K. Iyer. Power-aware deep learning model serving with u-Serve. In 2024 USENIX Annual Technical Conference (USENIX ATC 24), pages 75–93, Santa Clara, CA, July 2024. USENIX Association.
[22] Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, et al. Cachegen: Kv cache compression and streaming for fast large language model serving. In Proceedings of the ACM SIGCOMM 2024 Conference, pages 38–56, 2024.
[34] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Katrin Erk and Noah A. Smith, editors, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany, August 2016. Association for Computational Linguistics.
[23] Meta. meta-llama/Llama-2-7b. https://huggingfac e.co/meta-llama/Llama-2-7b, 2025. [24] Microsoft. Deepspeed, 2024. https://github.com /microsoft/DeepSpeed.
[35] SGLang. Sglang: Zero-overhead batch scheduler. http s://lmsys.org/blog/2024-12-04-sglang-v0-4/, 2024.
[25] OpenAI. OpenAI API Documentation. https://platform.openai.com/docs, 2025. [26] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
[36] SGLang. Sglang: Benchmark and profiling. https: //docs.sglang.ai/developer_guide/benchmark _and_profiling.html, 2025. [37] SGLang. Sglang documentation, 2025. https://docs .sglang.ai/index.html.
[27] David Patterson, Joseph Gonzalez, Urs Hölzle, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. The carbon footprint of machine learning training will plateau, then shrink, 2022.
[38] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. 13
[39] Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. Llumnix: Dynamic scheduling for large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 173–191, Santa Clara, CA, July 2024. USENIX Association.
[48] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023. [49] Jiaming Xu, Jiayi Pan, Yongkang Zhou, Siming Chen, Jinhao Li, Yaoxiu Lian, Junyi Wu, and Guohao Dai. Specee: Accelerating large language model inference with speculative early exiting. arXiv preprint arXiv:2504.08850, 2025. Accepted by ISCA 2025.
[40] Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025.
[50] Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, and Luis Ceze. Flashinfer: Efficient and customizable attention engine for llm inference serving. arXiv preprint arXiv:2501.01005, 2025.
[41] Qwen Team. Qwen2.5: A party of foundation models, September 2024. [42] Qwen Team. QwQ: Reflect deeply on the boundaries of the unknown. https://qwenlm.github.io/blog/qw q-32b-preview/, 2025. [43] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
[51] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for transformer-based generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521–538, 2022. [52] Ahmet Caner Yüzügüler, Jiawei Zhuang, and Lukas Cavigelli. Preserve: Prefetching model weights and kv-cache in distributed llm serving. arXiv preprint arXiv:2501.08192, 2025. [53] Hossein Entezari Zarch, Lei Gao, Chaoyi Jiang, and Murali Annavaram. Del: Context-aware dynamic exit layer for efficient self-speculative decoding. arXiv preprint arXiv:2504.05598, 2025. [54] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavyhitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:34661–34710, 2023.
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[55] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Sglang: Efficient execution of structured language model programs. arXiv preprint arXiv:2312.07104, 2024.
[45] vLLM. vLLM Asynchronous Scheduling. https: //github.com/vllm-project/vllm/pull/24799, 2025.
[56] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Distserve: Disaggregating prefill and decoding for goodputoptimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 193–210, Santa Clara, CA, July 2024. USENIX Association.
[46] vLLM. vllm: Optimization and tuning, 2025. https: //docs.vllm.ai/en/latest/configuration/opt imization.html. [47] vLLM. vllm: Parallelism and scaling, 2025. https: //docs.vllm.ai/en/latest/serving/paralleli sm_scaling.html. 14
[57] Rick Zhou. Benchmarking LLM inference backends: vLLM, LMDeploy, MLC-LLM, TensorRT-LLM, and TGI, 2024. https://bentoml.com/blog/benchmark ing-llm-inference-backends. [58] Kan Zhu, Yufei Gao, Yilong Zhao, Liangyu Zhao, Gefei Zuo, Yile Gu, Dedong Xie, Zihao Ye, Keisuke Kamahori, Chien-Yu Lin, et al. {NanoFlow}: Towards optimal large language model serving throughput. In 19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25), pages 749–765, 2025.
15
Stop Checking
Update Sequence
Seq Seq
…
Seq
Update Sequence
Stop Checking
Input Token ID(s) Token ID
Free Resources Output Processor
…
Decoding
Token
Text
Output Text Token ID Pair
Text
f()
h() ……
N
1&1
h(f()) ……
1&N
Single-token Lookup Table
De-tokenizer
h(f()) 1
Decoding is a crucial and time-consuming step in output processing for each sequence. However, because it depends on the thread-unsafe de-tokenizer, it must be executed sequentially. Albireo mitigates the time-consuming decoding process by using two fast lookup tables, thereby preventing frequent calls to the de-tokenizer. Let us first briefly explain the function of the de-tokenizer: Given a sequence s = ⟨id1 , id2 , . . . , idn ⟩, the de-tokenizer performs two decoding functions: f (⟨id1 , id2 , . . . , idn ⟩) = ⟨token1 ,token2 , . . . ,tokenn ⟩ and h(⟨token1 ,token2 , . . . ,tokenn ⟩) = text
Double-token Lookup Table
where f (⟨idi , id j ⟩) = ⟨ f (idi ), f (id j )⟩ and
Figure 18: Parallelized output processing.
A
h(⟨tokeni ,token j ⟩) ̸= h(tokeni ) + h(token j )
Parallel Output Processing
To reduce performance overhead, Albireo employs incremental decoding. Specifically, assuming the current output text of sequence s is textn , when a new sampled token ID idn+1 arrives, Albireo calculates the incremental text as:
Since existing inference frameworks [17, 24, 55] typically implement output processing via a python loop, they process each batch of sequences in a sequential manner. Consequently, the combined duration of scheduling, input processing, and output processing may, in some instances, exceed that of decoding forward, thereby impeding the effective overlap of CPU tasks with GPU operations. We break down the output processor’s sequence processing into four distinct steps and optimize each step individually. As illustrated in Figure 18, the output processor handles a sequence through four steps:
text ← h(⟨ f (idn ), f (idn+1 )⟩) − h( f (idn ))
(7)
and updates the output text by: textn+1 ← textn + text We derive two key insights from Equation 7: • Single-Token Lookup: The computation of h( f (idn )) can be reduced to an O(1) lookup in a table mapping token IDs to their corresponding text. This simplification is feasible because token IDs in LLMs are continuous and finite. For example, Qwen-2.5-14B has 152,064 token IDs ranging from 0 to 152,063, enabling Albireo to cache all tokens in a single-token lookup table (Figure 18).
• update sequence: The newly sampled token ID and its corresponding probability are appended to the sequence. • decoding: Each sequence is incrementally decoded using the new sampled token ID. • stop checking: The inference framework evaluates whether the sequence meets completion conditions, such as generating a specific string, encountering a special token ID, or reaching the maximum length.
• Double-Token Lookup: While the value space of idn and idn+1 pairs in h(⟨ f (idn ), f (idn+1 )⟩) is large and impractical to cache fully, Zipf’s law [28] suggests that many token pairs occur infrequently in natural language. Therefore, h(⟨ f (idn ), f (idn+1 )⟩) can be cached by storing commonly used pairs. In Albireo, we cache 109 pairs (take about 30 GB memory) for each model (doubletoken lookup table in Figure 18), achieving a hit rate of over 99%. When a lookup fails, the de-tokenizer is invoked to perform the decoding operation.
• free resources: Resources associated with finished sequences are released, and the generated results are returned. The update sequence and stop checking steps for different sequences are entirely independent, allowing Albireo to utilize thread pools for parallel execution of these calculations. In contrast, free resources depends on the KV cache management module and must be executed serially. However, since each sequence releases resources only once at the end of its lifecycle, the infrequent triggering of free resources has a negligible impact on overall performance.
These optimizations enable the output processing task to complete within 2 ms per iteration, ensuring seamless overlap between GPU and CPU tasks. 16