arXiv CSopen access

SLIM: Saturation-Aware Lightweight Performance Modeling for LLM Serving

SLIM: Saturation-Aware Lightweight Performance Modeling for LLM Serving Pol G. Recasens∗†‡ , Ferran Agullo∗†‡ , Yue Zhu§ , Chen Wang§ , Jordi Torres†‡ , Josep Ll. Berral‡† † Barcelona Supercomputing Center (BSC), {pol.garcia, ferran.agullo, jordi.torres}@bsc.es ‡ Universitat Politècnica de Catalunya - BarcelonaTech (UPC), {josep.ll.berral}@upc.edu

arXiv:2607.29575v1 [cs.DC] 31 Jul 2026

§ IBM Research, {Yue.Zhu, Chen.Wang1}@us.ibm.com

Abstract—Large language model (LLM) serving commonly increases batch size to improve throughput, but performance eventually reaches a deployment-dependent plateau beyond which larger batches provide marginal gains while increasing latency and GPU memory consumption. Previous studies have attributed this behavior to HBM/DRAM bandwidth limitations, but the underlying causes have primarily been supported by conceptual arguments or high-level performance observations. As our first contribution, we present a detailed GPU characterization using hardware profiling techniques, demonstrating that throughput saturation originates in the attention kernels during the decode phase. Specifically, we show that their nearly constant arithmetic intensity as active-context lengths increases—not merely larger batch sizes—drives DRAM-bandwidth saturation, while the achieved compute throughput remains far below the hardware limit. Building on this analysis, we present the Batching Configuration Advisor (BCA), which selects the highest-throughput batching configuration satisfying a target latency constraint and identifies up to 55 GB of GPU memory allocation that can be avoided for the evaluated OPT models with minimal throughput loss. To enable these recommendations, we introduce SLIM (Saturation-Aware Lightweight Performance Model), a semianalytical model that predicts LLM inference throughput and latency from analytical formulations of Transformer computation and memory traffic. Across the evaluated scenarios, SLIM outperforms representative performance-modeling baselines while successfully generalizing to previously unseen operating conditions.

I. I NTRODUCTION Large language models (LLMs) have become commoditized and are now widely used across personal and professional settings, supporting applications ranging from chatbots and voice interfaces to writing assistants and code-generation tools [1], [2], [3]. Despite their widespread adoption, the computational requirements of state-of-the-art LLMs remain beyond the capabilities of most end users, making cloud-based inference the dominant deployment model. Consequently, service providers must efficiently accommodate large numbers of concurrent requests exhibiting diverse characteristics, including varying context lengths, arrival rates, and latency requirements [4], [5]. At the same time, these providers are transitioning to heterogeneous hardware architectures with task-specific accelerators, while serving models of varying complexity across user tiers, further increasing system heterogeneity [6], [5]. * denotes equal contribution.

A key challenge in online LLM serving is understanding how this heterogeneity translates into accelerator-level performance. Among the factors affecting serving efficiency, the incoming request rate is particularly important because it determines the degree of batching that can be exploited during inference. Increasing batch size generally improves throughput by exposing greater request-level parallelism, albeit at the cost of higher request latency. However, throughput eventually reaches a saturation point beyond which additional parallelism provides only marginal performance gains, while latency continues to increase. At the same time, GPU memory consumption grows because additional requests require larger aggregate KV caches. Although this phenomenon, commonly referred to as the throughput plateau, has been widely observed in LLM serving systems, its underlying causes remain insufficiently explored. Several studies attribute this throughput saturation under large-batch workloads to the increasing cost of KV cache memory transfers, which are identified as the primary performance bottleneck [7], [8]. However, these conclusions are generally based on conceptual reasoning or on high-level performance observations, and lack detailed kernel-level analysis of the underlying execution behavior. In this work, we perform an in-depth characterization of LLM inference throughput using GPU kernel profiling and hardware-level analysis, demonstrating with low-level performance metrics that the throughput plateau is fundamentally caused by DRAM-bandwidth saturation in the attention kernels, resulting from excessive memory traffic during the decode phase. Furthermore, we show that this limitation is governed not only by batch size but also by the active context, defined as the total number of tokens involved at each processing step. Our analysis shows that matrix multiplication kernels increasingly benefit from larger active contexts, achieving higher hardware utilization as workload size grows. In contrast, attention kernels exhibit nearly constant arithmetic intensity, causing their bandwidth demands to increase proportionally with active context. Consequently, the memory-bandwidth demand of attention operations eventually saturates the available DRAM bandwidth, creating a critical knee point beyond which throughput can no longer scale. Identifying this memory-driven saturation mechanism enables a more principled approach to accelerator management.

Once DRAM bandwidth is saturated, further increases in batch size provide negligible throughput benefits despite consuming more GPU memory and increasing latency. Avoiding operating beyond this point can free GPU memory capacity for other workloads, model instances [9], or service tiers, improving overall system efficiency. We therefore introduce the Batching Configuration Advisor (BCA), a lightweight advisory framework that determines the most efficient operating point for a given accelerator. BCA selects the highest-throughput configuration that satisfies a target latency constraint and maintains a user-defined minimum batching efficiency. This criterion avoids operating deep within the throughput plateau, where additional concurrency provides diminishing returns while increasing latency and memory consumption. To support BCA, we further propose the Saturation-Aware Lightweight Performance Model (SLIM), a semi-analytical performance model for LLM serving. SLIM leverages the performance insights obtained from our characterization study to estimate throughput and latency across the heterogeneous operating conditions typical of service providers. The model combines interpretable analytical formulations of the computation and memory traffic generated by Transformer operations with a small set of deployment-specific constants calibrated through a lightweight profiling phase. This lightweight calibration enables BCA to reduce profiling time by 43.85% compared to exhaustive profiling. We compare SLIM against two state-of-the-art performance models, LLMVisor [10] and the model proposed by Imai et al. [11]. SLIM consistently outperforms both models when generalizing to previously unseen operating conditions, including different LLMs and activecontext lengths, reducing the mean absolute percentage error (MAPE) by an average of 79.3% across all evaluated scenarios. Beyond its role in BCA, SLIM may provide broader value for online service providers by enabling informed capacityplanning decisions and more efficient hardware utilization. Its latency predictions support service-tier configuration and quality-of-service management, while throughput estimates facilitate resource provisioning by determining the accelerator capacity required to meet workload demands and performance targets. In summary, our key contributions are: • We demonstrate that DRAM-bandwidth saturation in attention kernels is the primary driver of throughput saturation under large active-context workloads. Through detailed GPU-level characterization, we show that increasing the active context proportionally increases KV-cache memory traffic in the attention kernels, resulting in nearly constant arithmetic intensity. As a result, attention approaches the GPU memory-bandwidth limit, memory-related stalls increase, and compute resources remain underutilized, ultimately producing the throughput plateau. • We propose SLIM, a saturation-aware lightweight performance model for LLM serving. SLIM combines a semi-analytical formulation with a lightweight profiling phase to accurately predict serving throughput and

latency. It successfully generalizes to unseen operating conditions, including larger models and longer output lengths, achieving a throughput MAPE of 17.6% MAPE under combined model-and-output-length generalization and substantially outperforming the evaluated baselines. • We introduce the Batching Configuration Advisor (BCA), a lightweight advisory framework for accelerator operation. BCA selects the highest-throughput configuration that satisfies a target latency constraint and maintains a user-defined minimum batching efficiency. Experimental results show that BCA identifies configurations that reduce memory requirements by up to 55 GB for the tested OPT models, while requiring 43.85% less profiling time than exhaustive configuration search. Paper structure: The remainder of this paper is organized as follows. Sections II and III provide the necessary background and review the related work, with the former also illustrating the throughput plateau in practice. Section IV describes the methodology and experimental setup used throughout the paper. The three main contributions of this work are then presented and evaluated in separate sections. Section V presents the low-level performance analysis that identifies the causes of throughput saturation under large active-context workloads. Section VI introduces SLIM, the proposed performance model, and Section VII presents the configuration advisor for optimizing LLM serving. Finally, Sections VIII and IX discuss the implications of the results and conclude the paper. II. BACKGROUND A. LLM Inference and KV Caching Modern large language models such as OPT [12], GPT [13], and Llama [14] are trained using the next-token prediction objective, where output tokens are generated autoregressively conditioned on a prompt x. During inference, request processing consists of two phases. First, in the prefill phase, all prompt tokens are processed in parallel to generate the first output token. Subsequently, the model enters the decode phase, where new tokens are generated sequentially by conditioning on previously generated tokens. Generation terminates when an end-of-sequence token is produced or a predefined maximum length is reached. These models are built upon the Transformer architecture [15], which stacks multiple layers composed of a selfattention mechanism and a feed-forward network (FFN). Given an input representation X, the attention module first projects the input into queries Q, keys K, and values V using learnable weight matrices WQ , WK , and WV : Q = XWQ ,

K = XWK ,

V = XWV .

(1)

QK ⊤ √ dk

(2)

It then computes the output as:  Attention(Q, K, V ) = softmax

 V

Although the KV cache significantly reduces the computational cost of autoregressive generation, it introduces a memory-access bottleneck during the decode phase. For each newly generated token, the key and value tensors associated with all previously processed tokens must be retrieved from GPU main memory (HBM/DRAM) and transferred to the onchip memory hierarchy used by GPU execution kernels. Consequently, decoding exhibits low arithmetic intensity, defined as the ratio of floating-point operations (FLOPs) to bytes transferred from memory. Workloads with low arithmetic intensity are memory-bound, meaning that performance is constrained primarily by memory bandwidth rather than computational throughput. To alleviate the memory-bound nature of decoding, modern serving systems process multiple requests concurrently through batching [9], increasing the amount of computation performed per decoding step. However, traditional static batching is inefficient for LLM workloads, as requests can generate outputs of different lengths, causing some batch slots to remain idle while others continue decoding. To address this limitation, state-of-the-art serving systems employ continuous batching [16], [17], [18], [19], which dynamically admits new requests and removes completed ones between decoding iterations, maintaining high batch occupancy. The maximum batch size is ultimately constrained by GPU memory capacity. Since each active request maintains a growing KV cache, memory consumption increases with both sequence length and concurrency. To improve memory efficiency, modern serving frameworks such as vLLM [17] and S-LoRA [20] allocate KV-cache memory incrementally instead of reserving space for the maximum output length upfront, enabling higher request concurrency and better resource utilization. C. Throughput Saturation and the Latency Trade-off As discussed above, batching improves the performance of LLM serving systems by allowing multiple requests to be processed simultaneously. As illustrated on the left side of Figure 1, throughput increases with the batch size across all evaluated models. These gains, however, come at the cost of higher latency. Processing more requests in parallel increases

llama-2-13b llama-2-7b 10000

Latency (ms)

B. Memory Bottlenecks and Batching

overall system efficiency and throughput, but also prolongs the execution time experienced by each request. Depending on the target application, this throughput–latency trade-off may be acceptable or even desirable. However, throughput improvements diminish well before the GPU memory capacity is exhausted (marked by crosses in the figure), a phenomenon commonly referred to as the throughput plateau. For small batch sizes, throughput scales almost linearly with the number of requests processed concurrently. Beyond a certain point, however, the rate of improvement decreases significantly and eventually saturates, while latency continues to grow approximately linearly. As a result, allocating additional memory to support larger batches yields limited throughput benefits while incurring increasingly higher response times and GPU memory usage.

Throughput (tokens/s)

The resulting representations are then processed by the FFN, with residual connections and normalization applied around these sublayers. To avoid recomputing attention states for previously processed tokens during autoregressive decoding, modern LLM serving systems maintain a key-value (KV) cache in GPU memory. The KV cache stores the key and value tensors generated by each Transformer layer and allows attention computation for a newly generated token to reuse previously computed states. Consequently, the attention operation during decoding is reduced from a matrix-matrix multiplication QK T over the entire sequence to a matrix-vector multiplication qK T , where q is the query corresponding to the current token.

8000 6000 4000 2000 00

100

200

300

400

opt-1.3b opt-2.7b 60 40 20 00

100

200

300

400

Average Batch Size (reqs) Average Batch Size (reqs)

Fig. 1: Throughput (tokens/s) and latency (ms) evolution when setting the maximum batch size to values in range 1..512 across different models (OPT-1.3B, OPT-2.7B, Llama-2-7B and Llama-2-13B). The X-axis corresponds to the average batch size, rather than the configured maximum, and the crosses mark the point at which the KV cache capacity is exceeded due to the increased batch size. Results are obtained in the online mode described in Section IV. III. R ELATED W ORK A. LLM Inference Performance Analysis The performance of autoregressive LLM inference at low batch sizes is widely acknowledged to be constrained by memory bandwidth rather than computational throughput. While the prefilling phase is executed only once per inference request, the decoding phase, characterized by low arithmetic intensity, is repeatedly invoked for each generated token, making memory accesses the dominant factor affecting inference performance [21], [17]. This bottleneck is evidenced by the substantial body of work dedicated to increasing the arithmetic intensity of decoding. Representative approaches include chunked prefilling and speculative decoding, which improve hardware utilization by increasing the amount of computation performed per memory access [22], [23], [24]. Other techniques, such as multi-query and grouped-query attention [25], [26], as well as quantization methods [27], [28], seek to reduce KV cache memory traffic, thereby alleviating memory bandwidth constraints during inference. Nevertheless, limited attention has been devoted to understanding the performance limits of LLM inference at large

batch sizes and active context lengths. In particular, the fundamental causes of the throughput plateau regime discussed previously remain insufficiently characterized. Several studies suggest that KV-cache memory transfers become the primary bottleneck at large batch sizes [7], [8]. However, these explanations remain conceptual without a detailed lowlevel characterization. Consequently, the interplay between arithmetic intensity, memory traffic, and throughput saturation remains poorly understood and has not been systematically characterized at the kernel level. In this work, we take a step toward addressing this gap by providing a low-level analysis of modern LLM inference workloads under large batch sizes and active context lengths. Leveraging kernel-level measurements and roofline-based visualizations, we systematically analyze the factors governing throughput saturation and reveal the underlying causes of the observed plateau regime. B. Performance Modeling Recent studies have proposed performance models and simulators for LLM serving, although they target different optimization objectives. Vidur [29] is a large-scale simulation framework that models LLM inference operators using a combination of experimental profiling and predictive modeling, enabling end-to-end estimation of serving metrics across different workloads and deployment configurations. GenZ [30] models the compute, memory, and communication requirements of LLM inference across model, workload, and platform configurations, while LLMCompass [31] combines operatorlevel performance modeling with mapping exploration to evaluate accelerator designs. Similarly, AMALI [32] analytically models LLM inference on modern GPUs by accounting for architectural execution characteristics. Imai et al. [11] introduce a latency prediction model that combines roofline-based analytical estimates with regression models trained on serving information, but focus on small batch sizes and input lengths, without evaluating the model in the throughput saturation regime. Similarly, LLMVisor [10] introduces a roofline-guided latency attribution model for multi-tenant LLM serving, decomposing batch latency into per-request costs based on computation and memory-access features. These approaches are useful for simulating, estimating, or attributing serving performance, but they do not explicitly model the onset of the throughput plateau caused by decode-time attention bandwidth saturation or translate it into an active-context capacity recommendation. Sheng et al. [33] propose Virtual Token Counter (VTC), a fair scheduling policy that approximates service through token-based costs. However, token-count abstractions do not capture hardware bandwidth limits or the dependence of attention traffic on batch size and context length. Unlike these, SLIM extrapolates to unseen model scales and context lengths from a fixed calibration, without refitting. This enables SLIM to capture the hardware conditions under which batching stops providing proportional throughput gains, enabling knee-point estimation without exhaustively profiling the complete throughput curve.

C. Conference Extension A preliminary version of this work was presented at the IEEE International Conference on Cloud Computing [9]. This journal article substantially extends that prior work in two key directions. First, it generalizes the low-level analysis of throughput saturation from the batch-size dimension to large active-context scenarios, providing a more comprehensive characterization of the memory and computational bottlenecks that arise in modern LLM-serving workloads. Additionally, all evaluations have been repeated using a newer version of vLLM and a more recent set of LLMs. Second, it introduces SLIM, a performance model capable of predicting serving throughput and latency while requiring significantly fewer profiling measurements and generalizing effectively to previously unseen operating conditions. By enabling efficient exploration of the configuration space without exhaustive benchmarking, SLIM substantially reduces the profiling overhead associated with system tuning and serves as the foundation of the proposed BCA advisor. IV. M ETHODOLOGY We conduct our experiments using the vLLM framework [17], version 0.15.1. We use vLLM in two modes. In online mode, we follow a client-server architecture and send HTTP requests to a local server instance. In offline mode, used for the low-level GPU profiling in Section V with Nsight Systems and Nsight Compute, we instantiate vLLM directly in Python, injects synthetic requests into the scheduler and then runs the prefill and decode phases with explicit llm engine.step() calls. We retain the vLLM default setting but disable vLLM log statistics and prefix caching for controlled profiling runs. Workload. For the performance analysis, we generate synthetic prompts and independently vary the batch size, input length, and output length to isolate the impact of each factor on system performance. For the remaining evaluations, scenarios requiring fixed input and output lengths are generated using the random dataset generator provided by vLLM. In contrast, evaluations targeting realistic serving conditions use requests sampled from the cleaned ShareGPT dataset [34], thereby preserving the heterogeneous input and output length distributions observed in real-world workloads. Models. For the performance analysis, we evaluate two representative LLMs: Mistral-7B [35] and Granite-8B [36]. For all remaining experiments, we employ models from the OPT family [37], including OPT-125M, OPT-350M, OPT1.3B, OPT-2.7B, and OPT-6.7B. In addition, for large-scale multi-GPU settings, we employ models from the Qwen family [38], specifically Qwen-32B and Qwen-72B. Hardware. All experiments run on nodes equipped with four NVIDIA Hopper H100 (64GB HBM2), 512GB RAM memory, and 80 CPU cores. Single-GPU experiments request one GPU, while the Qwen-32B and Qwen-72B online experiments request two and four GPUs and use tensor parallelism. The launcher allocates 20 CPU cores per requested GPU. Accordingly, the SLIM performance model is parameterized

A. Impact of Active Context in Serving Performance We begin by revisiting the throughput plateau introduced in Figure 1. Whereas prior analyses have primarily examined this phenomenon as a function of batch size, we extend the characterization to include both input and output sequence lengths. Together, these dimensions determine the active context At , defined as the total number of tokens processed by the model at a given inference step. Formally, for a batch Bt at processing step t, we define the active context as: X At = (ILr + yr,t ) (3) r∈Bt

where ILr is the input length of request r and yr,t is the number of tokens already generated for that request. Figure 2 shows how throughput saturation evolves with batch size, input length, and output length. Although all configurations eventually reach a plateau, their saturation profiles differ, indicating that the plateau is governed not only by batch size but also by the total active context processed by the model. Increasing input length generally improves throughput because a larger fraction of execution is spent in the highly parallel prefill phase. In contrast, increasing output length extends the less efficient autoregressive decode phase, in which each generated token attends to a progressively longer context, thereby reducing execution efficiency and lowering throughput. B. System-level Analysis We continue the analysis by decomposing average inference time into the prefill and decode phases and examining how their contributions vary with batch size, input length, and output length. As shown in Figure 3, decode consistently

Throughput (tokens/s)

In this section, we present a low-level analysis of the throughput plateau observed under large active-context workloads. Unlike prior studies, our work is the first to provide a comprehensive explanation of this performance bottleneck using detailed GPU tracing data. We first examine how the different workload dimensions that determine the active context contribute to throughput saturation. We then investigate its underlying causes through a hierarchical analysis, beginning with the execution-time contributions of the prefill and decode phases and concluding with a kernel-level characterization of attention execution. Our findings reveal that DRAM-bandwidth saturation in attention kernels is the primary cause of the throughput plateau in large active-context scenarios. We observe that their arithmetic intensity remains nearly constant across increasing input lengths, output lengths, and batch sizes. This ultimately leads to memory-bandwidth saturation, leaving a significant portion of computational resources underutilized.

Throughput (tokens/s)

V. P ERFORMANCE A NALYSIS

128 512 OPT 1.3B: Input Length Sweep

1024 2048 OPT 1.3B: Output Length Sweep

30000 20000 10000 00

50

100

150

200

250

50

100

150

200

250

Batch size Mistral 7B: Input Length Sweep

20

40

60

80

100 120

150

200

Batch size Mistral 7B: Output Length Sweep

30000 20000 10000 00

Batch size

0

50

100

Batch size

250

Fig. 2: Throughput evolution under varying input lengths (left) and output lengths (right) across different batch sizes. The top row corresponds to OPT-1.3B, while the bottom row presents the results for Mistral-7B. Missing data points at larger batch sizes correspond to out-of-memory failures, as the available GPU memory was insufficient to accommodate the increased KV-cache footprint associated with longer active contexts. dominates total execution time across all evaluated scenarios, making it the primary contributor to inference latency and the most likely source of the observed throughput plateau. Although the relative contribution of prefill increases with batch size and input length, it remains substantially smaller than that of decode, even for the largest configurations. Increasing input length raises total execution time modestly (below 2s), because the additional prefill computation is highly parallelizable. In contrast, increasing the output length has a much larger effect, extending the execution time to 26s for 2048 generated tokens. As output length increases, the prefill contribution becomes negligible, further confirming decode as the dominant component of end-to-end inference time.

Elapsed time (s)

using the configured peak hardware capabilities of the H100 GPU, namely 989 TFLOP/s of tensor-core computational throughput and 1.62 TB/s of memory bandwidth.

30

IL=256, OL=256

Decode

Prefill

BS=16, OL=128

BS=16, IL=128

26.08s

20 10 0

3.16s 32

4.39s

5.91s

128

256

Batch size

12.30s 1.46s 256

1.59s

1.77s

2.93s

1024

2048

256

Input len

1024

Output len

2048

Fig. 3: Breakdown of total inference time into prefill and decode phases for the Mistral-7B model under increasing batch size, input length, and output length. In light of these observations, we focus the remainder of our analysis on the decode phase. Figure 4 presents GPU execution metrics for the full decode phase across workloads with varying batch sizes, input lengths, and output lengths. We consider three complementary metrics. First, Tensor Core Activity (%) measures the utilization of the Tensor Core execution

Utilization (%)

100

Tensor Core Activity IL=256, OL=256

DRAM read -0.5pp 50 Tensor core +15.3pp SM issue +5.1pp 0 32

128

Batch size

256

DRAM Read Bandwidth BS=16, OL=128

SM Issue Peak BS=16, IL=128

DRAM read +2.9pp Tensor core +1.3pp SM issue +0.5pp

DRAM read +2.8pp Tensor core +0.8pp SM issue +0.4pp

256

1024

Input len

2048

256

1024

Output len

2048

Fig. 4: Evolution of average Tensor Core activity, SM instruction-issue throughput, and DRAM read-bandwidth utilization during the decode phase of Mistral-7B as batch size, input length, and output length increase. Crosses denote the corresponding peak values observed during each execution. C. Kernel-level Analysis To further investigate the decode phase, we analyze how its execution time is distributed across GPU kernels. Figure 5 reports the average contribution of each kernel category. Decode time is dominated by the attention mechanism and the GEMM operations associated with the feed-forward network (FFN), the two main computational components of the Transformer block. As the active context increases, attention accounts for a progressively larger share of decode time, regardless of whether this increase is driven by batch size, input length, or output length. Moreover, in the batch-size regime where the throughput plateau emerges (i.e., batch sizes greater than 64, as shown in Figure 2), attention becomes the dominant execution component, exceeding the contribution of FFN computations.

These results identify attention as the primary source of the throughput plateau under large active-context workloads. 100

Time (%)

units, capturing the activity of the dense matrix multiplication operations that dominate Transformer execution. Second, SM Issue (%) quantifies the achieved instruction-issue throughput across the SM warp schedulers, providing a broader view of execution activity beyond operations executed on the Tensor Cores. Finally, DRAM Read Throughput measures the fraction of available memory bandwidth used to transfer data from GPU main memory to the on-chip cache hierarchy, providing a direct indication of memory pressure during execution. Overall, GPU execution remains far from its Tensor Core and instruction-issue limits, with average Tensor Core activity below 30% and average SM Issue below 10% across all evaluated configurations. In contrast, average DRAM readbandwidth utilization remains close to 80% and frequently reaches peaks near 100%, providing strong evidence that throughput saturation under large active-context workloads is primarily driven by memory-bandwidth limitations. At batch size 256, which already lies within the throughput plateau as shown in Figure 2, DRAM bandwidth is near saturation while Tensor Core activity and SM Issue remain substantially below their peak levels. Across the three scenarios, DRAMbandwidth utilization changes by only −0.5, 2.9, and 2.8 percentage points because it is already close to its limit. These results indicate that increasing the active context primarily intensifies memory pressure without producing a comparable increase in Tensor Core activity or instruction-issue throughput.

Attention GEMM Other IL=256, OL=256 BS=16, OL=5012

CPU/idle BS=16, IL=5012

32

256

50 0

128

Batch size

256

256

2048

Input len

5012

2048

Output len

5012

Fig. 5: Contribution of GPU kernels to the execution time of the decode phase as the active context increases for Mistral7B. The execution times of the attention and GEMM kernels (corresponding to the FFN block) are reported separately as the dominant contributors, while the remaining kernels are aggregated under Other. We next examine the attention kernel in greater detail. Figure 6 presents its roofline characterization across increasing active-context lengths for Mistral-7B and Granite-8B. The roofline model relates kernel performance to arithmetic intensity (AI), defined as the number of arithmetic operations performed per byte transferred from DRAM. Kernels with low AI operate in the memory-bound regime (red), where performance is limited by memory bandwidth, whereas kernels with high AI operate in the compute-bound regime (green), where performance is limited by computational throughput. The roofline boundaries are defined by the device’s peak singleprecision compute throughput and peak DRAM bandwidth, representing the accelerator’s theoretical maximum compute performance and memory-transfer capability, respectively. The complete quantitative results are reported in Table I. As shown in Figure 6, the attention kernel remains in the memorybound regime across all evaluated configurations, with arithmetic intensity varying only marginally as the active context increases. Furthermore, GPU compute capacity also remains largely underutilized. Across all evaluated configurations, the kernel sustains less than 6 TFLOP/s, more than two orders of magnitude below the hardware’s theoretical peak Tensor Core throughput of 989 TFLOP/s. These results demonstrate that increasing the active-context size —whether by increasing batch size, input length, or output length—does not shift the attention kernel toward the compute-bound regime, because the additional computation is accompanied by a proportional increase in DRAM traffic. Accordingly, as the active context increases, the attention kernel follows an almost vertical trajectory in the roofline model, reflecting its nearly constant arithmetic intensity while performance approaches the DRAM-bandwidth ceiling. Memory bandwidth therefore becomes the limiting resource, preventing further performance gains once saturation is reached. This behavior is evident at batch size 256, which lies within the throughput plateau regime: the attention kernel operates near the roofline’s memory-bandwidth limit and sustains approximately 90% average DRAM-bandwidth utilization. Although this average is computed over the full kernel execution,

Attention

IL = 256, OL = 256 4 34 258

B = 4, OL = 256 256 2048 16384

IL = 256, OL = 256 4 8 34

B = 8, OL = 256 256 2048

Peak memory bandwidth

B = 4, IL = 256 256 2048 16384

L1/TEX OL=128 OL=5012

2048

Input length

Hit rate (%)

10 8 6 4 2 0 256

5012

10 8 6 4 2 0 256

L2 OL=128 OL=5012

2048

Input length

5012

Fig. 7: L1/TEX and L2 cache hit rates of the attention kernels executed during the final decode step of Mistral-7B as input length increases at a fixed batch size of 8. Each curve corresponds to a different output length.

1013

1012

1014

B = 4, IL = 256 256 2048 16384

1013

1012 10 2

10 1

100

101

102

10 2

10 1

100

101

102

Arithmetic Intensity (FLOP/byte)

10 2

10 1

100

101

102

Fig. 6: Roofline characterization of the attention kernels executed during the final decode step under increasing batch size, input length, and output length. The top row corresponds to Mistral-7B, while the bottom row presents the results for Granite-8B. D. Low-level Memory Transfer Analysis We conclude the analysis by examining two additional aspects of the attention kernel’s memory behavior. Specifically, we analyze access locality across the GPU cache hierarchy and the fraction of memory-related stalls. Together, these metrics provide further evidence of DRAM-bandwidth saturation. Figure 7 reports the hit rates of the L1/TEX and L2 caches. These caches exploit temporal and spatial locality by serving memory requests before they reach DRAM. Since data is transferred in cache lines rather than individual bytes, effective locality enables data reuse and reduces DRAM traffic. However, as shown in Figure 7, both cache levels exhibit consistently low hit rates across the evaluated active-context sizes. The L1/TEX hit rate remains below 1%, while the L2 hit rate never exceeds 7%. Moreover, cache locality also decreases as the active context grows. These results indicate limited data reuse in the attention kernel, which further deteriorates as the context length increases, leading to repeated DRAM accesses and greater memory-bandwidth pressure, consistent with previous observations. Finally, Figure 8 reports the Long Scoreboard Stalls, which represent the percentage of active warps stalled per cycle on memory operations from the GPU memory hierarchy (L1/TEX, L2, or DRAM), and the Occupancy, which reports the percentage of active warps resident on a Streaming Multiprocessor (SM) relative to the hardware maximum. As the active context increases, we observe that the proportion of long scoreboard stalls increases substantially, ultimately

accounting for more than 50% of active warp cycles in the largest active-context configurations. On the other hand, occupancy remains constant and below 20%, suggesting limited latency-hiding capacity. Together with the previously observed DRAM-bandwidth saturation and low Tensor Core and SM utilization, these results indicate that many resident warps remain blocked on memory accesses while computational resources are underutilized. 100 75 50 25 0 4

Long scoreboard stall Occupancy IL=256, OL=128 BS=4, OL=128

BS=16, IL=256

Metric (%)

Performance (Flop/s) Granite 8B Mistral 7B

1014

Single precision roofline

Hit rate (%)

instantaneous utilization reaches 100%, as shown in Figure 4. The same behavior also emerges as input and output lengths increase, since all three workload dimensions raise memory traffic while leaving attention arithmetic intensity essentially unchanged. These results identify DRAM-bandwidth saturation as the primary cause of the attention-kernel performance bottleneck, explaining the increasing slowdown observed in Figure 5 under large active contexts.

8

Batch size

16 256

2048

Input len

16384 128

Output len

5012

Fig. 8: Percentage of Occupancy and Long Scoreboard Stalls for the attention kernels executed by Mistral-7B during the final decode step under increasing batch size, input length, and output length.

E. Concluding Remarks Collectively, these results show that the throughput plateau under large active-context workloads is driven by DRAMbandwidth saturation in attention kernels. As the active context increases, their arithmetic intensity remains nearly constant, causing memory traffic to scale with computation until the available DRAM bandwidth is saturated, as demonstrated by the roofline analysis in Figure 6. Because attention dominates decode time at large active contexts (Figure 5), and decode itself dominates overall execution time in this regime (Figure 3), this bottleneck propagates to the end-to-end throughput and produces the observed throughput plateau. The low cache hit rates in Figure 7 and the increasing memory-related stalls in Figure 8 provide further evidence of this behavior. VI. P ERFORMANCE M ODELING In this section, we present SLIM, a semi-analytical model for predicting LLM inference throughput and latency. SLIM builds on analytical formulations of the computation and memory traffic incurred by Transformer operations during the prefill and decode phases. These formulations are simplified to

TABLE I: Quantitative results corresponding to Figure 6 for Mistral-7B, reporting roofline measurements of the attention kernel during the final decode step under increasing batch size, input length, and output length. The active context is defined as B(IL + OL), where B is the batch size, IL the input length, and OL the output length. Sweep

Configuration (B, IL, OL)

Active context (tokens)

Arithmetic Intensity (FLOP/byte)

Performance (TFLOP/s)

DRAM Avg. Utilization (%)

Batch size Batch size Batch size

(4, 256, 256) (34, 256, 256) (258, 256, 256)

2,048 17,408 132,096

3.95 3.87 3.96

1.79 4.25 5.89

28.0 67.9 91.9

Input length Input length

(4, 2048, 256) (4, 16384, 256)

9,216 66,560

3.79 3.97

3.27 5.61

53.2 87.2

Output length Output length

(4, 256, 2048) (4, 256, 16384)

9,216 66,560

3.79 3.97

3.24 5.61

52.8 87.3

improve interpretability while preserving predictive accuracy, and are complemented with a small set of constants calibrated from empirical profiling data. A key advantage of SLIM is that it achieves accurate predictions from a small set of profiling measurements while generalizing to unseen execution conditions. This capability stems from explicitly modeling the key factors governing inference performance, including model architecture, sequence length, and batch size, rather than relying solely on data-driven fitting. SLIM serves as the foundation of the proposed BCA advisor, enabling configuration decisions across diverse LLM serving scenarios with limited profiling overhead. Beyond BCA, SLIM can also be applied to a broader range of optimization tasks, including autoscaling, system configuration exploration, and deployment optimization. A. Preliminaries We consider inference for decoder-only Transformer models characterized by the number of layers L, hidden dimension D, query heads Hq , key-value heads Hkv , and head dimension dh . Assuming D = Hq dh , we define the total key/value projection dimension as Hkv Dkv = Hkv dh = D . (4) Hq We estimate throughput and latency over a representative workload execution window. Specifically, the model considers a batch of size B composed of requests with fixed input length IL and output length OL. These parameters represent the average workload characteristics over the full execution and can be obtained from runtime profiling or external estimates. This abstraction simplifies the modeling of dynamic serving workloads while retaining predictive accuracy. B. Latency and Throughput As shown in Equation 5, we model end-to-end (E2E) batch execution latency as the sum of the prefill time Tpre and the decode time Tdec , which are described in the following sections, and a calibrated constant tfixed , which captures batchlevel system overheads not explicitly modeled, such as request scheduling and runtime management. E2E(IL, OL, B) = Tpre + Tdec + tfixed

(5)

Assuming that all requests in the batch complete simultaneously, this corresponds to their end-to-end latency. Total-token throughput is then computed by dividing the total number of processed tokens, B(IL + OL), by the total execution time. T (IL, OL, B) =

B(IL + OL) E2E(IL, OL, B)

(6)

C. Prefill Time We model the prefill time from its computational cost, as the prefill phase is predominantly compute-bound and memorytransfer overheads are comparatively small [22]. As shown in Equation 7, prefill time is estimated by dividing the total floating-point operations (FLOPs), Fpre , by the accelerator’s peak Tensor Core throughput, Fmax (FLOP/s), scaled by a calibrated compute-efficiency factor, ηc . This factor captures the gap between theoretical peak performance and the effective throughput achieved in practice due to kernel-level and implementation-specific inefficiencies. Tpre =

Fpre ηc Fmax

(7)

The total number of floating-point operations, Fpre , is obtained by summing the self-attention and feed-forward network (FFN) operations across all Transformer blocks, counting each multiply-accumulate as two floating-point operations: h i
Fpre = L B ∗ IL 4D2 + 4DDkv + 4rff D2 + 4BIL2 D (8) The 4D2 term accounts for the query and output projections, 4DDkv for the key and value projections, and 4rff D2 for the two FFN projections. Here, rff = Dff /D denotes the ratio between the FFN hidden dimension and the model dimension—for OPT models, rff = 4 [15]. Under the full-matrix attention formulation, computing QK ⊤ and multiplying the resulting attention probabilities by V each requires 2B IL2 D FLOPs, yielding the 4B IL2 D self-attention term. We use Dkv so that the projection cost reflects the reduced KV width of grouped-query attention (GQA) and multi-query attention (MQA) architectures [25], [26]. D. Decode Time For decode time, as shown in Equation 9, we model two components: the dense decode time Td , and the KV-cache

memory-transfer time, obtained by dividing the total memory traffic Mdec by the peak device bandwidth BWmax . Unlike prefill, decode becomes increasingly constrained by memory traffic under large-context workloads, as demonstrated in Section V. We further introduce a calibrated decode-efficiency factor, ηm , and a small batch offset, B0 , to capture the nonlinear scaling of decode execution at small batch sizes.

MQA reduce Dkv and therefore the corresponding memory traffic.

"

We evaluate SLIM using measured vLLM serving traces in two settings: accuracy on held-out batch sizes from the fitted serving curves, and generalization to unseen model sizes, input lengths, and output lengths without refitting. We estimate the parameters of the semi-analytical model on the training set by minimizing the sum of squared relative errors between measured and predicted throughput using nonlinear least squares. Each observation corresponds to a serving configuration defined by its model, input length, output length, and batch size. We evaluate prediction accuracy by reporting the mean absolute percentage error (MAPE) between the profiled and estimated total-token throughput and E2E latency. In our evaluation, we consider five generalization scenarios, summarized in Table II. First, the same-curve evaluation holds out batch-size points from each measured throughput curve. For every model and output-length curve, we use three points in the pre-knee regime for fitting and the remaining points for validation. This tests whether the estimator can recover the throughput-saturation shape from sparse profiling points on the same curve. Second, the model-transfer evaluation fixes the output length at OL=384 and fits on smaller OPT models, using OPT-125M, OPT-350M, OPT-1.3B, and OPT-2.7B for training and OPT-6.7B for validation. Third, the output-lengthtransfer evaluation fixes the input length and fits on shorter generations, using OL ∈ 32, 128, 256 for training and OL=512 for validation. Fourth, the input-length-transfer evaluation uses OPT-1.3B as the fixed model and the output length to OL=256, fits on IL ∈ 128, 512, and validates on IL=1024. Finally, the combined setting fits a single global parameterization on the union of the model-transfer and output-length-transfer training sets and evaluates this single fitted model on OPT-6.7B at OL=512, a configuration that is simultaneously unseen in both model scale and output length. This evaluation scenario tests whether one fitted model can generalize across both model scale and generation length. We compare SLIM against two higher-level predictors adapted to our profiling setting. First, we construct LLMVisorAgg, an adaptation of LLMVisor [10], which uses piecewise fitted latency functions for prefill and decode, with aggregate token-count features for batch size, context tokens, and squared prefill length. The original LLMVisor trains its piecewise functions from per-engine-step execution times. Our profiling interface instead provides one mean TTFT and one mean ITL measurement for each (model, B, IL, OL) configuration. We therefore fit the prefill and decode functions to mean TTFT and mean ITL, respectively, and reconstruct end-to-end latency. We additionally adapt the out-of-domain linear-regression approach of Imai et al. [11], labeled as

Tdec =

Mdec B + B0 Td + ηm BWmax

# (9)

The dense-computation term Td is modeled using an architecture-aware scaling formulation with calibrated coefficients α and γ:  Td = αOL + γL

D 1024

2 OL

(10)

where the first term captures per-output-token overhead through the calibrated coefficient α, accounting for residual costs such as decode-step scheduling and other lightweight operations. The second term models the execution time of the dense projection and feed-forward network (FFN) kernels through the calibrated coefficient γ. Its dependence on model architecture is expressed by the dominant scaling with the number of Transformer layers, L, and the square of the hidden dimension, D, which reflects the matrix-multiplication cost. Normalizing D by 1024 improves numerical stability during parameter calibration. Although both prefill and decode use calibrated parameters, their different parameterizations reflect their execution regimes. Prefill processes BIL tokens concurrently, producing large matrix multiplications with sufficient parallelism for performance to be modeled as a calibrated fraction of Fmax . In contrast, decode processes only one new token per active request at each step. A fixed fraction of Fmax is therefore less representative in this regime, so γ directly absorbs the associated runtime overheads. Finally, the memory-transfer workload, Mdec , is modeled as Mdec = 2LDkv bkv C(IL, OL),

(11)

which represents the total memory traffic generated by KVcache reads during decoding. The parameter Dkv denotes the effective KV width, accounting for the reduced KV-cache size of GQA and MQA architectures. The function C(IL, OL), defined in Equation 12, captures the dependence of KV-cache accesses on sequence length: traffic scales linearly with the input length and quadratically with the output length because decoding repeatedly scans a growing KV cache. Since each context position stores both a key and a value vector, with each element occupying bkv bytes, the total KV-cache traffic per request is given by 2LDkv bkv C(IL, OL). For standard multi-head attention (MHA), Dkv = D, whereas GQA and

C(IL, OL) =

OL X

(IL + j) = OL(IL + 1) +

j=1

OL(OL − 1) 2 (12)

E. Evaluation Methodology

Throughput (tokens/s) E2E latency (ms)

Input-Length Generalization

Train measured

MAPE 0.7%

40000

Held-out measured

Output-Length Generalization

MAPE 5.6%

SLIM prediction

Model Generalization

Model and Output-Length Generalization

MAPE 24.3%

MAPE 17.6%

20000 00

50

100

150

200

250

MAPE 0.7%

100

200

300

400

500

0

MAPE 5.5%

100

200

300

400

500

0

MAPE 16.5%

100

200

300

400

500

200

300

400

500

MAPE 13.4%

10000

1000

0

50

100

150

200

Mean batch size

250

100

200

300

400

Mean batch size

500

0

100

200

300

400

Mean batch size

500

0

100

Mean batch size

Fig. 9: Performance prediction accuracy when evaluating unseen input length, unseen output length, unseen model, and unseen model and output length. Each column reports one held-out scenario, with throughput shown on the top row and end-to-end latency on the bottom row as a function of mean batch size. Gray curves correspond to training measurements, while colored curves compare held-out measurements against SLIM predictions. TABLE II: Held-out validation settings used for SLIM’s prediction-accuracy evaluation. B is the defined batch size while IL/OL denote input and output length, respectively. Setting

Training set

Same curve

All fitted model–OL curves, B ∈ {1, 4, 8} Model OPT-125M, OPT-350M, OPT1.3B, OPT-2.7B at OL = 384 Input length OPT-1.3B, IL ∈ {128, 512} Output OPT-1.3B, OL ∈ length {32, 128, 256} Model Union of the Model and Output + output length training sets above length

Validation target Same curves, B ∈ {2, 16, 32, 64, 128, 512} OPT-6.7B, OL = 384 OPT-1.3B, IL = 1024 OPT-1.3B, OL = 512 OPT-6.7B, OL = 512

IMAI OOD-LR. This LR baseline predicts end-to-end latency from workload variables and numerical model characteristics, including batch size, input and output lengths, hidden dimension, layer and attention-head counts, and interaction features. Throughput is then derived from the predicted latency and total token count. F. Experimental Evaluation 1) Model Prediction Accuracy: Figure 9 compares measured and predicted throughput-latency curves under heldout serving configurations. Despite being calibrated from sparse measurements on smaller models and shorter generation lengths, SLIM captures the saturation trend for unseen input lengths, output lengths and model scales. Input- and outputlength generalization scenarios achieve throughput MAPE values below 6%, because these scenarios primarily extend the same computation and KV-cache scan mechanisms represented by the model. Model generalization is more challenging, yielding a throughput MAPE of 24.3%, because changing the model simultaneously affects architectural characteristics,

kernel behavior, and runtime overheads that are not fully captured analytically. Nevertheless, even in this setting, SLIM reconstructs the overall throughput-saturation curve, supporting its use to reduce the profiling effort required to identify efficient serving configurations. Table III quantifies the resulting prediction accuracy and compares SLIM with the two baselines. LLMVisor captures architectural and hardware effects only indirectly through fitted coefficients, but performs well when extrapolating across output lengths. Imai et al. encode model characteristics as regression features and achieve higher accuracy under inputlength generalization. In contrast, SLIM explicitly models the decode-time KV-cache traffic responsible for throughput saturation and the model-dependent dense-computation cost, yielding more accurate predictions under model and output length generalization. Across all scenarios, SLIM reduces average MAPE by 79.3% relative to the two baselines. This advantage is preserved in the most challenging scenario, where both the model and output length are unseen during training, with an average MAPE reduction of 78.7% over the baselines. Figure 10 further illustrates this trend, where baseline models generalize poorly to an unseen LLM, whereas SLIM closely follows the profiled throughput and latency curves. 2) SLIM Ablation: We assess the contribution of SLIM’s two decode components through the ablation study summarized in Table IV. Removing the memory-transfer term, Mdec , substantially degrades output-length generalization, increasing throughput MAPE from 5.6% to 33.0%. In contrast, removing the dense computation term, Td , primarily affects model generalization, increasing the throughput MAPE from 24.3% to 31.6%. These results show that Mdec effectively captures the context-dependent memory traffic incurred during decoding, whereas Td captures the architecture-dependent densecomputation cost. The two terms therefore make complemen-

TABLE III: Prediction error under held-out batch-size, sequence-length, and model-scale configurations. We report MAPE (%) for total throughput and end-to-end latency; lower values indicate higher prediction accuracy. SLIM (ours)

Held-out setting

Validation target

Same curve Input length Model Output length Model + output length

Held-out batch sizes IL=1024, OL=256 OPT-6.7B, OL=384 OPT-1.3B, OL=512 Combined

SLIM

20000

E2E latency (ms)

Throughput (tokens/s)

Held-out measured

15000 10000 5000 100

200

LLMVisor

E2E latency

Throughput

E2E latency

Throughput

E2E latency

15.1 0.7 24.3 5.6 17.6

17.5 0.7 16.5 5.5 13.4

55.8 68.1 141.8 7.7 95.8

81.2 255.2 59.9 8.9 43.7

55.3 1.2 39.9 38.9 91.7

42.9 1.2 72.8 25.9 60.2

IMAI-LR

15000 10000

300

5000 100

200

300

Mean batch size

Fig. 10: Held-out prediction for an unseen model. The estimators are fitted on OPT-125M, OPT-350M, OPT-1.3B, and OPT-2.7B at OL=384, then evaluated on OPT-6.7B at OL=384.

observed. We also include in Figure 11 the Mistral-7B model, whose architecture differs from the OPT models used to calibrate SLIM’s coefficients. In particular, Mistral-7B employs grouped-query attention rather than standard multi-head attention. By accounting for its reduced KV width through Dkv in Equation 4, SLIM closely follows the measured throughput and latency curves across the evaluated batch sizes, including their characteristic saturating behavior. These results suggest that SLIM generalizes not only to unseen model scales within the OPT family but also to an unseen attention architecture.

Model

Output length

Variant

Full SLIM w/o KV scan w/o dense decode Full SLIM w/o KV scan w/o dense decode

MAPE (%) Throughput

E2E latency

24.3 23.5 31.6 5.6 33.0 8.1

16.5 20.9 18.9 5.5 24.8 8.2

tary contributions to SLIM’s predictive accuracy. 3) Large Model Generalization and Limitations: We further evaluate SLIM’s ability to generalize to larger unseen models and tensor-parallel deployments. Figure 11 compares measured throughput and end-to-end latency with SLIM predictions for Qwen-32B and Qwen-72B, served on two and four H100 GPUs, respectively. The results show that SLIM captures the throughput-saturation trend as batch size increases, but overestimates the performance gap between the two models. Under tensor parallelism, Qwen-72B is sharded across four GPUs, resulting in a per-GPU parameter footprint of approximately 16.5 GB, close to the roughly 15 GB per GPU of Qwen-32B. Their measured throughput curves are therefore similar, suggesting that per-GPU workload dominates and that tensor-parallel communication overhead is limited in this setting. Because SLIM does not explicitly model tensor-parallel sharding, it treats Qwen-72B as a uniformly larger model and consequently predicts a larger throughput reduction than

qwen-72b

mistral-7b

E2E latency (ms)

Throughput (tokens/s)

qwen-32b

TABLE IV: Component ablation of SLIM under the held-out model-transfer and output-length-transfer settings. Values are MAPE (%) relative to the measured throughput and end-toend latency curves; lower is better. Transfer setting

IMAI OOD-LR

Throughput

20000

Mean batch size

LLMVisor-Agg

20000 15000 10000 5000 0 0

50

100

150

200

mean batch size

250

measured

SLIM

25000 20000 15000 10000 5000 0 0

50

100

150

200

mean batch size

250

Fig. 11: Predicted and measured throughput and end-to-end latency for large Qwen models under tensor-parallel serving and for Mistral-7B as an unseen grouped-query-attention architecture. Solid curves report measured performance, while dash-dotted curves report SLIM predictions using coefficients fitted only on single-GPU OPT measurements.

VII. M ODEL -G UIDED BATCHING C ONFIGURATION In this section, we introduce the SLIM-guided Batching Configuration Advisor (SLIM-BCA), an offline, modeldriven advisor that recommends a batch size cap Bcap by jointly accounting for the predicted throughput knee and a latency SLO. Rather than exhaustively profiling the serving system, SLIM-BCA evaluates candidate configurations using SLIM’s fitted throughput and latency estimates. As shown in Section V, increasing the batch size beyond a deploymentspecific knee provides diminishing throughput returns once decode-time attention approaches DRAM-bandwidth saturation, while latency and KV-cache requirements continue to grow. SLIM-BCA therefore selects the largest admissible batch size that remains within the efficient pre-saturation region and satisfies the SLO. This operating point preserves nearmaximum throughput while avoiding unnecessary KV-cache allocation, thereby releasing memory capacity that can be used

TABLE V: SLIM-BCA decision variation as more profiling curves are added. Throughput variation and SLO violation are evaluated against the full-profile BCA decision on measured data and ϵ=0.1. Target

Train

N

Thr. Var

SLO Viol.

OL-512 OL-512 OL-512 OPT-6.7B OPT-6.7B OPT-6.7B

16+32 16+32+48 16+32+48+64 125M 125M+350M 125M+350M+1.3B

5 8 13 9 18 29

34.6% 0.0% 0.0% -28.4% 15.1% 15.1%

0.0% 0.0% 0.0% 6.9% 0.0% 0.0%

to serve additional requests, model instances, or co-located workloads. A. Formulation Formally, Equation 13 defines Bcap as the batch size that maximizes throughput among configurations satisfying the latency SLO and minimum batching-efficiency constraint. SLIM-BCA uses the performance model to estimate T (IL, OL, B) directly from model, workload, and accelerator characteristics. Then, given a workload-dependent IL and OL, the advisor searches over batch size candidates B and returns the upper-bound estimate Bcap for the specified SLO and ϵ.

(13)

The search in Equation 13 is subject to two constraints. First, the predicted latency E2E(IL, OL, B) must not exceed the specified SLO. Second, the ratio T (IL, OL, B)/(B · T (IL, OL, 1)), which is the throughput at batch size B relative to what would be achieved if performance scaled linearly with B = 1, must remain above a user-defined threshold ϵ. This prevents the advisor from selecting a batch size that is in the plateau regime. Both SLO and ϵ are userdefined parameters that trade off latency guarantees against batching efficiency. In our experiments, we define a tight SLO as 2∗E2E(2, 128, 32), and a loose SLO as 4∗E2E(2, 128, 32), which is four times the latency at a specific profiling point. B. Evaluation of SLIM-BCA We evaluate SLIM-BCA along three dimensions: the quality of its recommendations relative to a fully profiled ground truth (Section VII-B1), the reduction in profiling effort required to obtain those recommendations (Section VII-B2), and the resulting savings in KV-cache allocation (Section VII-B3). 1) Profiling Performance: Table V evaluates SLIM-BCA as a low-cost alternative to full offline profiling, which requires profiling the system for each candidate batch size. Instead, SLIM-BCA fits the performance model on a small set of profiling points and uses the predicted curve to find the optimal batch size with the same BCA decision rule. We then compare

Model (opt-6.7b) SLO multiplier

Throughput retained (% of profiled peak)

KV-cache memory saved (GiB vs. max profiled context) 45

3×

76.4

63.3

49.3

41.8

70

3×

16.2

20.3

24.4

26.4

2×

49.3

41.8

41.8

29.1

60

2×

24.4

26.4

26.4

28.4

1.5×

41.8

41.8

29.1

24.0

50

1.5×

26.4

26.4

28.4

29.5

30

1.25×

29.1

29.1

24.0

24.0

1.25×

28.4

28.4

29.5

29.5

25

1×

24.0

24.0

24.0

17.0

1×

29.5

29.5

29.5

30.5

20

0.05

0.1

0.2

0.3

0.05

0.1

0.2

0.3

3×

75.9

75.9

75.9

57.0

70

3×

36.1

36.1

36.1

42.2

2×

75.9

75.9

75.9

57.0

60

2×

36.1

36.1

36.1

42.2

1.5×

57.0

57.0

57.0

57.0

50

1.5×

42.2

42.2

42.2

42.2

30

1.25×

37.3

37.3

37.3

37.3

1.25×

45.2

45.2

45.2

45.2

25

1×

37.3

37.3

37.3

37.3

1×

45.2

45.2

45.2

45.2

20

0.05

0.1

0.2

0.3

0.05

0.1

0.2

0.3

40 30 20

Throughput tolerance

Output length (OL-512) SLO multiplier

Bcap (IL, OL) = arg max T (IL, OL, B) B   E2E(IL, OL, B) ≤ SLO, subject to T (IL, OL, B)  > ϵ. B · T (IL, OL, 1)

the resulting batch size recommendation against full-profile BCA with ϵ = 0.1. Following the same experimental set-up as Section VI, we evaluate performance in scenarios with unseen models and output lengths while calibrating SLIM only on the smallest models and shortest output lengths, which incur the lowest profiling cost. For output-length transfer, profiling only OL 16 and 32 leads to a 34.6% throughput variation on OL-512 when comparing the full-profiling batch size decision, whereas adding OL-48 removes this discrepancy. For model transfer to OPT-6.7B, profiling on OPT-125M alone results in a 28.4% throughput discrepancy relative to the full-profile decision and a 6.9% SLO violation. Adding OPT-350M and OPT1.3B reduces the discrepancy to 15.1% and removes the SLO violation entirely. Overall, these results show that a small number of profiled curves is sufficient for SLIM-BCA to closely recover the operating point selected by full-profile BCA. Because the predicted curves preserve the shape of the fully profiled curves, SLIM-BCA produces similar batch-size recommendations. Nevertheless, the SLO violation observed under model generalization evaluation highlights the residual risk of relying on predicted latency rather than directly measuring the target configuration.

Throughput tolerance

40 35

Throughput tolerance

40 30 20

45 40 35

Throughput tolerance

Fig. 12: Sensitivity of SLIM-BCA to the throughput-tolerance threshold ϵ and latency-SLO multiplier under model generalization (top) and output-length generalization (bottom). The left column reports throughput retained relative to the profiled peak, while the right column reports KV-cache memory saved relative to the maximum profiled context. Figure 12 shows the trade-off induced by SLIM-BCA’s throughput threshold and latency constraint. Under model generalization, looser SLOs allow higher retained throughput, whereas increasing ϵ selects more conservative batch-sizes and yields larger memory savings. Under output-length generalization, the recommendation remains unchanged for ϵ ≤ 0.2 across most SLO levels, indicating limited sensitivity to the threshold. Only ϵ = 0.3 reduces throughput under the loosest

Setting Same curve O. length Model Model + O. length

Prof.

Pred.

Pt. save

Time save

3 15 25 40

6 5 8 13

66.7% 25.0% 24.2% 24.5%

56.9% 59.9% 27.2% 31.4%

SLOs, confirming that SLIM-BCA is generally robust while exposing the expected throughput–memory trade-off. 2) Profiling-cost Savings: Table VI quantifies the profiling effort avoided by fitting SLIM instead of exhaustively profiling every serving configuration. For each held-out setting, Prof. reports the number of configurations actually profiled to fit SLIM, whereas Pred. reports the number of held-out configurations whose throughput and latency are obtained from SLIM’s predictions. Pt. save denotes the resulting fraction of configurations that do not need to be profiled directly. Profiling-point savings range from 24.2% in the model-transfer setting, where 8 of 33 evaluated configurations are predicted by SLIM rather than directly profiled, to 66.7% in the samecurve setting, where only 3 of the 9 batch-size points on each curve need to be measured. Reducing the number of profiled configurations also reduces total profiling time. Because we evaluate SLIM-BCA under generalization to larger models and longer output lengths, the held-out configurations are often the most expensive to profile. For instance, by profiling 25.0% fewer configurations we reduce profiling time by 59.9%, because the held-out configuration (OL = 512) is also the most expensive, as longer generations require proportionally more decode steps. This indicates that SLIM’s benefit extends beyond the reduction in the number of profiling runs, as it specifically avoids the most expensive ones, which are concentrated in larger models and longer generations. 3) Memory Savings: Common LLM serving frameworks such as vLLM reserve the maximum available GPU memory for serving by default. While this is often a good practice for larger models, we show in Figure 13 the memory savings achieved by SLIM-BCA when serving smaller models on a high-end GPU. Instead of reserving the full KV-cache budget, SLIM-BCA allocates only enough KV-cache memory to support the batch size proposed by the model. In our setup, this corresponds to 1.52 GB for OPT-350M, 6.09 GB for OPT-1.3B, and 5.08 GB for OPT-2.7B, leaving 55.42 GB, 49.08 GB, and 47.49 GB of the default KV-cache allocation unused, respectively. These results show that SLIM-BCA can identify substantial memory savings while preserving its selected operating point. VIII. D ISCUSSION In this work, we identify the GPU bottlenecks underlying throughput saturation in LLM inference with large active contexts. Decode-time attention maintains nearly constant

Memory usage (GB)

TABLE VI: BCA profiling cost and recommendation quality when fitting SLIM and predicting held-out throughput curves instead of fully profiling every curve.

Model weights SLIM-BCA KV alloc.

60

Saved KV Other

40 20 0

OPT-350M

OPT-1.3B

OPT-2.7B

Fig. 13: Memory usage distribution for each model size in our 64GB GPU environment, considering Bcap under a strict SLO and ϵ = 0.1. By default, vLLM allocates 90% of available memory, leaving 10% for the other processes (Other).

arithmetic intensity as batch size or context length increases, causing KV-cache traffic to approach the DRAM-bandwidth limit while compute resources remain underutilized. This is consistent with the widening gap between accelerator compute and memory bandwidth, where peak compute has historically scaled much faster than DRAM and interconnect bandwidth [39]. We therefore expect attention-bandwidth bottlenecks to persist across future accelerators unless mitigated by substantially higher DRAM/HBM bandwidth or reduced KV-cache traffic. The low cache hit rates and increasing longscoreboard stalls further indicate that the bottleneck arises from streaming KV-cache data with limited temporal reuse, rather than from an isolated cache-management inefficiency. Our kernel-level profiling is limited to Mistral-7B and Granite8B on a single H100 GPU, so validation across accelerators and KV-cache formats remains future work. Nevertheless, the same mechanism is expected to occur across conventional decoder-only attention architectures, as the attention kernel’s computation-to-memory ratio is largely determined by the model’s attention structure itself. We propose SLIM as a lightweight semi-analytical model that translates these hardware observations into throughput and latency estimates. Its explicit memory-transfer term supports accurate generalization across input and output lengths, while a fitted dense-decode correction captures model-scale effects. Consequently, SLIM achieves a MAPE below 6% when generalizing to unseen sequence lengths by extrapolating the same memory behavior, while generalization across models is more challenging, yielding a MAPE of 24.3% due to differences in model architecture and runtime overheads. SLIM aims to minimize profiling rather than eliminate it entirely, since costs such as kernel launches, framework scheduling, and nonattention operators are difficult to model from first principles. The current formulation targets decoder-only autoregressive inference, reducing MAPE by more than 70% relative to the evaluated baselines, but would require recalibration to account for recent optimizations such as quantized KV caches, speculative decoding, and prefix caching. In most cases, however, these changes affect existing terms rather than the overall structure, as quantization reduces bkv , prefix caching reduces

the effective attended input context, and speculative decoding changes the effective per-step workload. Finally, we integrate SLIM with our proposed Batching Configuration Advisor (BCA) to select efficient batch-size and KV-cache configurations under latency constraints. Although BCA currently operates as an offline configuration advisor rather than an online scheduler and requires estimates of incoming request lengths, using average workload characteristics—readily obtainable in production environments—proves effective, reducing GPU memory usage by up to 55 GB for the evaluated OPT models with only minor throughput degradation. This makes it suitable for capacity planning, complementing reactive schedulers and autoscalers that can track its recommended operating point. In practice, BCA recommendations should include conservative throughput and latency margins, or be combined with lightweight runtime monitoring that adjusts the selected configuration when observed latency deviates from predictions. The reported memory reductions should similarly be interpreted as potential KVcache reservation savings rather than automatically reclaimed capacity. Finally, because SLIM is calibrated per accelerator, applying BCA across heterogeneous GPUs requires recalibration rather than direct reuse. Integrating these offline recommendations with adaptive runtime control is a natural next step toward multi-model orchestration under changing workloads and service-level objectives. IX. C ONCLUSION In this work, we conduct an in-depth GPU analysis to identify the performance bottlenecks responsible for throughput saturation in large active-context LLM inference. Our findings show that decode-time attention remains memory-bound, with KV-cache traffic driving DRAM-bandwidth saturation while substantial compute capacity remains underutilized. Building on this characterization, we introduce SLIM, a lightweight semi-analytical model that predicts throughput and latency across different model scales, batch sizes, and sequence lengths using only sparse profiling. We further instantiate SLIM through BCA, which selects efficient batch and KVcache configurations under latency constraints, reducing unnecessary memory allocation while preserving near-maximum throughput among SLO-feasible configurations. Overall, our results provide a hardware-grounded explanation of throughput saturation and a practical methodology for configuring LLMserving deployments to improve GPU resource efficiency. ACKNOWLEDGMENTS This work has been partially financed by grant agreement EU-HORIZON GA.101296742 (OptimAIse). Also, it has been partially financed by Generalitat de Catalunya (AGAUR) under grant agreement 2021-SGR-00478, by Severo Ochoa Center of Excellence CEX-2021-001148-S-20-3, and by the Spanish Ministry of Science (MICINN), the Research State Agency (AEI) and European Regional Development Funds (ERDF/FEDER) under grant agreement PID2021-126248OBI00, MCIN/AEI/10.13039/ 501100011033/ FEDER, UE.

The authors used AI generative models to assist with text rephrasing and stylistic refinement. All content was originally written and subsequently reviewed and validated by the authors. R EFERENCES [1] A. Bick, A. Blandin, and D. J. Deming, “The rapid adoption of generative ai,” National Bureau of Economic Research, NBER Working Paper 32966, Feb. 2025, revised February 2025. [Online]. Available: https://www.nber.org/papers/w32966 [2] A. Humlum and E. Vestergaard, “The adoption of chatgpt,” Becker Friedman Institute for Economics, University of Chicago, Working Paper 2024-50, Apr. 2024, available at SSRN. [Online]. Available: https://ssrn.com/abstract=4807516 [3] W. Liang, Y. Zhang, Z. Wu, H. Lepp, W. Ji, X. Zhao, H. Cao, S. Liu, S. He, Z. Huang, D. Yang, C. Potts, C. D. Manning, and J. Zou, “Quantifying large language model usage in scientific papers,” Nature Human Behaviour, vol. 9, no. 12, pp. 2599–2609, Dec 2025. [Online]. Available: https://doi.org/10.1038/s41562-025-02273-8 [4] B. Sun, Z. Huang, H. Zhao, W. Xiao, X. Zhang, Y. Li, and W. Lin, “Llumnix: Dynamic scheduling for large language model serving,” in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). Santa Clara, CA: USENIX Association, Jul. 2024, pp. 173–191. [Online]. Available: https: //www.usenix.org/conference/osdi24/presentation/sun-biao [5] S. Jaiswal, K. Jain, Y. Simmhan, A. Parayil, A. Mallick, R. Wang, R. S. Amant, C. Bansal, V. Ruhle, A. Kulkarni et al., “Sageserve: Optimizing llm serving on cloud data centers with forecast aware auto-scaling,” Proceedings of the ACM on Measurement and Analysis of Computing Systems, vol. 9, no. 3, pp. 1–24, 2025. [6] J. Li, J. Xu, S. Huang, Y. Chen, W. Li, J. Liu, Y. Lian, J. Pan, L. Ding, H. Zhou et al., “Large language model inference acceleration: A comprehensive hardware perspective,” arXiv preprint arXiv:2410.04466, 2024. [7] R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, J. Heek, K. Xiao, S. Agrawal, and J. Dean, “Efficiently scaling transformer inference,” Proceedings of Machine Learning and Systems, vol. 5, 2023. [8] M. Davies, N. Crago, K. Sankaralingam, and C. Kozyrakis, “Efficient llm inference: Bandwidth, compute, synchronization, and capacity are all you need,” arXiv e-prints, pp. arXiv–2507, 2025. [9] P. G. Recasens, F. Agullo, Y. Zhu, C. Wang, E. K. Lee, O. Tardieu, J. Torres, and J. L. Berral, “Mind the memory gap: Unveiling gpu bottlenecks in large-batch llm inference,” in 2025 IEEE 18th International Conference on Cloud Computing (CLOUD). IEEE, 2025, pp. 277–287. [10] S. Jin, X. Liu, J. Shan, L. Xu, T. Zhang, L. Xie, and Z. M. Mao, “Llmvisor: A real-time latency attribution model for multi-tenant llm serving,” 2025. [11] S. Imai, R. Nakazawa, M. Amaral, S. Choochotkaew, and T. Chiba, “Predicting llm inference latency: A roofline-driven ml method,” in Annual Conference on Neural Information Processing Systems, 2024. [12] S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin et al., “Opt: Open pre-trained transformer language models,” arXiv preprint arXiv:2205.01068, 2022. [13] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020. [14] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al., “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023. [15] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017. [16] G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, “Orca: A distributed serving system for transformer-based generative models,” in 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), 2022, pp. 521–538. [17] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” in Proceedings of the 29th Symposium on Operating Systems Principles, 2023, pp. 611–626.

[18] NVIDIA, “TensorRT-LLM,” GitHub repository, 2023–2025. [Online]. Available: https://github.com/NVIDIA/TensorRT-LLM [19] Microsoft, “DeepSpeed-MII,” GitHub repository, 2022–2025. [Online]. Available: https://github.com/deepspeedai/DeepSpeed-MII [20] Y. Sheng, S. Cao, D. Li, C. Hooper, N. Lee, S. Yang, C. Chou, B. Zhu, L. Zheng, K. Keutzer et al., “Slora: Scalable serving of thousands of lora adapters,” Proceedings of Machine Learning and Systems, vol. 6, pp. 296–311, 2024. [21] Z. Yuan, Y. Shang, Y. Zhou, Z. Dong, Z. Zhou, C. Xue, B. Wu, Z. Li, Q. Gu, Y. J. Lee et al., “Llm inference unveiled: Survey and roofline model insights,” arXiv preprint arXiv:2402.16363, 2024. [22] A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. Gulavani, A. Tumanov, and R. Ramjee, “Taming {Throughput-Latency} tradeoff in {LLM} inference with {Sarathi-Serve},” in 18th USENIX symposium on operating systems design and implementation (OSDI 24), 2024, pp. 117–134. [23] Y. Leviathan, M. Kalman, and Y. Matias, “Fast inference from transformers via speculative decoding,” in International Conference on Machine Learning. PMLR, 2023, pp. 19 274–19 286. [24] C. Chen, S. Borgeaud, G. Irving, J.-B. Lespiau, L. Sifre, and J. Jumper, “Accelerating large language model decoding with speculative sampling,” arXiv preprint arXiv:2302.01318, 2023. [25] N. Shazeer, “Fast transformer decoding: One write-head is all you need,” arXiv preprint arXiv:1911.02150, 2019. [26] J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebrón, and S. Sanghai, “Gqa: Training generalized multi-query transformer models from multi-head checkpoints,” arXiv preprint arXiv:2305.13245, 2023. [27] J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han, “Awq: Activation-aware weight quantization for on-device llm compression and acceleration,” Proceedings of machine learning and systems, vol. 6, pp. 87–100, 2024. [28] E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “Gptq: Accurate post-training quantization for generative pre-trained transformers,” arXiv preprint arXiv:2210.17323, 2022. [29] A. Agrawal, N. Kedia, J. Mohan, A. Panwar, N. Kwatra, B. S. Gulavani, R. Ramjee, and A. Tumanov, “Vidur: A large-scale simulation framework for llm inference,” in Proceedings of Machine Learning and Systems, 2024. [30] A. Bambhaniya, R. Raj, G. Jeong, S. Kundu, S. Srinivasan, S. Subramanian, M. Elavazhagan, M. Kumar, and T. Krishna, “Demystifying ai platform design for distributed inference of next-generation llm models,” arXiv preprint arXiv:2406.01698, 2024. [31] H. Zhang, A. Ning, R. B. Prabhakar, and D. Wentzlaff, “Llmcompass: Enabling efficient hardware design for large language model inference,” in 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 2024, pp. 1080–1096. [32] S. Cao, J. Wu, J. Chen, H. An, and Z. Yu, “Amali: An analytical model for accurately modeling llm inference on modern gpus,” in Proceedings of the 52nd Annual International Symposium on Computer Architecture, 2025, pp. 1495–1508. [33] Y. Sheng, S. Cao, D. Li, B. Zhu, Z. Li, D. Zhuo, J. E. Gonzalez, and I. Stoica, “Fairness in serving large language models,” in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), 2024, pp. 965–988. [34] anon8231489123, “Clean sharegpt dataset,” 2023. [Online]. Available: https://huggingface.co/datasets/anon8231489123/ShareGPT Vicuna unfiltered [35] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed, “Mistral 7b,” 2023. [Online]. Available: https://arxiv.org/abs/2310.06825 [36] M. Mishra, M. Stallone, G. Zhang, Y. Shen, A. Prasad, A. M. Soria, M. Merler, P. Selvam, S. Surendran, S. Singh et al., “Granite code models: A family of open foundation models for code intelligence,” arXiv preprint arXiv:2405.04324, 2024. [37] S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer, “Opt: Open pre-trained transformer language models,” 2022. [Online]. Available: https://arxiv.org/abs/2205.01068 [38] A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Tang, J. Wang, J. Yang, J. Tu, J. Zhang, J. Ma, J. Xu, J. Zhou, J. Bai, J. He, J. Lin, K. Dang,

K. Lu, K. Chen, K. Yang, M. Li, M. Xue, N. Ni, P. Zhang, P. Wang, R. Peng, R. Men, R. Gao, R. Lin, S. Wang, S. Bai, S. Tan, T. Zhu, T. Li, T. Liu, W. Ge, X. Deng, X. Zhou, X. Ren, X. Zhang, X. Wei, X. Ren, Y. Fan, Y. Yao, Y. Zhang, Y. Wan, Y. Chu, Y. Liu, Z. Cui, Z. Zhang, and Z. Fan, “Qwen2 technical report,” arXiv preprint arXiv:2407.10671, 2024. [39] A. Gholami, Z. Yao, S. Kim, C. Hooper, M. W. Mahoney, and K. Keutzer, “Ai and memory wall,” IEEE Micro, vol. 44, no. 3, pp. 33–39, 2024.