PACER: GPU Excursion Management for ML Workloads

arXiv:2611.17206v1 [cs.DC] 12 Nov 2026

PACER: GPU Excursion Management for ML Workloads RUTWIK JAIN∗ , University of Wisconsin-Madison, USA NATHANIEL MORRIS, Advanced Micro Devices, Inc., USA YIWEI JIANG, University of Wisconsin-Madison, USA SHIDHARTHA DAS, Advanced Micro Devices, Inc., USA GARY KAO, Advanced Micro Devices, Inc., USA INDRANI PAUL, Advanced Micro Devices, Inc., USA SHIVARAM VENKATARAMAN, University of Wisconsin-Madison, USA SRILATHA MANNE, Advanced Micro Devices, Inc., USA MATTHEW D. SINCLAIR, University of Wisconsin-Madison, USA Variability is an inherent characteristic of modern GPU systems, arising from a combination of process variation, dynamic environmental conditions, and power management decisions. Prior work has characterized inter-GPU variability and found consistent performance differences across GPUs. However, we observe that modern workloads, particularly large language models (LLMs), exhibit significant intra-GPU variability. Their short-running kernels interact poorly with the coarse granularity of dynamic power management mechanisms, leading to unpredictable performance. A key source of this intra-GPU transient variability is power excursions, also referred to as electrical design power peaks (EDP𝑝 ), where power temporarily exceeds the GPU’s rated power limits. While power management mechanisms are designed to ensure reliable operation despite the presence of EDP𝑝 , sustained power peaks cause throttling that detrimentally impacts overall application performance. To the best of our knowledge, our work is the first to characterize the magnitude and frequency of EDP𝑝 , showing that frequent, significant power overshoots are especially common in ML workloads, which combine high power draw, short kernels, and rapidly changing compute demands. To address this, we design and leverage hardware and firmware mechanisms to provide consistent power management behavior that reduces intra-GPU variability due to EDP𝑝 and improves overall performance. We evaluate our techniques across a range of popular workloads. Our approach improves performance by 3–12% over state-of-the-art GPU power management and comes within 3% of an oracular solution. CCS Concepts: • Hardware → Chip-level power issues; • Computer systems organization → Parallel architectures. Additional Key Words and Phrases: GPU Power Management, Power Excursions, Excursion Management

1

Introduction

Voracious scaling demands in domains like graph analytics, high performance computing (HPC), and machine learning (ML) have led system developers to push technology limits with large, highly complex, heterogeneous systems that operate at previously unheard of power limits. For example, recent systems such as AMD Instinct™ MI300A and AMD Instinct™ MI300X [1, 90] combine CPU/GPU solutions and stacked HBM using the latest in 2D and 3D chiplet technologies [42, 47, 59, 84]. Physically co-locating these components improves performance by shortening the distances ∗ Work done as intern at Advanced Micro Devices, Inc.

Authors’ Contact Information: Rutwik Jain, University of Wisconsin-Madison, Madison, USA, rnjain@wisc.edu; Nathaniel Morris, Advanced Micro Devices, Inc., Austin, USA, Nathaniel.Morris@amd.com; Yiwei Jiang, University of WisconsinMadison, Madison, Wisconsin, USA, jiang357@wisc.edu; Shidhartha Das, Advanced Micro Devices, Inc., Santa Clara, California, USA, Sid.Das@amd.com; Gary Kao, Advanced Micro Devices, Inc., Santa Clara, California, USA, Tzyyjuin.Kao@amd.com; Indrani Paul, Advanced Micro Devices, Inc., Santa Clara, California, USA, Indrani.Paul@amd.com; Shivaram Venkataraman, University of Wisconsin-Madison, Madison, Wisconsin, USA, shivaram@cs.wisc.edu; Srilatha Manne, Advanced Micro Devices, Inc., Santa Clara, California, USA, Srilatha.Manne@amd.com; Matthew D. Sinclair, University of Wisconsin-Madison, Madison, Wisconsin, USA, sinclair@cs.wisc.edu.

2

Jain et al.

Prefill Phase

Power (1ms) Power (50ms)

MI325X H100

(a)

MI300X

700 A100

CU Frequency

TDP (Watts)

B300

B200

1100

Power

MI355X

AMD Instinct™ NVIDIA Data Center

MI250X

MI50

300 V100

MI100

2018

2020

2022

Launch Year

(a)

2024

2026

CU Frequency (1ms) CU Frequency (50ms)

Time (b)

(b)

Fig. 1. (a) Thermal design power (TDP) of successive generations of AMD Instinct and NVIDIA datacenter GPUs by launch year. (b) Time series of power (top) and CU frequency set by the DPM (bottom) when running Qwen3-14B inference on an AMD Instinct MI300X GPU. The dotted black line indicates the GPU’s TDP.s

between components and enables finer, higher bandwidth connections [90]. Moreover, as shown in Figure 1a, these GPUs dissipate up to 750 Watts with newer GPUs reaching 1400W [2] and potentially 2300W [30]. As a result, compute-intensive workloads such as ML are often power constrained on modern systems [12, 20, 44, 55, 87, 89, 98, 110, 115]. To address these challenges, systems like the AMD Instinct MI300X have robust power management mechanisms that dynamically adjust components like compute and data fabric frequencies (and, consequently, their power) to operate the system at the highest performance while staying within thermal design power (TDP) constraints. The dynamic power manager (DPM) adjusts voltage and frequencies at short intervals, generally one to a few milliseconds [8, 16, 67, 74, 93, 117]. As long as application behavior is not highly variable across those time scales, most systems can successfully deliver the necessary performance within the device’s power budget by managing voltage and frequency appropriately. However, together power constrained systems and dynamic power management mechanisms expose inherent variability among GPU systems. Variability arises from various sources: process variation across chiplets, changing environmental conditions, and from the power-management decisions each GPU makes over time [19, 38, 89, 98]. We distinguish two forms of this variability, inter-GPU variability, which is the performance difference across GPUs running the same work, and intra-GPU or transient variability, which is variation over time when the same application or kernel runs on one GPU. Previous work [89, 87] characterized inter-GPU variability and found consistent variability across GPUs, with very little intra-GPU variability. However, in this work, we show that modern workloads, especially large language models (LLMs), have unique characteristics such as short kernels and variable compute requirements, which cause significant intra-GPU variability. Applications such as LLMs not only push the TDP limit of these systems, they also have rapidly fluctuating behavior varying between compute dense, high power phases and lower power phases at microsecond timescales. Figure 2 shows how the arithmetic intensity of the kernels varies within two Transformer layers of the prefill phase of LLM Inference, revealing fluctuations spanning over two orders of magnitude across kernels with sub-millisecond durations. The timescale is far shorter than the DPM’s adjustment interval (kernel fusion’s impact discussed further in Section 3.3). Moreover, modern ML workloads increasingly utilize diverse forms of parallelism and concurrency, from model and expert-level parallelism [88, 37, 43] to computation layer and kernel-level micro-batching [36,

PACER: GPU Excursion Management for ML Workloads

Normalized Intensity (log scale)

rms_norm gemm

3 rotary_embed tensor_copy

attn_fwd silu_and_mul

100

10−1

10−2 Time Slice during Inference Prefill Phase

Fig. 2. Time slice showing kernels from two transformer layers of the prefill phase for LLaMA3-8B inference with arithmetic intensity in operations/s, normalized between 0 and 1.

95]. These techniques are essential for scaling models beyond single-device memory capacity, improving hardware utilization, and reducing per-token serving costs. While these solutions improve performance, the microsecond timescales at which application characteristics change clash with the longer intervals for GPU dynamic power management solutions. The slower reactivity of the DPM relative to application behavior impacts transient kernel variability, and it also results in power peaks when the GPU is already operating near its power limits and the application rapidly switches between low and high activity phases [14, 15, 19, 38, 69]. To understand how frequently peak GPU power exceeds TDP for modern workloads, Figure 1b shows how GPU power and compute unit (CU)1 frequency vary for Qwen3-14B inference [21], run using vLLM [53]. The orange lines indicate power and DPM set frequency, respectively, averaged over 50 millisecond (ms) intervals. The blue lines show the same metrics at ≈1 ms intervals. As prior work observed [76, 75], inference power in Figure 1b (top) varies between the prefill and decode phases: prefill generally dissipates near TDP power, and decode operates below the TDP. Due to power constraints, the average CU frequency is lower during the prefill phase than during decode. By adjusting the CU frequency, the DPM optimizes performance under power constraints. Moreover, although power looks steady when averaged across the larger 50 ms intervals (orange lines) [109], smaller duration averages show the EDP𝑝 swinging beyond TDP (blue lines). These peaks occur frequently during an LLM’s prefill phase and less frequently during the decode phase. Each peak is followed by a frequency correction, so the same kernel does not see a stable frequency from one instance to the next. The lack of a stable operating frequency is the main cause of intraGPU variability. More generally, we find that EDP𝑝 occurs in applications with short kernels and significant compute variability across adjacent kernels (discussed further in Section 3). While previous works have observed EDP𝑝 (discussed in Section 7), a broader study is needed to understand how prevalent they are across applications, and how EDP𝑝 impacts application performance. Thus, we first perform a comprehensive analysis of EDP𝑝 on GPUs across modern graph analytics, HPC, and ML workloads to identify which application characteristics result in power overshoots. Our results show that EDP𝑝 events are triggered at especially high rates in ML workloads like Qwen3-14B and LLaMA3-8B inference. Moreover, EDP𝑝 will likely become even more challenging for future GPUs with further chiplet integration and higher TDPs. We also show that EDP𝑝 can be both beneficial and detrimental – properly managed EDP𝑝 enable applications to computationally sprint [80, 66, 65, 24] while excessive high-magnitude EDP𝑝 incur performance loss due to throttling mechanisms. We use the insights from this characterization to manage excursions so that EDP𝑝 stays in the range that preserves computational sprinting, without the throttling that produces intra-GPU variability. In hardware, EDP𝑝 clamping reduces peak magnitude, at a cost in performance. In 1Without loss of generality, we use AMD GPU terminology. NVIDIA GPUs have equivalent terminology (e.g., SMs vs. CUs).

4

Jain et al.

Table 1. (a) EDP𝑝 limits from the OCP specification and (b) the metrics and terminology used in this paper. (a) OCP [71] specification for EDP𝑝 . EDP𝑝

Duration

Management Solution

2 × TDP 1.7 × TDP 1.5 × TDP 1.25 × TDP 1.1 × TDP 1 × TDP

Up to 20𝜇 s Up to 2ms Up to 5ms Up to 10ms Up to 20ms Indefinitely

HW HW or SW SW SW SW SW

(b) Metrics and terminology glossary. Metric EDP𝑝

𝐶𝐴𝐶

Description Electrical Design Power peaks Active switching capacitance

Short time-scale power overshoots or spikes above TDP. GPU capacitive activity that drives dynamic power consumption.

firmware, a graphics frequency cap together with a fixed memory-clock state narrows the range over which the DPM can move, so successive kernels see more repeatable voltage and frequency decisions. We evaluate both mechanisms on an experimental system built atop production AMD Instinct MI300X GPUs, with no changes to the applications. Frequency limits improve prefill performance and reduce transient kernel variability for single-GPU inference. The same caps, chosen from offline profiling and applied based on dynamic request characteristics, improve multiGPU online serving. We analyze these EDP𝑝 –performance tradeoffs across popular ML workloads for both firmware and hardware management. Overall, our solutions improve performance over state-of-the-art, production-grade power management by 3–12% on AMD Instinct MI300X GPUs. Notably, our designs provide near-best-case performance: within 3% of an oracle power management mechanism. 2

Background

EDP𝑝 events occur when current consumption exceeds a certain threshold. EDP𝑝 can result in current increases that can drain package decoupling capacitors and push voltage regulators out of compliance, potentially triggering system protection mechanisms like power throttling. Power overshoots have thermal and reliability implications [6, 32, 39, 61, 72, 85, 86, 103, 112, 113] that, if not managed, may cause catastrophic hardware failures such as physical damage to VRs or even catching fire [11, 29, 35, 52, 101]. These challenges show how EDP𝑝 events require a robust and reliable hardware protection mechanism to ensure over-current protection and safe operation. Given the catastrophic nature of the problem, EDP𝑝 management must have a dependable backstop to ensure reliable operation. The Open Accelerator Infrastructure (OAI) OCP Accelerator Module specification provides an exemplar guide for the voltage regulator (VR) electrical design to handle power overshoots [71]. The OCP specification defines the tolerable peak power of the system as a function of the socket’s TDP. For example, Table 1a outlines the board rail of a hypothetical 700W system. For instance, this system can draw up to 2× the TDP power limit as long as that peak does not exceed 20 𝜇s. Meeting this specification, and not exceeding TDP over a period of 1 second [71, 16], ensures the VR will be reliable and operate as specified. Other GPU products are designed for a more relaxed specification with longer peak power tolerances (e.g., 50 ms duration). However, the more relaxed the specification, the higher the cost: additional board capacitance requirements and likely a higher power VR, which impacts the next level of the power delivery hierarchy, accruing further costs up the power delivery chain to the rack and potentially beyond [16]. EDP𝑝 response depends on how long the peak is allowed to last and the DPM’s capabilities. In modern systems, some of the listed overshoot time constants in Table 1a (e.g., 2×TDP) require hardware solutions to manage at the microsecond time scale. However, firmware management solutions that operate at millisecond scales can utilize dynamic voltage and frequency scaling (DVFS) to account for Table 1a’s longer durations (or those in prior work [16]).

PACER: GPU Excursion Management for ML Workloads

5

Table 2. Workloads used in characterization. Workload

Domain

Dataset/Config

Qwen3-14B [108] inference LLaMA3-8B [99, 21] inference LLaMA3-70B [99, 21] inference Mixtral-8x7B [43] inference Stable Diffusion [83, 79]

ML (LLM) ML (LLM) ML (LLM) ML (LLM) ML (Diffusion)

bsz 1, ip 4k, op 128 bsz 1/4/8/16, ip 1k/4k, op 128 bsz 1/4, ip 1k/4k, op 128 bsz 1/4, ip 1k/4k, op 128 bsz 1

MILC [7] LULESH [48, 34, 50, 49]

HPC HPC

su3_rhmd_hisq, 24 × 24 × 24 n 400, i 10

PageRank [104] Neighbor Sampling [105, 10, 9] Random Walk [105, 10, 9] K-Core [105, 10, 9]

Graph Analytics Graph Analytics Graph Analytics Graph Analytics

indochina, uk-2005 indochina, uk-2005, Orkut indochina, uk-2005, Orkut indochina, uk-2005, Orkut

Every fail-safe EDP𝑝 management mechanism must meet the VR’s specified requirements. The management solution cannot fail, so the resulting solution has built in margins to deal with power measurement inaccuracies, delays between measurement and evaluation, and response capabilities. For instance, a hardware solution that samples power from the board power supply will have very accurate data. However, there may be a relatively long latency to receive the data, whereas a digital power estimation solution can be computed quickly but will have some estimation error. The EDP𝑝 response may involve architectural solutions such as temporarily shutting down some sub-modules, reducing frequency [106, 27], or reducing both voltage and frequency (DVFS). Whatever solution is selected, it must be responsive enough to deal with the large overshoots that can rapidly occur in a highly parallel, highly synchronized, high switching capacitance system like a GPU. For one or more of these reasons, an EDP𝑝 management mechanism will likely have significant margining to ensure VR protection (e.g. conservative EDP𝑝 thresholds to trigger throttling) . If the frequency of EDP𝑝 is limited, the management solution’s efficiency will not matter. However, frequent, high-magnitude EDP𝑝 events that trigger the management solution and power throttling will degrade application performance. 3

EDP𝑝 Characterization

We characterize EDP𝑝 behavior across diverse applications and compare the magnitude and frequency of EDP𝑝 . We use the AMD Instinct MI300X GPU [1] which have a TDP of 750W. As defined in Section 1, EDP𝑝 refers to power peaks beyond the GPU’s TDP limit. Table 1b defines the metrics and terminology, including EDP𝑝 , that we use in the remainder of this paper. We measure GPU power using AMD GPU hardware counters, which provide 1ms granularity of power measurements [3]. Specifically, we profile the GPU’s Total Board Power2 and report measurements relative to a TDP of 750W. To ensure consistent results, we repeat all experiments three times and present either the median results or the distribution across all trials. We show cumulative distributions of this measured power, specifically showing the power distribution at and over TDP. Due to the 1 ms averaging of power, we cannot track smaller duration events with high amplitude, such as 2×TDP in Table 1a. Any EDP𝑝 events shorter than 1 ms in duration are dampened and absorbed into the 1 ms average power results. Nevertheless, the fact that we still observe EDP𝑝 even at this coarser granularity highlights the importance of the problem. Power variability occurs due to a combination of power management directives and application characteristics. The goal of power management firmware (PMFW) is to achieve the best performance possible for a given application. PMFW increases GPU clock frequency and the associated voltage 2 Total Board Power includes the GPU and HBM power as well as any power delivery overheads associated with the board

power supply.

6

Jain et al. Qwen3-14B bsz8 Inf K-Core (uk-2005) Neighbor Sampling (uk-2005)

Normalized Num of EDPp

CDF

1.0

0.5

1.1

1.2 xTDP

1.3

1.4

1.0

0.5

0.0

Q

LL

aM

0.0 1.0

Neighbor Sampling (Orkut) Random Walk (uk-2005) PageRank (indochina)

A3 w -7 en 0B K- LL 3-1 Co aM 4B re A3 (u k- 8B 20 05 ) Pa MI ge LC Ra nk N S( Lul uk es -2 h N 00 RW S(O 5) (u rku kt 20 ) 05 )

Lulesh (s300,i10) MILC (24x24x24) LLaMA3-70B bsz8 Inf LLaMA3-8B bsz8 Inf

Fig. 3. Cumulative Distribution (CDF) of EDP𝑝 above TDP for HPC, ML, and graph analytics applications on a single AMD Instinct MI300X GPU.

until either maximum board power or maximum frequency is reached. This results in higher frequencies when running medium- or light-density compute operations and lower frequencies when running compute intensive applications with high switching capacitance and high power. Similar to many CPU power management algorithms, GPU PMFW uses past behavior as an indicator of future requirements. If application behavior does not change or changes slowly over many milliseconds, the GPU PMFW can easily adjust DVFS states to adapt to application characteristics and hold power steady at the board power limit [8, 12, 17, 23, 26, 63, 67, 89]. However, as Figure 2 shows, the fluctuating load in ML applications in combination with a firmware DPM that operates on a larger time scale than microseconds can produce power overshoots due to a mismatch between the operating frequency and the switching capacitance. Thus, while previous work has observed some EDP𝑝 for specific workloads [16, 77], a broader study is necessary to understand the extent and frequency of EDP𝑝 events in modern systems. 3.1

Modern GPU Workload Characteristics

To characterize EDP𝑝 across diverse workloads and compare their behavior with LLMs, we profiled the power consumption and power peaks for popular graph analytics, HPC, and ML workloads from Gunrock [104], Skywalker [105], CORAL-2 and OLCF-6 [7, 48, 49, 50, 54], and MLPerf [62, 82]. These workloads provide wide coverage across a range of access patterns and compute intensities while also ensuring they are representative of workloads driving the requirements for future systems. For all applications, we either selected inputs that utilized the GPUs well or examined multiple inputs (e.g., LLaMA3-8B inference with different batch sizes). Table 2 summarizes these workloads and their inputs. Figure 3 shows the cumulative distribution function (CDF) of EDP𝑝 values for these workloads on an AMD Instinct MI300X GPU, as well as the number of EDP𝑝 events normalized to the application with the most EDP𝑝 (LLaMA3-70B inference). These overshoots are not unique to the system being analyzed: EDP𝑝 of up to 500W over TDP have been observed on other accelerators [75, 16]. Interestingly, all ML applications (LLaMA2 fine-tuning, LLaMA3 inference, and Stable Diffusion) exhibit very similar EDP𝑝 CDFs regardless of batch size or application: ≈80% of their peaks are between 1.0× and 1.2× of TDP, but their remaining peaks exhibit a long tail, and some of them can be quite significant: max 1.7×, although this max happens < 1% of the time. Of the graph analytics and HPC workloads (PageRank, Neighbor Sampling, Random Walk, K-Core, Lulesh, and MILC), only Lulesh shows a significant portion of EDP𝑝 above 1.2×TDP. All graph analytics workloads and MILC have much less variable behavior: over 90% of their EDP𝑝 is below 1.2×TDP. We also ran Neighbor Sampling, K-Core, and Random Walk workloads on the indochina dataset, but did not observe

PACER: GPU Excursion Management for ML Workloads

7

Prefill phase

1.0

1.0

0.0

0.5

1.0×

1.1× 1.2× 1.3× EDPp magnitude (×TDP) Decode phase

1.0

bsz=1 bsz=4 bsz=8 bsz=16

0.5 0.0

1.0×

CDF of durations

CDF of EDPp

0.5

1.1× 1.2× 1.3× EDPp magnitude (×TDP)

(a)

0.0 −3 10

Prefill phase bsz=1 bsz=4 bsz=8 bsz=16

10−2

10−1 100 Time (ms)

101

Decode phase

1.0

bsz=1 bsz=4 bsz=8 bsz=16

0.5 0.0 −3 10

102

10−2

10−1 100 Time (ms)

101

102

(b)

Fig. 4. (a) EDP𝑝 distribution and (b) cumulative distribution of kernel durations for prefill and decode phases of LLaMA3-8B inference with varying batch sizes (bsz) on an AMD Instinct MI300X GPU.

any excursions for them. Therefore, Figure 3 does not include these workload-input combinations. Moreover, although the relative fraction of EDP𝑝 shows that workloads like Lulesh might still see high EDP𝑝 magnitudes, these overshoots occur infrequently. Conversely, ML workloads have frequent and high magnitude EDP𝑝 events. Therefore, we focus our characterization and eventual evaluation of EDP𝑝 management solutions on ML workloads. 3.2

Inter-Kernel ML Workload Characteristics

ML workloads are characterized by extremely short duration kernels, on the order of a few 100s of microseconds to a few milliseconds, and high-power variability between adjacent kernels [41, 89, 110]. Short kernels make it difficult to effectively modify DVFS state from one kernel to the next [22, 23, 64, 67, 68, 97]. In addition, adjacent kernels that transition from lower to higher compute density mean that a DVFS state selected for a lower power kernel may result in overshoots for successive kernels with high compute intensity. We use LLaMA3-8B inference to highlight the relationship between short runtime kernels and EDP𝑝 characteristics. Figures 4a and 4b show the CDF of overshoot magnitudes and kernel durations, respectively, for LLaMA3-8B inference with varying batch sizes ranging from 1 to 32. Both figures average their results over three trials. EDP𝑝 is separated into prefill and decode phases. Prefill not only has the majority of overshoots, but it also has larger magnitude peaks. In prefill, small batch sizes produce larger EDP𝑝 due to short kernel durations (Figure 4b) resulting in high frequency fluctuations between high- and low power kernels. As the prefill batch size increases, the kernels increase in duration, enabling the DPM to adjust the DVFS state appropriately. In the case of decode, the opposite occurs – higher batch sizes result in larger peaks. Larger batch sizes in decode also increase kernel duration but not to the same extent as prefill. However, larger batch sizes increase decode power so that it begins to operate near the GPU’s TDP, increasing the likelihood of EDP𝑝 due to application load fluctuations. 3.3

Intra-kernel ML Workload Characteristics

Inter-kernel GPU power variability is common since natural shifts in activity and power may occur at kernel boundaries, leading to EDP𝑝 . As highlighted in Table 1b, we quantify activity using capacitive switching or C𝐴𝐶 .

8

Jain et al. GEMM1 (6144,8192,4096)

1.00 Norm CAC

0.75 0.50 0.25

0.50 0.25 0.00

1.00

1.00

0.75 0.50 0.25 0.00

Time

(a)

GEMM2 (28672,8192,4096)

0.75

0.00 Norm Intensity (ops/s)

Norm Intensity (ops/s)

Norm CAC

1.00

0.75 0.50 0.25 0.00

Time

(b)

Fig. 5. Intra-kernel variation in normalized C𝐴𝐶 and arithmetic intensity in operations/s for GEMMs within LLaMA3-8B inference.

While shorter kernels exacerbate EDP𝑝 due to fine-grained fluctuations in activity, making kernels longer through techniques like kernel fusion [4, 25, 56, 70, 102] or megaKernels [91, 92] does not resolve the problem. Longer kernels also incur intra-kernel power variability. For example, the computation in fused kernels (e.g., a compute-intensive GEMM and a memoryintensive elementwise operation) differs significantly. Moreover, even within the GEMM phase, compute requirements can vary. To illustrate this, consider GEMM kernels within LLM inference. At a high level, GEMMs appear uniform in their behavior, as shown in Figure 2. However, tail effects are an inherent aspect of GEMM tiling on modern GPU architectures due to the interaction between matrix dimensions (𝑀, 𝑁 , 𝐾) and the hardware’s organization of compute resources. When a GEMM kernel is tiled, the input matrices are partitioned into blocks (or tiles), each assigned to a workgroup (WG). These WGs are further decomposed into wavefronts (WFs), which are scheduled across the available CUs. The number of threads per WF and the mapping of WGs to CUs are determined by the architecture, but the sizes of 𝑀, 𝑁 , and 𝐾 are dictated by the application and rarely align perfectly with the chosen tile sizes [60, 73, 78]. As a result, the final tiles along each dimension – the tails – are almost always partial. Figures 5a and 5b show arithmetic intensity in TFLOP/s for two GEMMs within LLaMA3-8B inference when running on AMD Instinct MI300X – GEMM1 with (𝑀, 𝑁 , 𝐾) dimensions (6144, 8192, 4096) and GEMM2 with (𝑀, 𝑁 , 𝐾) dimensions (28672, 8192, 4096) respectively. Both GEMMs show periodic dips in arithmetic intensity: points within the kernel where one set of WGs completes while the next set of WGs is launched. For the smaller GEMM1, work is divided into three sets of WGs that can each run concurrently on the GPU’s 304 CUs.3 Here, the tail set of WGs has roughly the same arithmetic intensity as the previous two. Conversely, GEMM2 shows a long tail where arithmetic intensity steps down. In GEMM2 the tail WGs process fewer elements than the regular tiles, causing underutilization. This mismatch is not simply a software artifact, but a direct consequence of the discrete mapping between the logical decomposition of the GEMM workload and the GPU’s physical resources. This quantization effect is fundamental to the architecture. Even with careful tile size selection to match the preferred WF and WG configurations, the nature of matrix dimensions means that the last few tiles will often be incomplete. This reduction in compute requirements, combined with natural process variations across dies within a GPU that produces varying completion times across the SoC, reduces power at the end of a GEMM computation. The DPM detects the lower power and 3 For this hipBLAS GEMM, number of WGs is:

𝑀 𝑚𝑡𝑖𝑙𝑒

× 𝑛𝑁

𝑡𝑖𝑙𝑒

877 = 877 and 304CUs = 2.88. The tail still utilizes 88% of CUs.

Similar calculations for the other GEMM yield 23.5, which means the 24th set of WGs will utilize only about half the GPU.

PACER: GPU Excursion Management for ML Workloads

10

Abs Tracking Error Measured CAC Predicted CAC

llama2_train

sdxl 15

20

10

10

5

10 5 0

Time

0

Time

(a)

0

CAC

NRMSE (%)

llama3_inf

9

Workload

NRMSE (%)

LLaMA3-8B inference LLaMA2 training Stable Diffusion (SDXL)

5.7 12.3 5.6

5 Time

0

(b)

Fig. 6. (a) Timeseries showing absolute prediction error (left Y) and normalized measured and predicted C𝐴𝐶 (right Y) for LLaMA3-8B inference, LLaMA2 training, and Stable Diffusion (sdxl) and (b) % Normalized Root Mean Squared Errors (NRMSE) for C𝐴𝐶 predictions for each workload.

increases frequency accordingly, resulting in variability even within a kernel. Figure 5’s arithmetic intensity of GEMMs reflects this: the change at the end of the previous set of WGs allows the DPM to increase the frequency for a short duration before the next set of WGs is launched. This also means that the same GEMM computations may experience different power fluctuations and EDP𝑝 , depending on what kernel precedes it, and how it interacts with the DPM. 4

EDP𝑝 Management

Section 3’s characterization shows that ML applications exacerbate physical GPU constraints with high compute requirements and highly variable application behavior. Moreover, they can also impact performance via throttling if they occur too often and require significant management [5]. Therefore, we evaluate hardware (Section 4.1) and firmware (Section 4.2) solutions to mitigate EDP𝑝 by managing C𝐴𝐶 variability at different timescales. Overall, across a range of popular ML workloads, versus a state-of-the-art, production grade GPU power management scheme, our solution improves performance by up to 12% on production systems (which is within 3% of an oracular solution). 4.1

EDP𝑝 Management in Hardware

To strictly limit EDP𝑝 and reduce transient jitter, we developed a digital power estimator (DPE) using a hardware event-counter based power estimation mechanism. This is similar to existing counter-based runtime power estimators [51, 96, 107] that use regression models to tune weights from hardware event counters. However, unlike these solutions that use runtime power estimates for DVFS algorithms [51, 96, 107], we tune the counter weights for 𝐸𝐷𝑃𝑝 management to have high accuracy in medium to high 𝐶𝐴𝐶 scenarios, which is likely to cause EDP𝑝 . To do this, we use existing hardware counters that measure switching activity for different GPU pipelines, such as the matrix cores, vector and scalar ALUs, and caches. The DPE performs a regression to tune the weights of these hardware event-counters to estimate 𝐶𝐴𝐶 , and therefore dynamic power. Figure 6a shows the correlation between our DPE and measured data when running ML applications, and Figure 6b summarizes the Normalized Root Mean Squared Error (NRMSE) for each application. These results show how generalizable the DPE is across a variety of ML workloads, not just inference. The digital estimations and actual power track each other very well, with estimation error of 12% or less (7% geomean) across workloads. Using this DPE, we developed a boost management algorithm that monitors power at submicrosecond granularities and utilizes the AMD GPU’s adaptive clock response mechanism [81] to throttle power. At runtime, this mechanism estimates the total power consumption of the GPU using the DPE and compares it to a configurable threshold (e.g., 1.2×TDP). If the power estimate

10

Jain et al.

exceeds the threshold, the algorithm triggers the adaptive clock response to reduce the GPU’s graphics clock frequency [81] and reduce power. Our experimental prototype can respond to EDP𝑝 events at microsecond timescales of the DPE. Section 6.1 shows results and insights from using this boost management mechanism on various inference models. 4.2

EDP𝑝 Management in Firmware

The DPM varies the voltage and frequency of operation dynamically at a 1-2 ms cadence, which leads to two variability-inducing effects: (1) misalignment with application behavior since kernels (and therefore EDP𝑝 ) are much shorter than the firmware’s operating granularity, and (2) firmware DVFS is not synchronized across GPUs, so when ML workloads run across multiple GPUs, PM firmware on different GPUs can set different frequencies for graphics and memory clocks at the same time. Thus, firmware frequency modulation can cause transient jitter and degrade overall application performance. Therefore, we examine whether controlling graphics and memory clock frequency limits allows PM firmware to operate deterministically, reduce kernel jitter, and improve performance. For firmware EDP𝑝 management, we identified two control parameters: graphics frequency cap (𝑓𝑐𝑎𝑝 ) and memory clock state. A frequency cap is an upper bound on the graphics clock frequency that constrains the PM firmware’s DVFS range. Instead of allowing the firmware to boost freely up to the GPU’s maximum supported frequency and get throttled due to EDP𝑝 , the cap limits it to a specified frequency, reducing the range of frequency and indirectly limiting dynamic power and EDP𝑝 variation. Memory clock states are discrete operating points for the GPU’s memory subsystem, including the HBM clock and the memory controller/interconnect clock. Each state corresponds to a set of frequencies for these memory clocks. PM firmware dynamically switches between these states based on memory bandwidth demand as well as the power consumed by competing compute resources. In addition to setting a graphics frequency cap, we identify levels at which to fix the memory clock state so that we avoid transient variation in memory clocks affecting power, EDP𝑝 , and performance. We developed a two-stage algorithm, called PACER, for selecting optimal values for these control parameters when serving inference requests online, since the optimal frequencies are workloadand input-dependent. We categorize possible input lengths (number of batched tokens) for LLM inference prompts into buckets 𝑏, up to the maximum number of batched tokens for each model. We can then determine an optimal graphics frequency cap 𝑓cap as well as fixed memory clock state for each batched-token length bucket 𝑏 based on offline frequency profiling. This allows us to construct a lookup table for optimal 𝑓𝑐𝑎𝑝 s and memory clock states which can be used online without needing to re-profile incoming requests. Thus, PACER involves two stages: (1) developing an offline 𝑓cap model for graphics and memory clock states, and (2) deploying the frequency states dynamically based on input request characteristics (Figure 7). 4.2.1 Stage 1: Offline frequency profiling. For every batched-token bucket 𝑏, we run baseline inference at the GPU’s default (uncapped) frequency and collect kernel traces together with per-kernel effective frequency samples. From this profiling, we obtain the kernel composition of each transformer layer, showing which kernels dominate overall prefill time. We use this kernel composition to extract the top-𝐾 kernels in the workload (𝐾 is a per-model, heuristic value – we select 𝐾 such that the kernels together constitute > 50% of the workload’s runtime). Section 6.1 shows us that for frequency capping to not degrade performance, we must allow compute-intensive kernels to computationally sprint to completion. Therefore, from the profiled frequency samples for each of the top-𝐾 kernels, we compute the 90th-percentile frequency 𝑝90𝑘 (𝑏). Because a single kernel that still needs high frequency would otherwise become the latency bottleneck (degrading

PACER: GPU Excursion Management for ML Workloads Stage 1 — Offline frequency profiling

Stage 2 — Online 𝑓𝑐𝑎𝑝 deployment (per epoch)

Baseline inference run (default frequencies) per batched tokens bucket 𝑏 kernel tracing & frequency sampling

kernel composition

11

vLLM V1 scheduler epoch batched tokens 𝑏𝑡 (this step)

p90

Update EWMA 𝑚 ← 𝛼 𝑏𝑡 + (1 − 𝛼) 𝑚

per-kernel effective p90 frequency for top-𝐾 kernels cap lookup

bucket ← bucket_of (𝑚) cap ← predicted_cap[bucket]

𝑓𝑐𝑎𝑝 model table For bucket 𝑏, predicted_cap(𝑏) = max𝑘 𝑝90𝑘 (𝑏) mclk_state(𝑏) = majority[mclk state in prefill]

Apply cap

next epoch

Fig. 7. PACER’s two stages. Offline, a baseline inference run is traced per batched-token bucket to recover kernel composition and per-kernel p90 effective frequencies, which together yield an 𝑓𝑐𝑎𝑝 model table. Online, the vLLM scheduler’s batched-token count is mapped to a bucket, and used to look up and apply the corresponding cap per epoch.

performance), the predicted cap for the bucket is set by the most demanding of those kernels: predicted_cap(𝑏) = max 𝑝90𝑘 (𝑏),

(1)

𝑘 ∈ K (𝑏 )

where K (𝑏) is the top-𝐾 kernel set for bucket 𝑏. Alongside the frequency cap, we also record the memory clock state for each bucket, setting mclk_state(𝑏) to the memory clocks that the GPUs spend the majority of their time in during the profiled baseline run. The result is a static lookup table {𝑏 ↦→ (𝑓cap (𝑏), mclk_state(𝑏))} that is later consumed at serving time. 4.2.2 Stage 2: Online 𝑓cap deployment. At serving time the table is applied once per scheduler epoch. Let 𝑏𝑡 be the number of batched tokens scheduled in epoch 𝑡. We track a smoothed batch size with an exponentially weighted moving average to bias the input towards the latest epoch while avoiding firmware oscillation on short-lived batch-size spikes: 𝑚𝑡 = 𝛼 𝑏𝑡 + (1 − 𝛼) 𝑚𝑡 −1,

0 < 𝛼 ≤ 1,

(2)

We use 𝛼 = 0.3 for all our experiments, although 𝛼 can be tuned depending on how reactive the system should be to abrupt request length changes. We then map 𝑚𝑡 onto a bucket, and look up the corresponding cap and memory clock state: bucket ← bucket_of (𝑚𝑡 ), 𝑓cap ← predicted_cap[bucket], mclk_state ← mclk_state[bucket]. The cap is then applied to the GPUs, and this process repeats whenever a new request arrives. 5 5.1

Methodology Serving stack and hardware

All models are served with vLLM [53] running inside a rocm/vllm:latest docker container (ROCm 7.0.0, PyTorch 2.9.0a0, Ubuntu 22.04.5) on a single node of 8xAMD Instinct MI300X GPUs [1]. We built an experimental system on top of a production quality AMD Instinct MI300X. Table 3 provides details for both offline and online inference models and configurations we ran in Section 6. We ran offline inference on a single GPU across dense and MoE models, and across different batch sizes and input lengths. For online serving, we ran with tensor parallelism TP=8. The graphics frequency on MI300X can be varied up to 2100 MHz. Memory clock states can vary between 3 states corresponding to clock frequencies of 1300 MHz, 1550 MHz, and 1800 MHz. We drive load

12

Jain et al.

Table 3. Inference workloads. Offline inference ran on a single MI300X GPU, while online serving is run tensor parallel across 8 AMD Instinct MI300X GPUs with Poisson arrival at varying QPS rates. Scenario

Model

Type

dtype

Dataset

Config

TTFT SLO

Offline

Qwen3-14B [108] LLaMA3-8B [99, 21] LLaMA3-70B [99, 21] Mixtral-8x7B [43]

Dense Dense Dense MoE

BF16 BF16 BF16 BF16

– – – –

bsz 1, ip 4k, op 128 bsz 1/4/8/16, ip 1k/4k, op 128 bsz 1/4, ip 1k/4k, op 128 bsz 1/4, ip 1k/4k, op 128

N.A. N.A. N.A. N.A.

Online

Llama-3.1-405B [21] DeepSeek-V3.2 Mistral-Medium-3.5-128B

Dense MoE (MLA) Dense

FP8 FP8 FP8

LongBench LongBench LongBench

500 req, ip 8,894 tok (avg) 500 req, ip 4,528 tok (avg) 500 req, ip 11,112 tok (avg)

4.62 s 2.08 s 2.08 s

with vLLM’s bench serve harness, replaying a sampled set of 500 LongBench [111] prompts under a fixed seed so all runs have an identical request stream. For DeepSeek-V3.2, the maximum number of batched tokens is limited to 8k since we run out of memory (OOM) for weights and activations beyond this limit. While larger prompts work, vLLM will divide them into 8k chunks (chunked prefill), so analyzing up to 8k prompts is sufficient. Since EDP𝑝 primarily affects prefill behavior (Section 3), we run in a prefill-decode disaggregated scenario [76, 114] and requests arrive as a Poisson process at different request rates. 5.2

Request Rates

We sweep request rate or queries per second (QPS) up to 0.3 because that is the point at which the prefill pipeline approaches saturation for the long-context prompts from LongBench. Treating prefill as a queue, the offered load (utilization 𝜌) is the ratio of the arrival token rate to the sustained prefill token throughput of the MI300X node. Arrival token rate is QPS times the LongBench average input length (roughly 4.5k–11k tokens per request depending on model), while prefill token throughput sustained by the MI300X node is on the order of ∼3.4–3.8k tokens/s. Under this model, utilization grows roughly linearly with QPS and reaches 𝜌 ≈ 1 near 0.35 QPS. Therefore, request rates beyond 0.3 QPS drive the system into an unstable, backlog-dominated regime that no longer reflects steady-state serving, which is why we bound the sweep at 0.3. 5.3

Metrics

We perform measurements for the prefill portion of the run since it suffers the most from EDP𝑝 (Section 3). For single GPU offline inference, we show percentage improvements in Time-To-FirstToken (TTFT) and reduction in EDP𝑝 . We also show normalized kernel durations to highlight the effect of EDP𝑝 reduction on transient variability. For online serving, we report TTFT at three percentiles to account for bursty Poisson arrival of requests: the median (p50), which reflects typical user-facing latency; 90th percentile (p90), which captures latency for the slower tail of requests; and and 99th percentile (p99), which reflects worst-case behavior. We also report %SLO attainment, defined as the percentage of requests whose TTFT falls at or below an SLO threshold 𝜏. Following prior serving work [33, 57, 94], we set 𝜏 = 𝑘 · 𝑇mean . 𝑇mean is the model’s mean TTFT when serving a single LongBench request on an idle TP=8 deployment, and 𝑘 is a slowdown budget, set to 2 based on recommendation from prior work [33, 13] for long context workloads, since LongBench prompts average 4.5k–11k tokens for the models we serve. Table 3 lists the resulting per-model 𝜏. 6 6.1

Evaluation EDP𝑝 Management in Hardware

Using the DPE and hardware boost management mechanism outlined in Section 4.1, we determine the impact of EDP𝑝 clamping by varying the acceptable EDP𝑝 threshold from 1.0× to 1.5×TDP. We

PACER: GPU Excursion Management for ML Workloads 1.2x 1.3x

1.4x 1.5x

CDF

1.0

0.5

0.0

0.9x TDP

1.0x TDP

1.1x TDP Power

1-1.1×TDP 1.1-1.2×TDP

baseline

%EDPp (norm to baseline)

1x 1.1x

13

1.2x TDP

1.3x TDP

100 ↓36%

0

0.5 60 80 100 Sample Index

(c)

120

140

TTFT Improv (%)

Power (×TDP)

baseline 1x TDP

40

↑7%

↓100% 1x 1.1x

1.2x 1.3x 1.4x TDP Level (xTDP)

1.5x baseline

(b)

1.0

20

1.2-1.3×TDP >1.3×TDP

↑5%

50

(a)

0

↑8%

↑2%

0

-2%

-5%

0%

0%

-11% −20

-26% 1x

1.1x

1.2x

1.3x 1.4x TDP Level

1.5x baseline

(d)

Fig. 8. (a) Cumulative distribution of EDP𝑝 when boost management clamps power to different TDP levels; (b) Reduction in %EDP𝑝 at different clamping levels; (c) Timeseries showing varying power values for LLaMA3-8B inference with and without clamping; and (d) Performance impact relative to the unclamped baseline case (%). All results are for the prefill portion of LLaMA3-8B inference with 4096 input tokens and a batch size of 1.

first show results for LLaMA3-8B inference (4096 input tokens, 128 output tokens, and batch size 1) where the DPE estimation error is about 6%. Then we show results for LLaMA3-8B in different input configurations, as well as Qwen3-14B, Mixtral-8x7B, and LLaMA3-70B (Table 2). Figure 8a shows the power measurement CDF for LLaMA3-8B inference. We utilize the DPE to cap the boost from 1×TDP to 1.5×TDP and compare it to the baseline normal operation without clamping (labeled baseline). First, we observed that the boost management mechanism effectively clamps power based on the set threshold. The CDF of power events for each threshold shows that very few (<2%) events extend beyond the specified threshold value. Moreover, baseline’s EDP𝑝 behavior has higher amplitude power overshoots than those managed by our boost management mechanism. The lower boost thresholds (up to 1.1×TDP) have significantly fewer overshoots and fewer higher magnitude overshoots than baseline. At 1.2×TDP and above, the numbers of overshoots are similar to the baseline, but on average they have lower peak amplitudes. Figure 8c shows the timeseries of the power for the 32 Transformer layers in LLaMA3-8B inference for prefill for the unclamped and fully clamped (1×TDP) case. Power varies across each Transformer layer in the unclamped case either due to application behavior and/or EDP𝑝 mitigation. However, by using the boost cap, the fully clamped solution (1×TDP) removes these power peaks. Unfortunately, Figure 8d demonstrates that the performance impact of eliminating power peaks (by limiting boost to 1.0×TDP) is severe. The TTFT for inference degrades by 26% compared to normal operation when power is clamped to 750W (TDP). Conversely, adding more boost headroom improves performance until it matches the baseline at 1.4×TDP. This shows that boosting has computational sprinting benefits, allowing workloads to sprint during EDP𝑝 events. We also evaluate clamping across different models, input lengths, and batch sizes. Figure 9a shows clamping effects for different batch sizes and input configurations for LLaMA3-8B inference. Across different input configurations, LLaMA3-8B inference shows similar EDP𝑝 and performance trends with increasing clamping thresholds. As batch size increases, the proportion of high magnitude EDP𝑝 events (> 1.2×TDP) reduces in the prefill phase of the unclamped baseline. However, lower clamping thresholds lose out on sprinting benefits and suffer from worse performance. Figure 9b shows other models exhibit the same trend (geomean 28% loss at 1×TDP clamp). The proportions of EDP𝑝 in the unclamped baseline (labeled B) are different for different models, with Qwen3-14B and

14

Jain et al. 1-1.1×TDP

1.1-1.2×TDP

%EDPp (norm to baseline)

bs=1, ip=1024

100

50

0

1 1.1 1.2 1.3 1.4 1.5 B

0 TTFT Improv (%)

−5

B = baseline bs=8, ip=4096 100

100

75

75

50

50

50

25

25

25

0

0

1 1.1 1.2 1.3 1.4 1.5 B

-3%

-1% 0%

75

0

1 1.1 1.2 1.3 1.4 1.5 B

0 -3%

-2%

1 1.1 1.2 1.3 1.4 1.5 B 1%

0

0% 0%

0% 0%

-6%

-4%

−10

-8%

−10

>1.3×TDP bs=4, ip=4096

100

0

-1% -3%

1.2-1.3×TDP

bs=4, ip=1024

−10

-5%

-8%

−10

-13%

-9%

−20

−15 -16%

−20 -21%

-25%

-16% 1 1.1 1.2 1.3 1.4 1.5 B

1 1.1 1.2 1.3 1.4 1.5 B

−20 -21%

1 1.1 1.2 1.3 1.4 1.5 B

1 1.1 1.2 1.3 1.4 1.5 B

Clamping Threshold (×TDP)

(a) LLaMA3-8B inference for different input lengths/batch sizes 1-1.1×TDP

1.1-1.2×TDP

%EDPp (norm to baseline)

LLaMA3-8B 100

100

75

50

50

25

25

25

0

−10

−20

-1% -3% -4%-4%

1 1.1 1.2 1.3 1.4 1.5 B

0 0

0 -1%-1%

0% 0%

-3%

50

0

1 1.1 1.2 1.3 1.4 1.5 B

-4%-2%-1%-1% -18%

1 1.1 1.2 1.3 1.4 1.5 B

0 -1%

−5

−50

−5

-11%

LLaMA3-70B

100 75

0

B = baseline

100

50

1 1.1 1.2 1.3 1.4 1.5 B

>1.3×TDP Mixtral-8x7B

75

0

TTFT Improv (%)

1.2-1.3×TDP

Qwen3-14B

0% 0% 0%

-4%

−10 −100

−10 -13%

-23% 1 1.1 1.2 1.3 1.4 1.5 B

-127%

1 1.1 1.2 1.3 1.4 1.5 B

−15 -16%

1 1.1 1.2 1.3 1.4 1.5 B

1 1.1 1.2 1.3 1.4 1.5 B

Clamping Threshold (×TDP)

(b) bs1, ip4096 configuration for different models Fig. 9. %EDP𝑝 and performance impact at different clamp levels for (a) varying batch size and input lengths for LLaMA3-8B inference and (b) different models with bsz 1 and 4096 input tokens.

Mixtral-8x7B showing more high magnitude EDP𝑝 events than the LLaMA models. Mixtral-8x7B in particular suffers severe performance degradation when the threshold is limited to 1.0×TDP, reinforcing EDP𝑝 ’s sprinting benefits. Regardless, a clamping threshold of 1.2-1.3×TDP is sufficient to eliminate high-magnitude EDP𝑝 for all these models, while providing near-identical performance to the baseline. Thus, Figure 9b shows the DPE effectively clamps EDP𝑝 even for models it is not tuned on, like Qwen3-14B and Mixtral8x7B. Overall, these results show that EDP𝑝 ’s computational sprinting is crucial for highly variable applications like LLMs. However, we do not require a significant boost beyond the TDP. Instead, some degree of EDP𝑝 beyond TDP (e.g., 1.3×TDP) is sufficient.

PACER: GPU Excursion Management for ML Workloads

15

Lessons Learned: • Hardware clamping mechanisms can be tuned to strictly control EDP𝑝 to specific limits. • Low-magnitude EDP𝑝 allows workloads to sprint for short bursts of time by boosting their power. This boost leads to better performance for highly variable applications like LLMs. • Sprinting benefits diminish beyond a certain degree (1.3×TDP) – higher magnitude EDP𝑝 does not yield better performance over baseline.

6.2

EDP𝑝 Management in Firmware

6.2.1 Single GPU Offline Inference. To determine how well existing DPM solutions deal with the variable nature of ML applications and their associated overshoot behavior, we ran three solutions with various frequency caps: Oracle Capping: Oracle Capping shows the performance for an idealized, unattainable DVFS solution. It detects the best operating frequency for each kernel during the frequency cap sweep and analytically combines them with zero overhead. Realistic Oracle: This approach includes the overhead of predicting and transitioning to the best frequency, while it still uses oracle frequencies. Thus, it only modifies frequency approximately every two milliseconds – the granularity of traditional firmware V/f scaling [8, 74, 93, 117]. E2E Capping: E2E Capping uses the best operating frequency for each application when running with a maximum frequency cap. ML applications often operate at a lower effective frequency due to their high compute demand and resulting power draw [58, 5]. Thus, this approach determines if lower frequencies alleviate the negative impact of EDP𝑝 management, reduce power throttling, and improve performance. E2E Capping is the mechanism used for PACER. We compare these to the oracle solutions to identify how close PACER is to an idealized, unattainable solution. We use the methodology in Section 4.2.1 to identify optimal frequency caps for each model and input. Figure 10 shows how our three firmware solutions improve prefill performance relative to a default run with no frequency capping (Baseline). Although the power management algorithm does a reasonable job, there is still the potential for additional performance if we can identify the best operating frequency for each application. Oracle Capping’s results demonstrate that the existing power management algorithm is reactive and accurate enough to produce viable, effective DVFS results for each application. Oracle Capping provides the most performance improvement (≈7%) for workloads like Mixtral-8x7B, which has very short kernels. However, other applications with larger input sizes, and hence longer kernels, see diminishing performance benefits with Oracle Capping. Although Realistic Oracle cannot change quickly enough to capture all of Oracle Capping’s performance benefits, it still improves performance by 3% geomean (6% max) over Baseline. Finally, E2E Capping identifies a frequency cap that limits EDP𝑝 events and enables the DPM to operate more effectively. Thus, E2E Capping improves performance by up to ≈4% over Baseline on real hardware, albeit not to the same extent as Oracle Capping. Limiting the maximum frequency reduces the range over which DVFS has to operate, enabling it to react faster [100]. However, the biggest benefit is that it reduces the extreme EDP𝑝 events that require a substantial corrective solution (i.e., the ones that have the highest amplitude, and shortest duration requirements). By limiting the maximum frequency, we reduce the occurrence of events where the current operating frequency is considerably higher than the tolerable frequency for the kernel. Figure 11 shows the occurrence of EDP𝑝 events with and without frequency limits for a subset of applications in Figure 10. The results show that with frequency limits, the system continues to utilize lower power overshoots that facilitate computational sprinting but avoid the more significant events that require detrimental corrective action. For example, in Mixtral-8x7B with 4096 input tokens and a batch size of 1, the number of EDP𝑝 events increases, but they are

16

Jain et al.

TTFT Improvement over Baseline (%)

E2E Capping

Realistic Oracle

Oracle Capping

6 4

1400

1600 1400

1600

1500

2 1500

0

1500

1600

1600

1600

1600

1600

bs1 bs1 bs1 bs4 bs8 bs16 bs1 bs1 bs4 bs1 bs1 bs4 ip4k ip1k ip4k ip4k ip4k ip4k ip1k ip4k ip4k ip1k ip4k ip4k

Qwen3-14B

Llama-3.1-8B

Llama-3.1-70B

Mixtral-8x7B

Configuration (Batch Size, Input Length)

Fig. 10. %TTFT improvement over uncapped (2100 MHz 𝑓𝑚𝑎𝑥 ) baseline with E2E Capping and oracle solutions.

%EDPp (norm 2100MHz)

1.0-1.1xTDP 1.1-1.2xTDP

1.2-1.3xTDP >1.3xTDP

2100MHz E2E Best Cap

100 50 0

B B B B B 7B 7B -70 4 -70 6 1-8 4 1-8 6 -14 6 8x 4 8x 6 n3 409 a-3. 102 a-3. 409 -3.1 102 -3.1 409 ral- 102 ral- 409 e t t Qw 1 IP Llam 1 IP Llam 1 IP lama 1 IP lama 1 IP Mix 1 IP Mix 1 IP L BS L BS BS BS BS BS BS Configuration

Fig. 11. Reduction in high-magnitude EDP𝑝 (normalized to 2100MHz run) with E2E Capping for a subset of configurations from Figure 10. 1.1

gemm_1

1.2

attn_fwd

1.0

1.2

gemm_2

1.0

Normalized Duration

Normalized Duration

0.9

1.0 expert_2_fc

1.1 1.0 0.9

1300 1400 1500 1600 1700 1800 1900 2000 2100 Frequency Cap (MHz)

(a) LLaMA3-8B bs1

1.0

expert_1_fc

1.1 0.9

gemm_1

1.1

1.0

1.0

attn_fwd 1300 1400 1500 1600 1700 1800 1900 2000 2100 Frequency Cap (MHz)

(b) Mixtral-8x7B bs1

0.9 Normalized Duration

1.2

gemm_2

1.1 1.0 0.9

gemm_3

1.1 1.0 0.9

1300 1400 1500 1600 1700 1800 1900 2000 2100 Frequency Cap (MHz)

(c) LLaMA3-70B bs1

Fig. 12. Kernel duration scaling with frequency caps for the top 3 kernels (by kernel duration), normalized to uncapped 2100 MHz runs, for (a) LLaMA3-8B, (b) Mixtral-8x7B, and (c) LLaMA3-70B runs. The kernel distributions show durations for 96 kernel instances across 32 transformer layers and 3 trials for each model.

of lower magnitude and eliminating events over 1.3×TDP improves performance by 4%. Given the variable nature of these applications, capping the frequency reduces the maximum C𝐴𝐶 seen during the run, reducing the delta between the maximum and minimum C𝐴𝐶 in the workload and power variability across the workload. Setting graphics frequency caps also improves system stability, generating more deterministic results. Figure 12 shows the variability results for the three longest duration kernels, for a subset of the benchmarks in Figure 10, with different frequency limits. The dashed red line at 1.0 shows the

PACER: GPU Excursion Management for ML Workloads gemm

rccl

gemm

rccl

fwd_attn

layernorm

fwd_attn

layernorm

scaled_fp8_quant

act_and_mul

gemm

rccl

scaled_fp8_quant

act_and_mul

rotary

reshape

mqa_attn

gather

rotary

reshape

paged_attn_reduce

paged_attn

elementwise

act_and_mul

paged_attn_reduce

paged_attn

moe_sort

other

other

other

40

40

40

20

20

20

0

0

0

Input-length bucket

(a) LLaMA-3.1-405B

(b) DeepSeek v3.2

<

)

2k

4k

,3

,6

6k

2k

(1

(3

k,

k,

(4

(8

<2

k,

(2

4k

2k

,6

,3

2k

6k

(3

(1

k,

k,

(4

(8

<2

k,

(2

Input-length bucket

1 (1 k k, 2k (2 ) k, 4k (4 ) k, 8 (8 k) k, 1 (1 6k) 6k ,3 2k )

60

)

60

8k ) 16 k)

60

k

80

4k )

80

)

80

)

100

8k ) 16 k)

100

k

100

4k )

Share of prefill kernel time (%)

17

Input-length bucket

(c) Mistral-Medium-3.5-128B

Fig. 13. Prefill kernel composition by input-length bucket, normalized to the total prefill kernel time.

mean across all kernel instances across three trials (32 transformer layers per-trial) for a system with no frequency limits (i.e., it can attain a maximum frequency of 2100MHz). The solid line shows the median performance at that frequency cap relative to the performance of an uncapped system. Values below 1.0 indicate a reduction in runtime; values above 1.0 indicate an increase in runtime. Overall, Figure 12 shows lower graphics frequency limits marginally improve performance and provide more stable kernel operation. These complex GPUs, running modern ML applications, push the system’s physical limits, causing specification-imposed power throttling (Section 2). When and how the system is throttled is a complex algorithm that depends on intrinsic (process variation, power delivery network design, cooling solutions) and extrinsic (ambient temperature, application behavior) factors. Regardless, pushing the system’s physical limits increases performance variability. Given the frequency of synchronization across applications parallelized across many devices, this variability within and across GPUs could increase overall runtime [16, 41, 87, 89]. Lessons Learned: • Setting frequency limits reduces high-magnitude EDP𝑝 events, improving performance by 3-7% over baseline GPU power management, and is within 3% of an oracle per-kernel capping. • Setting frequency limits also provides stable, deterministic operations, reducing transient kernel variability.

6.2.2 Multi-GPU Online Serving. As outlined in Section 4.2, we can use the data from offline frequency profiling of models to create a lookup table for optimal 𝑓𝑐𝑎𝑝 s for each model and input length. When serving requests dynamically, we can use this lookup table to apply 𝑓𝑐𝑎𝑝 corresponding to the effective input length seen every scheduler epoch. Stage 1: Offline frequency profiling. Figure 13 shows the kernel composition for online serving models specified in Table 3, namely LLaMA3.1-405B, DeepSeek v3.2, and Mistral-Medium-3.5128B with varying number of input batched tokens. GEMMs dominate overall transformer layer runtime for the dense models (Figures 13a and 13c) across input lengths, with RCCL AllReduce (rccl) and attention (fwd_attn) being the next largest contributors. As input length increases for the dense models, attention kernels grow as a percentage of the total runtime, but GEMMs still

18

Jain et al.

Table 4. Stage 1 output: predicted 𝑓𝑐𝑎𝑝 (MHz) per batched-token bucket 𝑏, and per-cap measured TTFT improvement. We profile DeepSeek v3.2 up to 8k batched tokens due to memory capacity issues (OOM). Bucket 𝑏

<2k (2k, 4k) (4k, 8k) (8k, 16k) (16k, 32k) (32k, 64k)

LLaMA3.1-405B

DeepSeek v3.2

Mistral-Medium-3.5-128B

𝑓𝑐𝑎𝑝

%TTFT

mclk

𝑓𝑐𝑎𝑝

%TTFT

mclk

𝑓𝑐𝑎𝑝

%TTFT

mclk

1900 1500 1600 1400 1300 1300

7.5 9.5 7.5 7.9 6.7 1.5

1550 1550 1550 1550 1550 1550

2100 2000 1900

0.1 0.0 0.4

1800 1800 1800

2000 1900 1700 1600 1400 1400

1.0 0.8 3.7 3.1 4.4 5.9

default default default default default 1550

OOM

dominate overall as opposed to the dense models where 11 kernels make up a transformer layer, the MoE-based DeepSeek v3.2 model has 61 kernels per layer, with a much more fractured kernel distribution(Figure 13b). While GEMMs still make up part of the top-𝐾 kernels by runtime for DeepSeek v3.2, the distribution also includes less compute-intensive (lower C𝐴𝐶 ) kernels like gather, and elementwise kernels. As a result, the dense models benefit more from frequency capping since their top 𝐾 kernels have higher sensitivity to frequency changes. Based on the p90 frequencies of the top-𝐾 kernels for these models, Table 4 shows the predicted 𝑓𝑐𝑎𝑝 for each batched tokens bucket 𝑏 for LLaMA3.1-405B, DeepSeek v3.2, and Mistral-Medium3.5-128B. For the dense models, the predicted cap decreases linearly as prompts get longer, from 1900 MHz to 1300 MHz for LLaMA3.1-405B and from 2000 MHz to 1400 MHz for Mistral-Medium3.5-128B. This is because larger bucket sizes produce larger running GEMMs with higher EDP𝑝 , which makes the DPM throttle the graphics clock to lower frequencies. Both dense models also prefer the DPM to keep memory clock states to the default, middle value (1550 MHz). Conversely, DeepSeek v3.2 prefers both higher frequencies across buckets and a higher memory clock state. As shown in in Figure 13b, the MoE model’s top-𝐾 set of kernels includes memory-bound gather and elementwise kernels that do not saturate power. Therefore, their p90 frequency stays high, and the maximum frequency selected by Eq 1 is higher than that for dense models. Moreover, DeepSeek v3.2 also prefers the highest memory clock state since more of its kernels are memory-intensive. Table 4 also shows the percentage improvement in TTFT for each of these bucket lengths when the corresponding optimal frequency cap is applied. TTFT improvements show a bell curve: benefits initially increase as prompt length increases, but reduce for very large prompts. For a given model, as input length increases, kernels grow in computational demand to cause higher EDP𝑝 and consequently more throttling. Frequency capping alleviates the adverse performance impact of this throttling by limiting EDP𝑝 , improving TTFT over baseline as input length increases. However, as input lengths grow, the durations of these compute-intensive GEMM kernels also increases, allowing the PM firmware to mitigate EDP𝑝 in the baseline too. This means that for larger bucket lengths, such as >16k tokens for LLaMA3.1-405B, frequency capping has diminishing returns when compared to the baseline. The point at which we get diminishing returns depends on kernel durations and composition of each model. While these improvements are not necessary as inputs for Stage 2’s online serving algorithm, they help explain the results we obtain in Stage 2. Stage 2: Online 𝑓𝑐𝑎𝑝 deployment. Using the 𝑓𝑐𝑎𝑝 model table in Stage 1, we then deploy these frequency caps dynamically, as detailed in Section 4.2. Figure 14 shows online serving results at varying request rates, comparing PACER against baseline behavior, where all frequencies are uncapped. For LLaMA3.1-405B (Figures 14a and 14d), PACER reduces median TTFT by 9–12% across request rates. Frequency capping has unequal benefits depending on prompt length, so the dynamic batched token lengths at different request rates can affect how much TTFT improves. However, we observed that the p50 batched tokens per scheduler step is nearly rate-independent – the median token length per step grows from 8.5k to 10k tokens as QPS is increased from 0.1 to

PACER: GPU Excursion Management for ML Workloads p50

p90

p99

19

p50

p90

p50

p99

p90

p99

10 6

10 5

TTFT Improv (%)

TTFT Improv (%)

TTFT Improv (%)

15 5

0

4 2 0

−5

0 0.2

0.3

0.1

Request rate (QPS)

(a) LLaMA-3.1-405B

2.6 %

5.6 %

75 50 25 0

100

1.0 %

1.0 %

0.2

0.3

Request rate (QPS)

(d) LLaMA-3.1-405B

0.2

0.3

Request rate (QPS)

2.0 %

75 50 25 0

0.1

0.1

(c) DeepSeek-V3.2

PACER

Baseline

SLO Attainment (%)

SLO Attainment (%)

100

0.3

(b) Mistral-Medium-3.5-128B PACER

Baseline 0.2 %

0.2

Request rate (QPS)

PACER

Baseline

100 SLO Attainment (%)

0.1

−2

1.6 %

1.0 %

1.4 %

75 50 25 0

0.1

0.2

0.3

Request rate (QPS)

(e) Mistral-Medium-3.5-128B

0.1

0.2

0.3

Request rate (QPS)

(f) DeepSeek-V3.2

Fig. 14. Online serving results. Top row: reduction in TTFT with PACER over baseline at each percentile (p50/p90/p99); higher is better. Bottom row: fraction of requests meeting the TTFT SLO, annotated with PACER’s percentage gain over baseline.

0.3. Therefore, we only see a marginal improvement in p50 TTFT as request rate is increased. SLO attainment improves from 97.4% to 97.6% at 0.1 QPS and from 72.6% to 78.2% at 0.3 QPS. Although the per-request TTFT reduction is roughly constant with load, its effect on attainment grows with QPS. At 0.1 QPS, the baseline already meets the SLO target for 97.4% of requests and there is little headroom left to recover, whereas at 0.3 QPS, queueing pushes over a quarter of requests past the threshold and PACER improves attainment by 6%. Mistral-Medium-3.5-128B shows the same effect as LLaMA3.1-405B, improving median TTFT by 4–5% and SLO attainment by up to 2% (Figures 14b and 14e). The lower percentages of improvement track with the offline inference results in Table 4, which showed around 3–6% improvements for different buckets for the Mistral model. The tail (p99) of TTFT is iso-performance to the baseline or degrades with PACER because large requests operate in the diminishing returns region of the bell curve, and queueing delays dominate TTFT more than frequency changes. DeepSeek v3.2 benefits least, improving median TTFT by at most 1.2% and SLO attainment by about 2% (Figures 14c and Figures 14f). This follows directly from the 𝑓𝑐𝑎𝑝 model and Stage 1’s kernel composition. DeepSeek’s predicted caps sit at or near the uncapped 2100 MHz ceiling for the buckets it can serve across 8 GPUs, so PACER leaves the firmware close to its default operating state. With 61 kernels per layer and a large fraction of time running low-C𝐴𝐶 kernels, DeepSeek v3.2 does not suffer from as much EDP𝑝 events that would warrant aggressive frequency capping or EDP𝑝 management. MoE models with fractured kernel distributions are therefore poor candidates for frequency capping, and PACER accordingly retains performance close to the baseline behavior.

20

Jain et al.

Lessons Learned: • For dense models, PACER improves TTFT by up to 12% and SLO attainment by up to 6%. • Each model’s benefits track its distribution of frequency-sensitive kernels: GEMM-dominated dense models gain the most, while MoE models with fractured kernel distributions see much smaller improvements.

7

Related Work

Other works have also noted the existence of EDP𝑝 . Patel, et al. noted GPU EDP𝑝 complicate power oversubscription solutions, which rely on fixed power budgets to monitor and manage power oversubscription at the rack level or higher [77]. Later work from Patel and others shows power peaks interfere with power caps [75, 94], resulting in noticeable overshoots beyond the power cap. Notably, the work by Choukse, et al. [16] set a power floor of 0.9×TDP, their dynamic power range is 20% given a minimum EDP point of 1.1×TDP. Existing work identifies and qualitatively discusses the ramifications of power overshoots [16, 75, 77, 94, 101]. Conversely, our work performs a detailed quantification on the benefits and costs of peak power management on modern GPU workloads. Some works have identified frequency capping as a method to control power consumption of workloads, thereby reducing workload energy, with tolerable loss in performance. Kakolyris et al. [45] designed an offline profiling based frequency selection algorithm for inference serving on NVIDIA GPUs, but their objective was to reduce energy consumption while meeting TBT SLOs in decode, not to reduce variability or improve performance or SLO attainment during prefill. Thus, while they use frequency limits as a control parameter similar to PACER, our approach is complementary to theirs and identifies potential for performance improvements and transient jitter reduction by managing the same control variable, frequency limits, in prefill while they manage it in the decode phase. Other work [28, 31] also look at dynamic frequency selection, but for scientific workloads or smaller micro-benchmarks, not LLMs which we have impose unique constraints on the system’s power management (Section 3.2). Jain et al. [40] do quantify EDP𝑝 across workload types, but they use the power distribution of workloads to group them into classes, so known frequency settings from previously profiled workloads can be applied to a new, unseen workload. Moreover, they assume a single frequency cap works for a workload, while we show that frequencies are model- and input-dependent, with kernel distributions and their C𝐴𝐶 affecting EDP𝑝 and consequently the optimal frequencies for the workload. Instead, PACER proposes a one-time baseline profiling of kernel composition and frequencies to identify optimal frequencies for each model and input during online serving. Hyperscalers dedicate entire clusters to serving the same model, so we believe that PACER’s offline profiling step is a one-time cost. Other work uses intra-device GPU parallelism to increase existing hardware usage [18, 46, 116]. Although these works improve performance by more effectively sharing resources, they do not consider power management or EDP𝑝 . Thus, they focus on different problems. 8

Conclusion

Computing systems are being driven by the voracious scaling needs of workloads, especially ML. Unfortunately, these workloads frequently cause EDP𝑝 , leading to jittery power and frequency behavior that can impact system stability. To understand the impact of EDP𝑝 , we characterized it across graph analytics, HPC, and ML workloads. Our results show that fine-grained variations in compute requirements, especially for ML workloads, make stable power management extremely difficult. While vendors have added complex DPM support to prevent thermal meltdown and robust VRs for power delivery, running small intra- and inter-kernel phases with different power behavior in rapid succession still leads to EDP𝑝 , causing power and frequency variability, often degrading performance for these workloads. To manage these EDP𝑝 and power variations, we

PACER: GPU Excursion Management for ML Workloads

21

examined a fine-grained hardware mechanism to strictly clamp power within limits and a firmware frequency capping solution, PACER, that limits maximum operating frequency limits. Overall, across popular ML workloads, our approaches improve performance by 3-12% on state-of-the-art, production-grade systems and are within 3% of an oracular solution. More generally, our work highlights how power must be intelligently managed across peak power, average power, and EDP𝑝 to meet key performance goals across diverse workloads. References [1] [2] [3] [4] [5]

[6]

[7]

[8]

[9]

[10] [11] [12]

[13]

[14]

[15]

[16]

AMD. 2023. AMD CDNA™ 3 Architecture. (2023). https://www.amd.com/content/dam/amd/en/documents/instincttech-docs/white-papers/amd-cdna-3-white-paper.pdf. AMD. 2025. AMD CDNA™ 4 Architecture. (2025). https://www.amd.com/content/dam/amd/en/documents/instincttech-docs/white-papers/amd-cdna-4-architecture-whitepaper.pdf. AMD. 2024. ROCm SMI LIB Documentation, Release 7.2.0. Version 7.2.0. (2024). https://rocm.docs.amd.com/en/docs6.2.0/about/release-notes.html. Jeremy Appleyard, Tomás Kociský, and Phil Blunsom. 2016. Optimizing Performance of Recurrent Neural Networks on GPUs. CoRR, abs/1604.01946. http://arxiv.org/abs/1604.01946. Meena Arunachalam, Miro Hodak, Wei-Ting Liao, Karan Verma, Ean Garvey, Kumar Deepak, Giuseppe Franco, and Eliot Li. 2025. Reproducing the AMD Instinct™ GPUs MLPerf Inference v5.0 Submission. (2025). https://rocm.blogs. amd.com/artificial-intelligence/reproducing-amd-mlperf-inference-submission/README.html. Nick Baker, Vishesh Vikas, Andrew Lemmon, Chase Fortin, Justin Conzola, Arthur Boutry, and John Austin. 2024. Power Cycling of 1.7kV Multi-Chip Power Modules – SiC MOSFETs vs Silicon IGBTs. In PCIM Europe 2024; International Exhibition and Conference for Power Electronics, Intelligent Motion, Renewable Energy and Energy Management, 1918–1924. doi:10.30420/566262269. Claude Bernard, Michael C. Ogilvie, Thomas A. Degrand, Carleton E. Detar, Steven A. Gottlieb, A. Krasnitz, R.L. Sugar, and D. Toussaint. 1991. Studying Quarks and Gluons On MIMD Parallel Computers. Int. J. High Perform. Comput. Appl., 5, 4, (Dec. 1991), 61–70. doi:10.1177/109434209100500406. Srikant Bharadwaj, Shomit Das, Kaushik Mazumdar, Bradford M. Beckmann, and Stephen Kosonocky. 2023. Predict; Don’t React for Enabling Efficient Fine-Grain DVFS in GPUs. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 4 (ASPLOS). Tor M. Aamodt, Michael M. Swift, and Natalie D. Enright Jerger, (Eds.) ACM, 253–267. doi:10.1145/3623278.3624756. Paolo Boldi, Marco Rosa, Massimo Santini, and Sebastiano Vigna. 2011. Layered label propagation: a multiresolution coordinate-free ordering for compressing social networks. In Proceedings of the 20th international conference on World Wide Web. Sadagopan Srinivasan, Krithi Ramamritham, Arun Kumar, M. P. Ravindra, Elisa Bertino, and Ravi Kumar, (Eds.) ACM Press, 587–596. Paolo Boldi and Sebastiano Vigna. 2004. The WebGraph framework I: Compression techniques. In Proc. of the Thirteenth International World Wide Web Conference (WWW 2004). ACM Press, Manhattan, USA, 595–601. Cadence System Analysis. 2021. Power Electronics Reliability and Temperature Dependency. https://resources. system-analysis.cadence.com/blog/msa2021-power-electronics-reliability-and-temperature-dependency. (2021). Mithun Chandrasekhar. 2019. AMD Radeon Community Update: More Control Over GPU Power and Performance, Enhanced Thermal Monitoring, Maximized Performance. https://community.amd.com/t5/gaming/amd-radeoncommunity-update-more-control-over-gpu-power-and/ba-p/418629. (2019). Tzu-Tao Chang, Benjamin Yuanyang Hong, Kiet Pham, and Shivaram Venkataraman. 2027. TeDiServe: High SLO Attainment Serving for Diffusion Language Models. In 22nd European Conference on Computer Systems (EuroSys). doi:10.1145/3842654.3848543. Dimitrios Chasapis, Marc Casas, Miquel Moretó, Martin Schulz, Eduard Ayguadé, Jesus Labarta, and Mateo Valero. 2016. Runtime-Guided Mitigation of Manufacturing Variability in Power-Constrained Multi-Socket NUMA Nodes. In Proceedings of the 2016 International Conference on Supercomputing (ICS ’16) Article 5. Istanbul, Turkey, 12 pages. Dimitrios Chasapis, Miquel Moretó, Martin Schulz, Barry Rountree, Mateo Valero, and Marc Casas. 2019. Power Efficient Job Scheduling by Predicting the Impact of Processor Manufacturing Variability. In Proceedings of the ACM International Conference on Supercomputing (ICS ’19), 296–307. isbn: 9781450360791. doi:10.1145/3330345.3330372. Esha Choukse, Brijesh Warrier, Scot Heath, Luz Belmont, April Zhao, Hassan Ali Khan, Brian Harry, Matthew Kappel, Russell J. Hewett, Kushal Datta, Yu Pei, Caroline Lichtenberger, John Siegler, David Lukofsky, Zaid Kahn, Gurpreet Sahota, Andy Sullivan, Charles Frederick, Hien Thai, Rebecca Naughton, Daniel Jurnove, Justin Harp, Reid Carper, Nithish Mahalingam, Srini Varkala, Alok Gautam Kumbhare, Satyajit Desai, Venkatesh Ramamurthy, Praneeth Gottumukkala, Girish Bhatia, Kelsey Wildstone, Laurentiu Olariu, Ileana Incorvaia, Alex Wetmore, Prabhat Ram, Melur Raghuraman, Mohammed Ayna, Mike Kendrick, Ricardo Bianchini, Aaron Hurst, Reza Zamani, Xin Li, Michael

22

[17]

[18]

[19]

[20]

[21] [22]

[23]

[24]

[25]

[26]

[27]

[28] [29] [30] [31]

[32]

[33]

[34]

Jain et al.

Petrov, Gene Oden, Rory Carmichael, Tom Li, Apoorv Gupta, Pratikkumar Patel, Nilesh Dattani, Lawrence Marwong, Rob Nertney, Hirofumi Kobayashi, Jeff Liott, Miro Enev, Divya Ramakrishnan, Ian Buck, and Jonah Alben. 2025. Power Stabilization for AI Training Datacenters. arXiv preprint arXiv:2508.14318. https://arxiv.org/abs/2508.14318 arXiv: 2508.14318 [cs.AR]. J. Coplin and M. Burtscher. 2016. Energy, Power, and Performance Characterization of GPGPU Benchmark Programs. In IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW). (May 2016), 1190–1199. doi:10.1109/IPDPSW.2016.164. Xingqi Cui, Chieh-Jan Mike Liang, Jiarong Xing, and Haoran Qiu. 2025. From Models to Operators: Rethinking Autoscaling Granularity for Large Generative Models. (2025). https://arxiv.org/abs/2511.02248 arXiv: 2511.02248 [cs.DC]. Abhishek Das, Serkan Ozdemir, Gokhan Memik, Joseph Zambreno, and Alok Choudhary. 2007. Mitigating the Effects of Process Variations: Architectural Approaches for Improving Batch Performance. In Workshop on Architectural Support for Gigascale Integration (ASGI). Daniele De Sensi, Tiziano De Matteis, Konstantin Taranov, Salvatore Di Girolamo, Tobias Rahn, and Torsten Hoefler. 2022. Noise in the Clouds: Influence of Network Performance Variability on Application Scalability. Proc. ACM Meas. Anal. Comput. Syst., 6, 3, Article 49, (Dec. 2022), 27 pages. doi:10.1145/3570609. Abhimanyu Dubey et al. 2024. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783. https://arxiv.org/abs/ 2407.21783 arXiv: 2407.21783 [cs.AI]. Bishwajit Dutta, Vignesh Adhinarayanan, and Wu-chun Feng. 2018. GPU Power Prediction via Ensemble Machine Learning for DVFS Space Exploration. In Proceedings of the 15th ACM International Conference on Computing Frontiers (CF). Association for Computing Machinery, Ischia, Italy, 240–243. isbn: 9781450357616. doi:10.1145/ 3203217.3203273. Kaijie Fan, Biagio Cosenza, and Ben Juurlink. 2019. Predictable GPUs Frequency Scaling for Energy and Performance. In Proceedings of the 48th International Conference on Parallel Processing (ICPP 2019) Article 52. Association for Computing Machinery, Kyoto, Japan, 10 pages. isbn: 9781450362955. doi:10.1145/3337821.3337833. Songchun Fan, Seyed Majid Zahedi, and Benjamin C. Lee. 2016. The Computational Sprinting Game. In Proceedings of the Twenty-First International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS ’16). Association for Computing Machinery, Atlanta, Georgia, USA, 561–575. isbn: 9781450340915. doi:10.1145/2872362.2872383. Jiří Filipovič, Matúš Madzin, Jan Fousek, and Ludundefinedk Matyska. 2015. Optimizing CUDA Code by Kernel Fusion: Application on BLAS. The Journal of Supercomputing, 71, 10, (Oct. 2015), 3934–3957. doi:10.1007/s11227-0151483-z. R. Ge, R. Vogt, J. Majumder, A. Alam, M. Burtscher, and Z. Zong. 2013. Effects of Dynamic Voltage and Frequency Scaling on a K20 GPU. In 42nd International Conference on Parallel Processing (ICPP), 826–833. doi:10.1109/ICPP. 2013.98. Aaron Grenat, Sanjay Pant, Ravinder Rachala, and Samuel Naffziger. 2014. 5.6 adaptive clocking system for improved power efficiency in a 28nm x86-64 microprocessor. In 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), 106–107. doi:10.1109/ISSCC.2014.6757358. João Guerreiro, Aleksandar Ilic, Nuno Roma, and Pedro Tomás. 2019. DVFS-aware application classification to improve GPGPUs energy efficiency. Parallel Computing, 83, 93–117. doi:https://doi.org/10.1016/j.parco.2018.02.001. Chao Guo and Zhonghai Lu. 2023. Damage-based Lifetime Modeling for Power Electronic Devices. In PHM Society Asia-Pacific Conference number 1. Vol. 4. Hilbert Hagedoorn. 2025. Rubin Ultra GPU to Use Advanced Microchannel Cooling for its 2300W TDP. https: //www.guru3d.com/story/rubin-ultra-gpu-to-use-advanced-microchannel-cooling-for-its-2300w-tdp/. (Oct. 2025). Gabriel Hautreux and Etienne Malaboeuf. 2023. Reducing HPC energy footprint for large scale GPU accelerated workloads. In Proceedings of the SC ’23 Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis (SC-W ’23). Association for Computing Machinery, Denver, CO, USA, 1860–1865. isbn: 9798400707858. doi:10.1145/3624062.3624268. Felix Hoffmann, Stefan Schmitt, and Nando Kaminski. 2023. Lifetime Modeling of SiC MOSFET Power Modules During Power Cycling Tests at Low Temperature Swings. In 35th International Symposium on Power Semiconductor Devices and ICs (ISPSD), 294–297. doi:10.1109/ISPSD57135.2023.10147533. Ke Hong, Xiuhong Li, Lufang Chen, Qiuli Mao, Guohao Dai, Xuefei Ning, Shengen Yan, Yun Liang, and Yu Wang. 2025. SOLA: Optimizing SLO Attainment for Large Language Model Serving with State-Aware Scheduling. In Proceedings of the Eighth Conference on Machine Learning and Systems (MLSys). https://proceedings.mlsys.org/ paper_files/paper/2025/hash/bc82dbfbfa43232be85b8d9838f49c3e-Abstract-Conference.html. R. D. Hornung, Jeff A. Keasler, and M. B. Gokhale. 2011. Hydrodynamics Challenge Problem, Lawrence Livermore National Laboratory. Tech. rep. LLNL-TR-490254. Lawrence Livermore National Laboratory, Livermore, CA, 1–28.

PACER: GPU Excursion Management for ML Workloads

[35]

[36]

[37]

[38]

[39]

[40]

[41]

[42]

[43]

[44]

[45]

[46]

[47]

[48] [49]

[50] [51]

23

Farzad Hosseinabadi, Sajib Chakraborty, Sachin Kumar Bhoi, Guenter Prochart, Dino Hrvanovic, and Omar Hegazy. 2024. A comprehensive overview of reliability assessment strategies and testing of power electronics converters. IEEE Open Journal of Power Electronics, 5, 473–512. doi:10.1109/OJPEL.2024.3379294. Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. 2019. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. In Proceedings of the 33rd International Conference on Neural Information Processing Systems (NeurIPS). Ranggi Hwang, Jianyu Wei, Shijie Cao, Changho Hwang, Xiaohu Tang, Ting Cao, and Mao Yang. 2025. Pre-Gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference. In Proceedings of the 51st Annual International Symposium on Computer Architecture (ISCA). IEEE Press, Buenos Aires, Argentina, 1018–1031. isbn: 9798350326581. doi:10.1109/ISCA59077.2024.00078. Yuichi Inadomi, Tapasya Patki, Koji Inoue, Mutsumi Aoyagi, Barry Rountree, Martin Schulz, David Lowenthal, Yasutaka Wada, Keiichiro Fukazawa, Masatsugu Ueda, Masaaki Kondo, and Ikuo Miyoshi. 2015. Analyzing and Mitigating the Impact of Manufacturing Variability in Power-Constrained Supercomputing. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC) Article 78, 12 pages. isbn: 9781450337236. doi:10.1145/2807591.2807638. Intel Corporation. 2003. Voltage regulator module (vrm) and enterprise voltage regulator-down (evrd) 10.0 design guidelines. https://www.intel.com/content/dam/www/public/us/en/documents/design-guides/voltage-regulatormodule-enterprise-voltage-regulator-down-10-0-guidelines.pdf. Intel Design Guide. (2003). Rutwik Jain, Yiwei Jiang, Matthew D. Sinclair, and Shivaram Venkataraman. 2026. Minos: Systematically Classifying Performance and Power Characteristics of GPU Workloads on HPC Clusters. (2026). https://arxiv.org/abs/2604.03591 arXiv: 2604.03591 [cs.DC]. Rutwik Jain, Brandon Tran, Keting Chen, Matthew D. Sinclair, and Shivaram Venkataraman. 2024. PAL: A VariabilityAware Policy for Scheduling ML Workloads in GPU Clusters. In Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis (SC). (Nov. 2024). N. E. Jerger, A. Kannan, Z. Li, and G. H. Loh. 2014. NoC Architectures for Silicon Interposer Systems: Why Pay for more Wires when you Can Get them (from your interposer) for Free? In 47th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). (Dec. 2014), 458–470. doi:10.1109/MICRO.2014.61. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2024. Mixtral of Experts. (2024). https://arxiv.org/abs/2401.04088 arXiv: 2401.04088 [cs.LG]. Zhixing Jiang, Justin Garrigus, Allison Seigler, Ethan Syed, Yanlun Huang, Mehdi Sadi, Tawfik Rahal-Arabi, and Lizy Kurian John. 2026. Exploration of LLM Workload Reliability based on di/dt effects and Voltage Droops. In IEEE International Symposium on High-Performance Comp Architecture (HPCA). Andreas Kosmas Kakolyris, Dimosthenis Masouros, Sotirios Xydis, and Dimitrios Soudris. 2024. SLO-Aware GPU DVFS for Energy-Efficient LLM Inference Serving. IEEE Computer Architecture Letters, 23, 2, 150–153. doi:10.1109/ LCA.2024.3406038. Aditya K. Kamath, Ramya Prabhu, Jayashree Mohan, Simon Peter, Ramachandran Ramjee, and Ashish Panwar. 2025. POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS). Association for Computing Machinery, Rotterdam, Netherlands, 897–912. isbn: 9798400710797. doi:10. 1145/3676641.3715996. Ajaykumar Kannan, Natalie Enright Jerger, and Gabriel H Loh. 2015. Enabling Interposer-based Disintegration of Multi-core Processors. In 48th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). IEEE, 546–558. Ian Karlin. 2012. LULESH Programming Model and Performance Ports Overview. Tech. rep. LLNL-TR-608824. Lawrence Livermore National Laboratory, Livermore, CA, (Dec. 2012), 1–17. Ian Karlin, Abhinav Bhatele, Jeff Keasler, Bradford L. Chamberlain, Jonathan Cohen, Zachary DeVito, Riyaz Haque, Dan Laney, Edward Luke, Felix Wang, David Richards, Martin Schulz, and Charles Still. 2013. Exploring traditional and emerging parallel programming models using a proxy application. In 27th IEEE International Parallel & Distributed Processing Symposium (IPDPS). Boston, USA, (May 2013). Ian Karlin, Jeff Keasler, and Rob Neely. 2013. LULESH 2.0 Updates and Changes. Tech. rep. LLNL-TR-641973. Lawrence Livermore National Laboratory, Livermore, CA, (Aug. 2013), 1–9. Donggyu Kim, Jerry Zhao, Jonathan Bachrach, and Krste Asanović. 2019. Simmani: runtime power modeling for arbitrary rtl with automatic signal selection. In Proceedings of the 52nd Annual IEEE/ACM International Symposium

24

[52]

[53]

[54] [55]

[56] [57]

[58]

[59]

[60]

[61]

[62]

[63]

[64] [65]

[66]

[67]

[68]

Jain et al. on Microarchitecture (MICRO-52). Association for Computing Machinery, Columbus, OH, USA, 1050–1062. isbn: 9781450369381. doi:10.1145/3352460.3358322. Ivana F Kovacevic-Badstuebner, Johann W Kolar, Uwe Schilling, H Chung, H Wang, F Blaabjerg, and M Pecht. 2015. Modelling for the Lifetime Prediction of Power Semiconductor Modules. Reliability of Power Electronic Converter Systems, 103–140. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles (SOSP). Lawrence Livermore National Labs. 2020. CORAL-2 Benchmarks. https://asc.llnl.gov/coral-2-benchmarks. (2020). Jingwen Leng, Alper Buyuktosunoglu, Ramon Bertran, Pradip Bose, Yazhou Zu, and Vijay Janapa Reddi. 2021. Predictive Guardbanding: Program-Driven Timing Margin Reduction for GPUs. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 40, 1, 171–184. doi:10.1109/TCAD.2020.2992684. Ao Li, Bojian Zheng, Gennady Pekhimenko, and Fan Long. 2020. Automatic Horizontal Fusion for GPU Kernels. (2020). arXiv: 2007.01277 [cs.DC]. Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 663–679. https://www.usenix.org/conference/osdi23/presentation/li-zhuohan. Qunyou Liu, Darong Huang, Marina Zapater, and David Atienza. 2025. GreenLLM: SLO-Aware Dynamic Frequency Scaling for Energy-Efficient LLM Serving. arXiv preprint arXiv:2508.16449. https://arxiv.org/abs/2508.16449 arXiv: 2508.16449 [cs.PF]. Gabriel H. Loh, Natalie Enright Jerger, Ajaykumar Kannan, and Yasuko Eckert. 2015. Interconnect-Memory Challenges for Multi-chip, Silicon Interposer Systems. In Proceedings of the 2015 International Symposium on Memory Systems (MEMSYS ’15). ACM, Washington DC, DC, USA, 3–10. isbn: 978-1-4503-3604-8. doi:10.1145/2818950.2818951. Lingxiao Ma, Zhiqiang Xie, Zhi Yang, Jilong Xue, Youshan Miao, Wei Cui, Wenxiang Hu, Fan Yang, Lintao Zhang, and Lidong Zhou. 2020. Rammer: Enabling Holistic Deep Learning Compiler Optimizations with rTasks. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI). USENIX Association, (Nov. 2020), 881–897. isbn: 978-1-939133-19-9. https://www.usenix.org/conference/osdi20/presentation/ma. Alexandre Marie, Bernardo Cougo, Loic Renaudie, Trésor Kaounodji Koladoum, and Jean-Pierre Fradin. 2024. Precise 3D Modelling of SiC Die Temperature Oscillation for Lifetime Prediction of Power Modules used in DC/AC Power Converters. In 30th International Workshop on Thermal Investigations of ICs and Systems (THERMINIC), 1–6. doi:10.1109/THERMINIC62015.2024.10732697. Peter Mattson, Christine Cheng, Cody Coleman, Greg Diamos, Paulius Micikevicius, David A. Patterson, Hanlin Tang, Gu-Yeon Wei, Peter Bailis, Victor Bittorf, David Brooks, Dehao Chen, Debojyoti Dutta, Udit Gupta, Kim M. Hazelwood, Andrew Hock, Xinyuan Huang, Bill Jia, Daniel Kang, David Kanter, Naveen Kumar, Jeffery Liao, Guokai Ma, Deepak Narayanan, Tayo Oguntebi, Gennady Pekhimenko, Lillian Pentecost, Vijay Janapa Reddi, Taylor Robie, Tom St. John, Carole-Jean Wu, Lingjie Xu, Cliff Young, and Matei Zaharia. 2019. MLPerf Training Benchmark. CoRR, abs/1910.01500, 14 pages. http://arxiv.org/abs/1910.01500 arXiv: 1910.01500. Xinxin Mei, Ling Sing Yung, Kaiyong Zhao, and Xiaowen Chu. 2013. A Measurement Study of GPU DVFS on Energy Conservation. In Proceedings of the Workshop on Power-Aware Computing and Systems (HotPower) Article 10. Association for Computing Machinery, Farmington, Pennsylvania, 5 pages. isbn: 9781450324588. doi:10.1145/ 2525526.2525852. Ashish Mishra and Nilay Khare. 2015. Analysis of DVFS Techniques for Improving the GPU Energy Efficiency. http://file.scirp.org/pdf/ojee_2015121415504865.pdf. (2015). Nathaniel Morris, Siva Meenakshi Renganathan, Christopher Stewart, Robert Birke, and Lydia Chen. 2016. Sprint Ability: How Well Does Your Software Exploit Bursts in Processing Capacity? In IEEE International Conference on Autonomic Computing (ICAC), 173–178. doi:10.1109/ICAC.2016.61. Nathaniel Morris, Christopher Stewart, Robert Birke, Lydia Chen, and Jaimie Kelley. 2017. Early Work on Modeling Computational Sprinting. In Proceedings of the 2017 Symposium on Cloud Computing (SoCC ’17). Association for Computing Machinery, Santa Clara, California, 661. isbn: 9781450350280. doi:10.1145/3127479.3132691. Rajib Nath and Dean Tullsen. 2015. The CRISP Performance Model for Dynamic Voltage and Frequency Scaling in a GPGPU. In Proceedings of the 48th International Symposium on Microarchitecture (MICRO). Association for Computing Machinery, Waikiki, Hawaii, 281–293. isbn: 9781450340342. doi:10.1145/2830772.2830826. Cedric Nugteren, Gert-Jan van den Braak, and Henk Corporaal. 2014. Roofline-Aware DVFS for GPUs. In Proceedings of International Workshop on Adaptive Self-Tuning Computing Systems (ADAPT). Association for Computing Machinery, Vienna, Austria, 8–10. isbn: 9781450325141. doi:10.1145/2553062.2553067.

PACER: GPU Excursion Management for ML Workloads

[69] [70] [71]

[72]

[73]

[74]

[75]

[76]

[77]

[78]

[79]

[80]

[81]

[82]

[83] [84] [85] [86] [87]

25

NVIDIA. 2024. NVIDIA Blackwell Architecture Technical Brief. https://resources.nvidia.com/en-us-blackwellarchitecture/blackwell-architecture-technical-brief. (2024). NVIDIA. 2025. NVIDIA cuDNN: GPU Accelerated Deep Learning. https://developer.nvidia.com/cudnn. (2025). OCP Open Accelerator Infrastructure (OAI) Workstreams. 2022. OCP Universal Baseboard (UBB) Design Specification v1.5. https://www.opencompute.org/documents/universal-baseboard-design-specification-v1p5-final-20220223docx-pdf. (2022). Hiroshi Onodera, Nobuyuki Shishido, Daisuke Asari, Hiroshi Isono, and Wataru Saito. 2024. Effect of Solder Junction Void Variation in Power Semiconductor Package on Power Cycle Lifetime. Microelectronics Reliability, 161, 115471. doi:https://doi.org/10.1016/j.microrel.2024.115471. Muhammad Osama, Duane Merrill, Cris Cecka, Michael Garland, and John D. Owens. 2023. Stream-K: Work-centric Parallel Decomposition for Dense Matrix-Matrix Multiplication on the GPU. (2023). https://arxiv.org/abs/2301.03598 arXiv: 2301.03598 [cs.DS]. Sangyoung Park, Jaehyun Park, Donghwa Shin, Yanzhi Wang, Qing Xie, Massoud Pedram, and Naehyuck Chang. 2013. Accurate Modeling of the Delay and Energy Overhead of Dynamic Voltage and Frequency Scaling in Modern Microprocessors. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 32, 5, 695–708. doi:10.1109/TCAD.2012.2235126. Pratyush Patel, Esha Choukse, Chaojie Zhang, Inigo Goiri, Brijesh Warrier, Nithish Mahalingam, and Ricardo Bianchini. 2024. Characterizing Power Management Opportunities for LLMs in the Cloud. In ASPLOS. (Apr. 2024). https://doi.org/10.1145/3620666.3651329. Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2025. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. In Proceedings of the 51st Annual International Symposium on Computer Architecture (ISCA). IEEE Press, Buenos Aires, Argentina, 118–132. isbn: 9798350326581. doi:10.1109/ISCA59077.2024.00019. Pratyush Patel, Zibo Gong, Syeda Rizvi, Esha Choukse, Pulkit Misra, Thomas Anderson, and Akshitha Sriraman. 2023. Towards Improved Power Management in Cloud GPUs. IEEE Computer Architecture Letters, 22, 2, 141–144. doi:10.1109/LCA.2023.3278652. Suchita Pati, Shaizeen Aga, Nuwan Jayasena, and Matthew Sinclair. 2025. GOLDYLOC: Global Optimizations & Lightweight Dynamic Logic for Concurrency. ACM Trans. Archit. Code Optim., 22, 2, Article 78, (July 2025), 28 pages. doi:10.1145/3730584. Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. 2023. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. (2023). https: //arxiv.org/abs/2307.01952 arXiv: 2307.01952 [cs.CV]. Arun Raghavan, Yixin Luo, Anuj Chandawalla, Marios Papaefthymiou, Kevin P. Pipe, Thomas F. Wenisch, and Milo M. K. Martin. 2012. Computational Sprinting. In IEEE International Symposium on High-Performance Computer Architecture, 1–12. doi:10.1109/HPCA.2012.6169031. Tawfik Rahal-Arabi, Paul Van der Arend, Ashish Jain, Mehdi Saidi, Rashad Oreifej, Sriram Sundaram, Srilatha Manne, Indrani Paul, Rajit Seahra, Frank Helms, Esha Choukse, Nithish Mahalingam, Brijesh Warrier, and Ricardo Bianchini. 2024. Optimizing GPU Data Center Power. In IEEE Asia Pacific Conference on Circuits and Systems (APCCAS), 358–362. doi:10.1109/APCCAS62602.2024.10808935. Vijay Janapa Reddi, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, Carole-Jean Wu, Brian Anderson, Maximilien Breughe, Mark Charlebois, William Chou, Ramesh Chukka, Cody Coleman, Sam Davis, Pan Deng, Greg Diamos, Jared Duke, Dave Fick, J. Scott Gardner, Itay Hubara, Sachin Idgunji, Thomas B. Jablin, Jeff Jiao, Tom St. John, Pankaj Kanwar, David Lee, Jeffery Liao, Anton Lokhmotov, Francisco Massa, Peng Meng, Paulius Micikevicius, Colin Osborne, Gennady Pekhimenko, Arun Tejusve Raghunath Rajan, Dilip Sequeira, Ashish Sirasao, Fei Sun, Hanlin Tang, Michael Thomson, Frank Wei, Ephrem Wu, Lingjie Xu, Koichi Yamada, Bing Yu, George Yuan, Aaron Zhong, Peizhao Zhang, and Yuchen Zhou. 2020. MLPerf Inference Benchmark. In 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), 446–459. doi:10.1109/ISCA45697.2020.00045. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2021. High-resolution image synthesis with latent diffusion models. (2021). eprint: 2112.10752 (cs.CV). Skyler J Saleh, Samuel Naffziger, Milind S Bhagavat, and Rahul Agarwal. 2020. GPU Chiplets Using High Bandwidth Crosslinks. US Patent App. 16/456,287. (Dec. 2020). Vito Savino. 2025. Rethinking Power for AI Processors. In Power Systems Design. (Apr. 2025). https : / / www. powersystemsdesign.com/articles/rethinking-power-for-ai-processors/159/22717. Christian Schwabe, Niklas Seltner, and Thomas Basler. 2024. Reliability of SiC MOSFETs in the High Cycle Fatigue Regime under Fast Power Pulses. Solid State Phenomena, 361, 7–12. Thomas Scogland, Jonathan Azose, David Rohr, Suzanne Rivoire, Natalie Bates, and Daniel Hackenberg. 2015. Node Variability in Large-Scale Power Measurements: Perspectives from the Green500, Top500 and EEHPCWG. In

26

[88]

[89]

[90]

[91]

[92]

[93]

[94]

[95]

[96]

[97]

[98]

[99]

[100] [101]

[102]

[103] [104]

Jain et al. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’15) Article 74. Association for Computing Machinery, Austin, Texas, 11 pages. isbn: 9781450337236. doi:10.1145/ 2807591.2807653. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2020. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. (2020). https://arxiv. org/abs/1909.08053 arXiv: 1909.08053 [cs.CL]. Prasoon Sinha, Akhil Guliani, Rutwik Jain, Brandon Tran, Matthew D Sinclair, and Shivaram Venkataraman. 2022. Not All GPUs Are Created Equal: Characterizing Variability in Large-Scale, Accelerator-Rich Systems. In Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis (SC), 1–15. Alan Smith, Gabriel H. Loh, Michael J. Schulte, Mike Ignatowski, Samuel Naffziger, Mike Mantor, Nathan Kalyanasundharam, Vamsi Alla, Nicholas Malaya, Joseph L. Greathouse, Eric Chapman, and Raja Swaminathan. 2024. Realizing the AMD Exascale Heterogeneous Processor Vision : Industry Product. In 51st ACM/IEEE Annual International Symposium on Computer Architecture (ISCA). IEEE, 876–889. doi:10.1109/ISCA59077.2024.00068. Benjamin Spector, Jordan Juravsky, Stuart Sul, Dylan Lim, Owen Dugan, Simran Arora, and Chris Ré. 2025. We Bought the Whole GPU, So We’re Damn Well Going to Use the Whole GPU. https://hazyresearch.stanford.edu/ blog/2025-09-28-tp-llama-main. Accessed: 2025-11-16. (2025). Benjamin Spector, Jordan Juravsky, Stuart Sul, Dylan Lim, Dan Fu, Simran Arora, and Chris Ré. 2025. Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B. https://hazyresearch.stanford.edu/blog/2025-05-27no-bubbles. Accessed: 2025-11-16. (2025). Vasileios Spiliopoulos, Stefanos Kaxiras, and Georgios Keramidas. 2011. Green governors: A framework for Continuously Adaptive DVFS. In 2011 International Green Computing Conference and Workshops, 1–8. doi:10.1109/IGCC. 2011.6008552. Jovan Stojkovic, Chaojie Zhang, Inigo Goiri, Josep Torrellas, and Esha Choukse. 2025. DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency. In IEEE International Symposium on High Performance Computer Architecture (HPCA), 1348–1362. doi:10.1109/HPCA61900.2025.00102. Foteini Strati, Xianzhe Ma, and Ana Klimovic. 2024. Orion: Interference-aware, Fine-grained GPU Sharing for ML Applications. In Proceedings of the Nineteenth European Conference on Computer Systems (EuroSys ’24). Association for Computing Machinery, Athens, Greece, 1075–1092. isbn: 9798400704376. doi:10.1145/3627703.3629578. Bo Su, Junli Gu, Li Shen, Wei Huang, Joseph L. Greathouse, and Zhiying Wang. 2014. PPEP: Online Performance, Power, and Energy Prediction Framework and DVFS Space Exploration. In 2014 47th Annual IEEE/ACM International Symposium on Microarchitecture, 445–457. doi:10.1109/MICRO.2014.17. Zhenheng Tang, Yuxin Wang, Qiang Wang, and Xiaowen Chu. 2019. The Impact of GPU DVFS on the Energy and Performance of Deep Learning: An Empirical Study. In Proceedings of the Tenth ACM International Conference on Future Energy Systems (e-Energy). Association for Computing Machinery, Phoenix, AZ, USA, 315–325. isbn: 9781450366717. doi:10.1145/3307772.3328315. Burak Topcu, Deniz Karabacak, and Isil Oz. 2025. Demystifying Power and Performance Variations in GPU Systems through Microarchitectural Analysis. Computer Science and Information Systems, 22, 2, 533–561. doi:https://doi.org/ 10.2298/CSIS240722021T. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models. arXiv preprint arXiv:2302.13971. https://arxiv.org/abs/2302.13971 arXiv: 2302.13971 [cs.CL]. Daniel Velicka, Ondrej Vysocky, and Lubomir Riha. 2025. Methodology for gpu frequency switching latency measurement. (2025). https://arxiv.org/abs/2502.20075 arXiv: 2502.20075 [cs.DC]. Igor Wallossek. 2025. MSI GeForce RTX 5090 SUPRIM SOC review – When the gram costs one euro and puts the FE in the shade. https://www.igorslab.de/en/msi-geforce-rtx-5090-suprim-soc-in-the-test-when-the-gram-costs-oneeuro-and-puts-the-fe-in-the-shade/6/. (Jan. 2025). Guibin Wang, YiSong Lin, and Wei Yi. 2010. Kernel Fusion: An Effective Method for Better Power Efficiency on Multithreaded GPU. In Proceedings of the 2010 IEEE/ACM Int’l Conference on Green Computing and Communications & Int’l Conference on Cyber, Physical and Social Computing (GREENCOM-CPSCOM ’10). IEEE Computer Society, USA, 344–350. isbn: 9780769543314. doi:10.1109/GreenCom-CPSCom.2010.102. Huai Wang and Frede Blaabjerg. 2021. Power electronics reliability: state of the art and outlook. IEEE Journal of Emerging and Selected Topics in Power Electronics, 9, 6, 6476–6493. doi:10.1109/JESTPE.2020.3037161. Yangzihao Wang, Yuechao Pan, Andrew Davidson, Yuduo Wu, Carl Yang, Leyuan Wang, Muhammad Osama, Chenshan Yuan, Weitang Liu, Andy T. Riffel, and John D. Owens. 2017. Gunrock: GPU Graph Analytics. ACM Trans. Parallel Comput., 4, 1, Article 3, (Aug. 2017), 49 pages. doi:10.1145/3108140.

PACER: GPU Excursion Management for ML Workloads

[105]

[106]

[107]

[108]

[109]

[110]

[111]

[112]

[113]

[114]

[115]

[116]

[117]

27

Wang, Pengyu and Li, Chao and Wang, Jing and Wang, Taolei and Zhang, Lu and Leng, Jingwen and Chen, Quan and Guo, Minyi. 2024. Skywalker: Efficient Alias-method-based Graph Sampling and Random Walk on GPUs. In Proceedings of the 30th International Conference on Parallel Architectures and Compilation Techniques (PACT ’21). IEEE Press, Atlanta, GA, USA, 304–317. isbn: 9781665442787. doi:10.1109/PACT52795.2021.00029. Kathryn Wilcox, Robert Cole, Harry R. Fair III, Kevin Gillespie, Aaron Grenat, Carson Henrion, Ravi Jotwani, Stephen Kosonocky, Benjamin Munger, Samuel Naffziger, Robert S. Orefice, Sanjay Pant, Donald A. Priore, Ravinder Rachala, and Jonathan White. 2015. Steamroller Module and Adaptive Clocking System in 28 nm CMOS. IEEE J. Solid State Circuits, 50, 1, 24–34. doi:10.1109/JSSC.2014.2357428. Zhiyao Xie, Xiaoqing Xu, Matt Walker, Joshua Knebel, Kumaraguru Palaniswamy, Nicolas Hebert, Jiang Hu, Huanrui Yang, Yiran Chen, and Shidhartha Das. 2021. APOLLO: An Automated Power Modeling Framework for Runtime Power Introspection in High-Volume Commercial Microprocessors. In Proceedings of 54th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). (Oct. 2021). https://doi.org/10.1145/3466752.3480064. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. 2025. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388. Zeyu Yang, Karel Adamek, and Wesley Armour. 2024. Accurate and Convenient Energy Measurements for GPUs: A Detailed Study of NVIDIA GPU’s Built-In Power Sensor. In International Conference for High Performance Computing, Networking, Storage and Analysis (SC). IEEE Computer Society, Los Alamitos, CA, USA, (Nov. 2024), 307–323. doi:10.1109/SC41406.2024.00028. Xin You, Zhibo Xuan, Hailong Yang, Zhongzhi Luan, Yi Liu, and Depei Qian. 2024. GVARP: Detecting Performance Variance on Large-Scale Heterogeneous System. In Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis (SC). Yushi Bai and Shangqing Tu and Jiajie Zhang and Hao Peng and Xiaozhi Wang and Xin Lv and Shulin Cao and Jiazheng Xu and Lei Hou and Yuxiao Dong and Jie Tang and Juanzi Li. 2025. LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks. (2025). https://arxiv.org/abs/2412.15204 arXiv: 2412.15204 [cs.CL]. Boyi Zhang and Shuo Wang. 2020. A Survey of EMI Research in Power Electronics Systems With Wide-Bandgap Semiconductor Devices. IEEE Journal of Emerging and Selected Topics in Power Electronics, 8, 1, 626–643. doi:10.1109/ JESTPE.2019.2953730. Yi Zhang, Patrick Heimler, James Opondo Abuogo, Xinyue Zhang, Yichi Zhang, Dong Xie, Yuhao Zhang, Kaichen Zhang, Xiang Li, Haoze Luo, Ralf Schmidt, Frede Blaabjerg, Huai Wang, and Thomas Basler. 2026. Power Cycling Testing for Power Semiconductor Switches: Methods, Standards, Limitations, and Outlooks. IEEE Transactions on Power Electronics, 41, 1, 849–869. doi:10.1109/TPEL.2025.3595180. Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: disaggregating prefill and decoding for goodput-optimized large language model serving. In Proceedings of the 18th USENIX Conference on Operating Systems Design and Implementation (OSDI’24) Article 11. USENIX Association, Santa Clara, CA, USA, 18 pages. isbn: 978-1-939133-40-3. Zheng Zhong, Seyfal Sultanov, Michael Papka, and Zhiling Lan. 2025. Minimizing Power Waste in Heterogenous Computing via Adaptive Uncore Scaling. In Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis (SC). Kan Zhu, Yufei Gao, Yilong Zhao, Liangyu Zhao, Gefei Zuo, Yile Gu, Dedong Xie, Tian Tang, Qinyu Xu, Zihao Ye, Keisuke Kamahori, Chien-Yu Lin, Ziren Wang, Stephanie Wang, Arvind Krishnamurthy, and Baris Kasikci. 2025. NanoFlow: Towards Optimal Large Language Model Serving Throughput. (2025). https://arxiv.org/abs/2408.12757 arXiv: 2408.12757 [cs.DC]. Pengfei Zou, Ang Li, Kevin Barker, and Rong Ge. 2020. Indicator-Directed Dynamic Power Management for Iterative Workloads on GPU-Accelerated Systems. In 2020 20th IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing (CCGRID), 559–568. doi:10.1109/CCGrid49817.2020.00-37.

Related documents

• FliX: Flipped-Indexing for Scalable GPU Queries and Updates#120487
• AEGIS: Runtime-Guided GPU Collocation for Multi-Tenant Deep Learning Training#1217305
• VortexSynth: Profile-guided benchmark reconstruction for GPU architecture research#1296668
• GPUHammer: Rowhammer Attacks on GPU Memories are Practical#1011000
• Tile-Based Decompression in Compiled GPU Engines#1448211