arXiv CSopen access

Modeling and Chasing the Energy-Efficiency Sweet Spots in Modern GPUs

Modeling and Chasing the Energy-Efficiency Sweet Spots in Modern GPUs Ayesha Afzal1[0000−0001−5061−0438] , Markus Manfred Li2[0009−0004−0908−3460] , and Michael Panzlaff1[0009−0009−2993−7001] 1

arXiv:2607.00819v1 [cs.DC] 1 Jul 2026

2

Erlangen National High Performance Computing Center (NHR@FAU) Department of Computer Science, FAU Erlangen-Nürnberg, Germany

Abstract. Energy consumption is a key limitation in high-performance computing on heterogeneous CPU–GPU systems. This work studies how hardware configuration affects energy-to-solution under realistic workloads. We study energy efficiency regimes using molecular dynamics benchmarks (GROMACS and AMBER) and a stress-test benchmark (FIRESTARTER) on systems with A40, A100, H100, and H200 GPUs and Intel Ice Lake CPU, varying frequency scaling and power cap. We show that energy-to-solution exhibits workload- and architecture-dependent transitions between efficient and inefficient regimes, driven by nonlinear GPU power-frequency scaling. We introduce an interpretable analytical model that decomposes GPU power into linear and nonlinear components, identifying a workload- and architecture-dependent transition frequency beyond which efficiency degrades. The model fits empirical data with low error and highlights the role of baseline power, nonlinear power behavior, and transition frequency as the dominant parameters governing energy efficiency. Power capping is generally less effective for efficiency tuning than frequency reduction, especially for workloads that operate far from thermal design power. Overall, energy-efficient HPC execution is a configuration-dependent problem with identifiable regime shifts, and we provide model-driven guidance for selecting operating points.

Keywords: energy efficiency · molecular dynamics · frequency scaling · power cap · power modeling · stress test FIRESTARTER · GROMACS · AMBER

1

Introduction and related work

Energy efficiency has emerged as a primary limiting factor in high-performance computing (HPC), increasingly constraining system scalability under fixed power budgets rather than peak computational capability. This challenge is particularly pronounced in heterogeneous CPU–GPU systems, where multiple hardware control parameters jointly determine performance and energy to solution. A central difficulty arises from the high-dimensional configuration space exposed by modern HPC systems, including GPU dynamic voltage and frequency scaling (DVFS) [12], power caps [10], and empirical power modeling [11,15,14,13]. These strategies exhibit nonlinear GPU power scaling due to voltage–frequency coupling and workload dependence, yet most studies rely on synthetic kernels, limit-

2

A. Afzal et al.

ing their applicability to real-world applications. Few existing approaches implicitly assume smooth and monotonic relationships between performance, power, and energy efficiency. However, such assumptions frequently break down for scientific workloads with heterogeneous compute and memory behavior. Empirical or machine-learning-based energy models exist but often lack interpretability and hardware-level insight, creating a gap between predictive accuracy and physical understanding for realistic HPC workloads. In this work, we focus on molecular dynamics (MD) applications, represented by GROMACS [1,6] and AMBER [7], complemented by the synthetic stress-test benchmark FIRESTARTER [8]. These workloads are executed on multiple GPU architectures (A40, A100, H100, H200) and an Intel Ice Lake CPU platform. Collectively, they span compute-bound, memory-bound, and thermally constrained regimes, enabling a unified analysis of energy behavior across diverse operating conditions. Contributions This paper makes the following contributions: – We systematically evaluate energy-to-solution using molecular dynamics workloads (GROMACS, AMBER) and a synthetic stress-test (FIRESTARTER), covering GPU frequency scaling and power cap configurations. – We introduce an analytic GPU power–frequency model fitted to real application data with low error across GPU architectures (A40, A100, H100, H200), decomposing power into baseline, linear, and nonlinear terms; it identifies workload- and architecture-dependent transition frequencies beyond which efficiency collapses due to quadratic power growth, while CPUs (Intel Ice Lake) exhibit near-linear scaling. – We integrate empirical analysis with analytic modeling to derive predictive operating regimes for energy-efficient HPC, showing that energy increases nonlinearly beyond workload-dependent transition points despite diminishing performance gains. – We show that frequency scaling consistently outperforms power capping for energy optimization, particularly for compute-light workloads, and show that optimal operating points require workload-aware hardware configuration. Our results establish energy-efficient HPC execution as a configuration-dependent, regime-based problem, providing a principled foundation and model-driven guidance for selecting operating points for MD and stress-test workloads. Overview The remainder of this paper is organized as follows: In Sec. 2, we describe the benchmark workloads, hardware testbed, software configuration, and experimental methodology used in our study. Sec. 3 presents the analytical GPU and CPU power-frequency model. Sec. 4 presents the evaluation of energyto-solution across architectures, comparing frequency tuning, and power cap strategies. Sec. 5 summarizes the main findings and outlines future work.

2

Workloads, testbed and experimental setup

This section describes the experimental platforms, workloads, measurement methodology, and parameter exploration used to analyze energy behavior in heterogeneous CPU–GPU systems. Workloads We evaluate two workload classes: (1) molecular dynamics (MD) application workloads GROMACS and AMBER and (2) the synthetic

Modeling and Chasing the Energy-Efficiency Sweet Spots in Modern GPUs

3

Table 1. Selected MD benchmarks across a range of biomolecular systems and sizes. GROMACS GPU Benchmark Atoms 2md_start0 20,248 rnanvt 31,889 PI_large_test 80,289 FL_md1_berendsen 170,320 eag1 615,924 stmv_pme_nvt 1,066,628 ⋆

AMBER GPU Benchmark Atoms TRPCage 304 Myoglobin 2,492 JAC⋆ 23,558 Nucleosome 25,095 Factor IX⋆ 90,906 Cellulose⋆ 408,609 STMV⋆ 1,067,095

AMBER CPU Benchmark Atoms Thioredoxin 14,093 JAC 23,558 JAC (pmemd) 23,558 DHFR 23,930 Factor IX 90,906

GPU runs use standard NPT or NVE ensembles following typical MD workflows.

stress-test benchmark FIRESTARTER. The MD benchmark suite spans workloads from small, GPU-underutilized benchmarks to large, throughput- and memory-bound simulations, differing in parallelization strategies and compute intensities. This enables analysis of heterogeneous performance and energy behavior across architectures. Detailed configurations and atom counts are summarized in Table 1. We evaluate six publicly available GROMACS3 benchmarks including solvent boxes, membrane proteins, and viral benchmarks. Larger benchmarks are increasingly sensitive to memory bandwidth. Inputs are available in the Artifact Description10 , and simulations follow benchmark settings. We evaluate eleven GPU and five CPU standard AMBER4 benchmarks spanning small peptides to large biomolecular assemblies. This span enables the comprehensive CPU–GPU energy analysis from latency-dominated workloads (e.g., smaller TRPCage and Myoglobin benchmarks, where kernel launch overhead and per-step latency dominate) to throughput- and memory-bound regimes (e.g., larger Cellulose and STMV benchmarks, where long-range electrostatics and memory bandwidth become limiting). The CPU benchmarks represent standard MPI/OpenMP workloads that are predominantly compute-bound on Ice Lake CPU architectures, and primarily probing parallel efficiency and cache/memory hierarchy behavior. To study worst-case sustained power behavior, we additionally use FIRESTARTER5 , a synthetic stress benchmark based on architecturespecific low-level kernels designed to maximize utilization of compute units, caches, and memory subsystems. Unlike application workloads, it targets nearTDP operation and is commonly used for infrastructure validation and energyefficiency studies. HPC platforms and architectures Experiments were conducted on heterogeneous HPC systems with Intel Ice Lake CPUs and NVIDIA GPUs (A40, A100, H100, and H200); key specifications are summarized in Table 2. Three clusters were at our disposal: 1. Helma6 comprising eight NVIDIA H100 or H200 GPUs per node 2. Alex7 comprising eight NVIDIA A40 or A100 GPUs per node 3

https://manual.gromacs.org/documentation/current https://ambermd.org/GPUPerformance.php 5 https://tu-dresden.de/zih/firestarter 6 https://doc.nhr.fau.de/clusters/helma 7 https://doc.nhr.fau.de/clusters/alex 4

4

A. Afzal et al. Table 2. Key specifications of NVIDIA GPUs used in the experiments. GPUs

A40 A100 H100 H200

MEM clock std. [GHz] 7.251 1.215 1.593 1.41

Graphics clock min–max [GHz] 0.21 / 1.74 0.21 / 1.41 0.345 / 1.98 0.40 / 1.98

MEM bandwidth [GB/s] 798 1,555 3,352 4,800

Power cap min–max [W] 100–300 100–400 350–700 400–700

TDP

[W] 300 400 700 † 700 †

MEM type

MEM size

GDDR6 HBM2 HBM3 HBM3e

[GB] 48 40 80 141

†

Thermal Design Power (TDP) denotes the typical full-load power draw without enforced power limits. Default power cap on the Helma6 configuration is 500 W.

3. Fritz8 comprising dual-socket Intel Xeon Platinum 8360Y (Ice Lake) nodes with 72 cores per node, base frequency 2.4 GHz (up to 3.5 GHz turbo), 256 GB DDR4-3200 memory, AVX-512 vector units, and SMT disabled. The systems span multiple architectural generations and differ in compute throughput, memory bandwidth, thermal characteristics, and powermanagement behavior, allowing us to assess whether energy-efficiency trends generalize across platforms. Even for GPU-accelerated MD codes, CPU-side configuration and CPU–GPU interaction significantly influence performance and energy efficiency (more so for GROMACS than AMBER) [3]. Metrics We analyze four metrics: (1) Performance in ns/day9 (physical time simulated per wall-clock day) for MD workloads and FLOPS (floating-Point Operations per Second) for FIRESTARTER, (2) average power consumption in W, (3) total Energy-to-solution in J, and (4) Energy-delay product (EDP) in J · s for a fixed workload. We use EDP, though other formulations, such as ED2 P, are also valid. Performance is defined as work ω normalized by delay D (runtime), and energy efficiency as useful work per energy E. Configurations are analyzed using Pareto-optimal performance–energy trade-offs (Fig. 1). We distinguish three operating regimes: (1) Energy efficiency-improving (green): higher performance increases efficiency; (2) Balanced (blue): the Pareto-optimal knee region; and (3) Energy-inefficient (red): further performance gains incur disproportionate energy use. Optimal operating points are identified using (1) maximum energy efficiency η (minimum Energy-to-solution; red point), and (2) minimum EDP (energy–performance trade-offs; blue point. It enables systematic identification of workload-dependent optimal operating points and transition boundaries across architectures. Configuration parameters sweeps We systematically vary CPU and GPU control parameters to study their impact on energy-to-solution. (1) Frequency scaling: GPU graphics and CPU core frequencies are varied across their supported ranges using nvidia-smi –-lock-gpu-clocks and –-cpu-freq:performance, respectively. GPU memory clocks remain at their default operational values and not reduced to idle-state (P8) levels, as the latter would result in impractically low memory bandwidth. Clocks are fixed during execution and no explicit power cap is applied. (2) Power cap: GPU power 8 9

https://doc.nhr.fau.de/clusters/fritz Performance is normalized by atom count for cross-benchmark comparability.

Modeling and Chasing the Energy-Efficiency Sweet Spots in Modern GPUs

5

max η min EDP ⇔ max(η · P ) for constant ω

0 > d dP η

∆P

r

∆P

tie on

0

fr

<

to

∆η

re Pa

∆η

dη P d

ω Energy Efficiency (η = E )

iso-EDP: ηP = const

High P

ω ) Performance (P = D

Fig. 1. Performance–energy efficiency trade-off, illustrating iso-EDP contours and the Pareto frontier (lying between the red and grey points). The minimum-EDP (blue point) corresponds to the point of tangency between an iso-EDP contour and the Pareto frontier.

limits are controlled using nvidia-smi –-power-limit, across the supported range while retaining default clock frequencies. In contrast to frequency scaling, power caps impose an upper power bound to help with thermal constraints or datacenter power budgeting and trigger throttling only when limits are reached. Values below minimum cap are invalid. Software configuration All experiments used reproducible HPC software stacks summarized in Table 3. GROMACS 2024.4 was executed in a hybrid MPI+OpenMP configuration (one MPI rank per GPU, 16 OpenMP threads) with full GPU offloading for all major MD kernels using binary .tpr inputs (preprocessed with grompp). AMBER24 GPU runs used pmemd.cuda, while AMBER20 CPU baselines used MPI-enabled pmemd and sander. FIRESTARTER executed a standalone CUDA stress kernel with fixed computational intensity to generate deterministic GPU load under controlled conditions. Experimental methodology All codes were compiled with architecturespecific optimization flags targeting the native capabilities of the respective GPUs. Experiments were executed sequentially across frequency settings and benchmarks on reserved SLURM nodes with dedicated GPU allocation per job (CUDA_VISIBLE_DEVICES=0) to avoid resource contention and scheduling interference. GPU power and utilization were sampled via nvidia-smi at 100 ms intervals, while CPU power was measured using likwid-perfctr -g ENERGY. Thread and core affinity were enforced for all MPI/OpenMP workloads to ensure deterministic CPU scheduling and for reproducibility. Each configuration was repeated three times and evaluated under stable runtime conditions. Power measurements were averaged only during steady-state utilization phases (85% GPU utilization for MD workloads and 100% for FIRESTARTER) to avoid startup and transient effects. Observed variability remained below 5% in most cases, with slightly higher fluctuations in selected FIRESTARTER runs on Helma due to presumably thermal conditions, hardware variability, and architecture-specific DVFS behavior. Our analysis focuses on stable and reproducible behavioral trends governing energy efficiency across workloads and hardware generations, rather than recovering exact optimal frequencies at single-watt precision. Execution timestamps and application runtime logs were recorded for all configurations. GPU-only and CPU-only measurements reflect accelerator and on-chip power consumption and exclude system baseline power [5,4]. During GPU-only execution on FIRESTARTER, CPU contributions were negligible, and GPU

6

A. Afzal et al. Table 3. Software stacks, execution models, and workload configurations. Component Execution model

GROMACS Single-node GPU; 1 MPI rank/GPU + 16 OMP threads

AMBER GPU: single-GPU serial; CPU: single-node MPI (OMP=1)

FIRESTARTER Single-node GPU; no MPI/OpenMP; single CUDA kernel

Workload config

gmx mdrun; .tpr inputs; 200k steps; mixed precision; full GPU offload (PME, bonded, nonbonded, update); 0.2 h runtime limit

GPU: pmemd.cuda, .mdin/.prmtop/.inpcrd inputs; explicit solvent; GPU-offloaded PME/forces; CPU: pmemd.MPI/sander.MPI; MPI-only; walltime controlled

CUDA stress kernel; non-MD workload; deterministic FP loop; fixed 60 s runtime

Software stack

GROMACS 2024.4; GCC 11.2 + MKL + CUDA; SLURM

AMBER 24 (GPU), AMBER 20 (CPU); CUDA pmemd.cuda; Intel MPI/OpenMPI

FIRESTARTER CUDA build; CUDA 12.9; standalone execution

utilization was 100%. To capture system-level effects, IPMI telemetry was sampled for node-level power, thermal, and cooling measurements. On Alex under mixed workloads, approximately 27% of total node power was attributed to noncompute subsystems, including 6% for fans and the remainder to network components, storage, memory, motherboard subsystems, and peripherals. On newer architectures like H200, memory can significantly contribute to total power. Open-source dataset artifact All data required to reproduce this work are available in our performance–power–energy artifact repository10 ; Zenodo DOI to be added upon acceptance. The repository contains datasets, input files, machine-state files documenting hardware and software environments, and scripts for experimental design, methodology, and figure generation. Results (including runtime, power logs, and derived metrics) are stored in uniquely named directories indexed by GPU model, job ID, and clock or power-cap settings to enable fully reproducible energy-efficiency analysis across architectures.

3

Analytical power-frequency modeling

The dynamic power draw of a compute device is commonly approximated by W (f ) ∝ C · V (f )2 · f,

(1)

where f is the clock frequency and V (f ) is the supply voltage [11,15]. In general, one expects V (f ) ∝ f , which would lead to cubic power-frequency behavior. Current designs show substantial deviations from this ideal due to leakage currents, hard lower limits for the supply voltage, and “smart” power management algorithms baked into the hardware [14,13]. Empirical measurements from MD and stress-test workloads show that GPU power cannot be described by a single smooth law over the full frequency range; instead, distinct regimes exist; see Fig. 3. We model GPU power consumption using a piecewise function: ( W0 + b1 f, f < ft W (f ) = (2) W0 + a1 f + a2 (f − ft )2 , f ≥ ft Here, W0 denotes the baseline and leakage-dominated power, b1 is the linear dynamic power scaling, a1 the residual linear scaling at high frequency, a2 the 10

https://github.com/AyeshaAfzal91/GPU-Energy-Sweetspot

Modeling and Chasing the Energy-Efficiency Sweet Spots in Modern GPUs

Power P(f) Efficiency (f) f* * fEDP ft * fEDP f*

0.6 0.4

300

1400

250

1200

200

1000

0

200

400

600

800

1000

1200

1400

Frequency f (MHz) (a) Frequency operating landscape

600

100 50

0.2

800

150

f*

0.8

Baseline power W0

Normalized value

1.0

7

400

f * = ft

0 0.00000 0.00005 0.00010 0.00015 0.00020 0.00025 0.00030 Quadratic curvature a2

200

(b) Optimal frequency phase space

Fig. 2. Piecewise GPU power–frequency model and energy-optimal regime structure. (a) Normalized power P (f ) (black) and normalized energy efficiency η(f ) (green) vs. ∗ in blue) and transition point ft in black. frequency, showing optima (fη∗ in red, fEDP ∗ (b) Phase diagram of fη as a function of baseline power P0 and quadratic curvature a2 . The color map shows numerically obtained efficiency-optimal frequency fη∗ (Eq. 3), from high (yellow) to low (blue). Shaded regions indicate whether the optimum lies in the linear or quadratic power regime, with the horizontal boundary at fη∗ = ft .

nonlinear high-frequency voltage-frequency and thermal scaling, and ft is the transition frequency between linear and nonlinear regimes. The fitted coefficients are obtained through nonlinear least-squares regression on measured (f, W ) samples for each workload and architecture combination (see below). The model is not necessarily continuous at f = ft , but in general we observe b1 ≈ a1 . The transition frequency ft marks the onset of superlinear power growth. Its location depends on architecture and workload characteristics, particularly computebound and memory-bound behavior. In the linear power scaling regime f < ft , increasing the frequency is energetically favorable as long as W0 > 0 if we assume that the code performance is proportional to f . This is because the runtime decreases faster than the power increases. In the quadratic regime f ≥ ft , power grows faster than performance, leading to diminishing energy efficiency. Assuming near-linear performance scaling, P (f ) ∝ f , energy efficiency in this regime becomes P (f ) f η(f ) = ∝ (f > ft ). W (f ) W0 + a1 f + a2 (f − ft )2 The stationary condition dη df = 0 yields the well-known approximate optimum r W0 ∗ f ≈ , (3) a2 which is independent of a1 [2,9]. This shows that a large base power W0 shifts the optimum toward higher frequencies, whereas a larger a2 shifts it toward lower frequencies. This analysis also shows that, under the assumptions used to derive Eq. (3), the optimal frequency f ∗ cannot be smaller than ft . Model implications The model provides a useful interpretation of GPU frequency scaling behavior: (1) efficiency is mainly governed by regime transitions, not smooth scaling, and (2) W0 and a2 dominate optimal operation. In

8

A. Afzal et al.

Fig. 3. Piecewise power–frequency fitted model of five GROMACS GPU workloads on H100. Plots for the remaining workloads and GPUs are available at [Link].

Fig. 2(a) we show a typical scenario for a compute-bound code (i.e., performance is linear in the clock speed) with a piecewise linear-to-quadratic GPU power-frequency relation. The model reproduces the DVFS landscape observed empirically (see Fig. 3 and Fig. 5 for real data). It can be used to identify two distinct optima: the energy-efficiency maximum at fη∗ and the EDP minimum ∗ . For compute-bound code, the former is always at or to the right of the at fEDP frequency transition point ft . The phase diagram (Fig. 2(b)) explores the dependence of f ∗ on baseline power W0 and quadratic curvature a2 , confirming Eq. 3. Increasing W0 shifts f ∗ upward (toward yellow), as higher performance is needed to amortize fixed baseline power, whereas increasing a2 shifts f ∗ downward (toward blue) due to stronger superlinear growth in the high-frequency regime. The linear term a1 is negligible for compute-bound workloads but can become relevant when memory-bound effects distort performance linearity, causing increased idle cycles in the cores. A boundary at f ∗ = ft separates linear- (bottom) and curvature-dominated (top) regimes, illustrating that energy-optimal GPU operation is a multi-parameter, regime-based problem. Fitting to GPU workloads GPU power is modeled using benchmarkspecific piecewise regression on (f, W ) data. The legend in Fig. 3 shows model coefficients and errors for all six GROMACS benchmarks on H100. The ranges of transition frequency ft and corresponding quadratic curvature a2 across GPUs and workloads are summarized in Fig. 4(a). The transition point ft is identified via exhaustive search over interior indices of each frequency sweep, splitting the low- and high-frequency regimes. Observed ft ranges with AMBER and GROMACS workloads exhibit distinct sub-ranges within each GPU architecture. For each candidate split, a linear model is fitted to the low-frequency regime and a quadratic model to the high-frequency regime. Parameters are estimated via nonlinear least squares (Levenberg–Marquardt), and the optimal split minimizes the combined sum of squared errors: SSEtotal = SSElin + SSEquad . Fits are discarded if convergence fails or if fewer than six samples are available per segment. To prevent unphysical extrapolation, the quadratic model is optionally compared against the maximum power envelope. Model accuracy is quantified using the root-mean-square error (RMSE), computed from the total residual error of the

Modeling and Chasing the Energy-Efficiency Sweet Spots in Modern GPUs 1.2 1e 3

9

A40 GROMACS A40 AMBER A100 GROMACS A100 AMBER H100 GROMACS H100 AMBER H200 GROMACS H200 AMBER

Quadratic curvature a2

1.0 0.8 0.6 0.4 0.2 0.0

900

1000

1100

1200

Transition frequency ft [MHz]

1300

(a) GPU MD workloads

(b) CPU MD workloads

Fig. 4. (a) Ranges of high-frequency GPU power curvature a2 (vertical dash lines) and transition frequency ft (horizontal dotted lines), with workload midpoints as markers (circle: AMBER, square: GROMACS) and colors indicating GPU models. (b) CPU power–frequency scaling on Intel Ice Lake for AMBER, mostly linear (a2 = 0).

selected piecewise fit: r

SSElin + SSEquad , (4) N where N is the total number of samples and the errors are computed from the total residual error at the optimal split. Across benchmarks and architectures, RMSE remains within a few Watts (typically < 1% relative error, with maximum deviations of < 1.8% only for large benchmarks on H100 due to smooth saturation effects), confirming that the piecewise formulation with its linear and quadratic branches is a useful approximation. Fig. 4(a) shows clear trends: old-generation GPUs exhibit higher a2 , indicating stronger superlinear power growth at high frequencies than later-generation GPUs. Transition frequencies vary widely across workloads and a2 is always positive, implying that high-frequency nonlinear effects may slightly amplify or suppress power depending on workload and GPU. RMSE =

Comparison with CPU power-frequency behavior Modern CPU power consumption exhibits near-linear dependence on frequency as shown by the five compute-bound AMBER benchmarks on an Intel Ice Lake CPU across the evaluated DVFS range, as shown in Fig. 4(b). No pronounced superlinear regime is observed, indicating negligible DVFS curvature and weak thermal or voltage-frequency nonlinearities under workloads. CPU power is modeled as: PCPU (f ) = W0c + αf + ∆Pu (fu ),

(5)

where W0c is the static baseline power, α the linear dynamic scaling coefficient.

∆Pu is a discrete offset induced by the uncore frequency fu which vanishes when the uncore frequency is fixed, reducing the model to a purely linear form. A reproducible power jump occurs at fu = 2.3 GHz (Fritz). This power jump is consistent across all workloads and disappears when the uncore frequency is fixed [4]. Since CPU DVFS power is dominated by uniform switching activity rather than workload-dependent nonlinear effects, CPU energy-to-solution is weakly sensitive to frequency scaling.

10

4

A. Afzal et al.

Energy-efficiency sweet spots

In this section, we examine how frequency scaling and power caps interact differently with GPU workloads and power regimes, although both aim to reduce energy compared to default operation. Energy efficiency depends primarily on workload intensity relative to GPU power regimes, which can be grouped into three classes: (1) small workloads underutilize GPUs, yielding poor efficiency; (2) medium workloads balance utilization and efficiency; (3) large workloads enter power-limited regimes with diminishing returns. These align with Sec. 3, where energy degradation begins above the transition frequency ft . Frequency scaling and energy-efficiency regimes Figure 5 relates energy efficiency to code performance, with GPU frequency as a parameter. Horizontal lines indicate constant efficiency, vertical lines indicate constant performance, with the peak marking the efficiency optimum fη∗ . Panels (a)–(h) show that frequency scaling exposes a broad energy–performance space across all architectures. Reducing frequency lowers power faster than performance degrades, improving efficiency until the workload-specific optimum fη∗ is reached. Beyond this point, runtime growth dominates, increasing energy-to-solution. In (a)–(d), the smaller AMBER benchmark (Red, Factor IX, 90K atoms) generally performs worse than the larger one (Orange, Cellulose, 400K atoms), except on A40, which has the highest ft (Fig. 4(a)), reflecting architectural differences. We investigated whether the maximum-efficiency point coincides with the transition frequency ft . Results show that fη∗ (black) typically clusters near ft but does not necessarily coincide. The transition frequency marks the onset of nonlinear power growth, while the quadratic curvature a2 in Eq. 2 governs efficiency collapse. Thus, ft acts as a first-order indicator of regime transition, while a2 controls the rate of efficiency deterioration, consistent with Eq. 3, where higher curvature a2 on early generation A40/A100 GPUs shifts the optimum to lower frequencies. For compute-bound workloads, the optima remain close to the transition frequency ft due to near-linear performance scaling (e.g., fη∗ ≈ ft = 1005 MHz for FIRESTARTER; Fig. 9). Few MD workloads show fη∗ –ft separation, indicating partial memory-bound execution. Frequency scaling thus enables systematic navigation of GPU regimes, moving workloads between efficiency-improving, balanced, and energy-inefficient regions of the Pareto frontier. Power cap and energy-efficiency regimes The effectiveness of a power cap depends on the interaction between the hardware limit and the workload’s natural power demand. Figure 6 shows average GPU power as a function of the enforced cap for A40, A100, H100, and H200 GPUs across AMBER, GROMACS, and FIRESTARTER workloads. A linear increase in measured power with increasing cap, most clearly seen for FIRESTARTER (Fig. 6(i)), indicates continuous hardware throttling, i.e., the workload remains power-limited across the tested range. In contrast, MD applications in panels (a)–(h) exhibit workloaddependent saturation, where power saturates, showing that the cap does not constrain the workload. GROMACS (e)–(h) occupies an intermediate regime, while AMBER (a)–(d) spans a broad spectrum from “cold” to “hot” benchmarks. Cold AMBER benchmarks, such as TRPCage, saturate at lower power than compute-

Modeling and Chasing the Energy-Efficiency Sweet Spots in Modern GPUs

(a) A40-AMBER

(b) A100-AMBER

(c) H100-AMBER

(d) H200-AMBER

(e) A40-GROMACS

(f) A100-GROMACS

(g) H100-GROMACS

(h) H200-GROMACS

11

Fig. 5. Performance–energy efficiency Pareto space under frequency scaling. The max∗ imum energy efficiency (fη∗ ) and minimum EDP (fEDP ) frequencies are labeled.

intensive cases like Cellulose or STMV. On modern GPUs (H100/H200), many workloads remain below the hardware power envelope even at default settings, including 5 of the 11 AMBER benchmarks. Figure 7 demonstrates that power capping is generally less effective than frequency scaling for improving energy efficiency on modern GPUs and is strongly

12

A. Afzal et al.

(a) A40-AMBER

(b) A100-AMBER

(c) H100-AMBER

(d) H200-AMBER

(e) A40-GROMACS

(f) A100-GROMACS

(g) H100-GROMACS

(h) H200-GROMACS

(i) FIRESTARTER

Fig. 6. Impact of hardware power capping on average GPU power across architectures and workloads: (a)–(d) AMBER, (e)–(h) GROMACS, and (i) FIRESTARTER synthetic stress-test baseline on A40, A100, H100, and H200 GPUs.

workload-dependent. The range of optimization through power caps is much smaller than achievable via frequency tuning. This difference is more pronounced on newer architectures: on older A100 GPUs (Fig. 7(a)), power capping AMBER behaves similarly to frequency tuning in terms of energy efficiency and performance (Fig. 5(b)), with the minimum-energy configuration at fη∗ = 990 MHz/ Wη∗ = 160 W. It show mixed AMBER behavior: larger benchmarks sometimes enter power-limited regions, while smaller benchmarks remain largely insensitive to cap reduction. This is most pronounced for compute-light workloads, which underutilize GPU resources and remain below the hardware power limit. Consequently, power caps offer limited optimization leverage, whereas frequency reduces dynamic power and improves efficiency across all regimes. On H100 (Fig. 7(a)), workloads rarely reach the hardware limit, so reducing the cap seldom triggers throttling or affects execution. Unlike power capping, frequency tuning reshapes the entire power–performance trajectory, enabling workloads to traverse operating regimes. This explains why frequency mostly provides larger energy-efficiency gains at minimum-energy configurations. Overall, power-cap tuning primarily affects the high-performance points, whereas frequency scaling enables broader movement toward energy-efficient operating points. Power capping vs. frequency scaling under stress On modern GPUs, power capping primarily affects thermally intensive (“hot”) workloads such as

Modeling and Chasing the Energy-Efficiency Sweet Spots in Modern GPUs

(a) A100-AMBER

13

(b) H100-AMBER

Fig. 7. Performance–energy efficiency Pareto space under power-cap settings for AMBER benchmarks on A100 and H100. Benchmarks are ordered from small to larger size, and optimal power caps for maximum energy efficiency and minimum EDP are labeled. Plots for the remaining workloads and GPUs are available at [Link].

(a) Power-frequency

(b) Frequency-efficiency

(c) Powercap-efficiency

Fig. 8. (a) Piecewise power–frequency behavior and energy efficiency versus performance under (a) frequency scaling and (b) power capping for the FIRESTARTER GPU stress-test benchmark.

FIRESTARTER, while having moderate effects on AMBER hot benchmarks and negligible impact on “cold” (MD) workloads. Figure 8(a) highlights the powerfrequency behavior and saturation across architectures. Panel (b) demonstrates that frequency scaling consistently achieves the highest peak energy efficiency under high-stress conditions across all GPU generations (A40, A100, H200). Overall, frequency scaling yields the best performance-per-watt trade-off (e.g., 100 GF/W vs. 80 GF/W on H200; see Panel (b)-(c)). Consequently, H200 maintains performance above 48 TFLOPS with stable efficiency under power-capped operation. For thermally intensive workloads, frequency scaling remains the most effective strategy: these workloads push the GPU toward its power envelope, invoking the cap mechanism to regulate frequency and voltage. In this regime, power capping yields only minor performance degradation and smaller energyefficiency gains compared to frequency scaling. Impact of optimization on the Pareto frontier Figure 9 evaluates whether optimization improves the energy-performance trade-off by comparing energy-efficiency gain ∆η against performance loss ∆P . Points above the diagonal (∆η > ∆P ) indicate favorable trade-offs where efficiency improvements outweigh performance loss. Frequency scaling exposes a broader optimization space than power capping, enabling larger gains but with stronger workload- and architecture-dependent variability, as it reshapes the full power–performance tra-

A. Afzal et al. Frequency - MaxEff

70

Frequency - MinEDP

A40 A100 H100 H200

0

Powercap - MaxEff

70

Powercap - MinEDP

60

0.4

50 40 30

0.2

20 10 0

1e6 1.0

50 40 0.8

30 A40 A100 H100 H200

20 10

0.6

0

0.6

Powercap - MaxEff

70

Powercap - MinEDP

60

Efficiency Gain (%)

20

Efficiency Gain (%)

0.8

Frequency - MinEDP

60

Number of atoms

Efficiency Gain (%)

40 30

Frequency - MaxEff

70

50

10

Efficiency Gain (%)

1e6 1.0

60

Number of atoms

14

0.4

50 40 30

0.2

20 10

0

20

40

60

Performance Loss (%)

0

20

40

60

0

Performance Loss (%)

(a) AMBER

0

20

40

60

Performance Loss (%)

0

20

40

60

Performance Loss (%)

(b) GROMACS

Fig. 9. Efficiency–performance trade-off for frequency scaling and power-cap settings ∗ ∗ ) points. Each point , WEDP at maximum-efficiency (fη∗ , Wη∗ ) and minimum-EDP (fEDP shows efficiency gain (∆η) vs. performance change (∆P ) relative to default. Markers denote GPU type; color encodes workload size from small (blue) to large (yellow).

jectory. In contrast, power capping delivers smaller, more stable improvements by limiting only the upper power envelope. Older GPUs (A40/A100) generally achieve more favorable trade-offs across MD workloads, while newer GPUs (H100/H200) exhibit narrower efficient operating regions and greater sensitivity to nonlinear high-frequency effects and baseline/memory power. Larger workloads (yellow) consistently benefit from optimization, as fixed system costs are better amortized at high utilization.

5

Conclusion and outlook

Across molecular dynamics and synthetic stress workloads, we show that energy efficiency is fundamentally a workload–hardware matching problem. We propose a two-stage methodology: an analytic, piecewise GPU power model is first used to extract interpretable regime structure, which is then correlated with empirical measurements across real workloads and architectures. The model is fitted directly from application-level DVFS measurements, achieves low prediction error, and yields physically meaningful parameters that generalize across workloads. In particular, baseline power W0 , nonlinear curvature a2 , and transition frequency ft define the energy landscape. The transition frequency ft marks the onset of inefficient operation, while a2 controls the rate of efficiency degradation beyond this point. Across workloads, this structure consistently matches measurements: small workloads underutilize GPUs, medium workloads operate near optimal efficiency, and large workloads become bandwidth- or saturation-limited. While numerical values vary, the regime structure remains stable and transferable across architectures. This shows that energy efficiency depends on aligning workload behavior with GPU regime boundaries. Frequency scaling is the dominant mechanism for navigating these regimes, reshaping the full power–performance frontier and enabling access to energy-optimal regions. In contrast, power capping

Modeling and Chasing the Energy-Efficiency Sweet Spots in Modern GPUs

15

provides limited benefit unless workloads approach sustained thermal or power limits on modern GPUs that often do not fully saturate hardware power envelopes. Overall, energy-efficient HPC execution is a regime identification task defined by workload characteristics and a small set of interpretable hardwaredependent parameters. Future work will extend this analysis to other accelerators such as AMD and Intel and emerging workloads such as large language models. Acknowledgments The authors acknowledge support from the German Research Foundation (DFG), project number 545776403, FOR 5880: Holistic Energy and Performance Modeling for Sustainable Computing (Mod4Comp), as well as the EEC initiative within the national high-performance computing framework at German universities. The authors gratefully acknowledge the HPC resources provided by the Erlangen National High Performance Computing Center (NHR@FAU) at FAU Erlangen-Nürnberg. NHR funding is provided by the German Federal Ministry of Education and Research and the state governments participating on the basis of the resolutions of the GWK for the national high-performance computing at universities by federal and Bavarian state authorities. The NHR@FAU hardware is partially funded by the German Research Foundation (DFG), grant number 440719683.

References 1. Abraham, M.J., Murtola, T., Schulz, R., Páll, S., Smith, J.C., Hess, B., Lindahl, E.: GROMACS: High performance molecular simulations through multilevel parallelism from laptops to supercomputers. SoftwareX 1-2, 19–25 (2015). https://doi.org/10.1016/j.softx.2015.06.001 2. Afzal, A.: The cost of computation: Metrics and models for modern multicorebased systems in scientific computing. Master’s thesis, Department Informatik, Friedrich Alexander Universität Erlangen-Nürnberg (2015). https://doi.org/10. 13140/RG.2.2.35954.25283 3. Afzal, A., Hager, G., Wellein, G.: SPEChpc 2021 Benchmarks on Ice Lake and Sapphire Rapids Infiniband Clusters: A Performance and Energy Case Study. In: 14th IEEE/ACM Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS) (2023). https://doi.org/10.1145/3624062. 3624197 4. Afzal, A., Hager, G., Wellein, G.: Analytic Roofline Modeling and Energy Analysis of LULESH Proxy Application on Multi-Core Clusters. International Journal of High Performance Computing Applications (IJHPCA) (2025). https://doi.org/ 10.1177/10943420251363711 5. Afzal, A., Hager, G., Wellein, G.: Wattlytics: An Interactive Web Platform for Smarter Energy Decisions (2026). https://doi.org/10.48550/arXiv.2604.08182 6. Afzal, A., Kahler, A., Hager, G., Wellein, G.: GROMACS Unplugged: How Power Capping and Frequency Shapes Performance on GPUs. Euro-Par 2025: Parallel Processing Workshops Volume in the Springer Lecture Notes in Computer Science (LNCS) series (2025). https://doi.org/10.48550/arXiv.2412.08792

16

A. Afzal et al.

7. Case, D.A., Cheatham III, T.E., Darden, T., Gohlke, H., Luo, R., Merz Jr, K.M., Onufriev, A., Simmerling, C., Wang, B., Woods, R.J.: The amber biomolecular simulation programs. Journal of computational chemistry 26(16), 1668–1688 (2005). https://doi.org/10.1002/jcc.20290 8. Hackenberg, D., Oldenburg, R., Molka, D., Schöne, R.: Introducing firestarter: A processor stress test utility. In: 2013 International Green Computing Conference Proceedings. pp. 1–9. IEEE (2013). https://doi.org/10.1109/IGCC.2013. 6604507 9. Hager, G., Treibig, J., Habich, J., Wellein, G.: Exploring performance and power properties of modern multi-core chips via simple machine models. Concurrency and Computation: Practice and Experience 28(2), 189–210 (2016). https://doi. org/https://doi.org/10.1002/cpe.3180 10. Han, Y., Nan, Z., Zhou, S., Niu, Z.: Dvfs-aware dnn inference on gpus: Latency modeling and performance analysis. In: ICC 2025-IEEE International Conference on Communications. pp. 1274–1279. IEEE (2025). https://doi.org/10.1109/ ICC52391.2025.11161883 11. Horowitz, M., Alon, E., Patil, D., Naffziger, S., Kumar, R., Bernstein, K.: Scaling, power, and the future of cmos. In: IEEE InternationalElectron Devices Meeting, 2005. IEDM Technical Digest. pp. 7–15. IEEE (2005). https://doi.org/10.1109/ IEDM.2005.1609253 12. Krzywaniak, A., Czarnul, P.: Performance/energy aware optimization of parallel applications on gpus under power capping. In: Wyrzykowski, R., Deelman, E., Dongarra, J., Karczewski, K. (eds.) Parallel Processing and Applied Mathematics. pp. 123–133. Springer International Publishing, Cham (2020). https://doi.org/ 10.1007/978-3-030-43222-5_11 13. Leng, J., Hetherington, T., ElTantawy, A., Gilani, S., Kim, N.S., Aamodt, T.M., Reddi, V.J.: GPUWattch: Enabling energy optimizations in GPGPUs. ACM SIGARCH computer architecture news 41(3), 487–498 (2013). https://doi.org/ 10.1145/2485922.2485964 14. Li, S., Ahn, J.H., Strong, R.D., Brockman, J.B., Tullsen, D.M., Jouppi, N.P.: Mcpat: an integrated power, area, and timing modeling framework for multicore and manycore architectures. In: Proceedings of the 42nd Annual IEEE/ACM International Symposium on Microarchitecture. p. 469–480. MICRO 42, Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/ 1669112.1669172 15. Weste, N.H., Harris, D.: CMOS VLSI design: a circuits and systems perspective. Addison-Wesley Publishing Company, United States (2015), https://www.pearson.com/en-us/subject-catalog/p/ cmos-vlsi-design-a-circuits-and-systems-perspective/P200000003427/ 9780137981076