Evaluating MFU as a Proxy for GPU Power for Energy-Aware Simulation of LLM Training

arXiv:2608.03880v1 [cs.PF] 4 Aug 2026

Evaluating MFU as a Proxy for GPU Power for Energy-Aware Simulation of LLM Training Niklas Enskat

Philipp Wiesner

Technische Universität Berlin enskat@campus.tu-berlin.de

Technische Universität Berlin wiesner@tu-berlin.de

Abstract—High-fidelity performance simulators are essential for designing and configuring efficient AI systems, yet today’s tools lack the ability to predict power consumption. Established GPU power models rely on hardware utilization counters, which do not exist until the workload has actually run. This work evaluates whether Model FLOPs Utilization (MFU)—an analytical, software-defined metric relating achieved throughput to peak hardware capability—can serve as a portable, software-defined predictor of GPU power for LLMs. We benchmark almost 3000 single-device training runs across six GPUs, covering different model families, numerical precisions, batch sizes, and contextwindow lengths. We find that a linear MFU-based power model fits every tested GPU as long as the workload is computebound, as in production LLM training. Fitting per-(GPU, dtype, batch size) instead of per-GPU drops the within-cell mean error from around 10 % to around 1 %, matching the cross-repeat measurement-noise floor. Index Terms—GPU power modeling, Model FLOPs Utilization, LLM simulation, energy-efficient AI, performance modeling

I. I NTRODUCTION The rapid progress of artificial intelligence (AI) has led to increasingly larger and more energy-demanding models [1]. Training and deploying such models relies heavily on GPUaccelerated infrastructures, where GPUs have become the backbone of modern AI systems [2]. As AI workloads contribute a growing share of global data-center electricity use [3], there is an intensifying need for energy-aware system design and predictive performance modeling [4], [5]. High-fidelity simulators and analytical models are the primary tools used by researchers and system architects to explore ”what-if” scenarios for future AI clusters, such as optimizing scheduling policies or evaluating different hardware parallelization strategies [6]–[8]. However, predicting the power consumption of simulated AI workloads remains a fundamental challenge. Existing power models typically rely on hardware-level telemetry, such as GPU Utilization, which represents an aggregate measure of device activity reported by the hardware during execution [9], [10]. While useful for operational monitoring, these metrics present a telemetry gap for simulation and performance modeling. First, hardware-defined counters are only available after code execution on physical hardware, making them inaccessible during the design or simulation phase. Second, these metrics are vendor-specific (e.g., NVIDIA’s NVML vs. AMD’s ROCm), limiting the portability of derived power models across different providers or architectural generations.

A recently introduced yet already widely adopted performance metric for generative AI systems is Model FLOPs Utilization (MFU) [11]. Unlike hardware counters, MFU is an analytical metric that relates observed throughput to the theoretical peak compute capacity required per token. Because MFU is derived from the model structure and high-level execution parameters, it can be calculated by simulators (e.g., Microsoft’s Vidur [12]) without access to physical hardware. While MFU has been proposed as a candidate input for power modeling in these settings [13], its predictive value and robustness across diverse hardware have not been systematically evaluated. This paper investigates whether MFU can serve as a reliable, portable, and software-defined proxy for GPU power consumption. For LLM training, which is predominantly computebound, MFU captures the compute saturation that drives power consumption. We validate MFU against ground-truth hardware measurements, providing a portable methodology for energyaware simulation without vendor-specific telemetry. To do so, we build a cross-architectural benchmarking framework and execute a controlled single-GPU training sweep on six accelerators spanning two vendors and four hardware tiers (NVIDIA A100, L40, L4, Quadro RTX 5000, RTX 4070 Ti, and AMD MI210). The sweep crosses three model families (Qwen2.5, GPT-2, DialoGPT), three numerical precisions (fp32, fp16, bf16), seven batch sizes (1–128), and two context-window lengths (512, 2048 tokens), yielding 126 configurations per device, each repeated four times with telemetry sampled until statistical convergence. For every configuration, we record MFU, vendor-reported GPU Utilization, and GPU power, and use this dataset to evaluate MFU and GPU Utilization as predictors of power across hardware platforms and workload regimes. We use internal GPU telemetry (NVML on NVIDIA, ROCm on AMD) as the reference power signal throughout. Appendix A documents agreement with external wall-power measurements for two devices. Our key findings are: • A linear MFU-power model fits every GPU we measure, including AMD MI210 where vendor-reported GPU Utilization is a binary activity flag and admits no linear fit at all. • The linear relationship holds across vendors and precisions, but each device needs its own fit because MFU is normalized by peak FLOPS. Supporting a new accelera-

Preprint. Published at the 34th International Symposium on the Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS 2026).

tor therefore takes only a one-shot per-GPU calibration sweep. • Conditioning a separate linear fit per (GPU, dtype, batch size) collapses MAPE from ∼10 % per-GPU to ∼1 % per cell, which is statistically indistinguishable from the cross-repeat measurement-noise floor. The practical recipe for an MFU-based power proxy is therefore to condition on these three axes and on nothing else. • At batch 1, MFU collapses below 1.3 % while power stays at ∼60 % of the batch-128 level. Further-out memory-bound regimes need a complementary memorybandwidth signal. Our benchmarking pipeline and data are publicly available: https://github.com/dos-group/gpu power benchmark.

precision (dtype) used. Unlike a hardware-defined activity counter, MFU is derived purely from the model architecture and measured throughput, making it a software-defined metric that can be computed inside a simulator without access to physical hardware. Concretely, MFU [11], [16] is defined as

II. P ERFORMANCE AND P OWER M ETRICS FOR GPU S

where N is the number of model parameters, Lnum the number of layers, Hnum the number of attention heads, Qdim the head dimension, and Tseq the sequence length [11]. In practice the exact operation count is more involved due to architectural and implementation details, so we use the integrated profiler CalFLOPS [17] to obtain Creq directly. Whether MFU can act as a reliable predictor of GPU power consumption depends on the execution regime. In computebound regions, performance is limited by arithmetic throughput, which MFU captures by construction. In memory-bound regions, the bottleneck shifts to data transfer between memory and processors, so GPU Utilization may remain high while MFU drops. A linear MFU–power relationship is therefore not expected to hold in this regime, and we restrict our study to the compute-bound case typical of production LLM training. Because MFU reflects effective training progress rather than total executed operations, it is also less sensitive to implementationspecific recomputation effects than counter-based metrics. This makes MFU an attractive abstraction for comparing training efficiency across devices and configurations. Yet, as a power predictor, it has not been systematically validated, motivating the study in the remainder of this paper.

Modern LLM training workloads impose sustained high computational demand and correspondingly high GPU power draw, so understanding how workload performance metrics relate to energy behavior is central to both efficient operation and predictive modeling. Floating-point operations per second (FLOPS) are the conventional starting point for describing GPU compute capability, with vendors reporting theoretical peaks under idealized conditions. Real training workloads rarely approach these peaks because of memory effects, scheduling overhead, communication latency, and kernel structure, so the relevant practical question is which fraction of peak compute a workload actually realizes and how that fraction relates to power. The remainder of this section reviews two utilization metrics commonly used to answer this question (Section II-A, II-B) and then outlines how GPU power is measured and estimated in practice (Section II-C). A. GPU Utilization GPU Utilization is commonly reported via tools such as nvidia-smi or rocm-smi and is typically defined as the fraction of time during which the GPU is actively executing kernels within a sampling interval [14], [15]. It is widely used for operational monitoring due to its simplicity and availability. However, GPU Utilization is a time-based activity metric rather than a measure of compute-efficiency. High utilization may correspond to memory-bound execution, synchronization overhead, or stalled kernels, without implying productive throughput. It is also a vendor-defined counter whose semantics differ across vendors and architectural generations, so equal values can correspond to substantially different throughput and power behavior. These characteristics limit GPU Utilization as a universal efficiency metric and raise the question about its suitability as a predictor of energy consumption. It is nevertheless frequently used as an input to GPU power models [9], [10]. B. Model FLOPs Utilization (MFU) MFU relates observed throughput (tokens per second) to the theoretical compute required per token, normalized by the peak compute capacity of the GPU for the specific numerical

MFU =

Tokens/s · Creq , FLOPSpeak

where Creq is the floating-point operation count required for a forward and backward pass of a single token, and FLOPSpeak is the theoretical peak throughput (operations per second) for the used precision and hardware generation. For dense transformer models, Creq is commonly approximated as Creq = 6N + 12 Lnum Hnum Qdim Tseq ,

C. GPU Power Measurement and Estimation Modern data-center and workstation GPUs expose onboard power telemetry through vendor management interfaces (NVML on NVIDIA, ROCm SMI on AMD), reporting boardlevel power at sub-second granularity [18]. These signals are convenient and ubiquitous, and we use them as the groundtruth power signal throughout this study. Appendix A validates them against external node-level wall-power measurements for two of our devices. External hardware power meters can provide reference measurements at higher accuracy and at component or node level [19], [20], but their deployment requires dedicated instrumentation and does not scale easily to large clusters. Where direct measurement is impractical (most prominently in simulation and design-space exploration) analytical and statistical models estimate power from hardware activity counters and performance metrics [21]. Such models depend on calibration data and typically rely on architecturespecific telemetry, which limits their portability across vendors

and generations [19] and prevents their use in simulators that have no hardware counters to read in the first place. This portability gap is exactly what an MFU-based power model aims to close. III. E XPERIMENTAL F RAMEWORK To validate MFU as a simulation proxy, we develop a crossarchitectural benchmarking framework that maps high-level model parameters to empirical power consumption. We organize the framework into the workloads we run (Section III-A), the protocol used to measure them (Section III-B), and the analysis pipeline that turns the raw logs into the regressions reported in Section IV (Section III-C). A. Workloads Each measurement run drives the GPU with a causallanguage-modelling training step (forward pass, backward pass, and AdamW optimizer step), so that observed power reflects the full compute graph a training simulator would target. Input is a single fixed English prompt replicated to the configured batch size, tokenized with the model’s own tokenizer, and padded or truncated to the requested context length. This keeps the data pipeline out of the measurement and isolates on-device compute. We sweep the following axes: • 6 GPUs: NVIDIA A100, L40, L4, Quadro RTX 5000, RTX 4070 Ti, and AMD MI210. • 3 model families (Qwen2.5, GPT-2, DialoGPT), with sizes matched to GPU tier: commodity (RTX 4070 Ti, Quadro RTX 5000): 0.5B / 124M / medium; mid (L40, L4): 1.5B / GPT-2-XL / large1 ; large (A100, MI210): 3B / GPT-2-XL / large. 2 • 3 numerical precisions: float32, float16, bfloat16 . • 7 batch sizes: 1, 4, 8, 16, 32, 64, 128. • 2 context windows: 512 and 2048 tokens. The cross product yields 126 workload configurations per GPU (84 on the Quadro RTX 5000). Each configuration is repeated four times. We use the same optimizer (AdamW, learning rate 10−4 , weight decay 0.01), attention implementation (eager, with flash and memory-efficient SDPA kernels disabled), sampling interval (1 s of GPU-event time), warmup (5 training steps), cooldown (5 s between configurations), and single-GPU execution across all runs. Gradient scaling is disabled across all precisions for consistency. Clocks, persistence mode, and power limits are left at driver defaults, so devices are free to adjust frequency and voltage as they would in production. B. Execution and Telemetry Per-step duration is measured with paired CUDA events, and a telemetry sample is emitted roughly once per second 1 Qwen2.5-1.5B in float32 with AdamW does not fit in the L4’s 24 GB and is excluded from that cell; the L4’s float16 and bfloat16 Qwen runs and the L40’s float32 Qwen runs are unaffected. 2 The Quadro RTX 5000 lacks hardware bfloat16 support.

TABLE I GPU MEMORY AND PEAK FLOPS ( USED FOR MFU NORMALIZATION ) TAKEN FROM THE OFFICIAL VENDOR DATASHEETS FOR EACH DEVICE . Memory

Peak FLOPS (TFLOPS)

GPU

(GB)

FP32

FP16

BF16

NVIDIA A100 NVIDIA L40 NVIDIA L4 NVIDIA Quadro RTX 5000 NVIDIA RTX 4070 Ti AMD Instinct MI210

80 48 24 16 12 64

156 90.5 30.3 89.2 40.1 181

312 181.05 121 89.2 80.2 181

312 181.05 121 — 80.2 181

of accumulated step time, with in-window GPU Utilization weighted by per-step duration. Per-step training FLOPs are approximated as 3× the forward FLOPs reported by CalFLOPS [17], following the standard 1:2 forward-to-backward ratio for transformer training [22]. Peak per-device, perprecision FLOPS values were pulled from the official vendor datasheets (Table I). The FP32 peaks reflect the Tensor/Matrix-Core path that PyTorch takes under set_float32_matmul_precision("high") with allow_tf32=True (TF32 Tensor Core on Ampere, FP16 Tensor Core on Turing, BF16 Matrix Core on CDNA2), not IEEE-strict FP32 vector throughput. For each configuration we sample until the 95 % confidenceinterval half-width of both MFU and GPU Utilization falls within 5 % of their running mean (i.e. 1.96 · SEM/x̄ ≤ 0.05), with a minimum of 50 samples and a hard cap of 500. Power is sampled jointly but excluded from the stopping rule. It is the smoothest of the three signals, and once both predictors are stable it is stable by extension. Models are executed with PyTorch [23] and Hugging Face Transformers [24]. Configuration sweeps are managed with Hydra [25]. On-device telemetry uses NVML3 on NVIDIA and ROCm4 [18] on AMD. FLOP profiling uses CalFLOPS [17]. C. Aggregation and Metrics Per-sample logs are aggregated to per-configuration means across the four repetitions. All linear fits in Section IV are ordinary least squares of GPU-reported power on the empirical (CalFLOPS-based) MFU, fitted on the per-configuration means. Only the per-sample error distribution in Figure 3 is computed on the raw samples. R2 and MAPE = mean(|y − ŷ|/|y|) are reported with 95 % confidence intervals from a row-level bootstrap with 2000 resamples. IV. R ESULTS We evaluate how well MFU and GPU Utilization predict GPU power consumption across hardware platforms and workload configurations. We report predictor–power relationships, error characteristics, and sensitivity to experimental variables. 3 https://developer.nvidia.com/management-library-nvml 4 https://www.amd.com/en/products/software/rocm.html

TABLE II P ER -GPU LINEAR MFU- BASED POWER FITS OVER ALL CONFIGURATIONS . E RRORS ARE 95 % BOOTSTRAP CI S . MFU

NVIDIA A100 NVIDIA L40 NVIDIA L4 NVIDIA Quadro 5000 NVIDIA RTX 4070 Ti AMD MI210

GPU Util

R2

MAPE

R2

0.74 ± 0.03 0.80 ± 0.03 0.37 ± 0.03 0.84 ± 0.02 0.68 ± 0.06 0.77 ± 0.05

11.0 ± 0.7 % 7.8 ± 0.5 % 3.5 ± 0.5 % 12.5 ± 1.4 % 12.5 ± 1.1 % 13.7 ± 0.9 %

0.87 0.82 0.17 0.85 0.67 —

GPU

dtype bfloat16

float16

MFU (%)

float32

GPU Utilization (%)

NVIDIA A100

300 200 100 0

NVIDIA L40

300

200

100

0

NVIDIA L4

60 40

0 250

NVIDIA Quadro RTX 5000

GPU Power (W)

20

200 150 100 50 0

NVIDIA RTX 4070 Ti

200 150 100 50 0 200

AMD MI210

150

Not available

100 50 0 0

10

20

30

MFU (%)

40

40

60

80

100

GPU Utilization (%)

Fig. 1. MFU and GPU Utilization versus GPU-reported power. The MI210 appears only in the MFU panel because its GPU Utilization signal is effectively binary.

A. A Linear MFU-Based Power Model Fits Every GPU Across all six devices, a linear MFU-based power model fits the GPU-reported power signal. Table II reports the per-GPU fits alongside the GPU Utilization counterpart for context. The per-GPU MFU fits explain 68–84 % of variance in GPUreported power on training-class NVIDIA cards, dropping to 37 % on the inference-class L4, and 77 % on the AMD MI210. Mean absolute percentage errors (MAPE) range from 3.5 % on the L4 to 13.7 % on MI210. This is sufficient precision for the “what-if” energy comparisons targeted by training simulators. GPU Utilization fits are tighter than MFU on NVIDIA hardware (0.67–0.87), but MFU is the only one of the two predictors that yields a fittable model on every device: on the MI210, vendor-reported GPU Utilization saturates at 100 % under any sustained training workload (Section IV-B), so no linear power–utilization fit is defined there. a) Each device needs its own fit: The fitted slopes range from 1.21 W per percent of MFU on the MI210 to 4.49 on the A100, a 3.7× spread that mirrors the spread in peak per-precision FLOPS across these devices. Because MFU normalizes throughput by peak FLOPS, equal MFU values correspond to different raw arithmetic work on different chips, and power scales with raw arithmetic work. The portability claim is therefore that the model class (linear, MFU-based) transfers across vendors and precision modes, not that a single slope does. A simulator targeting a new device needs a oneshot per-GPU calibration sweep, after which power follows from MFU alone. This is precisely the input simulators already produce as an analytical metric [12], [13]. b) Power scales linearly with MFU within each precision: Holding numerical precision and batch size fixed, MFU explains 98 % of power variance on the MI210 (median R2 = 0.997 across 21 cells; mean 0.980, std 0.038). The corresponding NVIDIA medians range from 0.89 (RTX 4070 Ti) to 0.98 (A100, Quadro 5000), with the exception of the L4 with 0.43 due to its low power range. Figure 1 shows the underlying scatter: within each dtype, power scales nearlinearly with MFU on every device, with per-dtype slopes that reflect the peak-FLOPS spread noted above. The per-GPU R2 values in Table II are lower because they average over multiple regimes, not because the underlying relationship is noisy. c) The L4 is a partial exception: Its 72 W TDP leaves only ∼10 W above a 64 W idle floor (87 % of TDP, versus 60– 80 % on the data-center cards). With this little signal to fit, R2 drops to 0.37, yet the L4’s MAPE is the lowest in the fleet (3.5 % per-GPU, 0.6 % per cell): in absolute terms the linear model still predicts L4 power accurately, simply because there is little to predict. Takeaway. A linear MFU-based power model fits every device we measured, including the AMD MI210 where GPU Utilization is a binary activity flag and admits no linear fit. The linear relationship holds across vendors and precisions, but each device needs its own fit because MFU is normalized by peak FLOPS. Supporting a new accelerator therefore takes only a one-shot per-GPU calibration sweep.

GPU Utilization (%)

100

40 30 20 10 0

80 60 40 20 0

1

4

8

16 32 64 128

1

Batch size

4

8

16 32 64 128

Batch size

Fig. 2. Per-GPU mean MFU (left) and GPU Utilization (right) as a function of batch size. MFU tracks arithmetic throughput and ramps smoothly with batch on every device. GPU Utilization rises shallowly on NVIDIA but pins at 100 % on the MI210 from batch 1 onward. Shaded bands are ±1 SEM across configurations.

B. Conditioning on Dtype and Batch Size We established that a linear MFU-based power model fits every GPU. Two questions remain: which workload variables shift the slope of that fit, and how far does conditioning on them reduce prediction error before the data themselves run out of signal? a) Impact of dtype and batch size: Of the workload axes we vary, only numerical precision and batch size materially shift the MFU-power relationship. Figure 1 shows the perdtype OLS fits: reduced-precision formats shift the predictor distributions and change the slope of the line, with MFU exhibiting stronger dtype-dependent scaling than GPU Utilization. This is expected, since MFU is normalized against a dtype-specific peak FLOPS rate. Figure 2 illustrates the effect of batch size on MFU and GPU Utilization. MFU rises smoothly from around 1 % at batch 1 to 28–44 % at batch 128, tracking arithmetic throughput. GPU Utilization sits at 40– 55 % already at batch 1 and saturates near 90 % well before MFU does. On the MI210, GPU Utilization is pinned at 100 % across every batch size: ROCm reports utilization from the GRBM_COUNT counter, which acts as a binary activity flag rather than a throughput proxy and saturates at 100 % under any sustained training workload, so the linear fit is undefined on that device. The tested context window lengths (512 versus 2048 tokens) had no measurable effect on either predictor or on power. b) Conditioning cuts MAPE from ∼10 % to ∼1 %: Fitting a separate linear MFU-power model per (GPU, dtype, batch) cell collapses the prediction error from ∼10 % per-GPU to ∼1 % per cell. Figure 3 illustrates the per-sample absolute relative error from MFU or GPU Utilization to power draw. Figure 4 compares the two fits directly. Each point’s y value is the power predicted by an OLS regression from MFU, with the left column using a single per-GPU slope and intercept and the right column using a separate slope and intercept fit within each (dtype, batch size) cell of that GPU. The perGPU fit (left) shows a clear batch-1 cluster sitting off the diagonal as a flat horizontal band, reflecting the memory-

conditioned on GPU

Relative prediction error (%)

50

MFU (%)

80

NVIDIA Quadro RTX 5000 NVIDIA RTX 4070 Ti AMD MI210

conditioned on (GPU, dtype, batch size)

NVIDIA A100 NVIDIA L40 NVIDIA L4

60

MFU GPU Utilization

40 20 0 80 60 40 20 0 NVIDIA A100

NVIDIA L40

NVIDIA L4

NVIDIA NVIDIA Quadro RTX 5000 RTX 4070 Ti

AMD MI210

Fig. 3. Distribution of per-sample absolute relative error for linear regressions from MFU or GPU Utilization to power draw, across all tested hardware. Conditioning on (dtype, batch size) substantially reduces error for both predictors on all devices.

bound regime that the single-slope model cannot capture (Section IV-C). Under the (GPU, dtype, batch)-conditioned fits (right) predictions collapse onto the diagonal across all batch sizes and dtypes. The tail behavior of the per-GPU fit is informative on its own: GPU Utilization’s 95th-percentile error stays below 26 % on every NVIDIA device, while MFU can exceed 30 % (worst case 57.9 % on the Quadro 5000), with the heavier-tail outliers clustering under reduced precision. The RTX 4070 Ti reverses this ordering: both predictors fit it worst overall, reflecting the workstation environment in which it was measured (Section V). c) The conditioned model is at the data limit: The ∼1 % MAPE quoted above is an in-sample number. We investigate whether the remaining error is model misspecification or measurement noise. Each of the 504 (336 on the Quadro 5000; 392 on the L4) per-GPU configurations is measured across four independent repeats separated by large time gaps. The crossrepeat standard deviation of mean power therefore bounds any predictor’s achievable accuracy. Table III compares the per-GPU MFU model’s residual SD, the per-(GPU, dtype, batch) model’s residual SD, and the cross-repeat measurementnoise SD, each as a percentage of the GPU’s mean power. Conditioning cuts the per-GPU residual by 4–7× on every device, and on every NVIDIA GPU the conditioned residual is within 1.3–3.7× of the noise floor. On the RTX 4070 Ti and L4 it is statistically indistinguishable from it. The MI210 is the lone outlier in relative terms: its AMD power telemetry is unusually quiet (cross-repeat SD of 0.20 % against ≥0.35 % on every NVIDIA device), but its conditioned residual is in line with the rest of the fleet in absolute terms. Takeaway. Within compute-bound regimes, conditioning the MFU power model on dtype and batch size reduces MAPE from ∼10 % per-GPU to ∼1 % per (GPU, dtype, batch) cell, and the resulting residual matches the cross-repeat measurement noise on every NVIDIA GPU. In-sample, the remaining error is therefore dominated by sampling noise rather than model misspecification, leaving little headroom for a more elaborate predictor within a calibrated cell.

Batch size 1

4

dtype

16

64

128

float32

conditioned on GPU

float16

bfloat16

conditioned on (GPU, dtype, batch size)

NVIDIA A100

300 250 200 150 150

200

250

300

350

150

200

250

300

NVIDIA L40

300 250

200

250

300

350

200

250

NVIDIA L4

Predicted power (W)

65 60 55 55

NVIDIA Quadro RTX 5000

per-GPU

per-(GPU, dtype, batch)

Repeat noise

NVIDIA A100 NVIDIA L40 NVIDIA L4 NVIDIA Quadro 5000 NVIDIA RTX 4070 Ti AMD MI210

12.50 % 9.02 % 4.46 % 12.81 % 16.46 % 17.81 %

2.13 % 1.29 % 1.00 % 1.95 % 2.21 % 2.87 %

1.08 % 0.35 % 0.71 % 1.23 % 1.66 % 0.20 %

300

70

60

65

70

55

60

65

70

GPU

MFU1 MFU128 P1 (W) P128 (W)

NVIDIA A100 NVIDIA L40 NVIDIA L4 NVIDIA Quadro 5000 NVIDIA RTX 4070 Ti AMD MI210

0.4 % 0.4 % 0.5 % 0.8 % 0.3 % 1.2 %

28.1 % 31.8 % 22.3 % 35.5 % 14.9 % 44.3 %

189 200 63 111 97 72

307 315 70 201 169 148

200 150 100

100

150

200

250

100

150

200

200

NVIDIA RTX 4070 Ti

GPU

TABLE IV BATCH -1 ( MEMORY- BOUND ) VERSUS BATCH -128 ( COMPUTE - BOUND ), AVERAGED OVER MODELS , PRECISIONS , AND CONTEXT WINDOWS .

200

150

100

100

150

200

100

150

200

200

AMD MI210

TABLE III MFU POWER MODEL RESIDUAL SD VERSUS THE CROSS - REPEAT MEASUREMENT- NOISE FLOOR , AS A PERCENTAGE OF MEAN GPU POWER .

150 100 50 50

100

150

200

GPU-reported power (W)

50

100

150

200

GPU-reported power (W)

Fig. 4. Predicted versus measured GPU power, where the prediction is the fitted value of an OLS regression from MFU to power. Left: a single perGPU fit; batch-1 configurations (yellow) cluster off the diagonal because the memory-bound regime breaks the single-slope fit. Right: a separate fit per (dtype, batch) cell; predictions collapse onto the diagonal across batch sizes and dtypes.

C. MFU Underestimates Power in Memory-Bound Regimes MFU normalizes throughput by peak FLOPS, so any workload bottlenecked on memory bandwidth rather than arithmetic will execute with low MFU even when power and GPU Utilization remain high. Our search grid already brackets this boundary at its low-arithmetic-intensity end: the batch-1 band visible in Figure 4 is its signature. Table IV quantifies the

gap. Going from batch 1 to batch 128, MFU grows by 36– 72× across the six GPUs while power changes by only 1.1– 2.0×. At batch 1, every device runs below 1.3 % MFU yet still draws 49–64 % of its batch-128 power—and 90 % on the L4, where the idle floor dominates the device’s envelope. A single linear MFU-power slope fit on the broader sweep therefore extrapolates downward through this regime and substantially under-predicts power, with a 70–120 W absolute error on the data-center cards. Batch 1 with 2048-token contexts is the smallest-arithmeticintensity workload in our sweep, but still well inside the LLMtraining scope. Workloads with even lower arithmetic intensity (decode-style inference, or training with very small models at very long contexts) would extrapolate further outside the regime where a linear MFU-based power model is informative. A quantitative characterization of that extrapolation, ideally paired with a complementary memory-bandwidth signal, is left to future work. Takeaway. The MFU-based power model holds inside the compute-bound regime that dominates production LLM training. At its low-arithmetic-intensity edge (batch 1) MFU already collapses below 1.3 % while power stays at around 60 % of the batch-128 level. A single linear slope cannot serve both regimes, and very-low-batch or decode-style workloads need a complementary memory-bandwidth signal. V. L IMITATIONS In this study, each measurement drives a full forward, backward, and optimizer step (Section III-A), so the fitted MFU-power relation describes training cost. Autoregressive inference differs along two axes: First, the decode phase

generates one token at a time against a growing KV cache, so per-step arithmetic intensity is low and execution falls into the memory-bound regime where MFU loses predictive power. Second, production serving additionally introduces requestlevel dynamics (continuous batching, prefill/decode interleaving, variable prompt and generation lengths) that a fixed training step cannot represent. We leave an inference evaluation, together with the complementary memory-bandwidth signal it likely requires, to future work. A second limitation is that all configurations run on a single accelerator, so the fit does not see the inter-device communication component (collectives, pipeline bubbles, FSDP allgathers) that contributes to power in distributed training. This is intentional: we isolate the per-device compute-to-power relation, which is the per-node building block that a multi-GPU simulator composes with its own communication model. Prior work on distributed transformer training already characterizes the scaling and communication side of the problem [26]. A cluster simulator reusing our fit must still model communication power and pipeline-bubble idling, and the accuracy of the composed estimate remains to be established. Lastly, we disabled flash and memory-efficient SDPA kernels in this study and run attention in eager mode across all devices (Section III-B). Production stacks typically rely on fused attention kernels, which raise arithmetic intensity and shift the achieved MFU. We adopt eager attention so that the MFU-topower relation is not confounded by kernel-implementation variance across the NVIDIA and AMD toolchains. We still expect the linear relationship to hold under fused kernels, as any change in slope is exactly what a one-shot per-GPU calibration is supposed to absorb. VI. R ELATED W ORK GPU power consumption is closely tied to workloaddependent hardware activity. Previous work has analyzed this relationship through hardware-exposed utilization metrics, telemetry at system scale, and compute-centric abstractions. However, existing approaches often rely on vendor-specific instrumentation, limiting portability across architectures and preventing direct use in performance modeling or simulation frameworks. A. Utilization Metrics and Power Correlation Elvinger et al. [27] analyze GPU utilization under kernel colocation and show that coarse metrics such as achieved occupancy fail to capture resource interference, while finegrained indicators such as instructions per cycle and pipeline utilization are more expressive. Their study targets performance interference rather than energy, but illustrates that broad vendor-defined counters can obscure the activity that actually matters. Islam et al. [28] bridge utilization and energy more directly, showing through NVML and CUPTI traces that streamingmultiprocessor activity correlates strongly with power draw during compute phases. Similarly, AccelWattch [29] models GPU power from microarchitectural activity counters with

high fidelity. Both depend on architecture-specific instrumentation, however, which limits portability across devices and is unavailable in high-level simulation frameworks. At system scale, Fernandez et al. [26] study distributed transformer training and use MFU as a performance indicator, finding that compute efficiency degrades under scaling regimes due to communication overhead. While MFU is treated as a throughput metric, the work supports the view that computecentric measures offer a stable abstraction of workload efficiency across configurations. B. Compute-centric Energy Abstractions An alternative approach models energy consumption from computational cost. Desislavov et al. [30] estimate inference energy demand by combining model FLOPs with FLOPSper-Watt efficiency figures across hardware generations. Their analysis highlights long-term trends in compute growth and hardware efficiency improvements. This illustrates the appeal of compute-based abstractions as FLOPs are independent from hardware, derivable from model structure, and independent of vendor instrumentation. However, their estimates rely largely on theoretical efficiency assumptions rather than empirical system-level power measurements, leaving the predictive accuracy of FLOPs-based abstractions uncertain in practical workloads. C. Portability Challenges in Simulation-Based Modeling The limits of hardware-dependent utilization metrics are particularly evident in performance modeling and simulation. Systems such as Vidur [12] for inference and existing LLM training simulators and performance-modeling toolkits [7], [8], [31] model LLM execution from workload structure and parallelization strategies, without access to hardware-level utilization counters. Vendor-defined metrics such as GPU Utilization or SM activity therefore cannot feed into the modeling process. However, training simulators often report MFU as a performance metric, which makes it a natural integration point for the proposed power model. Özcan et al. [13] extend Vidur with an MFU-based GPU power model to support energy-aware optimization and for coupling the simulator to co-simulation testbeds like Vessim [32]. However, their MFU-based power model is not validated against hardware power measurements. We close this gap for compute-bound LLM training. VII. C ONCLUSION In this paper, we evaluated MFU as a portable, softwaredefined power proxy for LLM training across six NVIDIA and AMD GPUs and nearly 3000 runs. A linear MFU-power model fits every device in the compute-bound regime, with one fit per (GPU, dtype, batch size) cell. Because MFU is an analytical quantity derivable from model structure and execution parameters, this result establishes MFU as a predictor for computebound LLM training simulation. Future work should extend the approach to LLM inference simulators such as Vidur [12], whose autoregressive decode regime is memory-bound rather than compute-bound.

A PPENDIX A P OWER M EASUREMENT VALIDATION Reliable power measurements are a prerequisite for utilization-based modeling. For the Quadro RTX 5000 and NVIDIA L40, we recorded external node-level power readings and compared them against GPU-reported power. As shown in Figure 5, GPU-reported power follows a stable, strongly linear relationship with external measurements (R2 > 0.99). The constant offset represents the host system’s base power, which remains invariant under stable GPU load. This alignment justifies using internal telemetry as the primary modeling target throughout the experiments.

External meter power (W)

NVIDIA Quadro RTX 5000

NVIDIA L40

500 400 300 200 100 0 0

50

100

150

200

250

300

350

0

50

GPU-reported power (W)

100

150

200

250

300

350

GPU-reported power (W)

Fig. 5. GPU-reported vs. external hardware-based power measurements for NVIDIA L40 and Quadro RTX 5000.

R EFERENCES [1] Z. Yang, K. Adamek, and W. Armour, “Double-exponential increases in inference energy: The cost of the race for accuracy,” arXiv preprint arXiv:2412.09731, 12 2024. [2] A. Jahanshahi, H. Z. Sabzi, C. Lau, and D. Wong, “GPU-NEST: Characterizing energy efficiency of multi-GPU inference servers,” IEEE Computer Architecture Letters, vol. 19, no. 2, pp. 139–142, 7 2020. [3] I. E. Agency, “Energy and AI,” International Energy Agency, Tech. Rep., 4 2025. [Online]. Available: https://www.iea.org/reports/energy-and-ai [4] A. Katsenou, X. Wang, D. Schien, and D. Bull, “Comparative study of hardware and software power measurements in video compression,” in 2024 Picture Coding Symposium (PCS), 2 2024. [5] Google, “Environmental report 2024,” Google, Tech. Rep., 7 2024. [Online]. Available: https://www.gstatic.com/gumdrop/sustainability/g oogle-2024-environmental-report.pdf [6] A. Agrawal, N. Kedia, J. Mohan, A. Panwar, N. Kwatra, B. Gulavani, R. Ramjee, and A. Tumanov, “Vidur: A large-scale simulation framework for llm inference,” Proceedings of Machine Learning and Systems, vol. 6, pp. 351–366, 2024. [7] Moore Threads, “SimuMax: A simulator for distributed LLM training,” https://github.com/MooreThreads/SimuMax, 2025. [8] A. Zhebrak, “LLM Cluster Simulator: Interactive distributed training and inference planning,” 2026, browser-based simulator for GPU cluster parallelism strategies, calibrated against published benchmarks from Meta, DeepSeek, and NVIDIA. [Online]. Available: https: //github.com/zhebrak/llm-cluster-simulator [9] Antarctica, “The one-token model,” 12 2025, whitepaper. [Online]. Available: https://antarctica.io/research/one-token-model [10] R. A. Bridges, N. Imam, and T. M. Mintz, “Understanding gpu power: A survey of profiling, modeling, and simulation methods,” ACM Computing Surveys, vol. 49, no. 3, pp. 41:1–41:27, 2016. [11] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, and P. Barham, “PaLM: scaling language modeling with pathways,” Journal of Machine Learning Research, vol. 24, no. 1, pp. 11 324– 11 436, 1 2023. [12] A. Agrawal, N. Kedia, J. Mohan, A. Panwar, N. Kwatra, B. S. Gulavani, R. Ramjee, and A. Tumanov, “Vidur: A large-scale simulation framework for LLM inference,” in Proceedings of Machine Learning and Systems (MLSys), 2025.

[13] M. Özcan, P. Wiesner, P. Weiß, and O. Kao, “Quantifying the energy consumption and carbon emissions of LLM inference via simulations,” in Workshop on Performance and Energy Efficiency in Concurrent and Distributed Systems (PECS) at Euro-Par ’25, 2025. [14] M. Cornelius, G. Cross, S. Shilpika, M. T. Dearing, and Z. Lan, “Extracting practical, actionable energy insights from supercomputer telemetry and logs,” in SC ’25: International Conference for High Performance Computing, Networking, Storage and Analysis, 11 2025. [15] NVIDIA, nvidia-smi Documentation, NVIDIA Corporation, 2025. [Online]. Available: https://docs.nvidia.com/deploy/nvidia-smi/index.ht ml [16] J. Ray, “Using model FLOPs utilization (MFU),” 5 2025, medium article. [Online]. Available: https://medium.com/better-ml/using-modelflops-utilization-mfu-7b17de07faec [17] X. Ye. (2023) calflops: a flops and params calculate tool for neural networks in pytorch framework. [Online]. Available: https: //github.com/MrYxJ/calculate-flops.pytorch [18] L. M. Weakley, S. Michael, L. Huber, A. Thota, B. Fulton, and M. Kusz, “Monitoring and characterizing GPU usage,” Concurrency and Computation: Practice and Experience, vol. 37, no. 3, 1 2025. [19] Z. Yang, K. Adamek, and W. Armour, “Accurate and convenient energy measurements for GPUs: A detailed study of NVIDIA GPU’s built-in power sensor,” in SC ’24: International Conference for High Performance Computing, Networking, Storage and Analysis, 11 2024, pp. 1–17. [20] M. Jay, V. Ostapenco, L. Lefevre, D. Trystram, A.-C. Orgerie, and B. Fichel, “An experimental comparison of software-based power meters: Focus on CPU and GPU,” in Proceedings of the 2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing (CCGrid), 5 2023, pp. 106–118. [21] J. Lang and G. Rünger, “High-resolution power profiling of GPU functions using low-resolution measurement,” in Euro-Par 2013 Parallel Processing, ser. Lecture Notes in Computer Science, vol. 8097. Springer, 8 2013, pp. 801–812. [22] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” arXiv preprint arXiv: 2001.08361, 2020. [23] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al., “Pytorch: An imperative style, high-performance deep learning library,” Advances in neural information processing systems, vol. 32, 2019. [24] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush, “Transformers: State-of-the-art natural language processing,” in EMNLP, 2020. [25] O. Yadan, “Hydra - a framework for elegantly configuring complex applications,” Github, 2019. [Online]. Available: https://github.com/fac ebookresearch/hydra [26] J. Fernandez, L. Wehrstedt, L. Shamis, M. Elhoushi, K. Saladi, Y. Bisk, E. Strubell, and J. Kahn, “Hardware scaling trends and diminishing returns in large-scale distributed training,” arXiv preprint arXiv:2411.13055, 11 2024. [27] P. Elvinger, F. Strati, N. E. Jerger, and A. Klimovic, “Understanding gpu resource interference one level deeper,” in ACM Symposium on Cloud Computing (SoCC), 2026. [28] T. Z. Islam, A. Marathe, H. Schutte, and M. Zaeed, “Data-driven analysis to understand GPU hardware resource usage of optimizations,” arXiv preprint arXiv:2408.10143, 8 2024. [29] V. Kandiah, S. Peverelle, M. Khairy, J. Pan, A. Manjunath, T. G. Rogers, T. M. Aamodt, and N. Hardavellas, “Accelwattch: A power modeling framework for modern gpus,” in 54th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), 2021. [30] R. Desislavov, F. Martı́nez-Plumed, and J. Hernández-Orallo, “Trends in AI inference energy consumption: Beyond the performance-vsparameter laws of deep learning,” Sustainable Computing: Informatics and Systems, vol. 38, p. 100857, 2 2023. [31] M. Liang, H. T. Kassa, W. Fu, B. Coutinho, L. Feng, and C. Delimitrou, “Lumos: Efficient performance modeling and estimation for largescale LLM training,” in Proceedings of Machine Learning and Systems (MLSys), 2025. [32] P. Wiesner, I. Behnke, P. Kilian, M. Steinke, and O. Kao, “Vessim: A testbed for carbon-aware applications and systems,” ACM SIGENERGY Energy Informatics Review, vol. 4, no. 5, 2024.

Related documents

• Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC#2544
• RAPID-LLM: Resilience-Aware Performance analysis of Infrastructure for Distributed LLM Training and Inference#609852
• KernelGenBench: Can LLMs and Agents Write Efficient Kernels Across Operator Sources and Hardware Platforms?#987764
• A Multi-Dimensional, Per-Pass Empirical Study of the LLVM Optimization Pipeline#324969
• Rotary GPU: Exploring Local Execution Paths for Large Mixture-of-Experts Models Under Limited GPU Memory#238571