The Energy Consumption of Transformer Fine-Tuning: A Roofline-Inspired Scaling Model

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

The Energy Consumption of Transformer Fine-Tuning: A Roofline-Inspired Scaling Model

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

cloud distributedcomputing parallelcomputing

distributed computing, parallel computing, cloud

T HE E NERGY C ONSUMPTION OF T RANSFORMER F INE -T UNING : A ROOFLINE -I NSPIRED S CALING M ODEL

Mansour Zoubeirou a Mayaki

arXiv:2606.23546v1 [cs.LG] 22 Jun 2026

Université Lumière Lyon 2, CNRS, Ecole Centrale de Lyon, INSA Lyon, Universite Claude Bernard Lyon 1, LIRIS, UMR5205, 69007 Lyon, France

mansour.mayaki at liris.cnrs.fr

June 23, 2026

A BSTRACT Transformer-based models underpin modern natural language processing but incur rapidly growing computational and energy costs. As training scales in both model size and parallelism, accurately predicting energy consumption has become critical for sustainable and cost-aware system design. We present a framework for modeling the energy consumption of Transformer training on multiple GPUs. Using controlled architectural sweeps of BERT models, we relate measured energy to lightweight proxies for compute, memory trafﬁc, and hardware efﬁciency. Inspired by rooﬂine models, our approach incorporates a speedup-based hardware-efﬁciency factor that captures the effects of tensor parallelism and fully sharded data parallelism. We derive a scaling law model that accurately predicts training energy across heterogeneous conﬁgurations.

Introduction

The rapid growth of deep learning has driven remarkable progress across natural language processing (NLP), computer vision, and multimodal reasoning. In particular, Transformer-based architectures have become the dominant paradigm for language modeling and representation learning, underpinning widely deployed models such as BERT, T5, and GPT. This progress, however, has come at the cost of rapidly increasing computational and energy demands. Training modern Transformer models requires substantial compute resources, leading to high operational costs and a growing environmental footprint. Recent work has raised serious concerns about the sustainability of large-scale machine learning Wu et al. (2022); Mehlin et al. (2023); Yarally et al. (2023). As models scale in depth, width, and training data, energy consumption grows nonlinearly and becomes increasingly sensitive to architectural choices, parallelization strategies, and hardware efﬁciency. While several studies report aggregate energy or carbon footprints for individual large models, a systematic understanding of how training energy scales with compute, memory trafﬁc, and parallel efﬁciency remains limited. Existing approaches often conﬂate model size with hardware utilization and provide little guidance for predicting energy consumption under new architectures or training conﬁgurations. This paper addresses this gap by developing a predictive framework for training energy consumption in Transformer models. Rather than treating energy as a blackbox outcome, we decompose it into interpretable components driven by (i) total compute, (ii) memory trafﬁc, and (iii) hardware efﬁciency. Our approach is inspired by classical rooﬂine models from high-performance computing and is augmented with an empirically grounded treatment of parallel efﬁciency derived from execution-time speedup. We conduct controlled architectural sweeps of BERT models using the Hugging Face implementation, varying depth, width, feed-forward dimension, and attention conﬁguration. From these sweeps, we construct lightweight proxies for parameter count, ﬂoating-point operations, and activation memory trafﬁc. To capture parallelization effects, we introduce a speedup-based hardware-efﬁciency factor for tensor parallelism and fully sharded data parallelism. This work focuses on operational training energy under controlled architectural scaling, with empirical evaluation on BERT-style ﬁne-tuning workloads and common distributed training strategies (data parallelism, FSDP, and tensor parallelism). Our goal is not to provide a universal carbon or energy accounting framework, but to identify the dominant

The Energy Consumption of Transformer Fine-Tuning

scaling mechanisms governing training energy once system-level efﬁciency is taken into account. While our experiments are conducted on encoder-based Transformers, the proposed rooﬂine-inspired formulation and speedup-based efﬁciency factor rely only on generic compute–memory interactions and execution-time scaling, and therefore extend conceptually to other Transformer architectures. Empirical generalization beyond the studied regime is intentionally framed as trend-level consistency rather than direct validation. Contributions. The main contributions of this paper are: • We propose a mechanistic, rooﬂine-inspired decomposition of Transformer training energy into compute, memory, and hardware-efﬁciency components. • We introduce a speedup-based hardware-efﬁciency model for tensor and data parallelism, enabling energy estimation without direct multi-GPU energy measurements. • We derive and empirically validate a scaling law that predicts training energy across a wide range of BERT conﬁgurations.

Related Work

We review three strands of work most relevant to this paper: (i) measurement and reporting of energy and carbon in machine learning, (ii) empirical evidence on how scale, architecture, and parallelization affect efﬁciency, and (iii) rooﬂine-style frameworks for reasoning about compute–memory trade-offs. We conclude by positioning our contribution relative to these lines of work. 2.1

Measuring and Reporting Energy and Carbon

Early empirical studies emphasized the need for standardized reporting of energy consumption and carbon emissions in machine learning, highlighting substantial variation across measurement tools and experimental protocols Henderson et al. (2020). Subsequent analyses documented the rapid growth of energy use in state-of-the-art training and the importance of system-level factors such as cooling, power usage effectiveness (PUE), and geographic siting Cottier et al. (2024); Luccioni et al. (2024). Methodological work has proposed approaches for estimating emissions from public artifacts Luccioni et al. (2023), as well as telemetry pipelines that attribute energy across CPU, GPU, and memory subsystems Yarally et al. (2023); Ogbogu et al. (2023). In NLP, Strubell et al. (2020) popularized reporting carbon alongside accuracy, while more recent lifecycle-aware frameworks combine operational and embodied emissions to support forward estimation under varying hardware and grid assumptions Faiz et al. (2024). 2.2

Scale, Architecture, and Efﬁciency

Classical scaling laws relate parameter count, data, and compute to model quality Kaplan et al. (2020); Brown et al. (2020), but they do not directly characterize training energy, which is mediated by utilization, memory trafﬁc, and parallel efﬁciency. Systems studies have shown that realized throughput for large models depends critically on parallelization strategy and communication behavior Narayanan et al. (2021). Complementary case studies further show that conﬁguration choices such as batching, numerical precision, and interconnect topology can induce large differences in energy consumption for comparable training objectives Patterson et al. (2021). Recent work has also begun to study energy scaling more directly, including efﬁciency-oriented analyses of local LLM deployment and operation Alvarez et al. (2025). In particular, Alvarez et al. (2025) study CPU-only local inference for LLMs and VLMs on consumer and embedded devices, showing that inference cost scales approximately linearly with token length for LLMs and exhibits preprocessing-dependent threshold effects for VLMs. Their results highlight compression and input preprocessing as practical levers for sustainable edge inference, whereas our work focuses on distributed training, where parallelization strategy and execution efﬁciency are the primary determinants of energy scaling. 2.3

Rooﬂine Models

Rooﬂine models from high-performance computing provide a principled framework for reasoning about the interaction between compute throughput, memory bandwidth, and achieved performance Williams et al. (2009). Extensions of the rooﬂine approach have been used to analyze GPU kernels and deep learning workloads, highlighting the roles of arithmetic intensity, memory trafﬁc, and utilization in determining performance and energy efﬁciency Choi et al. (2013); Jouppi et al. (2017); Wang et al. (2020); Owens et al. (2008). More recent studies apply rooﬂine-style analyses to neural network training to explain when workloads are compute-bound versus memory-bound and how architectural choices shift this balance Czaja et al. (2020); Owens et al. (2007). However, most rooﬂine-based analyses focus on performance or power at the kernel or single-device level, rather than on training energy under distributed parallelism. 2

The Energy Consumption of Transformer Fine-Tuning

2.4

Positioning of This Work

Most prior work falls into three categories: (i) reporting aggregate energy or emissions for speciﬁc training runs Strubell et al. (2020); Patterson et al. (2021), (ii) modeling end-to-end emissions without explicitly isolating execution efﬁciency Faiz et al. (2024), or (iii) studying scaling and parallelization primarily through throughput and time-to-solution rather than energy Kaplan et al. (2020); Brown et al. (2020). Recent work by Zoubeirou a Mayaki & Charpenay (2025) proposes an empirical energy model based on power-law relationships among energy, FLOPs, and a hardware-efﬁciency factor, validated on single-GPU measurements across LSTM, GRU, and Transformer models. Their approach decomposes Transformer layers into elementary operations and ﬁts operation-level efﬁciency curves per GPU, yielding accurate predictions under ﬁxed hardware and execution conditions. Our work is complementary but distinct. We target distributed Transformer training and explicitly study how training energy scales with parallelization. Drawing inspiration from rooﬂine theory, we decompose training energy into compute- and memory-related components, then augment this decomposition with a speedup-derived efﬁciency proxy that captures the effects of multi-GPU execution. The resulting model is not intended as a universal energy law across all architectures and hardware platforms; rather, it provides an interpretable and predictive framework for reasoning about training energy within a controlled regime, with explicit attention to tensor parallelism and fully sharded data parallelism.

Methodology

This section details our framework for predicting BERT Devlin et al. (2019) ﬁne-tuning energy. Our methodology combines controlled architectural sweeps with closed-form proxies for parameter count, compute, and memory trafﬁc, together with an explicit efﬁciency term for parallel execution. The goal is not to causally isolate perfectly orthogonal factors, but to obtain sufﬁcient variation in architecture and system-level quantities to support an interpretable regression model of training energy. Model map. Eq. (4) gives a mechanistic decomposition of training energy into compute, memory, and efﬁciency terms; Eq. (13) gives a reduced-form scaling law ﬁtted to these components. 3.1

Tasks, Models and Training Protocol

Tasks and model suite. We ﬁne-tune a family of BERT models on a suite of standard NLP tasks, including sentence classiﬁcation and extractive question answering. These models are implemented using the Hugging Face transformers library, speciﬁcally BertForSequenceClassification. To study scaling behavior, we vary four core architectural hyperparameters: the number of Transformer layers L, hidden dimension d, number of attention heads h, and feed-forward dimension. Controlled sweeps. The architectural sweeps are designed to provide broad coverage of the conﬁguration space rather than strict orthogonal control of compute and memory. Changing depth, width, sequence length, and batch size necessarily affects multiple quantities simultaneously; this is inherent to Transformer scaling. Our aim is therefore to generate sufﬁcient variation across conﬁgurations to statistically disentangle the contributions of compute, memory proxy, and execution efﬁciency in the downstream regression model. Unless otherwise stated, each conﬁguration is ﬁne-tuned for a ﬁxed number of epochs under a consistent optimizer and learning-rate schedule. We log step-level telemetry (time, power/energy counters when available) and aggregate metrics within and across epochs. Full training hyperparameters are reported in Appendix G. 3.2

Compute, Parameter and Memory Proxies

To connect architectural choices to energy, we use closed-form proxies for model size, compute, and activation/memory trafﬁc Devlin et al. (2019); Brown et al. (2020). These proxies are lightweight, depend only on (L, d, ﬀ) and sequence length, and are intended as minimal sufﬁcient statistics for comparative scaling analysis rather than exact kernel- or byte-level measurements. Parameter proxy. We approximate the dominant non-embedding parameter terms per Transformer layer (attention projections and MLP weights), yielding N (L, d, ﬀ) ≈ L 12d2 + 2d ﬀ . (1) Compute proxy (FLOPs per token). We approximate training compute per token using a simpliﬁed decomposition into projection/MLP cost and attention-related cost, giving Ctok (L, d, S) ≈ 12Ld2 + 3LSd, (2) 3

The Energy Consumption of Transformer Fine-Tuning

where S denotes sequence length. Total training compute over a run is then C ≈ Ctok × (tokens processed). Activation/memory trafﬁc proxy. We use a proxy proportional to activation volume across layers, Mproxy (B, S, d, L) ≈ B S d L,

(3)

where B is batch size. This proxy captures the leading dependence of activation/gradient trafﬁc on batch size, sequence length, width, and depth, and serves as a practical substitute for direct bandwidth measurements. We emphasize that Mproxy is an aggregate architectural proxy: it does not attempt to separate activation trafﬁc, optimizer-state movement, or communication volume at a kernel level. These proxies are used throughout the paper to relate measured energy to architecture-dependent compute and memory demands under controlled sweeps. 3.3

Instrumentation and Energy Accounting

We measure training energy using complementary telemetry from Nsight Systems, CodeCarbon, and systemutilization logs. Measurements are converted to kWh, temporally aligned, and integrated over the active training interval after discarding warm-up and cool-down phases. For each conﬁguration, we report mean energy over 5 epochs with standard error. Additional instrumentation details are provided in Appendix G. 3.4

Compute–Memory Energy Model

We model batch energy per training step as the sum of compute, memory trafﬁc, and ﬁxed overheads: E = ϵcomp · C + ϵmem · M + E0 ,

ϵcomp , ϵmem > 0.

(4)

Here, C denotes ﬂoating-point operations and M denotes bytes moved to/from high-bandwidth memory. The constant E0 captures host, leakage, and unmodeled overheads. Proposition 3.1 (Rooﬂine energy lower bounds). Let C be the FLOPs executed in one training step, M the DRAM/HBM bytes moved, AI ≜ C/M the arithmetic intensity, Fpeak the device peak FLOP/s, and let Pdyn denote the dynamic power during the active portion of the step. Deﬁne τmem as the effective memory service rate (bytes/s), i.e., the rate at which memory trafﬁc is serviced under the workload. The classical rooﬂine argument implies that the achieved throughput is upper bounded by min{Fpeak , τmem · AI}, which characterizes whether computation or memory trafﬁc limits performance and energy. Then the step time T satisﬁes C , (5) T ≥ min{Fpeak , τmem · AI} which yields the energy lower bounds E ≥

Pdyn ·C Fpeak

and E ≥

Pdyn · M. τmem

(6)

Full details for the proof of Proposition 3.1 can be found in Appendix F.1. Theorem 3.2 (Expected energy scaling bands). Assume (i) quasi-stationary power during a step; (ii) a stable op-mix as hyperparameters vary; and (iii) over the hyperparameter ranges studied there exist k1 , k2 > 0 and an exponent β ∈ [ 12 , 1) such that the dominant DRAM trafﬁc co-scales with compute as k1 C β ≤ M ≤ k2 C β . Then for device-speciﬁc positives ϵcomp , ϵmem and a ﬁxed overhead E0 ≥ 0, min{ϵmem M, ϵcomp C} ≲ E ≲ ϵcomp C + ϵmem M + E0 . Proof sketch. See Appendix F.2 for details. Estimated compute–memory energy model. While Eq. (4) captures the physical contributions of compute and memory to energy consumption, the strategy-dependent offsets in Appendix B indicate that parallelization alters energy beyond what can be explained by C and M alone. In particular, tensor and sharded data parallelism introduce systematic energy effects that are not fully absorbed into per-FLOP or per-byte costs. This motivates an explicit hardware-efﬁciency factor, modeled in the next subsection via execution-time speedup. 4

The Energy Consumption of Transformer Fine-Tuning

3.5

Hardware Efﬁciency via Empirical Speedup Models

The strategy-dependent energy offsets observed in Section 3.4 motivate modeling parallel efﬁciency explicitly. Rather than modeling communication bandwidth, collective latency, or microbatch-level synchronization costs in detail, we represent hardware efﬁciency through execution-time speedup. This yields a compact, dimensionless proxy that captures the combined effects of communication, synchronization, kernel fragmentation, and utilization under a ﬁxed hardware class and operating regime. Let T (N ) denote the execution time per training step (or epoch) when using N GPUs under a ﬁxed workload. The corresponding speedup is T (1) S(N ) ≜ . (7) T (N ) Because execution time is often easier to obtain and reproduce than direct multi-GPU energy measurements, we use S(N ) as the basis for an energy-relevant efﬁciency proxy. We emphasize that this proxy is not a direct physical measurement of efﬁciency; rather, it summarizes how parallel execution modiﬁes the energy cost of a ﬁxed workload. Deﬁnition. We restrict attention to two parallelism modes: tensor parallelism (degree t) and data parallelism (degree d). In our setup, t is the number of GPUs that jointly compute shards of each layer’s linear algebra (typically within a node), while d is the number of GPUs that process distinct mini-batches and periodically synchronize gradients. Let N = t d denote the total number of devices and g the number of GPUs per node. We factor ϕ(t, d) =

ηTP (t) | {z }

ηDP (d) | {z }

(8)

tensor-parallel efﬁciency data-parallel efﬁciency

with t ∈ {1, 2, 4, . . . , g} and d ∈ {1, 2, . . . , ⌊g/t⌋} in the single-node regime. Here, ϕ ∈ (0, 1] accounts for the effective efﬁciency loss induced by tensor-parallel (TP) and data-parallel (DP/FSDP) execution. Any effects of interconnect topology (e.g., NVLink vs. PCIe), synchronization pattern, or collective implementation are absorbed into the ﬁtted efﬁciency curves below. Tensor-parallel efﬁciency. Tensor parallelism introduces per-layer collective communication and can reduce kernel efﬁciency as the tensor-parallel degree t increases. Empirically, TP exhibits rapid initial gains followed by saturation once communication and kernel fragmentation dominate. We therefore model TP speedup using a saturating exponential law: STP (t) = Smax 1 − e−kt , (9) where Smax is the asymptotic speedup limit and k > 0 controls the rate of saturation. The corresponding tensorparallel efﬁciency is STP (t) . (10) ηTP (t) = t This parameterization captures diminishing returns with increasing TP degree without introducing explicit bandwidth or latency terms. Data-parallel efﬁciency. Data parallelism incurs gradient synchronization once per global batch, with cost depending on communication volume and overlap. Following empirical observations from large-scale distributed training benchmarks, we model DP speedup using a generalized Amdahl-like law: SDP (d) =

d , 1 + α · dβ

(11)

where α ≥ 0 captures the effective communication- or serial-bound fraction and β ≥ 0 allows synchronization overhead to grow nonlinearly with scale. The corresponding data-parallel efﬁciency is ηDP (d) =

SDP (d) 1 . = d 1 + α · dβ

(12)

This model captures the observed sublinear scaling of FSDP/DP and explains the efﬁciency degradation at larger device counts. Empirical estimation of speedup. Figure 1 illustrates the empirical scalability of data-parallel (FSDP/DP) and tensorparallel (TP) training as a function of GPU count Nichols et al. (2022). For data parallelism, measured speedup is well captured by Eq. (11), with ﬁtted parameters α = 0.005 and β = 1.266. The exponent β > 1 indicates that 5

The Energy Consumption of Transformer Fine-Tuning

DDP Scalability (Generalized Amdahl Mod l)

TP Scalability (E.(on ntial d cay law)

Amdahl fit: α = 0.005, β = 1.266 M as,) d

Sp d Up

6 y = 9.79(1 − e −0.76x)

Measu)ed

15 20 25 30 2 4 6 8 10 12 14 16 Number of GPUs N,mb ) of GPUs Figure 1: Execution-time speedup as a function of GPU count for data-parallel (FSDP/DP, left) and tensor-parallel (TP, right) training. Points show measured speedups; curves denote ﬁtted models. The gap from ideal linear scaling motivates explicit modeling of parallel efﬁciency in energy estimation.

synchronization and communication overheads grow faster than linearly at scale, leading to diminishing returns as the number of devices increases. In contrast, tensor parallelism exhibits rapidly saturating speedup that is well approximated by Eq. (9), with Smax ≈ 9.8 and k ≈ 0.76. This reﬂects strong initial gains from partitioning within-layer computation, followed by saturation once collective communication and kernel fragmentation dominate. TP therefore appears most effective for relatively small device counts, particularly in intra-node settings. Lemma 3.3 (From speedup to energy efﬁciency). Consider a ﬁxed training workload executed on N devices, and let T (N ) denote the corresponding execution time. Let P (t) be the instantaneous power draw and E(N ) = R T (N ) P (t) dt the total energy consumed. Assume that: 0 (A1) The workload is ﬁxed across device counts, i.e., the total amount of computation and memory trafﬁc does not change with N . (A2) The average power draw

Z T (N ) 1 P (t) dt T (N ) 0 varies slowly with N within a ﬁxed hardware class and operating regime. P̄ (N ) ≜

(A3) Changes in execution time dominate changes in average power as N varies. Then energy scaling with respect to N is primarily governed by execution-time scaling, and the parallel speedup S(N ) = T (1)/T (N ) induces a corresponding hardware-efﬁciency proxy S(N ) . N In particular, any parametric model that accurately captures T (N ) (and hence S(N )) provides a consistent proxy for the dependence of energy consumption on parallelization within the studied regime. Proof sketch. See Appendix F.3 for details. ηh (N ) ∝

3.6

Estimation and Identiﬁability

Theoretical bounds from Theorem 3.2 imply that, in regimes where the baseline offset E0 is negligible, the effective log–log slope of energy with respect to compute lies between the memory-dominated and compute-dominated limits. This motivates the reduced-form scaling law α

1 2 < αC ≤ 1,

E = κ · C αC M αM ηh η , 6

αη < 0.

(13)

The Energy Consumption of Transformer Fine-Tuning

Here, C denotes total compute (FLOPs), M is a proxy for memory trafﬁc, and ηh is a hardware-efﬁciency factor derived from empirical speedup. We stress that ηh is an efﬁciency proxy, not a direct physical measurement: it summarizes the combined effects of communication, synchronization, and execution inefﬁciency under a given hardware class, interconnect topology, and parallel strategy. The reduced-form model can thus be viewed as a compact representation of the additive compute–memory decomposition in Eq. (4), augmented with a multiplicative efﬁciency correction. We estimate both the mechanistic decomposition in Eq. (4) and the reduced form in Eq. (13) using constrained log-linear regression with architecture- and strategy-speciﬁc ﬁxed effects. We do not assume strict independence between compute, memory, and hardware efﬁciency; rather, they are treated as correlated predictors in an empirical scaling model. To assess stability, we report model diagnostics and robustness analyses in Appendix D, including checks of linearity and functional form, residual behavior, feature-speciﬁc bias, error structure and normality, inﬂuence, and numerical stability.

Energy Scaling with Hyperparameters

We evaluate how training energy scales with compute, hardware resources, and parallelization strategy. The experimental platform consists of a single 8-GPU node with GeForce RTX 2080 Ti accelerators. All energies are averaged over epochs and reported in kilowatt-hours (kWh). FLOP counts correspond to a full training step (forward and backward passes) unless otherwise stated. Energy vs. compute. Figure 5 (in Appendix A.2) shows that training energy increases monotonically with total compute (FLOPs) but does so sub-linearly. The large dispersion at ﬁxed compute conﬁrms that FLOPs alone are insufﬁcient to predict energy consumption. Across all regimes, fully sharded data parallelism (FSDP) consistently consumes less energy than tensor parallelism (TP), with the gap widening at higher compute.

Figure 2: Training energy consumption as a function of model size for tensor parallelism (TP) and fully sharded data parallelism (FSDP), shown separately for 2 and 4 GPUs. Energy vs. model size. Figure 2 shows that energy increases with model size (number of parameters), but the rate depends strongly on the parallelization strategy. FSDP scales more favorably than TP, particularly for larger models. Depth-speciﬁc analysis are reported in Appendix A.5. Overall, training energy is governed by a three-way interaction between compute, parallelization strategy, and hardware efﬁciency. These results motivate the energy model introduced in Section 3, which explicitly accounts for hardware efﬁciency via ηh . Energy vs. number of GPUs. As shown in Figure 3, increasing the number of GPUs raises total energy consumption despite reducing wall-clock time. This reﬂects higher instantaneous power draw and communication overheads that are not fully offset by speedup. FSDP achieves lower energy than TP at the same device count, while TP exhibits greater variance. 7

The Energy Consumption of Transformer Fine-Tuning

E ergy Co sumptio (kWh)

0.08

framework Baseline FSDP TP

0.06

0.04

0.02

0.00 1

2 Number of GPUs

Figure 3: Training energy consumption as a function of the number of GPUs, shown for the single-GPU baseline, fully sharded data parallelism (FSDP) and tensor parallelism (TP). Energy vs. training duration. Figure 6 in Appendix highlights a clear decoupling between runtime and energy. Faster conﬁgurations often consume more energy, demonstrating that minimizing time-to-solution does not necessarily minimize energy usage. This motivates explicitly modeling hardware efﬁciency rather than relying on runtime as a proxy.

Energy Model Estimation

5.1

Implementation Details

All regressions are performed in log space using ordinary least squares with heteroskedasticity-robust (HC3) standard errors. The reduced-form model in Eq. (13) is estimated with explicit regressors for compute, memory trafﬁc, and hardware efﬁciency. Parallelization strategy effects are modeled using categorical ﬁxed effects, with single-GPU baseline runs serving as the reference category. Train–validation protocol. Conﬁgurations are stratiﬁed by architecture and randomly partitioned into a 70/30 train– validation split. This procedure is repeated over ﬁve random seeds, and we report medians and interquartile ranges (IQR) for all metrics. Model ﬁtting is performed exclusively on the training subset; validation results are strictly out of sample. Primary metrics. We report (i) out-of-sample R2 (R2 ), (ii) RMSE, and (iii) calibration slope and intercept obtained by regressing observed energy on predicted energy in the validation set. We further compute 95% prediction intervals using a residual bootstrap on the training data and evaluate empirical coverage on held-out conﬁgurations. 5.2

Final Estimated Energy Model

Unless otherwise stated, all reported results use the reduced-form model Eq. (13), with the hardware-efﬁciency term ηh parameterized via empirical speedup models (Eq. (8)). Estimated coefﬁcients and conﬁdence intervals are summarized 8

The Energy Consumption of Transformer Fine-Tuning

2 in Table 3 in Appendix C. The model achieves an adjusted coefﬁcient of determination Radj = 0.853 over 410 independent conﬁgurations, indicating strong explanatory power across heterogeneous architectures and parallelization strategies. As illustrated in Figure 4, the model demonstrates high predictive ﬁdelity across three orders of magnitude of energy consumption (10−3 to 10−1 kWh). The observations are tightly clustered around the 1:1 perfect prediction line, conﬁrming that the log-linear formulation effectively captures the underlying physical energy dynamics.

The ﬁnal estimated energy model for training is E ≈ e−10.52 · C 0.20 · M 0.10 · ηh−1.06 · Ψ

(14)

where C denotes total compute (FLOPs), M is the memory-trafﬁc proxy, ηh is the hardware-efﬁciency factor derived from speedup, and Ψ is a strategy-dependent constant. Estim ted vs. Me su.ed Ene.gy Consumption

Es0ima0ed Energy ((Wh)

10−1

st. tegy B seline FSDP TP GPU model NVIDIA GeFo.ce RTX 2080 Ti Tesl V100-PCIE-16GB Q1 dro RTX 6000 Perfec0 Predic0ion (1:1)

10−2

10−3

10−2 Measured Energy ((Wh)

10−1

Figure 4: Estimated vs. Measured energy consumption in kWh. The dashed red line represents the 1:1 ideal prediction. The model maintains high accuracy across diverse GPU architectures and distributed strategies, successfully capturing 2 the scaling behavior of the system (Radj = 0.853). 5.3

Interpretation of Energy Scaling Results

We interpret the estimated coefﬁcients of the energy model in terms of their implications for increases or decreases in training energy. Because the model is speciﬁed in log space, coefﬁcients correspond to elasticities or multiplicative effects and can be interpreted quantitatively, conditional on a ﬁxed workload and hardware class. Effect of compute. The estimated elasticity of energy with respect to total compute is α̂C = 0.20, indicating strongly sub-linear scaling. A 10% increase in compute leads to an approximate 2% increase in energy, while doubling compute increases energy by only about 15%. This behavior suggests that larger workloads beneﬁt from improved utilization and amortization of ﬁxed overheads, consistent with rooﬂine-style efﬁciency effects. Effect of memory trafﬁc. The coefﬁcient on the memory-trafﬁc proxy is positive but modest (α̂M = 0.10). Doubling memory trafﬁc increases energy consumption by approximately 7%, indicating that memory intensity contributes to energy use, but to a lesser extent than compute or efﬁciency effects in the regimes studied. This is consistent with partially memory-bound execution on modern GPUs. 9

The Energy Consumption of Transformer Fine-Tuning

Effect of hardware efﬁciency. Hardware efﬁciency has the strongest impact on energy consumption. The coefﬁcient is α̂η = −1.06, implying that improvements in parallel efﬁciency translate almost directly into energy savings. A 10% increase in efﬁciency reduces energy by approximately 10%, while a twofold increase reduces energy by about 52%. This result empirically conﬁrms the central role of execution-time scaling in determining training energy. Effect of parallelization strategy. Parallelization strategy introduces substantial ﬁxed effects beyond those captured by compute, memory trafﬁc, and efﬁciency. Relative to the single-GPU baseline, both fully sharded data parallelism (FSDP) and tensor parallelism (TP) are associated with large positive multiplicative shifts (Ψ = 1 for the baseline, Ψ ≈ e5.38 for FSDP, and Ψ ≈ e7.65 for TP), reﬂecting additional communication and coordination overheads inherent to distributed execution. Parallelism therefore reduces energy only when efﬁciency gains are sufﬁcient to offset these strategy-speciﬁc costs. Overall, the results show that training energy is far more sensitive to hardware efﬁciency than to raw increases in compute or memory trafﬁc. While larger models and workloads do increase energy consumption, their impact is sublinear and can be mitigated by effective parallelization. In contrast, poorly utilized parallel execution can substantially increase energy despite reduced time-to-solution, underscoring the importance of jointly modeling compute, memory, and efﬁciency when evaluating the energy cost of training.

Validation, Diagnostics, and Ablations

No-ηh baseline. To quantify the value of modeling hardware efﬁciency, we compare a baseline power-law model, E = κ, C αC M αM , against the proposed full model that augments compute and memory with a speedup-derived hardware-efﬁciency term and architecture effects. Including ηh yields a clear improvement in predictive accuracy (Appendix Table 4): R2 increases from 0.7768 to 0.8543 and RMSE decreases by 19%. Calibration remains unchanged (unit slope), indicating reduced variance rather than rescaling. A likelihood-ratio test strongly favors the full model, showing that a substantial portion of energy variation arises from implementation-speciﬁc throughput and parallelization effects beyond compute and memory scaling alone. Estimation on state-of-the-art models. We estimate training energy for several prominent language models (T5, Meena, GShard, Switch Transformer, and GPT-3) under a ﬁxed-data regime that isolates architectural and parallelization effects. Because the original models were trained on TPU clusters, these results should be interpreted as orderof-magnitude consistency checks rather than direct validation. Still, the estimates preserve reported scaling trends and relative efﬁciency differences across strategies, suggesting that the framework captures dominant training-energy drivers in this controlled setting. Results are shown in Table 1, with additional details in Appendix E. Table 1: Comparison of reported and estimated energy consumption for SOTA models. Model

Dev.

Params FLOPs Tokens C_epoch M_proxy Rep. Energy FSDP Est. (MWh) TP Est. (MWh) (B) (ZF) /Epoch (FLOPs) (Proxy) (MWh) Epoch. Tot. Epoch. Tot.

GPT-3 Brown et al. (2020) GShard Lepikhin et al. (2020) Meena Adiwardana et al. (2020) Switch Fedus et al. (2022) T5 Raffel et al. (2020)

OpenAI Google Google Google Google

175.0 619.0 2.6 1500.0 11.0

314.0 133.3 112.0 82.2 40.5

1e14 1e14 1e14 1e14 1e14

1.81e25 2.57e23 3.21e22 1.71e23 3.40e22

2.42e23 5.03e21 6.29e20 3.36e21 1.26e21

1287.0 24.1 232.0 179.0 85.7

0.12 0.04 0.02 0.03 0.02

17.33 1.72 6.90 8.40 2.65

4.88 1.45 0.79 1.29 0.85

696.13 69.08 276.94 337.34 106.47

Discussion

By decomposing training energy into contributions from compute, memory trafﬁc, and execution efﬁciency, we explain why energy scaling departs from FLOPs-only intuition and identify the system-level factors that most strongly inﬂuence training efﬁciency. Our results show that energy inefﬁciency arises not only from compute intensity, but also from memory pressure and parallelization overheads, motivating system-aware energy models rather than purely model-centric ones. Beyond FLOPs and runtime-based scaling. Across all experiments, hardware efﬁciency is the dominant driver of training energy variation. While total compute remains a necessary descriptor of workload size, its effect on energy is strongly sub-linear once utilization is accounted for. Faster conﬁgurations often operate at higher instantaneous power, offsetting runtime reductions and showing that neither FLOPs nor runtime alone is a sufﬁcient proxy for energy cost. Parallelism and diminishing returns. Parallelism introduces a fundamental trade-off between time-to-solution and energy efﬁciency. Although tensor parallelism and fully sharded data parallelism reduce runtime, they also incur coordination, communication, and synchronization overheads that can increase total energy unless parallel efﬁciency remains high. As a result, scaling to larger device counts can be counterproductive from an energy perspective if the chosen strategy is poorly matched to model structure, hardware, or interconnect regime. 10

The Energy Consumption of Transformer Fine-Tuning

Architectural scaling and strategy choice. Fully sharded data parallelism consistently achieves lower training energy than tensor parallelism, particularly for larger and deeper models. This suggests that sharding parameters and optimizer state is often more energy efﬁcient than partitioning computation across devices. These differences amplify with model scale, indicating that architectural scaling and parallelization strategy must be co-designed for energy-efﬁcient training. Limitations. Our empirical evaluation is limited to BERT-style Transformer ﬁne-tuning under controlled architectural sweeps. While the modeling structure is not tied to BERT speciﬁcally, we do not validate it on decoder-only pretraining, mixture-of-experts models, long-context attention variants, or pipeline-parallel execution; results on larger public models are therefore presented as trend-level consistency checks rather than direct validation. The model also relies on coarse-grained proxies for compute and memory demand, chosen as minimal sufﬁcient statistics rather than kernel-level attributions. Accordingly, ﬁtted coefﬁcients and strategy-speciﬁc constants should be interpreted as aggregate effects reﬂecting communication, synchronization, and execution overheads. Finally, the speedup energy linkage assumes operation within a ﬁxed hardware class and operating regime, where average device power varies more slowly than execution time; it does not account for embodied emissions, data-center level effects, or heterogeneous power-management behavior. Broader implications and outlook. Our ﬁndings suggest that reducing the environmental footprint of large-scale training requires a shift from model-centric metrics toward system-aware efﬁciency considerations. Reporting execution time and parallel efﬁciency alongside FLOPs and parameter counts would enable more meaningful comparisons across models and hardware platforms. Future work could extend this framework to multi-node and heterogeneous settings, incorporate ﬁner-grained communication and attention-kernel models, and integrate carbon-intensity aware scheduling for more comprehensive energy-aware system design.

Conclusion

This work shows that the energy cost of Transformer training is governed not only by total compute, but also by hardware efﬁciency and parallelization strategy. While distributed training reduces time-to-solution, it can increase total energy when parallel scaling is imperfect or coordination overheads dominate. By modeling training energy as a function of compute, memory trafﬁc, and a speedup-derived hardware-efﬁciency term, we move beyond FLOPs-only scaling laws and provide an interpretable framework for energy prediction. Our results further show that executiontime scaling can serve as a useful proxy for energy efﬁciency within a ﬁxed hardware class and operating regime, enabling out-of-sample energy estimation even when direct power measurements are unavailable. Overall, the framework suggests that improving parallel efﬁciency and utilization can be as important as reducing nominal compute, offering practical guidance for the energy-aware design and evaluation of distributed Transformer training systems.

Impact Statement This paper studies the operational energy consumption of Transformer training and proposes an interpretable framework for predicting training energy from compute, memory trafﬁc, and execution efﬁciency. A potential positive impact is to support more energy-aware design and reporting of machine learning systems, including better comparison of training conﬁgurations beyond FLOPs or runtime alone. At the same time, this work focuses only on operational training energy and does not account for embodied emissions, electricity mix, or downstream inference costs. As a result, it should not be used in isolation as a complete measure of environmental impact. More generally, improved energy prediction could also make large-scale training easier to optimize, which may reduce energy per run without necessarily reducing total energy use. We therefore view this work as a tool for improving transparency in system-level trade-offs rather than as a complete sustainability assessment.

The Energy Consumption of Transformer Fine-Tuning

Pre-Modeling Exploratory Data Analysis

A.1

Energy Consumption vs. Model Size

Figure 2 illustrates the relationship between training energy consumption and model size for Transformer models under different parallelization strategies and GPU counts. Across all conﬁgurations, energy consumption increases monotonically with the number of parameters, conﬁrming that model size is a ﬁrst-order driver of training energy. However, the slope and dispersion of this increase depend strongly on the parallelization strategy. For a ﬁxed number of GPUs, FSDP consistently exhibits lower energy consumption than TP at comparable model sizes. This gap widens as the model grows, indicating that tensor parallelism incurs increasing overheads such as additional collective communications and reduced kernel efﬁciency that are not fully amortized by larger models. In contrast, FSDP scales more smoothly with parameter count, suggesting better preservation of hardware utilization as model size increases. Increasing the number of GPUs from 2 to 4 raises the absolute energy consumption for both strategies, despite reducing time-to-solution. This highlights a key trade-off: while additional devices improve throughput, they also increase instantaneous power draw and communication costs, leading to higher total energy usage when parallel efﬁciency is imperfect. Notably, the energyparameter relationship is sub-linear on a loglog scale, consistent with the scaling exponent αC < 1 estimated in Section 3.6. This behavior reﬂects the combined effects of rooﬂine-style utilization improvements and the amortization of ﬁxed overheads at larger model sizes. Overall, the ﬁgure underscores that model size alone is insufﬁcient to predict energy consumption; parallelization strategy and hardware efﬁciency play a decisive role in determining the energy cost of training. A.2

Energy Consumption vs. Model Compute

Figure 5 shows how total training energy scales with model compute across parallelization strategies and GPU counts. As expected, energy consumption increases monotonically with the total number of ﬂoating-point operations, conﬁrming compute as a primary driver of training energy. However, the relationship is markedly sub-linear and exhibits substantial dispersion, indicating that FLOPs alone are insufﬁcient to predict energy usage. For a ﬁxed compute budget, FSDP consistently consumes less energy than TP across both 2-GPU and 4-GPU conﬁgurations. This gap becomes more pronounced at higher compute levels, suggesting that tensor parallelism incurs increasing coordination and communication overheads that are not fully amortized as compute grows. In contrast, FSDP better preserves hardware utilization by reducing redundant parameter storage and communication, leading to improved energy efﬁciency. Increasing the number of GPUs from 2 to 4 shifts the curves upward for both strategies, despite reducing wall-clock training time. This highlights a key system-level trade-off: additional devices improve throughput but increase instantaneous power draw and synchronization overhead, resulting in higher total energy when parallel efﬁciency is imperfect. Finally, the curvature observed on a log–log scale aligns with the sub-linear compute elasticity estimated in Section 3.6. This behavior is consistent with rooﬂine-style utilization effects and reinforces the necessity of explicitly modeling hardware efﬁciency via ηh to explain energy scaling beyond raw compute counts. A.3

Energy Consumption vs. Training Duration

Figure 6 compares training duration across baseline, fully sharded data parallelism (FSDP), and tensor parallelism (TP) as the number of GPUs increases. As expected, increasing the number of GPUs generally reduces training duration for both parallel strategies, reﬂecting improved throughput and higher aggregate compute capacity. However, the reduction in duration does not translate proportionally into lower energy consumption. While FSDP achieves the shortest median durations for 2 and 4 GPUs, TP exhibits substantially longer and more variable runtimes, particularly at higher GPU counts. This variability indicates sensitivity to communication patterns and synchronization overheads introduced by tensor parallelism. Crucially, these results highlight that minimizing time-to-solution is not equivalent to minimizing energy. Faster conﬁgurations often operate at higher instantaneous power due to increased device utilization and communication activity, which can offset or even outweigh the beneﬁts of reduced runtime. This decoupling between duration and 12

The Energy Consumption of Transformer Fine-Tuning

0.05

gpus = 2

gpus = 4

Framework FSDP TP

Framework FSDP TP 0.08

Total E erg( (kWh)

0.04

0.06

0.03

0.04

0.02

0.01

0.00

0.00 0.0

0.5

1.0 1.5 2.0 Model Compute (FLOPs)

2.5

3.0

1e14

4 6 Model Compute (FLOPs)

1e14

Figure 5: Training energy consumption as a function of total model compute (FLOPs) for fully sharded data parallelism (FSDP) and tensor parallelism (TP), shown separately for 2 and 4 GPUs.

energy motivates the explicit inclusion of a hardware-efﬁciency factor in our energy model, as duration alone cannot capture the full cost of parallel execution. Overall, Figure 6 reinforces the need to jointly consider runtime, power draw, and parallel efﬁciency when evaluating the sustainability of training conﬁgurations. Energy vs. training duration. Figure 6 highlights a clear decoupling between runtime and energy. Faster conﬁgurations often consume more energy, demonstrating that minimizing timeto-solution does not necessarily minimize energy usage. This motivates explicitly modeling hardware efﬁciency rather than relying on runtime as a proxy.

0.05

Energy C nsumpti n (kWh)

0.04

gpus = 2

gpus = 4

Framew rk FSDP TP FSDP TP

0.08

Framew rk FSDP TP FSDP TP

0.06

0.03

0.04

0.02

0.01

0.00

0.00 50

100

150

200 250 300 duration (in sec nds)

350

400

450

100

200

300 400 500 durati n (in sec nds)

600

700

Figure 6: Distribution of training duration (seconds per epoch) across parallelization strategies and GPU counts. B

The Energy Consumption of Transformer Fine-Tuning

A.4

Energy Consumption vs. Computational Resources

Figure 3 illustrates how total energy consumption varies with the number of GPUs for different parallelization strategies. Moving from a single GPU to multiple GPUs increases total energy consumption across all conﬁgurations, even though multi-GPU training substantially reduces wall-clock time. This conﬁrms that additional computational resources incur higher instantaneous power draw and coordination overheads that are not fully offset by speedup gains. Among multi-GPU strategies, FSDP consistently exhibits lower median energy consumption than tensor parallelism at the same GPU count. This difference becomes more pronounced at four GPUs, where TP shows both higher medians and substantially larger variance. The increased dispersion under TP reﬂects sensitivity to communication patterns and reduced kernel efﬁciency as tensors are partitioned across devices. Notably, the single-GPU baseline remains the most energy-efﬁcient conﬁguration in absolute terms for small- and medium-scale models, despite being the slowest in runtime. This highlights a central trade-off in distributed training: optimizing for time-to-solution does not necessarily minimize energy consumption. Overall, these results emphasize that the energy cost of training grows with the number of devices unless parallel efﬁciency is sufﬁciently high. This motivates the explicit inclusion of a device-countdependent hardware-efﬁciency factor ηh in our energy model to capture diminishing returns from additional GPUs. A.5

Energy Consumption vs. Number of Layers

Figure 7 examines how training energy scales with model depth under different parallelization strategies and GPU counts. For all conﬁgurations, energy consumption increases monotonically with the number of layers, conﬁrming depth as a direct driver of training cost through its linear effect on both parameter count and total compute. At ﬁxed depth, increasing the number of GPUs leads to higher total energy consumption across all strategies. Although multi-GPU execution reduces wall-clock time, the increase in instantaneous power draw and synchronization overheads outweighs these gains, resulting in higher overall energy usage. This effect becomes more pronounced as depth increases, reﬂecting the compounding impact of deeper models on communication and memory trafﬁc. Across all depths and GPU counts, FSDP consistently achieves lower energy consumption than tensor parallelism. The gap between FSDP and TP widens for deeper models, indicating that TP incurs depth-dependent overheads such as per-layer collective operations and reduced kernel efﬁciency that scale poorly with model depth. In contrast, FSDP better amortizes communication costs by sharding parameters and optimizer states, leading to more stable energy scaling. Notably, for shallow models (4–6 layers), the single-GPU baseline remains the most energy-efﬁcient conﬁguration in absolute terms. As model depth increases, however, the baseline becomes increasingly impractical due to long runtimes, making multi-GPU strategies necessary despite their higher energy cost. This highlights a fundamental trade-off between feasibility and energy efﬁciency in training deep models. Overall, Figure 7 reinforces that model depth, parallelization strategy, and device count interact non-trivially to determine energy consumption. These results motivate modeling depth-driven compute separately from hardware efﬁciency effects, as captured by the ηh factor in our energy model.

Estimated Compute–Memory Energy Model

This appendix reports the estimated parameters of the linear compute–memory energy decomposition introduced in Eq. (4). The model expresses per-step training energy as an additive combination of compute cost,memory-trafﬁc cost, and ﬁxed overheads, and serves as a mechanistic baseline for the reduced-form scaling laws used in the main text. Table 2: Estimated coefﬁcients of the compute–memory energy model (Eq. (4)) using HC3 robust standard errors. Variable E0 (Intercept) Strategy: FSDP Strategy: TP Compute (C, centered) Memory (Mproxy , centered)

Coef.

Std. Err.

P > |z|

95% CI

0.0028 0.0041 0.0163 0.0027 3.0 × 10−4

0.0010 0.0010 0.0012 0.0010 4.4 × 10−5

3.72 4.59 14.07 4.38 5.96

0.000 0.000 0.000 0.000 0.000

[0.001, 0.004] [0.002, 0.006] [0.014, 0.019] [0.001, 0.004] [2.1 × 10−4 , 3.9 × 10−4 ]

The Energy Consumption of Transformer Fine-Tuning

4 ayers

6 ayers

Energy Consumption (kWh)

0.025

8 ayers

0.035

0.025

0.030

0.020

0.025

0.015

0.020

0.015

0.010

0.005

10 ayers

12 ayers

0.030

0.035

0.025

0.030

frame)ork Base ine FSDP TP

16 ayers

0.05

Energy Cons(mption (kWh)

0.04

0.025

0.020

0.03

0.020

0.015

0.02 0.015

0.010 0.010

0.01

0.005 1

2 N(mber of GPUs

Figure 7: Distribution of training energy consumption as a function of model depth (number of Transformer layers) and GPU count. Each panel corresponds to a ﬁxed depth; box plots show medians and interquartile ranges across baseline, fully sharded data parallelism (FSDP), and tensor parallelism (TP).

The results conﬁrm that both compute and memory trafﬁc contribute positively and signiﬁcantly to training energy, consistent with the additive structure of the model. The memory-trafﬁc proxy exhibits a statistically signiﬁcant coefﬁcient, indicating that off-chip data movement contributes non-negligibly to energy consumption even after controlling for total compute. Parallelization strategy introduces substantial additive offsets. Relative to the single-GPU baseline, both fully sharded data parallelism (FSDP) and tensor parallelism (TP) are associated with higher per-step energy, reﬂecting additional communication, synchronization, and coordination overheads that are not captured by raw compute or memory proxies alone. The substantially larger coefﬁcient for TP is consistent with its heavier reliance on ﬁne-grained collective communication. While the compute–memory model explains a majority of the variance in per-step energy, its adjusted R2 of 0.694 is signiﬁcantly lower than that of the reduced-form model incorporating hardware efﬁciency. This gap motivates the introduction of the speedup-based hardware-efﬁciency factor ηh in the main text, which captures execution-time effects and parallel efﬁciency beyond static compute and memory costs.

C Final Estimated Energy Model Unless otherwise stated, all reported results use the reduced-form model Eq. (13), with the hardware-efﬁciency term ηh parameterized via empirical speedup models (Eq. (8)). Estimated coefﬁcients and conﬁdence intervals are summa2 rized in Table 3. The model achieves an adjusted coefﬁcient of determination Radj = 0.853 over 410 independent 15

The Energy Consumption of Transformer Fine-Tuning

conﬁgurations, indicating strong explanatory power across heterogeneous architectures and parallelization strategies. As illustrated in Figure 4, the model demonstrates high predictive ﬁdelity across three orders of magnitude of energy consumption (10−3 to 10−1 kWh). The observations are tightly clustered around the 1:1 perfect prediction line, conﬁrming that the log-linear formulation effectively captures the underlying physical energy dynamics. The estimated coefﬁcients reveal a strongly sub-linear dependence of energy on total compute, with elasticity α̂C = 0.196This reduced elasticity reﬂects the fact that a substantial portion of energy scaling with workload size is mediated through changes in hardware utilization, which are explicitly captured by the parallel-efﬁciency term. Once utilization is accounted for, the residual sensitivity of energy to raw FLOP count is markedly attenuated. Memory trafﬁc also contributes positively to energy consumption. The coefﬁcient on log(Mproxy ) is positive and statistically signiﬁcant, indicating that conﬁgurations with larger activation footprints incur higher energy costs, consistent with memory-bound regimes predicted by rooﬂine analysis. Hardware efﬁciency plays a dominant role. The coefﬁcient on log(parallel efﬁciency) is α̂η = −1.06, implying that improvements in parallel efﬁciency translate directly into reduced energy consumption. This result empirically validates Lemma 3.3 and conﬁrms that execution-time scaling is a ﬁrst-order determinant of training energy. Parallelization strategy effects are substantial. Relative to the single-GPU baseline, both fully sharded data parallelism (FSDP) and tensor parallelism (TP) exhibit signiﬁcant upward shifts in energy consumption when controlling for compute, memory, and efﬁciency. Speciﬁcally, the estimated multiplicative factors are ΨFSDP ≈ exp(5.38),ΨTP ≈ exp(7.65), reﬂecting the additional communication and coordination overheads introduced by distributed execution beyond what is captured by speedup alone. These effects highlight the importance of explicitly modeling strategydependent constants in energy prediction. Table 3: Estimated coefﬁcients for the Transformer energy model. The last two columns report 95% conﬁdence 2 intervals. The model attains Radj = 0.853 over 410 conﬁgurations.

Variable Intercept Strategy: FSDP Strategy: TP log(compute) log(Mproxy ) log(parallel efﬁciency)

Coef. -10.5283 5.3839 7.6487 0.1955 0.0980 -1.0601

Std. Err. 0.449 0.466 0.581 0.025 0.037 0.099

z -23.472 11.556 13.165 7.881 2.637 -10.678

P> |z| 0.000 0.000 0.000 0.000 0.008 0.000

[0.025 -11.407 4.471 6.510 0.147 0.025 -1.255

0.975] -9.649 6.297 8.787 0.244 0.171 -0.865

Notes: Standard errors are HC3 robust. Continuous variables are mean-centered. Adjusted R2 = 0.8525; Durbin-Watson = 2.125; Condition Number = 86.5.

D Model Diagnostics and Robustness No-ηh baseline. To isolate the contribution of hardware-efﬁciency modeling, we compare a baseline power-law model, E = κ C αC M αM , to the proposed full speciﬁcation that augments compute and memory with a speedupderived hardware-efﬁciency term and architecture-speciﬁc effects. As shown in Appendix Table 4, including ηh substantially improves predictive performance: explained variance increases from R2 = 0.7768 to R2 = 0.8543 (+ 0.078), while RMSE decreases by 0.091kwh (a 19% reduction). Calibration remains unchanged, with a unit slope in both models, indicating that the gains arise from reduced dispersion rather than rescaling. A likelihood-ratio test strongly favors the full model, demonstrating that a substantial fraction of energy variation is driven by implementationspeciﬁc throughput and parallelization effects rather than compute and memory scaling alone. To validate the reliability of our energy scaling laws, we conducted a multi-dimensional diagnostic analysis of the Ordinary Least Squares (OLS) residuals. The results are reported in ﬁgures 8. Linearity and Functional Form: The Residuals vs. Fitted plot reveals a slight U-shaped curvature in the LOWESS trend line. While the log-log speciﬁcation captures the primary scaling trend (Adj. R2 = 0.853), these marginal deviations at the extreme ends of the energy spectrum suggest that the scaling exponent (α) may undergo subtle shifts as models transition between small-scale and large-scale regimes. Residual diagnostics To evaluate the reliability of the energy scaling law, a comprehensive diagnostic analysis was performed on the model’s residuals, as illustrated in Figure 8 in appendix. The model demonstrates a robust log-log ﬁt 16

The Energy Consumption of Transformer Fine-Tuning

Table 4: Model comparison: baseline power law vs. full energy model

Metric Power Law (A) Full Model (B) R2 0.7768 0.8543 RMSE (kwh) 0.4761 0.3846 Calibration slope 1.0000 1.0000 Likelihood-ratio test statistic: 174.90 LRT p-value: 6.30 × 10−40

∆ (B–A) +0.0775 −0.0914 0.0000

with an Adjusted of 0.853 and a Durbin-Watson statistic of 2.125, conﬁrming the absence of ﬁrst-order autocorrelation. While the Residuals vs. Fitted plot reveals minor non-linearity at the extreme ends of the compute scale and the Normal Q-Q plot indicates heavy-tailed residuals typical of hardware power spikes, the error distribution remains stable across different training strategies and architectural proxies. Feature-Speciﬁc Bias: The Residuals vs. log Compute and Residuals vs. log Mproxy (Centered) plots show relatively balanced dispersion across the centered compute range. However, a minor upward trend in residuals is observed for the highest model-size proxies, indicating that for exceptionally large architectural dimensions, the model may slightly underestimate energy consumption. Strategy-Level Stability: The Residuals by Strategy boxplot demonstrates that prediction error is consistent across Baseline, FSDP, and TP. While distributed strategies exhibit slightly higher variance and a few isolated outliers, the median residual for each category remains centered at zero, justifying our use of strategy-speciﬁc intercept shifts to account for parallelism overhead. Error Structure and Normality: The Normal Q-Q Plot displays heavy-tailed residuals (leptokurtosis), where extreme energy outcomes deviate from the Gaussian line at the tails. This is consistent with hardware benchmarking data subjected to transient power spikes. Furthermore, a Durbin-Watson statistic of 2.125 conﬁrms that our aggregation strategy successfully eliminated ﬁrst-order autocorrelation. Inﬂuence and Numerical Stability: The Cook’s Distance plot identiﬁes two speciﬁc clusters of inﬂuential observations (near indices 60 and 265). Despite these clusters, all distances remain below 0.08, well within safe thresholds for regression stability. Finally, mean-centering continuous variables reduced the Condition Number to 86.5, ensuring that the effects of compute and parallel efﬁciency are numerically decoupled and individually interpretable.

Energy Estimation for State-of-the-Art Models

Table 5 reports a comparison between energy consumption ﬁgures reported in the literature for several state-of-the-art language models and energy estimates obtained using our proposed framework. The models considered include GPT3, GShard, Meena, Switch Transformer, and T5, covering a wide range of parameter counts, architectural depths, and training regimes. Reconstruction methodology. For each model, we reconstruct the quantities required by our energy model from publicly available information, including parameter count, number of layers L, hidden width W , and reported training compute. To enable a controlled comparison across heterogeneous systems, we evaluate all models under a ﬁxed-data regime, holding the number of training tokens per epoch constant (1014 tokens) and ﬁxing the number of GPUs to 100. From these quantities, we derive the per-epoch compute Cepoch and a memory-trafﬁc proxy Mproxy based on activation volume. Estimated energy under parallel strategies. Using the reconstructed inputs, we estimate energy consumption per epoch and total energy under two parallelization strategies: fully sharded data parallelism (FSDP) and tensor parallelism (TP). Estimates are produced by applying the reduced-form energy model with the corresponding speedupbased hardware-efﬁciency factors. Reported energy values from the original publications are shown for reference. 17

The Energy Consumption of Transformer Fine-Tuning

Interpretation and limitations. Absolute energy values are not expected to match reported ﬁgures, as the original models were trained on large-scale TPU or GPU clusters with different accelerators, interconnects, and system-level overheads. Accordingly, the estimates in Table 5 should be interpreted as order-of-magnitude reconstructions rather than replications. Despite these differences, the estimates preserve qualitative trends across models and parallelization strategies. In particular, the results highlight the strong sensitivity of energy consumption to parallel efﬁciency: for a ﬁxed token budget, TP consistently yields higher estimated energy than FSDP, reﬂecting the dominant role of execution-time scaling captured by the hardware-efﬁciency term. Overall, this analysis demonstrates how the proposed framework can be used to extrapolate training energy for large models from limited experimental calibration, providing a practical tool for comparative and predictive energy analysis when direct measurements are unavailable. Table 5: Comparison of reported and estimated energy consumption for SOTA models. Model

Dev.

Params FLOPs (B) (ZF)

GPT-3 Brown et al. (2020) GShard Lepikhin et al. (2020) Meena Adiwardana et al. (2020) Switch Fedus et al. (2022) T5 Raffel et al. (2020)

OpenAI Google Google Google Google

175.0 619.0 2.6 1500.0 11.0

96 12288 48 2048 24 1024 32 2048 24 1024

314.0 133.3 112.0 82.2 40.5

GPUs Tokens C_epoch M_proxy Rep. Energy FSDP Est. (MWh) TP Est. (MWh) /Epoch (FLOPs) (Proxy) (MWh) Epoch. Tot. Epoch. Tot. 100 100 100 100 100

1e14 1e14 1e14 1e14 1e14

1.81e25 2.57e23 3.21e22 1.71e23 3.40e22

2.42e23 5.03e21 6.29e20 3.36e21 1.26e21

1287.0 24.1 232.0 179.0 85.7

0.12 0.04 0.02 0.03 0.02

17.33 1.72 6.90 8.40 2.65

4.88 1.45 0.79 1.29 0.85

696.13 69.08 276.94 337.34 106.47

The Energy Consumption of Transformer Fine-Tuning

Proofs

F.1 Detailed Proof of Proposition 3.1 Proposition F.1 (Rooﬂine energy lower bounds). Let C be the number of FLOPs executed in one training step, M the number of DRAM/HBM bytes transferred, AI ≜ C/M the arithmetic intensity, Fpeak the device peak FLOP/s, B the sustained memory bandwidth (bytes/s), and let Pdyn denote the dynamic power during the active portion of the step. Then the step time T satisﬁes C , T ≥ min{Fpeak , B AI} and consequently the energy satisﬁes E ≥

Pdyn C Fpeak

E ≥

and

Pdyn M. B

Proof. (Throughput bound.) The rooﬂine model states that the achieved compute throughput is bounded by Perf ≤ min{Fpeak , B AI} ≜ Feﬀ . The term Fpeak is a hard upper bound imposed by the devices functional units and clock frequency, while the term B AI reﬂects the maximum sustained compute rate achievable when memory bandwidth limits execution. (Time lower bound.) By deﬁnition, Perf = C/T , hence C/T ≤ Feﬀ , which implies T ≥

C C = . Feﬀ min{Fpeak , B AI}

RT (Energy lower bounds.) Energy is given by E = 0 P (t) dt. During the active portion of the step, the power draw is at least a positive dynamic level Pdyn (idle phases can only increase T for ﬁxed C). Therefore, E ≥ Pdyn T ≥ Pdyn

C . min{Fpeak , B AI}

Using min{a, b} ≤ a and min{a, b} ≤ b with a = Fpeak and b = B AI yields E ≥

Pdyn C Fpeak

and

E ≥

Pdyn Pdyn C = M. B AI B

F.2 Detailed Proof of Theorem 3.2 Theorem F.2 (Expected energy scaling bands). Assume (i) quasi-stationary power during a training step; (ii) a stable operation mix as hyperparameters vary; and (iii) over the hyperparameter ranges studied there exist constants k1 , k2 > 0 and an exponent β ∈ [ 12 , 1) such that the dominant DRAM trafﬁc co-scales with compute as k1 C β ≤ M ≤ k2 C β . Then, for device-speciﬁc constants ϵcomp , ϵmem > 0 and a ﬁxed overhead E0 ≥ 0, min{ϵmem M, ϵcomp C} ≲ E ≲ ϵcomp C + ϵmem M + E0 . Moreover, over any range where E0 is negligible, the effective log–log slope of E with respect to C lies in (β, 1), yielding the reduced form α E ≈ κ C αC N αN ηh η , with αC ∈ ( 21 , 1], αη < 0. Proof. (Sandwich bounds.) Let a ≜ ϵcomp and b ≜ ϵmem . Using the upper co-scaling bound on M , E ≤ a C + b M + E0 ≤ a C + b k 2 C β + E0 . Using the lower co-scaling bound together with Proposition 3.1, Pdyn Pdyn E ≥ max C, M ≥ max a′ C, b′ C β , Fpeak B 19

The Energy Consumption of Transformer Fine-Tuning

dyn where a′ = Fpeak and b′ =

Pdyn B k1 are positive constants. Hence, for C in the experimental range,

max{a′ C, b′ C β } ≤ E ≤ a C + b k2 C β + E0 .

(15)

(Log–log slope bounds.) Consider f (C) = a C + b̃ C β + E0 with b̃ = b k2 > 0 and 0 < β < 1. The instantaneous log–log slope is d log f (C) Cf ′ (C) a C + b̃ β C β s(C) ≜ = = . d log C f (C) a C + b̃ C β + E0 All terms are positive, hence s(C) ∈ (0, 1). When C is large enough that E0 is negligible, s(C) lies strictly between β and 1, approaching β in the memory-dominated regime and 1 in the compute-dominated regime. Therefore, any log– log regression of E on C over a compact range with a non-degenerate mix yields an effective exponent αC ∈ (β, 1). (From β to ( 12 , 1].) Architectural co-scalings determine β. For RNNs (LSTM/GRU), C = Θ(BT h2 ) while M = √ Θ(BT h) (dominant state and activation trafﬁc), implying M = Θ( C) and β = 12 . For Transformers, compute grows as Θ(BL2 d) + Θ(BLd2 ) while M = Θ(BLd) (activations), yielding M = Θ(C β ) with β ∈ ( 21 , 1) over typical depth and width sweeps. Consequently, αC ∈ ( 12 , 1] across these model families. (Incorporating N and ηh .) Dependence on model size N and hardware efﬁciency ηh introduces multiplicative factors that vary slowly relative to C. In log space, these contribute additive terms αN log N and αη log ηh . Since higher ηh (greater utilization) reduces time and thus energy for ﬁxed work, one has αη < 0, yielding the stated reduced form. F.3 Detailed Proof of Lemma 3.3 Lemma F.3 (Transferability of speedup to energy efﬁciency). Consider a ﬁxed training workload executed on N devices, and let T (N ) denote the corresponding execution time. Let P (t) be the instantaneous power draw and R T (N ) E(N ) = 0 P (t) dt the total energy consumed. Assume that: (A1) The workload is ﬁxed across device counts, i.e., the total amount of computation and memory trafﬁc does not change with N . R T (N ) (A2) The average power draw P̄ (N ) ≜ T (N )−1 0 P (t) dt is a slowly varying function of N and is bounded above by a constant multiple of the single-device power draw. (A3) Changes in execution time dominate changes in average power as N varies. Then the energy scaling with respect to N is primarily governed by the execution time scaling, and the parallel speedup S(N ) = T (1)/T (N ) induces a corresponding hardware-efﬁciency factor S(N ) . N In particular, any parametric model that accurately captures T (N ) (and hence S(N )) provides a consistent proxy for the dependence of energy consumption on parallelization. ηh (N ) ∝

Proof. Fix the workload and consider any device count N ≥ 1. Deﬁne the average power Z T (N ) 1 P̄ (N ) ≜ P (t) dt, T (N ) 0 so that, by the deﬁnition of an average,

Z T (N ) P (t) dt = P̄ (N ) T (N ).

E(N ) =

(16)

Equation (16) is an identity and holds without assumptions. Step 1: Relating energy ratios to time ratios.

Consider the energy ratio between N devices and a single device:

P̄ (N ) T (N ) P̄ (N ) E(N ) P̄ (N ) T (N ) 1 = = . = · · E(1) P̄ (1) T (1) P̄ (1) T (1) P̄ (1) S(N )

(17)

Thus, for any N , energy scaling is exactly determined by the product of a power ratio and an inverse speedup factor. 20

The Energy Consumption of Transformer Fine-Tuning

Step 2: Bounding and controlling the power term. Assumption (A2) states that P̄ (N ) is slowly varying in N and is bounded above by a constant multiple of P̄ (1). Concretely, this implies the existence of a constant c ≥ 1 such that for all N in the regime of interest, P̄ (N ) 1/c ≤ ≤ c, (18) P̄ (1) possibly after restricting to a ﬁnite range of N used in experiments. (The lower bound is not essential for the conclusion, but it simpliﬁes the interpretation: average power cannot collapse to zero as N grows under ﬁxed workload execution.) Combining (17) and (18) yields 1 1 E(N ) 1 · ≤ ≤ c· . (19) c S(N ) E(1) S(N ) Therefore, up to a multiplicative constant that does not scale rapidly with N , energy is inversely proportional to the speedup. Step 3: Formalizing “time dominates power”. Assumption (A3) states that changes in execution time dominate changes in average power as N varies. One formal way to express this is via log-derivatives (elasticities): d log P̄ (N ) d log T (N ) ≪ , (20) d log N d log N over the device-count range considered. Using (16), d log E(N ) d log P̄ (N ) d log T (N ) = + . d log N d log N d log N Under (20), the ﬁrst term is negligible relative to the second, hence d log E(N ) d log T (N ) ≈ , d log N d log N which makes precise the statement that energy scaling is primarily governed by time scaling. Equivalently, combining (A2) and (A3) implies that, to leading order, E(N ) ≈ P̄ (1) T (N ) and thus E(N ) ∝ T (N ), where the proportionality absorbs the slowly varying power factor.

(21)

Step 4: Induced hardware-efﬁciency factor. Deﬁne the ideal (perfectly efﬁcient) time on N devices as T (1) Tideal (N ) ≜ , N corresponding to linear speedup and no parallel overhead. A standard notion of parallel efﬁciency is then T (1)/N S(N ) Tideal (N ) = = . ηpar (N ) ≜ T (N ) T (N ) N This quantity equals 1 under perfect scaling and decreases when communication, synchronization, or load imbalance reduce speedup. Since (21) gives E(N ) ∝ T (N ) up to a slowly varying factor, normalizing energy by the ideal scaling yields the same efﬁciency structure. In particular, relative deviations from ideal energy scaling are captured (up to constants from P̄ (N )/P̄ (1)) by ηpar (N ). Thus we may deﬁne a device-count hardware-efﬁciency factor S(N ) ηh (N ) ∝ ηpar (N ) = , N where the proportionality constant can absorb the bounded power-ratio term in (17). Step 5: Transferability from time models. Finally, let Tb(N ) be any parametric model that accurately predicts execution time (or speedup) over the device counts considered. Then the induced efﬁciency proxy b b ) S(N b ) = T (1) , , with S(N ηbh (N ) ≜ N Tb(N ) provides a consistent approximation to ηh (N ) in the sense that it captures the leading-order dependence of energy on N via (21). Therefore, time-based benchmark results can be transferred into the energy model through ηh (N ), completing the proof.

The Energy Consumption of Transformer Fine-Tuning

Model Conﬁgurations

In our experiments, we train Transformer models for a ﬁxed number of ﬁve epochs across a broad range of architectural and training hyperparameters in order to systematically characterize their computational and energy behavior. We vary batch size and key architectural parametersincluding model depth, hidden dimensionality, feedforward width, and the number of attention headsto capture how design choices affect training dynamics, hardware utilization, and energy consumption. Fixing the number of training epochs across all conﬁgurations ensures a fair and controlled comparison between models. This design isolates the effects of architectural scaling and hyperparameter selection on energy efﬁciency, independent of convergence speed or ﬁnal task performance. As a result, differences in measured energy consumption can be attributed primarily to changes in compute intensity, memory trafﬁc, and parallel utilization rather than optimization dynamics. This controlled setup enables a systematic analysis of trade-offs between training time, computational cost, and resource utilization. In particular, it allows us to study how increases in model capacity and throughput translate into practical energy costs under realistic training workloads. The resulting measurements provide a comprehensive empirical basis for the energy models developed in the main text. The hyperparameter ranges explored in our experiments are summarized in Table 6. Table 6: Transformer model hyperparameters and explored ranges. Parameter Batch size Number of layers (ℓ) Model dimension (dmodel ) Number of attention heads (nheads ) Feedforward dimension (dﬀ )

Range and purpose 64 to 640, to study the impact of throughput and data parallelism on training energy 2 to 12, to evaluate the effect of model depth on compute, memory trafﬁc, and energy scaling 128 to 1280, corresponding to the dimensionality of embeddings and internal representations 2 to 12, to analyze the energy implications of multi-head self-attention 512 to 4096, controlling the width of Transformer MLP blocks and their compute and memory intensity

References Adiwardana, D., Luong, M.-T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y., et al. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977, 2020. Alvarez, A., Genuardi, A., Sinha, N., Tiene, A., Okyay, M., Ryskulov, B., Montero, D., Mugel, S., and Orús, R. Scaling laws for energy efﬁciency of local llms, 2025. URL https://arxiv.org/abs/2512.16531. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901, 2020. Choi, J. W., Bedard, D., Fowler, R., and Vuduc, R. A rooﬂine model of energy. In 2013 IEEE 27th International Symposium on Parallel and Distributed Processing, pp. 661–672, 2013. doi:10.1109/IPDPS.2013.77. Cottier, B., Rahman, R., Fattorini, L., Maslej, N., and Owen, D. The rising costs of training frontier ai models. arXiv preprint arXiv:2405.21015, 2024. Czaja, J., Gallus, M., Wozna, J., Grygielski, A., and Tao, L. Applying the rooﬂine model for deep learning performance optimizations. CoRR, abs/2009.11224, 2020. URL https://arxiv.org/abs/2009.11224. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp. 4171–4186, 2019. Faiz, A., Kaneda, S., Wang, R., Osi, R. C., Sharma, P., Chen, F., and Jiang, L. LLMCarbon: Modeling the end-to-end carbon footprint of large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=aIok3ZD9to. Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and efﬁcient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022. Henderson, P., Hu, J., Romoff, J., Brunskill, E., Jurafsky, D., and Pineau, J. Towards the systematic reporting of the energy and carbon footprints of machine learning. Journal of Machine Learning Research, 21(248):1–43, 2020. 22

The Energy Consumption of Transformer Fine-Tuning

Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th annual international symposium on computer architecture, pp. 1–12, 2017. Kaplan, J., McCandlish, S., OpenAI, T. H., OpenAI, T. B. B., OpenAI, B. C., OpenAI, R. C., OpenAI, S. G., OpenAI, A. R., OpenAI, J. W., and OpenAI, D. A. Scaling laws for neural language models, 2020. Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020. Luccioni, A. S., Viguier, S., and Ligozat, A.-L. Estimating the carbon footprint of bloom, a 176b parameter language model. Journal of Machine Learning Research, 24(253):1–15, 2023. Luccioni, S., Jernite, Y., and Strubell, E. Power hungry processing: Watts driving the cost of ai deployment? In The 2024 ACM Conference on Fairness, Accountability, and Transparency, pp. 85–99, 2024. Mehlin, V., Schacht, S., and Lanquillon, C. Towards energy-efﬁcient deep learning: An overview of energy-efﬁcient approaches along the deep learning lifecycle. arXiv preprint arXiv:2303.01980, 2023. Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., et al. Efﬁcient large-scale language model training on gpu clusters using megatronlm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1–15, 2021. Nichols, D., Singh, S., Lin, S.-H., and Bhatele, A. A survey and empirical evaluation of parallel deep learning frameworks, 2022. URL https://arxiv.org/abs/2111.04949. Ogbogu, C., Abernot, M., Delacour, C., Todri-Sanial, A., Pasricha, S., and Pande, P. P. Energy-efﬁcient machine learning acceleration: From technologies to circuits and systems. In 2023 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED), pp. 1–8, 2023. doi:10.1109/ISLPED58423.2023.10244360. Owens, J. D., Luebke, D. P., Govindaraju, N. K., Harris, M. J., Krüger, J. H., Lefohn, A. E., and Purcell, T. J. A survey of generalpurpose computation on graphics hardware. Computer Graphics Forum, 26, 2007. URL https: //api.semanticscholar.org/CorpusID:15634829. Owens, J. D., Houston, M., Luebke, D., Green, S., Stone, J. E., and Phillips, J. C. Gpu computing. Proceedings of the IEEE, 96(5):879–899, 2008. doi:10.1109/JPROC.2008.917757. Patterson, D., Gonzalez, J., Le, Q., Liang, C., Munguia, L.-M., Rothchild, D., So, D., Texier, M., and Dean, J. Carbon emissions and large neural network training. arXiv preprint arXiv:2104.10350, 2021. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a uniﬁed text-to-text transformer. Journal of machine learning research, 21(140): 1–67, 2020. Strubell, E., Ganesh, A., and McCallum, A. Energy and policy considerations for modern deep learning research. In Proceedings of the AAAI conference on artiﬁcial intelligence, volume 34, pp. 13693–13696, 2020. Wang, Y., Yang, C., Farrell, S., Zhang, Y., Kurth, T., and Williams, S. Time-based rooﬂine for deep learning performance analysis. In 2020 IEEE/ACM Fourth Workshop on Deep Learning on Supercomputers (DLS), pp. 10–19, 2020. doi:10.1109/DLS51937.2020.00007. Williams, S., Waterman, A., and Patterson, D. Rooﬂine: an insightful visual performance model for multicore architectures. Commun. ACM, 52(4):6576, April 2009. ISSN 0001-0782. doi:10.1145/1498765.1498785. URL https://doi.org/10.1145/1498765.1498785. Wu, C.-J., Raghavendra, R., Gupta, U., Acun, B., Ardalani, N., Maeng, K., Chang, G., Aga, F., Huang, J., Bai, C., et al. Sustainable ai: Environmental implications, challenges and opportunities. Proceedings of Machine Learning and Systems, 4:795–813, 2022. Yarally, T., Cruz, L., Feitosa, D., Sallou, J., and Van Deursen, A. Uncovering energy-efﬁcient practices in deep learning training: Preliminary steps towards green ai. In 2023 IEEE/ACM 2nd International Conference on AI Engineering–Software Engineering for AI (CAIN), pp. 25–36. IEEE, 2023. Zoubeirou a Mayaki, M. and Charpenay, V. Modeling Energy Consumption in Deep Learning Architectures Using Power Laws. In Frontiers in Artiﬁcial Intelligence and Applications, Frontiers in Artiﬁcial Intelligence and Applications, Bologne, Italy, October 2025. IOS Press. doi:10.3233/FAIA250900. URL https://hal.science/ hal-04977474.

The Energy Consumption of Transformer Fine-Tuning

Resid1als vs log Compute

Resid1als vs Fitted (log E) 1.0

0.5

0.5 None

1.0

0.0

−0.5

−8

−7

−6

Fitted log E

−5

−4

−3

−6

−4

−2 0 )og_com−u0e_cen0ered

Resid1als vs log M_proxy

Resid1als by Str tegy

0.5

0.5 None

1.0

None

1.0

0.0

−0.5

−3

−2

−1 0 1 log_M_−ro3y_cen0ered Norm l Q-Q Plot

B seline

FSDP str tegy

Cook's Dis0 nce (Leverage) 0.07

Sam−le Qu ntiles

1.0

0.06 0.05

0.5

0.04 0.0

0.0

0.02

−0.5

0.01

−1.0

0.00

−2

−1

0 Theore0ical Qu ntiles

100

150 200 Obser2a0ion

250

300

350

400

Index

Figure 8: Comprehensive diagnostic suite for the energy scaling law model (N = 410). Top Row: Residuals vs. Fitted and Centered Log Compute show overall linearity and consistency. Middle Row: Residuals vs. Architectural Proxy (log Mproxy ) and the Strategy boxplot verify stability across model types and frameworks. Bottom Row: The Q-Q plot and Cook’s Distance conﬁrm that while the data is heavy-tailed due to hardware power spikes, no individual observations exert undue inﬂuence on the scaling coefﬁcients.

Record · ID 299876 · SHA-256 140c9cfbfab43273
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.