From Tokens to Watt-hours: Analytical Energy Estimation for LLM Inference on Modern GPUs Tina Vartziotis1,2,3,*,† , Rodopi Kosteli4,† , Elli Danae Vartziotis3,6 , George Dasoulas2 , Michael Keckeisen3 , Konstantinos Skianis5 , Sotirios Kotsopoulos1,7 and Francesca Dominici2 1
National Technical University of Athens, Patission Complex 42, 10682 Athens, Greece Harvard University, 1350 Massachusetts Avenue, 02138 Cambridge, MA, USA 3 TWT GmbH Science & Innovation, Industriestraße 6, 70565 Stuttgart, DE 4 NIKI Ltd Digital Engineering, 205 National Resistance Street, 45500 Ioannina, Greece 5 University of Ioannina, Campus, 451 10 Ioannina, Greece 6 National and Kapodistrian University of Athens, Panepistimiou 30, 106 79 Athens, Greece 7 Massachusetts Institute of Technology, 77 Massachusetts Avenue, 02139 Cambridge, MA, USA 2
Abstract
The operational energy consumption of large language model (LLM) inference is becoming an increasingly important component of the environmental footprint of deployed AI systems. However, direct measurement of inference energy often requires hardware telemetry, power instrumentation, or infrastructure-specific monitoring, limiting its applicability in comparative studies, early-stage system design, and sustainability reporting. This report presents an analytically structured, empirically calibrated, GPU-level methodology for estimating LLM inference energy on NVIDIA H100-class accelerators without direct runtime measurement. The proposed estimator combines parameter-scaled transformer FLOP accounting, calibrated memory-traffic factors, and hardware-specific energy coefficients for FP16/BF16 tensor-core computation and high-bandwidth-memory movement. It explicitly separates prompt prefill from autoregressive decoding, enabling energy estimates for input tokens, output tokens, and complete inference requests. The methodology further decomposes total energy into compute, parameter-access, key–value-cache write, and attention-read components, allowing the scaling behavior with model size, context length, and generated-token count to be analyzed. The resulting estimates are not intended to replace physical power measurements; rather, they provide transparent, reproducible, and assumption-explicit approximations suitable for model comparison, green-coding analysis, and design-time evaluation of LLM inference workloads.
Keywords
Large Language Models, GPU Inference, Inference Energy Modeling, Transformer Systems, Computing Emissions, Tensor-Core Computing, Energy-Efficient AI, Green AI, Sustainable Computing, High-Performance Computing
1. Introduction As large language models (LLMs) scale to serve millions of users across cloud, edge, and on-premise deployments, their cumulative inference energy has emerged as a first-order sustainability concern, driven by the substantial computational and environmental costs of large-scale AI inference. Prior work in Green AI has shown that advances in model capability are often accompanied by increases in energy consumption and carbon emissions, motivating energy-aware approaches to machine learning research and deployment [1, 2]. While early work focused primarily on training costs [3], broader Green AI research highlighted the importance of computational efficiency, energy-aware reporting, and environmental impact across the ML lifecycle [2, 4, 5, 6], including recent efforts to extend energy-aware optimization to model selection [7]. Attention has since shifted toward inference, which is becoming an increasingly important operational component of deployed LLM systems. Corresponding author. These authors contributed equally. $ [email protected] (T. Vartziotis); [email protected] (R. Kosteli); [email protected] (E. D. Vartziotis); [email protected] (G. Dasoulas); [email protected] (M. Keckeisen); [email protected] (K. Skianis); [email protected] (S. Kotsopoulos); [email protected] (F. Dominici) 0000-0002-0877-7063 (T. Vartziotis); 0000-0001-7116-9338 (R. Kosteli); 0009-0006-1309-8598 (E. D. Vartziotis); 0000-0002-0562-5136 (G. Dasoulas); 0000-0002-9421-8566 (K. Skianis); 0000-0002-9421-8566 (S. Kotsopoulos); 0000-0002-9421-8566 (F. Dominici)
* †
© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
Unlike training, which is performed occasionally, inference workloads run continuously and scale with user demand through autoregressive token generation, accumulating substantial energy costs over a model’s deployment lifetime [8]. Their energy footprint depends strongly on generated-token count, sequence length, model size, hardware configuration, and batching behavior [9, 10, 11]. This makes inference energy a practical concern not only for large cloud providers, but also for applied AI teams, research groups, and small-to-medium organizations deploying LLMs on local or on-premise GPU infrastructure. For such deployments, accelerator-side energy does not capture the full service-level cost, since system and facility overheads, including cooling, networking, and power delivery, also contribute [12, 13]. Nevertheless, GPU-side energy is often a major controllable component of inference cost in GPU-dense systems [14], making GPU-level estimation useful for comparing model and workload choices, assessing green-coding interventions [15], and making deploymenttime decisions when full datacenter-level instrumentation is not available. Existing approaches for estimating or reporting AIrelated energy consumption often rely on hardware telemetry, external power measurement, infrastructurespecific monitoring, or coarse-grained carbon-accounting tools [6, 5]. Measurement-based studies provide valuable empirical evidence across hardware configurations and task types[16, 17], but their results are tied to specific hardware, serving systems, and runtime configurations. Carbon-aware inference work such as SPROUT further demonstrates that autoregressive generation can itself be optimized for sustainability [18]. Conversely, transformer scaling-law studies and GPU microarchitectural benchmarks provide the
basis for analytical FLOP and hardware-energy modeling [19, 20, 21], but do not by themselves provide request-level or token-level inference energy estimates. This leaves a gap for transparent, reproducible estimators that combine transformer compute modeling, memory-traffic attribution, and hardware-specific energy coefficients to estimate request-level and normalized token-level GPU inference energy without requiring runtime instrumentation. We address this gap by presenting a semi-analytical methodology for estimating accelerator-side GPU compute and memory-movement energy for LLM inference. The estimator combines parameter-scaled transformer FLOP modeling, calibrated HBM memory-traffic estimation, and hardware-specific energy coefficients. It separates prompt prefill from autoregressive decode, reports normalized input-token, output-token, and request-level energy estimates, and decomposes total energy into compute, parameter-access, KV-cache-write, and attention-read components. Beyond comparative estimation, the framework also highlights actionable mechanisms for reducing GPU inference energy, including smaller models, shorter generations, KV-cache quantization, prompt compression, and improved batching efficiency.
𝐸in/token =
𝐸pre , 𝑇in
𝐸out/token =
𝐸dec . 𝑇out
This section defines the analytically structured, empirically calibrated methodology used to estimate GPU-level energy consumption for LLM inference. The estimator is designed for settings in which direct runtime power measurement is unavailable, impractical, or not comparable across systems. It therefore provides assumption-explicit energy estimates rather than replacing hardware-level measurement. The scope of the estimator is accelerator-side operational energy. We model only the energy associated with GPU-side computation and GPU memory movement during inference. System-level and datacenter-level contributions, including CPU execution, host memory, networking, storage, powersupply losses, cooling, and power usage effectiveness (PUE), are outside the scope of this work and are not included in the reported estimates. Similarly, this paper does not estimate carbon emissions; all reported quantities are expressed as GPU-level energy. Each inference request is separated into two phases: prompt prefill and autoregressive decode. During prefill, the model processes the input prompt and constructs the initial key–value (KV) cache. During decode, the model generates output tokens sequentially, with each step attending to the accumulated context through the KV cache. This phase separation is required because input-token and output-token costs have different compute and memory-access patterns.
2.1. Request-Level Energy Decomposition Let 𝑇in denote the number of input tokens, i.e., the tokenized prompt provided to the model, and let 𝑇out denote the number of generated output tokens. The total GPU energy of one request, denoted 𝐸GPU , is decomposed into prefill and decode energy: (1)
where 𝐸pre is the energy consumed during the prefill phase and 𝐸dec is the energy consumed during the autoregressive
(2)
where 𝐸in/token and 𝐸out/token denote the average energy per input token and per generated output token, respectively. These quantities are not intrinsic constants of a model. They depend on prompt length, generated-token count, inference precision, batching, cache reuse, hardware characteristics, and serving implementation. At the GPU level, request energy is modeled as the sum of tensor-core compute energy and high-bandwidth-memory (HBM) movement energy: 𝐸GPU = 𝐸compute + 𝐸memory .
(3)
where 𝐸compute denotes energy consumed by tensor-core computation, and 𝐸memory denotes energy consumed by memory movement. This decomposition follows common GPU energy modeling approaches, which treat computation and data movement as the dominant contributors to energy consumption [21]. The compute term is proportional to the number of tensor-core floating-point operations: 𝐸compute = 𝛼TC 𝐶TC ,
2. Methodology
𝐸GPU = 𝐸pre + 𝐸dec .
decode phase. The reported input-token and output-token energy values are defined as:
(4)
where 𝐶TC denotes tensor-core FLOPs and 𝛼TC is the hardware-specific energy per tensor-core FLOP. The memory term is proportional to the number of bits transferred through HBM: 𝐸memory = 𝑒HBM 𝑄HBM ,
(5)
where 𝑄HBM denotes HBM traffic in bits and 𝑒HBM is the energy per transferred HBM bit. Combining these terms gives: 𝐸GPU = 𝛼TC 𝐶TC + 𝑒HBM 𝑄HBM . (6) This formulation separates workload-dependent quantities (𝐶TC , 𝑄HBM ) from hardware-specific coefficients (𝛼TC , 𝑒HBM ) [21].
2.2. Compute Model For dense decoder-only transformers, we approximate the dominant tensor-core FLOPs using the standard parameterscaled transformer estimate: 𝐶pre,dense = 𝐾𝑁 𝑇in ,
𝐶dec,dense = 𝐾𝑁 𝑇out ,
(7)
where 𝑁 is the number of model parameters and 𝐾 = 6 is the FLOP coefficient per parameter per token. For long contexts, architecture-aware attention corrections are added through 𝐶pre = 𝐶pre,dense +𝐶pre,attn , so that
𝐶dec = 𝐶dec,dense +𝐶dec,attn , (8)
𝐶TC = 𝐶pre + 𝐶dec .
(9)
The full attention correction terms are provided in Supplementary Section A.
2.3. Memory-Movement Model In addition to tensor-core computation, LLM inference requires substantial data movement through the GPU memory hierarchy. We approximate the dominant off-chip memory traffic as high-bandwidth-memory (HBM) traffic and decompose it into parameter-access traffic, KV-cache write traffic, and attention-related KV-cache read traffic: Bitstotal = Bitsparams + BitsKV + Bits′attn .
(10)
This decomposition captures the primary sources of memory traffic in autoregressive transformer inference, including parameter access, KV-cache storage, and attention-related reads, as observed in modern LLM serving systems [22]. The corresponding memory energy is modeled as 𝐸memory = Bitstotal · 𝑒HBM · 𝜂(𝑁 ),
(11)
where 𝑒HBM is the energy per bit transferred from HBM and 𝜂(𝑁 ) is a calibrated memory-inefficiency factor. The individual memory-traffic terms are defined in Section 3.4, with full derivations provided in Supplementary Section B.
2.4. Total Request and Token-Level Energy The total GPU energy for a request, denoted 𝐸request , is obtained by combining the tensor-core compute term and the calibrated HBM memory-movement term: 𝐸request = 𝛼TC (𝐶pre + 𝐶dec ) + Bitstotal · 𝑒HBM · 𝜂(𝑁 ). (12) where 𝐶pre and 𝐶dec denote the total prefill and decode tensor-core FLOPs, obtained from the dense terms in Section 2.2 and, when applicable, the attention correction terms in Supplementary Section A. The term Bitstotal denotes total HBM traffic (Equation 10). The factor 𝜂(𝑁 ) is a dimensionless inefficiency multiplier applied to the memory term, representing non-ideal HBM behavior. The average energy per processed token is: 𝐸avg/token =
𝐸request . 𝑇in + 𝑇out
(13)
where 𝑇in and 𝑇out are the numbers of input and generated output tokens, respectively. For phase-level reporting, we compute:
or datacenter-level energy. They are GPU-level compute and HBM-transfer coefficients. Additional coefficients for other accelerator classes, such as A100, are provided in the Supplementary Table 6.
2.6. Simplified Parameter-Only Estimator For model inventories where detailed architecture information is unavailable, we also define a simplified estimator. In this case, output-token energy is approximated as: ̂︀out/token = 𝛼eff 𝐾𝑁, 𝐸
(15)
where 𝛼eff is an effective energy-per-FLOP coefficient that absorbs compute, memory, and utilization effects. This parameter-scaled approximation is consistent with prior analyses showing that transformer compute scales linearly with model size [19, 20]. Input-token energy is modeled as a prompt-length-dependent multiple of output-token energy: ̂︀in/token = 𝑀 (𝑇in )𝛼eff 𝐾𝑁. 𝐸
(16)
The request-level estimate is: ̂︀request = 𝑇in 𝐸 ̂︀in/token + 𝑇out 𝐸 ̂︀out/token . 𝐸
(17)
where 𝑀 (𝑇in ) is a prompt-length-dependent prefill multiplier. Its functional form is defined in Section 3.4. The simplified estimator is useful when only parameter counts and token counts are known. The architecture-aware estimator in equation 12 should be preferred whenever layer count, hidden dimension, KV-cache dimension, precision, and workload assumptions are available.
3. Evaluation Setup and Assumptions This section specifies the hardware configuration, model set, workload assumptions, and fixed estimator parameters used in the analytical evaluation. The purpose is to make the reported energy estimates reproducible and to separate methodological assumptions from the numerical results.
3.1. Target Hardware and Precision
The evaluation 𝐸dec = 𝐸compute,dec +𝐸 . is instantiated for NVIDIA H100-class acmemory,dec celerators, corresponding to the inference hardware consid(14) ered in this work. Unless otherwise stated, all models are where 𝐸compute,pre and 𝐸compute,dec denote the tensorassumed to run using FP16 or BF16 tensor-core inference. core compute energy consumed during the prefill Model weights and KV-cache entries are therefore assumed and decode phases, respectively, and 𝐸memory,pre and to use 16-bit precision: 𝐸memory,dec denote the corresponding memory-movement energy. The input-token and output-token metrics are then 𝑏𝑤 = 𝑏kv = 16. (18) obtained using Equation 2. 𝐸pre = 𝐸compute,pre +𝐸memory,pre ,
2.5. Hardware Coefficients The hardware coefficients convert tensor-core FLOPs and HBM traffic into energy. In the main evaluation, we instantiate the estimator for H100-class FP16/BF16 inference using the fixed coefficients reported in Table 1: 𝛼TC = 0.52 pJ/FLOP and 𝑒HBM = 11.68 pJ/bit. These values are based on microarchitectural accelerator-energy measurements of GPU tensor-core computation and HBM access reported by Antepara et al. [21]. These coefficients represent microarchitectural accelerator energy rather than wall-plug
For the H100-class configuration, we use the acceleratorlevel energy coefficients summarized in Table 1. Here, 𝛼TC denotes the energy per FP16/BF16 tensor-core FLOP and 𝑒HBM denotes the energy per bit transferred through highbandwidth memory. These coefficients represent GPU-level microarchitectural energy and do not include CPU, cooling, networking, or datacenter-level overheads.
Table 1 Fixed constants and hardware coefficients used in the evaluation. Parameter
Value
Description
𝐾 𝛼TC 𝑒HBM 𝑏𝑤 𝑏kv
6 0.52 pJ/FLOP 11.68 pJ/bit 16 16
FLOPs per parameter per token (transformer constant) H100 tensor-core energy coefficient HBM energy per bit transferred Bits per model weight (FP16/BF16) Bits per KV-cache element
3.2. Model Set We evaluate a representative set of transformer-based LLMs that cover small (< 3𝐵), medium (3𝐵 − 30𝐵), and large (> 30𝐵) parameter regimes as summarized in Supplementary Table 5. The model set includes embedding models, general-purpose decoder-only LLMs, code models, reasoning models, and vision-language models. For each model, the estimator requires at minimum the parameter count 𝑁 and the input/output token counts. When available, additional architecture-specific quantities such as the number of layers 𝑛ℓ , hidden dimension 𝑑model , Kv and effective KVcache dimension 𝑑kv are used by the architecture-aware estimator. If full architecture metadata is unavailable, the simplified parameter-only estimator is used. In this case, the model is represented by its parameter count and token workload only. This enables consistent comparison across heterogeneous model inventories while preserving explicit assumptions.
Here, 𝛾(𝑁 ) captures effective parameter reuse, while 𝑠attn (𝑁 ) captures non-ideal KV-cache read overhead; their calibrated forms are given in Table 2. Full derivations and interpretation are provided in Supplementary Section B. For the simplified parameter-only estimator, the prefill multiplier 𝑀 (𝑇in ) is selected according to the input length bucket: ⎧ 1.2, 𝑇in ≤ 2048, ⎪ ⎪ ⎪ ⎨ 1.8, 2048 < 𝑇in ≤ 5120, 𝑀 (𝑇in ) = (23) ⎪ 3.0, 5120 < 𝑇in ≤ 10240, ⎪ ⎪ ⎩ 4.0, 𝑇in > 10240.
3.3. Workload Scenarios
We estimate the model parameters using a data-driven calibration procedure based on reported energy measurements for LLM inference [23]. For each model size, we first decompose the total energy into compute and memory components, using the analytical expressions derived in the previous sections. The remaining memory contribution is then further decomposed into parameter, KV-cache, and attention terms. Using this decomposition, we obtain approximate estimates of the effective parameter-access factor 𝛾(𝑁 ) for each model:
The analytical evaluation considers request-level inference workloads defined by the number of input tokens 𝑇in and generated output tokens 𝑇out . The default workload used for model-size comparisons is: 𝑇in = 500,
𝑇out = 500.
(19)
Additional experiments vary 𝑇out while holding model size and input length fixed in order to study how inference energy scales with generated sequence length. This isolates the effect of autoregressive decoding and attention-related KV-cache reads. Unless otherwise stated, the evaluation assumes single-request execution and does not explicitly model continuous batching. Effects of cache reuse, locality, and batching on parameter-access traffic are represented through the parameter-access factor 𝛾.
This multiplier is used only in the simplified parameteronly estimator.
3.5. Calibration Procedure
𝛾(𝑁 ) =
Bitsparams , 𝑏𝑤 𝑁
(24)
where Bitsparams is inferred from measured energy after subtracting compute and other memory contributions. The parameters 𝛾0 and 𝛽 are then determined by fitting the power-law model to these inferred values, while also ensuring consistency with the overall energy estimates. The pa3.4. Fixed Model Parameters rameters of 𝑠attn (𝑁 ) (Equation 22) and 𝜂(𝑁 ) (Equation 11) are calibrated by minimizing the deviation between model We adopt standard constants for transformer inference, inpredictions and measurement-based reported energy valcluding the dense transformer FLOP coefficient 𝐾, along ues across the evaluated models. In practice, we perform with hardware-related energy parameters. All fixed modela low-dimensional parameter search over the coefficients ing constants and hardware coefficients are summarised in of the scaling functions, selecting values that minimize the Table 1. relative error while preserving the expected scaling trends The individual memory-traffic terms in Equation 10 are with model size. specified by the parameter-access, KV-cache-write, and This procedure is a constrained least-squares fitting over a attention-read terms below: (︂ )︂𝛽 small number of parameters, where the objective is to match 𝑁 both the magnitude and growth behavior of measured enBitsparams = 𝑏𝑤 𝑁 𝛾(𝑁 ), 𝛾(𝑁 ) = 𝛾0 , (20) 𝑁0 ergy consumption rather than exactly fitting individual data BitsKV = 2𝑏kv 𝑑model 𝑛ℓ 𝑇out , (21) points. Due to the small number of calibration parameters )︂]︂ and limited data points, the calibration is intentionally kept [︂ (︂ 𝑇out (𝑇out − 1) Bits′attn = 2𝑏kv 𝑑model 𝑛ℓ 𝑇out 𝑇in + 𝑠attn (𝑁 ).to avoid overfitting and to retain generalizability simple 2 (22)
Table 2 Calibrated model parameters used in the analytical estimator. Parameter
Value
Description
𝛾0 𝑁0 𝛽
0.10 24B 0.8
Baseline parameter-access factor at reference model size 𝑁0 Reference model size for scaling relationships Exponent controlling degradation of parameter reuse with model size
)︂ 𝑁 0.9 (︂ 𝑁0 )︂0.8 𝑁 1 + 0.8 𝑁0 (︂
𝑠attn (𝑁 ) 𝜂(𝑁 )
1 + 1.5
across models and workloads. After calibration, these factors are kept fixed across the scaling and decomposition experiments. This calibration makes the estimator an analytically structured, empirically calibrated model: transformer FLOP counts and memory-traffic terms provide the analytical structure, while the calibrated factors represent non-ideal reuse, attention-access overhead, and global HBM inefficiency. The factors should therefore be recalibrated when applying the methodology to a different GPU generation, inference engine, batching regime, or serving configuration.
Attention scaling factor capturing KV-cache read overhead Global memory-inefficiency factor for HBM traffic
4.1. Scaling with Generated Output Length We first evaluate how inference energy scales with the number of generated tokens. For this experiment, we fix the model size to 32B parameters and use an input length of 𝑇in = 100 tokens. We then vary the number of generated output tokens 𝑇out .
3.6. Reported Metrics For each model–workload pair, the estimator reports prefill energy 𝐸pre , decode energy 𝐸dec , total request energy 𝐸request , input-token energy, output-token energy, and average energy per processed token. The full description of these metrics is provided in Supplementary Section D.
3.7. Interpretation of Estimates All reported values are analytical GPU-level estimates and are scenario-dependent approximations rather than direct measurements or intrinsic properties of the models. Differences between measured and estimated energy may arise from batching, tensor-parallel communication, framework overheads, kernel fusion, quantization, cache behavior, and runtime GPU utilization. These effects are discussed further in the limitations section.
4. Results and Evaluation This section presents the analytical energy estimates obtained using the setup defined in Section 3. We evaluate the estimator along four dimensions: scaling with generated sequence length, scaling with model size, decomposition of energy into compute and memory components, and comparison against measurement-based results from prior work. We also report token-level energy estimates for the model inventory considered in this study. Unless otherwise stated, all results assume FP16/BF16 inference on H100-class accelerators, a dense transformer FLOP constant 𝐾 = 6, the calibrated parameter-access factor 𝛾(𝑁 ), the attention-access factor 𝑠attn (𝑁 ), the memory-inefficiency factor 𝜂(𝑁 ), and the hardware coefficients 𝛼TC = 0.52 pJ/FLOP and 𝑒HBM = 11.68 pJ/bit. The set of models considered in this study and their key architectural characteristics are summarized in Table S5, which can be found in the Supplementary material.
Figure 1: Energy breakdown as a function of generated output length 𝑇out for a 32B-parameter model with 𝑇in = 100 input tokens. Compute and KV-cache write costs scale almost linearly with 𝑇out , while attention-related KV-cache reads exhibit quadratic growth and become increasingly significant at larger output lengths.
Figure 1 shows that compute energy grows approximately linearly with 𝑇out , consistent with the autoregressive decoding process in which each generated token requires one forward pass through the model. KV-cache write traffic also grows linearly, because each generated token contributes one new key and value entry per layer. In contrast, scaled attention-related memory traffic grows super-linearly. This is due both to the quadratic growth of KV-cache reads with output length and to the attention-access factor 𝑠attn (𝑁 ) used in the calibrated memory model. During decode, each newly generated token attends over the accumulated context, so the total number of KV-cache reads increases as: 𝑇out 𝑇in +
𝑇out (𝑇out − 1) . 2
(25)
4.2. Scaling with Model Size We next evaluate the dependence of request-level energy on model size. The workload is fixed to: 𝑇in = 500,
𝑇out = 500.
(26)
This isolates the effect of model parameter count while holding the token workload constant.
Figure 2: Energy per request as a function of model size for a fixed workload of 500 input tokens and 500 output tokens. Total energy, compute energy, and memory energy are shown on a log–log scale.
Figure 2 shows that total request energy increases approximately linearly with model size on a log–log scale. This behavior is expected from the parameter-scaled compute model: 𝐶dec,dense = 𝐾𝑁 𝑇out ,
𝐶pre,dense = 𝐾𝑁 𝑇in . (27)
For fixed input and output lengths, the dominant dense matrix operations scale linearly with the number of parameters 𝑁 . Consequently, the compute component follows: 𝐸compute ∝ 𝑁.
(28)
The memory component increases with model size through the calibrated factor 𝛾(𝑁 ), the attention-access factor 𝑠attn (𝑁 ), and the global memory-inefficiency factor 𝜂(𝑁 ). Under the fixed 500-token/500-token workload, compute remains the dominant component for most evaluated model sizes, although memory energy increases nonlinearly with the calibrated memory factors. While the overall scaling trend is approximately linear in model size, the memory-energy component exhibits non-monotonic behavior for certain models. This is due to differences in architectural configurations, rather than parameter count alone. In particular, key contributors to memory traffic, such as the hidden dimension 𝑑model and the number of layers 𝑛ℓ , do not scale uniformly with 𝑁 across different models. As a result, some medium-sized models may exhibit lower memory traffic than nearby models with slightly different parameter counts, leading to localized deviations (e.g., a reduction in the memory-energy curve). This highlights that memory costs are sensitive to architectural design choices, not solely parameter count.
4.3. Energy Decomposition To understand which terms dominate request energy, we decompose total energy into compute, parameter movement, KV-cache writes, and attention-related KV-cache reads. The decomposition is evaluated for a 32B-parameter model while varying 𝑇out . Figure 3 shows the absolute energy decomposition as a function of generated output length for a 32Bparameter model. The stacked representation highlights the additive structure of the energy model, where total energy is the sum of compute and memory components. Compute energy increases approximately linearly with 𝑇out , reflecting the constant per-token cost of autoregressive decoding. In contrast, attention-related memory energy
Figure 3: Fractional energy contribution as a function of generated output length for a 32B-parameter model. Compute dominates at short and moderate output lengths, while attentionrelated memory traffic increases with sequence length due to quadratic KV-cache read scaling.
exhibits super-linear growth, due to the quadratic scaling of KV-cache reads with sequence length. At small output lengths, total energy is dominated by compute. As 𝑇out increases, the attention-related component grows rapidly and becomes a substantial contributor to the overall energy. This results in an upward curvature of the total energy trend, indicating the increasing impact of memory movement at longer sequence lengths. The contributions from parameter access and KV-cache writes are negligible relative to compute and attentionrelated memory traffic across the evaluated range. In particular, the KV-cache write component, although included in the model, is not visually distinguishable in the stacked representation. This is because KV writes scale linearly with 𝑇out , while attention-related KV-cache reads scale quadratically and dominate memory traffic at larger sequence lengths.
4.4. Token-Level Energy Estimates for the Model Inventory Table 3 reports simplified token-level estimates for the model inventory using the parameter-only computedominated estimator: ̂︀in/token ≈ 1.2 𝐸 ̂︀out/token , 𝐸 (29) where the input-token multiplier corresponds to the short-prompt setting. These values are simplified, computedominated estimates; further interpretation is provided in Supplementary Section D. Table 3 shows the expected linear dependence of tokenlevel energy on model size. Sub-billion-parameter embedding models have estimated token-level costs below approximately 2.3 mJ/input token under the simplified estimator, whereas 70B- and 120B-parameter models require substantially higher per-token energy. For example, the simplified estimate for LLaMA 3.3 70B is 218.4 mJ/output token and 262.1 mJ/input token under the short-prompt assumption. The 120B model reaches 374.4 mJ/output token and 449.3 mJ/input token. These values are useful for comparative model selection because they expose the energy scaling of energy consumption with parameter count. The analysis isolates the cost of token processing and does not account for task performance. Models with identical parameter counts yield identical ̂︀out/token = 𝛼TC 𝐾𝑁, 𝐸
Table 3 Estimated energy consumption per token and per request across evaluated models. Model EmbeddingGemma MXBAI Embed Large Qwen3 Embedding Qwen3 (1.7B) Granite 3.2 Vision Qwen3 (8B) Granite 3.3 Ministral 3 (14B) DeepSeek-Coder V2 GPT-OSS (20B) Qwen3 (32B) Qwen2.5-Coder (32B) Qwen3-VL (32B) DeepSeek-R1 Llama 3.3 (70B) GPT-OSS (120B)
Params (B)
𝐸out/token (mJ/token)
𝐸in/token (mJ/token)
𝐸request (Wh)
0.308 0.334 0.600 1.700 2.530 8.000 8.170 14.000 16.000 20.000 32.000 32.000 32.000 32.000 70.000 120.000
0.961 1.042 1.872 5.304 7.894 24.960 25.490 43.680 49.920 62.400 99.840 99.840 99.840 99.840 218.400 374.400
1.153 1.250 2.246 6.365 9.472 29.952 30.588 52.416 59.904 74.880 119.808 119.808 119.808 119.808 262.080 449.280
0.000599 0.000696 0.001027 0.002641 0.004923 0.011795 0.012465 0.021313 0.017425 0.022281 0.052721 0.052721 0.052721 0.052721 0.170747 0.155204
Table 4 Analytical versus measured energy per prompt for a 500-token input and 500-token output workload. Model size (B parameters) 8 24 70 72
Analytical energy (Wh/request)
Measured energy (Wh/request)
Error (%)
0.011795 0.032427 0.170747 0.170747
0.009270 0.026280 0.179230 0.233260
27.23 23.39 4.73 26.80
token-level estimates under the parameter-scaled compute model, since per-token FLOPs depend only on the number of parameters. Architectural differences influence memoryrelated energy and full request-level costs, but are not reflected in these simplified token-level estimates. Finally, modality-specific models such as vision-language architectures may incur additional costs that are not captured by the parameter-only formulation.
4.5. Comparison with Measurement-Based Results Finally, we compare the analytical estimates against the measurement-based study of Caravaca et al. [23]. The comparison uses the same nominal workload: 𝑇in = 500,
𝑇out = 500.
(30)
Table 4 shows that the analytical estimator matches measurement-based values within approximately 5–27% for the compared cases. The lowest error is observed for the 70B model, where the analytical estimate differs from the measured value by 4.73%. For the 8B, 24B, and 72B cases, the relative error remains below 30%. The remaining discrepancies are expected. The analytical model estimates GPU-level compute and memory movement under controlled assumptions, whereas measurement-based studies include additional effects from inference engines, batching policies, runtime scheduling, tensor-parallel execution, kernel fusion, GPU utilization, and system-level overheads. Differences may also arise from comparing models of similar size but different architecture. Therefore,
the comparison shows that the estimator captures the correct order of magnitude and scaling behavior, rather than as exact per-deployment energy accounting. Overall, the results indicate that the proposed analytical estimator provides a transparent and reproducible approximation of LLM inference energy. It captures the expected linear scaling with model size, the super-linear effect of attention-related memory traffic in long generations, and the distinction between input-token, output-token, and request-level energy.
5. Limitations and Conclusion The proposed estimator is limited to accelerator-side operational energy. It excludes CPU execution, host memory, networking, storage, power-supply losses, cooling, and datacenter-level power usage effectiveness. Consequently, the reported values are GPU-level operational energy estimates, not end-to-end service energy, datacenter energy, carbon emissions, or lifecycle emissions. Nevertheless, accelerator-side energy is a major controllable component in GPU-dense on-premise inference deployments: H100 SXM-class accelerators have a thermal design power of approximately 700 W, and multi-GPU servers can therefore be dominated by accelerator power under high utilization [14]. Facility-level energy remains larger because of cooling and power-delivery overheads; for example, Uptime Institute reports an average data-center PUE of approximately 1.56, and MLPerf Power treats inference energy as a system-level measurement problem [12, 13, 24].
The estimator also abstracts away several architectureand system-specific effects. The parameter-scaled FLOP model does not fully capture feed-forward expansion ratios, grouped-query attention, multi-query attention, mixture-ofexperts routing, or modality-specific processing in visionlanguage models. The memory model approximates weight access, KV-cache writes, and attention reads using calibrated factors 𝛾(𝑁 ), 𝑠attn (𝑁 ), and 𝜂(𝑁 ), which summarize parameter reuse, attention-access overhead, and HBM inefficiency. These factors do not explicitly model the complete GPU memory hierarchy, kernel scheduling, cache residency, batching behavior, or inference engine implementation, and should be recalibrated for other hardware platforms, serving engines, or batching regimes. Despite these limitations, the methodology provides a reproducible framework for comparing GPU-level inference energy across models and workloads when direct instrumentation is unavailable. It also identifies actionable inference-stage levers for reducing energy: smaller or task-specialized models reduce parameter-scaled compute; shorter outputs reduce autoregressive decoding cost; prompt compression and retrieval filtering reduce context length; quantized weights and KV caches reduce memory traffic; and batching, prefix caching, efficient attention kernels, speculative decoding, and model routing can improve serving efficiency. The method is therefore best understood as a complementary tool for green-coding analysis, comparative model selection, and design-time evaluation, while precise deployment accounting still requires system-level power measurement. Future work should integrate measured serving traces, extend the estimator to quantized and mixture-of-experts models, and incorporate tensor-parallel communication and batching effects.
Declaration on Generative AI During the preparation of this work, the authors used OpenAI ChatGPT for language editing, restructuring, consistency checking, and drafting assistance. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content.
Acknowledgments The authors gratefully acknowledge the ITEA GreenCode research project for fostering discussions on energy-aware AI systems and for funding parts of this work. The authors also thank their colleagues at TWT Science & Innovation, Stefanos Papanikolaou and Michael Herrnberger, for their valuable discussions and insights.
References [1] E. Strubell, A. Ganesh, A. McCallum, Energy and policy considerations for deep learning in nlp, in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Florence, Italy, 2019, pp. 3645–3650. URL: https://aclanthology.org/P19-1355/. doi:10.18653/v1/P19-1355. [2] R. Schwartz, J. Dodge, N. A. Smith, O. Etzioni, Green ai, Communications of the ACM 63 (2020) 54–63.
URL: https://doi.org/10.1145/3381831. doi:10.1145/ 3381831. [3] D. Patterson, J. Gonzalez, Q. V. Le, C. Liang, L.-M. Munguia, D. Rothchild, D. So, M. Texier, J. Dean, Carbon emissions and large neural network training, 2021. URL: https://arxiv.org/abs/2104.10350. arXiv:2104.10350. [4] P. Henderson, J. Hu, J. Romoff, E. Brunskill, D. Jurafsky, J. Pineau, Towards the systematic reporting of the energy and carbon footprints of machine learning, Journal of Machine Learning Research 21 (2020) 1–43. URL: http://jmlr.org/papers/v21/20-312.html. [5] A. Lacoste, A. Luccioni, V. Schmidt, T. Dandres, Quantifying the carbon emissions of machine learning, 2019. URL: https://arxiv.org/abs/1910.09700. arXiv:1910.09700. [6] L. Lannelongue, J. Grealey, M. Inouye, Green algorithms: Quantifying the carbon footprint of computation, Advanced Science 8 (2021) 2100707. URL: https://doi.org/10.1002/advs.202100707. doi:10.1002/ advs.202100707. [7] F. Betello, V. Vineis, A. Purificato, G. Tolomei, F. Silvestri, One search fits all: Pareto-optimal eco-friendly model selection, arXiv preprint arXiv:2505.01468 (2025). arXiv:2505.01468. [8] E. Lim, Z. Pan, Y. Zhou, Characterizing the carbon impact of llm inference, Final course project report for 15-642: Machine Learning Systems (2024). Inference characterization. [9] J. Fernandez, C. Na, V. Tiwari, Y. Bisk, S. Luccioni, E. Strubell, Energy considerations of large language model inference and efficiency optimizations, in: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, 2025. URL: https:// aclanthology.org/2025.acl-long.1563/. [10] Z. Fu, F. Chen, S. Zhou, H. Li, L. Jiang, Llmco2: Advancing accurate carbon footprint prediction for llm inferences, 2024. URL: https://arxiv.org/abs/2410.02950. arXiv:2410.02950. [11] T. Vartziotis, M. Schmidt, G. Dasoulas, I. Dellatolas, S. Attademo, V. D. Le, A. Wiechmann, T. Hoffmann, M. Keckeisen, S. Kotsopoulos, Carbon footprint evaluation of code generation through llm as a service, in: A. C. Kulzer, H.-C. Reuss, A. Wagner (Eds.), 2024 Stuttgart International Symposium on Automotive and Engine Technology, Springer Fachmedien Wiesbaden, Wiesbaden, 2024, pp. 230–241. [12] Uptime Institute, Uptime Institute Global Data Center Survey 2024, Technical Report, Uptime Institute Intelligence, 2024. Reports average data-center PUE of approximately 1.56. [13] MLCommons, MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems, 2024. arXiv:2410.12032. [14] NVIDIA, NVIDIA H100 Tensor Core GPU Datasheet, https://resources. nvidia.com/en-us-hopper-architecture/ nvidia-tensor-core-gpu-datasheet, 2023. Accessed 2026-05-24. [15] T. Vartziotis, I. Dellatolas, G. Dasoulas, M. Schmidt, F. Schneider, T. Hoffmann, S. Kotsopoulos, M. Keckeisen, Learn to code sustainably: An empirical study on green code generation, in: Proceedings of the 1st International Workshop on Large Language Mod-
els for Code, LLM4Code ’24, Association for Computing Machinery, New York, NY, USA, 2024, p. 30–37. URL: https://doi.org/10.1145/3643795.3648394. doi:10. 1145/3643795.3648394. [16] S. Samsi, D. Zhao, J. McDonald, B. Li, A. Michaleas, M. Jones, W. Bergeron, J. Kepner, D. Tiwari, V. Gadepally, From words to watts: Benchmarking the energy costs of large language model inference, in: 2023 IEEE High Performance Extreme Computing Conference (HPEC), IEEE, ????, pp. 1–9. URL: https://ieeexplore.ieee.org/document/10363447/. doi:10.1109/HPEC58863.2023.10363447. [17] S. Luccioni, Y. Jernite, E. Strubell, Power hungry processing: Watts driving the cost of AI deployment?, in: The 2024 ACM Conference on Fairness Accountability and Transparency, ACM, ????, pp. 85–99. URL: https://dl.acm.org/doi/10.1145/3630106.3658542. doi:10.1145/3630106.3658542. [18] B. Li, Y. Jiang, V. Gadepally, D. Tiwari, Sprout: Green generative ai with carbon-efficient llm inference, in: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 2024, pp. 21799–21813. URL: https://aclanthology.org/2024.emnlp-main.1215/. [19] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, D. Amodei, Scaling laws for neural language models, 2020. URL: https://arxiv.org/abs/2001.08361. arXiv:2001.08361. [20] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, L. Sifre, Training compute-optimal large language models, in: Advances in Neural Information Processing Systems, volume 35, 2022, pp. 30016–30030. [21] O. Antepara, Z. Zhao, B. Austin, N. Ding, L. Oliker, N. J. Wright, S. Williams, Benchmark-driven models for energy analysis and attribution of gpu-accelerated supercomputing, in: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’25, Association for Computing Machinery, St. Louis, MO, USA, 2025. URL: https://doi.org/10.1145/3712285.3759815. doi:10.1145/3712285.3759815. [22] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, I. Stoica, Efficient memory management for large language model serving with pagedattention, in: Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP), ACM, 2023. doi:10.1145/3600006.3613165. [23] F. Caravaca, Á. Cuevas, R. Cuevas, From prompts to power: Measuring the energy footprint of llm inference, 2025. URL: https://arxiv.org/abs/2511.05597. arXiv:2511.05597. [24] MLCommons, MLPerf Inference: Power Measurement Documentation, https://docs.mlcommons.org/ inference/power/, 2024. Accessed 2026-05-24. [25] T. Dao, D. Y. Fu, S. Ermon, A. Rudra, C. Ré, Flashattention: Fast and memory-efficient exact attention with io-awareness, in: Advances in Neural Information Processing Systems, volume 35, 2022. URL: https:// proceedings.neurips.cc/paper_files/paper/2022/hash/ 67d57c32e20fd0a7a302cb81d36e40d5-Abstract-Conference.
html. [26] NVIDIA, Ministral 3 model documentation, 2024. URL: https://docs.nvidia.com/nemo/megatron-bridge/ 0.2.0/models/vlm/ministral3.html. [27] Google, Embeddinggemma-300m, 2024. URL: https: //huggingface.co/google/embeddinggemma-300m. [28] Mixedbread AI, mxbai-embed-large-v1, 2024. URL: https://www.aimodels.fyi/models/huggingFace/ mxbai-embed-large-v1-mixedbread-ai. [29] Alibaba, Qwen3 embedding 0.6b, 2024. URL: https: //huggingface.co/Qwen/Qwen3-Embedding-0.6B. [30] Alibaba, Qwen3-32b model configuration, 2024. URL: https://huggingface.co/Qwen/Qwen3-32B. [31] IBM, Granite 3.3 8b instruct, 2024. URL: https:// huggingface.co/ibm-granite/granite-3.3-8b-instruct. [32] IBM, Granite 3.2 2b instruct, 2024. URL: https:// huggingface.co/ibm-granite/granite-3.2-2b-instruct. [33] DeepSeek AI, Deepseek coder v2 lite instruct, 2024. URL: https://huggingface.co/deepseek-ai/ DeepSeek-Coder-V2-Lite-Instruct. [34] DeepSeek AI, Deepseek r1 distill qwen 32b, 2024. URL: https://huggingface.co/deepseek-ai/ DeepSeek-R1-Distill-Qwen-32B. [35] Alibaba, Qwen3 8b model configuration, 2024. URL: https://huggingface.co/Qwen/Qwen3-8B. [36] Alibaba, Qwen3 32b model configuration, 2024. URL: https://huggingface.co/Qwen/Qwen3-32B. [37] Alibaba, Qwen3 1.7b model configuration, 2024. URL: https://huggingface.co/Qwen/Qwen3-1.7B. [38] Alibaba, Qwen2.5 technical report, 2024. URL: https: //arxiv.org/html/2409.12186v2. [39] Unsloth AI, Llama 3.3 overview, 2024. URL: https:// unsloth.ai/blog/llama3-3. [40] OpenAI, Gpt-oss 120b model configuration, 2024. URL: https://huggingface.co/openai/gpt-oss-120b. [41] OpenAI, Gpt-oss 20b model configuration, 2024. URL: https://huggingface.co/openai/gpt-oss-20b.
Supplementary Material
B.1. Parameter-Access Traffic
A. Derivation of the Compute Model This section provides the architecture-aware compute terms used by the estimator. The main paper reports the compact parameter-scaled form, while the full attention correction terms are given here for reproducibility. For dense decoder-only transformers, the dominant computation arises from matrix multiplications in attention projections and feed-forward layers. When only the model parameter count is available, the dense decode-phase FLOPs per generated token, denoted 𝐶dec/token , are approximated as: 𝐶dec/token ≈ 𝐾𝑁, (31) where 𝑁 is the number of model parameters and 𝐾 is a transformer FLOP constant. Following standard transformer FLOP accounting, we use: 𝐾 = 6,
(32)
which corresponds to a commonly used approximation that dense transformer inference requires on the order of six FLOPs per parameter per token for the forward pass [19, 20]. The dense compute cost of the decode phase is therefore: 𝐶dec,dense = 𝐾𝑁 𝑇out .
(33)
Similarly, the dense compute cost of the prefill phase is: 𝐶pre,dense = 𝐾𝑁 𝑇in .
(34)
For long contexts, additional attention terms can be included. If architecture-level parameters are available, the prefill attention correction is approximated as: 2 𝐶pre,attn ≈ 2𝑛ℓ 𝑑model 𝑇in ,
(35)
where 𝑛ℓ is the number of transformer layers and 𝑑model is the hidden dimension. During decode, the token generated at step 𝑡 attends to a context of length 𝑇in + 𝑡 − 1. The attention-related decode cost is approximated as: (︂ )︂ 𝑇out (𝑇out − 1) . 𝐶dec,attn ≈ 4𝑛ℓ 𝑑model 𝑇out 𝑇in + 2 (36) The total tensor-core compute estimate is: 𝐶TC = 𝐶pre,dense + 𝐶dec,dense + 𝐶pre,attn + 𝐶dec,attn . (37) When detailed architectural parameters are unavailable, the attention correction terms are omitted and the parameter-scaled approximation is used.
B. Derivation of the Memory-Movement Model This section provides the full memory-traffic decomposition used by the architecture-aware estimator. The total HBM traffic is decomposed into parameter access, KV-cache writes, and attention-related KV-cache reads: Bitstotal = Bitsparams + BitsKV + Bits′attn ,
(38)
where Bitsparams denotes parameter-access traffic, BitsKV denotes KV-cache write traffic, and Bits′attn denotes scaled attention-related KV-cache read traffic.
Model weights are stored in GPU memory and accessed during the forward passes required for inference. A naive upper-bound formulation would assume that the full set of model weights is transferred from HBM at every decoding step, leading to memory traffic proportional to the number of generated tokens. However, modern GPU implementations reduce this cost through on-chip caching, kernel fusion, memory locality, and overlap between memory access and computation. To account for imperfect parameter reuse, we introduce a model-size-dependent parameter-access factor 𝛾(𝑁 ): Bitsparams = 𝑏𝑤 𝑁 𝛾(𝑁 ),
(39)
where 𝑏𝑤 is the number of bits per model weight and 𝑁 is the number of model parameters. The parameter-access factor 𝛾(𝑁 ) ∈ (0, 1] represents the effective fraction of model parameters retrieved from HBM throughout the inference request. We model its dependence on model size as: )︂𝛽 (︂ 𝑁 , (40) 𝛾(𝑁 ) = 𝛾0 𝑁0 where 𝑁0 is a reference model size, 𝛾0 is the baseline reuse factor at 𝑁0 , and 𝛽 approximates the degradation of effective reuse as the model working set exceeds on-chip memory capacity.
B.2. KV-Cache Write Traffic For each generated token, the model stores key and value vectors in each transformer layer. The KV-cache write traffic is approximated as: BitsKV = 2𝑏kv 𝑑model 𝑛ℓ 𝑇out ,
(41)
where 𝑏kv is the number of bits per KV-cache element, 𝑑model is the hidden dimension, 𝑛ℓ is the number of transformer layers, and 𝑇out is the number of generated output tokens. The factor of two accounts for storing both keys and values.
B.3. Attention-Related KV-Cache Read Traffic During autoregressive decoding, each newly generated token attends over the input prompt and the previously generated tokens through the KV cache. For an input length 𝑇in and an output sequence of length 𝑇out , the baseline attention-related memory traffic is: (︂ )︂ 𝑇out (𝑇out − 1) Bitsattn = 2𝑏kv 𝑑model 𝑛ℓ 𝑇out 𝑇in + . 2 (42) This term includes a linear prompt-attention component and a quadratic generated-token component. To account for non-ideal memory behavior, including irregular access patterns, limited locality, cache contention, and increasing memory pressure at larger model sizes, we apply an attention-specific scaling factor: Bits′attn = Bitsattn · 𝑠attn (𝑁 ),
(43)
where 𝑠attn (𝑁 ) ≥ 1 is a dimensionless scaling factor.
and to generate the output sequence, respectively. The total request energy is defined as
Table 5 Overview of Model Architectural Configurations. Model
(45)
Params
Layers
𝑑𝑚𝑜𝑑𝑒𝑙
𝐸request = 𝐸pre + 𝐸dec .
14B 308M 334M 0.6B 32B 8.17B 2.53B 16B 32B 8B 32B 1.7B 32B 70B 120B 20B
40 26 24 28 64 40 32 27 64 36 64 28 64 80 36 24
5120 768 1024 1024 5120 4096 4096 2048 5120 4096 5120 2048 5120 8192 2880 2880
To compare requests with different prompt and generation lengths, we normalize phase-level energy by the corresponding token counts. The input-token energy is defined as 𝐸pre 𝐸in/token = , (46) 𝑇in where 𝑇in is the number of prompt tokens. Analogously, the output-token energy is defined as
Ministral 3 (14B) [26] EmbeddingGemma [27] MXBAI Embed Large [28] Qwen3 Embedding (0.6B) [29] Qwen3-VL (32B) [30] Granite 3.3 (8B) [31] Granite 3.2 Vision [32] DeepSeek-Coder V2 (16B) [33] DeepSeek-R1 (32B) [34] Qwen3 (8B) [35] Qwen3 (32B) [36] Qwen3 (1.7B) [37] Qwen2.5-Coder (32B) [38] Llama 3.3 (70B) [39] GPT-OSS (120B) [40] GPT-OSS (20B) [41]
The attention-related terms follow the quadratic dependence on sequence length characteristic of standard selfattention, along with the associated memory-access costs on GPU memory hierarchies [25]. The importance of KV-cache memory traffic in LLM serving has also been emphasized in prior system-level work, including PagedAttention and vLLM [22].
B.4. HBM-Dominated Memory Energy The memory-energy term is computed as: 𝐸memory = Bitstotal · 𝑒HBM · 𝜂(𝑁 ),
(44)
where 𝑒HBM is the energy per bit transferred from HBM and 𝜂(𝑁 ) ≥ 1 is a global memory-inefficiency factor. The factor 𝜂(𝑁 ) approximates bandwidth saturation, memorycontroller overhead, cache contention, and pipeline stalls under high memory pressure. The factors 𝛾(𝑁 ), 𝑠attn (𝑁 ), and 𝜂(𝑁 ) are empirical calibration terms fitted against measurement-based LLM inference energy results reported by Caravaca et al. [23].
C. Model Architecture Details In this section we present key architectural details of the models that have been used in this study. Table 5 shows the parameter, layer, and hidden dimension (𝑑model ) of the model architectures.
D. Reported Metrics For each model–workload pair, the estimator reports energy at three levels of granularity: phase-level energy, tokennormalized energy, and aggregate request-level energy. This separation is necessary because LLM inference is not a homogeneous operation: prompt prefill and autoregressive decoding differ in their compute structure, memory-access pattern, and dependence on sequence length. The phase-level quantities are the prefill energy, 𝐸pre , and the decode energy, 𝐸dec . These terms represent the estimated GPU energy required to process the input prompt
𝐸out/token =
𝐸dec , 𝑇out
(47)
where 𝑇out is the number of generated tokens. We additionally report the average energy per processed token: 𝐸request 𝐸avg/token = . (48) 𝑇in + 𝑇out This aggregate metric mixes prefill and decode costs and is therefore sensitive to the input/output token ratio. Finally, to analyze the dominant sources of energy consumption, the request energy is decomposed into compute and memory contributions: 𝐸request = 𝐸compute + 𝐸memory .
(49)
The memory component is further attributed to parameter access, KV-cache writes, and KV-cache reads. Reporting these components enables the evaluation to distinguish compute-dominated regimes from memory-influenced or long-context regimes.
D.1. Interpretation of Simplified Token-Level Estimates The model-inventory table in the main paper reports simplified token-level energy estimates based on a parameter-only, compute-dominated approximation: ̂︀in/token ≈ 1.2 𝐸 ̂︀out/token . 𝐸 (50) These values do not include the calibrated memory factors 𝛾(𝑁 ), 𝑠attn (𝑁 ), or 𝜂(𝑁 ) used in the architectureaware evaluation. They are first-order comparative estimates across the model inventory, rather than full computeplus-memory request-level estimates. In contrast, the scaling and decomposition figures in the main paper use the full calibrated compute-plus-memory model. ̂︀out/token = 𝛼TC 𝐾𝑁, 𝐸
E. Implementation and Execution Assumptions This section describes the implementation details and system assumptions underlying the analytical energy estimator. Estimator implementation. The estimator is implemented as a Python-based analytical tool that computes energy consumption from the model and the workload parameters. Given the number of model parameters 𝑁 , number of layers 𝑛ℓ , hidden dimension 𝑑model , and token counts, the tool calculates FLOPs and memory traffic using the analytical formulas described in the main text.
The implementation does not execute neural networks or perform runtime profiling. Instead, it deterministically estimates energy based on compute and memory abstractions. No direct GPU telemetry, wall-plug power instrumentation, or runtime energy measurement APIs (e.g., NVML or nvidiasmi) are used during estimation. Hardware assumptions. The main evaluation uses H100class coefficients, while additional accelerator coefficients are reported here for completeness. Energy conversion coefficients for tensor-core operations (𝛼TC ) and HBM traffic (𝑒HBM ) are taken from measurement-based prior work [21]. Table 6 Accelerator-level energy coefficients for FP16/BF16 tensor-core inference. Hardware
𝛼TC (pJ/FLOP)
𝑒HBM (pJ/bit)
H100 / GH200 A100
0.52 0.70
11.68 13.11
Software stack assumptions. The estimator assumes inference execution on optimized tensor-core GPU kernels using FP16/BF16 arithmetic and HBM-resident model weights. It is intended to approximate inference behavior of optimized GPU-based implementations built on CUDA and highperformance libraries such as cuBLAS and cuDNN, as well as modern LLM inference frameworks (e.g., TensorRT-LLM, Megatron-LM, and vLLM). The model does not explicitly simulate kernel-level execution or software-specific optimizations such as kernel fusion, scheduling, or memory tiling. Instead, these systemlevel effects are treated implicitly and are approximated through the empirical scaling factors 𝛾(𝑁 ), 𝑠attn (𝑁 ), and 𝜂(𝑁 ), which collectively capture deviations from ideal compute and memory behavior observed in optimized inference systems. Calibration reference. Model parameters are calibrated using reported energy measurements for optimized LLM inference from prior work [23]. These measurements serve as reference values for matching the magnitude and scaling behavior of energy consumption. Limitations. The estimator does not model GPU execution at the kernel or instruction level. In particular, it does not simulate thread-level parallelism, CUDA scheduling, or detailed memory hierarchy behavior. Instead, such effects are approximated through calibrated scaling factors. As a result, the model is an analytical approximation rather than a cycle-accurate simulation or direct hardware measurement.