Improving the Energy Efficiency of High Throughput Computing: A Measurement-Based Case Study

1

Improving the Energy Efficiency of High Throughput Computing: A Measurement-Based Case Study Damu Ding

, Xinpeng Hong

, Alastair Dewhurst

arXiv:2608.06622v1 [cs.NI] 6 Aug 2026

Greenwood

, James Walder, Daniel Schien

, David

, Noa Zilberman

Abstract—The significant energy consumed by data centers has become a concern both for costs and associated carbon emissions. In particular, the energy efficiency of servers is a key consideration for data center operators, and understanding servers’ power consumption under different operating conditions is an important aspect of it. In this paper we present a measurement-based case study of high-throughput computing. We analyze power usage information of an operational data center, combined with focused measurements of power reduction techniques for a representative high-throughput workload. The study points out the obstacles encountered by data center operators in their efforts to minimize energy consumption and carbon emissions, and discusses the impact of server configuration adjustments on the energy consumption of processing jobs. We offer actionable recommendations for decreasing the energy usage of servers, while considering both performance and carbon emissions. Index Terms—Green computing, Data center, Power reduction, Energy efficiency, Virtualization, Measurements

I. Introduction The electricity consumption of data centers has been increasing over time with the growing demand for computing power, storage capacity, and data processing capabilities. According to the International Energy Agency (IEA), data centers’ global electricity consumption was 460TWh as of 2022, and this number is expected to continuously increase [1]. The increased energy usage leads to rising financial burdens and a negative impact on the environment. In recent years, numerous methods have been proposed to enhance data centers’ energy efficiency, such as consolidating multiple servers into a single physical unit using virtualization[2], adjusting server power usage by workload [3], improving cooling efficiency [4], and utilizing energy-efficient hardware like Solid State Drives (SSD)[5]. Rather than upgrading data center infrastructure, which is associated with additional embodied carbon, a simpler approach to saving energy is improving computing-related energy efficiency. In this work, we focus on High Throughput Computing (HTC), which aims to maximize job throughput without prioritizing task-level latency [6]. This is not like High Performance Computing (HPC), which seeks to minimize time-to-solution (i.e., the average time to finish jobs) by distributing tasks across multiple workers Affiliations as of the time of writing: Damu Ding, Xinpeng Hong, and Noa Zilberman were with University of Oxford, UK. Alastair Dewhurst and James Walder were with Science and Technology Facilities Council, Harwell Campus, UK. Daniel Schien was with University of Bristol, UK. David Greenwood was with University of Newcastle, UK.

at the cost of parallelization. Both kinds of computing lead to high servers utilization under heavy workloads, but given the independent nature of jobs, HTC is operating under different constraints, such as the number of jobs to be finished within a given period of time. To balance the requirements of performance and energy efficiency, it is necessary to investigate the trade-off between power consumption and job execution performance, and to understand how they vary with different configurations. Furthermore, this needs to be assessed on accumulated energy-to-performance scale, rather than average power, to account for hidden overhead costs. This measurement study is designed for this purpose, covering a crucial aspect of the broader effort to make data centers more sustainable and cost-efficient. Although measurement-based analyses of data center server energy consumption have been conducted in the past, there has been a gap in recent studies, with the most recent analyses dating back to 2023 [7], [8], [9]. This means that there is a scarcity of empirical studies examining energy trends in current servers. As both workloads and servers have come a long way since, it is important to get a fresh look on the subject, and to provide the community with up to date data and new insights. Understanding the handling of configuration parameters and their impact on various variables may result in substantial cost and energy savings. In this paper, we present a case study of improving the energy efficiency of high throughput computing, based on power measurements from a UK Science and Technology Facilities Council (STFC) data center, and focused power optimization measurements in a local cluster. As access to renewable energy varies between data center operators, our study provides an insight into some of the challenges faced by data center operators when trying to reduce the operational carbon emissions of their facilities. It further discusses how tuning server configurations affects the energy consumption of processing jobs, ranging from the number of active CPU cores and the number of jobs to performance boosting technologies. Finally, we provide practical recommendations for energy reduction of servers, with a combined view of both performance and energy efficiency. In summary, this paper makes the following contributions: • We analyze a dataset of racks power consumption within a real data center and discuss power consumption trends.

2

We propose a simple but effective model to monitor the power and energy consumption of data center on service level, based on our power analysis. • We measure the power, energy consumption, and job execution time in two kinds of data center servers and analyze their sensitivity to different configurations and operating scenarios. • Based on our measurement results, we provide recommendations to balance the energy consumption and performance in data center servers for green computing. The rest of the paper is organized as follows. Section II introduces background information and discusses related work. In Section III, we present a study of power consumption in a real data center. Section IV describes our energy consumption modelling. In Section V, we present the experimental setup of our testbed and report measurement results. Section VI provides recommendations for reducing data center energy consumption based on our measurement results. Finally, Section VII concludes the paper. •

II. Background and related work A. Background This subsection provides the background of the concepts discussed in this paper. Definition of power and energy Power consumption P is an instantaneous value that indicates the energy per unit time, and its unit is Watt. Energy E is the cumulative power consumption over a given period of T seconds. Thus, energy consumption can be formulated as: E = P̄ · T

(1)

where P̄ is the average power consumption during period T . Note that the unit of power (P ) is Watt (W), and the unit of energy (E) can be kWh (kilo Watt hour) or Joule (J). Since 1 kWh is equal to 3.6 × 106 Joules, they are interchangeable. If not otherwise specified, kWh is the energy unit in the remainder of this paper. Energy consumption and carbon emissions Cloud data centers contribute significantly to environmental impact, particularly due to their substantial energy consumption [1].Carbon intensity is a measure of the ratio between energy consumption and carbon emissions: when the carbon intensity is high, energy consumption results in higher CO2 emissions. It is a property of the mix of energy sources that are being used. Data centers aim to reduce their carbon footprint by utilizing renewable energy sources (e.g., wind, hydro, or solar power), adopting low-carbon procurement standards, implementing measurements to prevent deforestation, and other sustainable practices. While some data centers (especially hyperscalers [10]) can prioritize renewable energy procurement, others rely on the energy mix available from regional or national grids, depending on local regulations and infrastructure. Despite ongoing efforts, the average carbon emission of modern data center is still very high: according to the IEA [10], [11], data centers generated 330 Megatons of CO2 equivalent in 2020, constituting 0.6% of total global greenhouse gas (GHG) emissions and 0.9% of energyrelated GHG emissions. Power Usage Effectiveness Power Usage Effectiveness (PUE) [12] is a metric to quantify the ratio between

overall data center energy and IT equipment energy. Additional energy consumption beyond IT equipment energy is attributed to overheads such as cooling, and an energy-efficient data center should have a PUE close to 1. However, PUE does not consider the utilization of computational resources [13]. High Throughput Computing versus High Performance Computing High Performance Computing (HPC) and High Throughput Computing (HTC) are two computing paradigms that address different types of workloads with distinct characteristics. While both aim to achieve high performance, their metrics for performance are different. HPC aims to minimize time-to-solution for complex computational tasks (e.g., floating point operations (FLoPS)), while HTC focuses on maximizing the number of jobs completed within a given time frame [6]. This dictates different types of optimizations and resource allocation schemes. Large Hadron Collider beauty experiment (LHCb) is a typical scientific computing example of HTC [14]: it is a particle physics detector experiment that needs to maintain high real-time experimental data throughput in each server, and the server needs to send important filtered physical information to a collector within the servers’ cluster for further processing. Cores, Threads and Jobs Many chip multiprocessors (CMPs) today have multiple processing cores, often referred to simply as cores. Each core is an independent processing unit, with its own dedicated hardware resources, and running a program. All CPU cores are integrated into a single chip, typically residing on the same silicon die. Each core typically has one or two levels of dedicated cache, while higher-level caches and external memory are generally shared among multiple cores. Data center servers have tens to hundreds of cores, meaning that they can execute many programs in parallel. In addition, each core typically handles one hardware thread at a time, although Intel and AMD CPUs allow two threads to be ’pinned’ to a core. Moreover, a program may also support multiple threads through software level parallelism. In this paper, we refer only to the hardware-level threads. In addition, a thread is sometimes referred to as a logical core or a virtual CPU (vCPU). We consider each isolated running application as a job. The application can be executed either directly on the physical server or in a virtualized environment (e.g., a virtual machine or a docker container). Using virtualization is the preferred option in most data centers, as it provides isolation between users, enables easy resource management and migration. DVFS and performance boosting Dynamic Voltage and Frequency Scaling (DVFS) is a technique that can dynamically adjust the voltage alongside the frequency of a CPU. In general, power grows with the square of voltage and linearly with frequency: P =C ·V2·f

(2)

where C is the capacitance, V is the supply voltage, and f is the operating frequency. DVFS monitors the activity of the CPU, and based on the measurements decreases voltage and frequency when the device is idle, and increases them when the utilization is high to maximize performance. This leads in turn to power savings in time of low activity.

3

Performance-boosting technologies, such as Intel’s Turbo Boost and AMD’s Precision Boost, are forms of DVFS designed to temporarily increase CPU frequency to optimize job execution performance. These technologies boost the frequency for short bursts, typically during high-demand moments, but cannot sustain the increased frequency over long workloads. As such, they are focused on enhancing performance rather than saving energy. Figure 1. PUE in STFC over the last 2 years

B. Related work This subsection presents recent related work on sustainable computing in data centers. Power and energy consumption models and measurements Modelling the power consumption of data centers servers was the focus of many works [15], [16]. Most works modelled the power consumption as an additive model [17], measuring the sum of the power consumption all of server components. However, these works rarely considered the combined impact of other parameters (e.g., the number of CPU cores, the number of jobs, DVFS) on power consumption. In this work, a non-linear model built on top of Baseline-active model [17] is used to investigate the effect of multiple parameters on the energy consumption of highly-utilized data center servers. Jordi et al. [7] conducted measurement-based experiments on energy consumption of data center servers. They only considered the impact of frequency, number of cores and I/O. This work, beyond exploring energy consumption in newer generations of servers, explores more parameters and provides further sensitivity analysis. Server virtualization Server virtualization (e.g., virtual machines and containers), which enables running multiple isolated applications on a single server, can help reduce energy usage if suitable virtualization is deployed. Jin et al. [18] empirically investigated the energy efficiency using server virtualization. It has been shown that an optimum number of virtual machines can maximize power savings. In [19], Zhang et al. proposed a method to estimate the power consumption of containers and virtual machines for better resource scheduling and green computing. This work builds upon previous research to explore the benefits of server virtualization and containerization, aiming to identify the optimal strategy for minimizing the energy consumption of containers running on a single server. By doing so, it seeks to reduce the overall energy consumption of the data center. Power consumption adjustment based on workload Power consumption can be reduced by adjusting the voltage and clock frequency through DVFS or by reducing the number of active cores in a server [3]. Usually, DVFS is managed by keeping track of the CPU utilization. The frequency is increased when the utilization is high and becomes low when idle. However, Google [20] indicated that when the machines are heavily utilized, DVFS has limited effectiveness in terms of energy savings. Other similar solutions, like reducing the number of active cores, rely on slowing down the workload to reduce real-time power consumption, but this may significantly affect the user experience and result in even longer execution time. As we show in this paper, reducing the number of active cores has negative impacts on energy saving in data center servers.

Figure 2. IT power consumption in STFC over the last 2 years

Additional approaches There are many additional approaches to increase energy efficiency, such as improving cooling efficiency [4] and using energy-efficient hardware [5]. For example, it has been shown that replacing aircooled system with liquid-cooled system can help reduce the PUE from 1.48 to 1.14 [4]. It is also possible to save energy by using more energy-efficient components (e.g., Solid State Drive (SSD)) in data center servers. Hard Disk Drives (HDD) consume 85% of overall energy when the server is idle [21], [22], whereas in SSD the energy consumption is proportional to I/O operations per second. This means that upgrading storage devices to SSD can reduce energy usage [5]. Unlike aforementioned approaches, the goal of this work is not to minimize the PUE or upgrade IT infrastructure, but to decrease the energy consumption of existing IT equipment, especially when highly utilized. III. A Study of Power Consumption at STFC RAL We consider R89, a data center operated by STFC [23] at the Rutherford Appleton Labs (RALs) as a real world use case for analysis purposes. The goal is to minimize the energy consumption of the data center. As a reference, from 2019 to 2020, their utilities generated 33,885 tonnes of CO2 [24]. Hence, diminishing the electricity consumption of servers in data centers is still valid, and can facilitate a reduction in carbon emissions. The CERN particle collider is the largest high energy physics experimental site in the world, and STFC RAL provides Tier-1 computing infrastructure for CERN experiments, being responsible for processing around 10% of CERN data. The majority of compute is required for Monte Carlo analysis, reconstruction and filtering. The type of compute is High Throughput Computing (HTC), meaning that many jobs by different uses are executed simultaneously within the data centers, with multiple users sharing the same server. This indicates that the data center servers have high utilization but low dependence between servers, even within the same rack. It is critical to study the energy usage of HTC because most scientific computing, like particle collider for nuclear

4

Table I Monitored Racks’s Specifications and Median and 75th Percentile Power Measurements. Server and core power are normalized. Rack ID

Type

#Servers

CPU

113 124

Dell16 Dell17 Dell17 XMA17 XMA20 XMA20 XMA21 XMA20 XMA20 XMA21 Dell19 Dell19

48 48 24 24 20 24 12 20 24 12 48 48

Intel Xeon E5-2630 V4 2.30GHz Intel Xeon Gold 6130 2.10GHz Intel Xeon Gold 6130 2.10GHz Intel Xeon Gold 5120 2.20GHz AMD EPYC 7452 2.35GHz AMD EPYC 7452 2.35GHz AMD EPYC 7763 2.45Ghz AMD EPYC 7452 2.35GHz AMD EPYC 7452 2.35GHz AMD EPYC 7763 2.45Ghz AMD EPYC 7452 2.35GHz AMD EPYC 7452 2.35GHz

125 169 170 177 179 180 224 406 407

Logical cores per server 40 64 64 56 128 128 256 128 128 256 128 128

research, is running HTC intensive workloads. Almost 100% of STFC’s computing workloads are HTC, because Monte Carlo simulations require very large computational resources over a long period of time. Minimizing STFC’s energy consumption can effectively reduce the carbon emissions. STFC is continuously monitoring its PUE and power usage over time. As shown in Figure 1, from 2020 to 2022, the PUE of STFC ranges from 1.28 to 1.45, and the small variations are caused by the seasons. Figure 2 shows the IT power usage of the STFC data center (without cooling and other overheads) over the last two years. It shows that the data center consumes about 1.2MW, and this power consumption does not vary significantly over time, except when new equipment is introduced. This power consumption is also coupled with high utilization of the computing resources, of over 92%. Changes in the data center’s overall energy consumption were mostly due to PUE changes, i.e., changes in cooling efficiency over the seasons, rather than changes in IT power consumption. The insight from these results is that for this data center there is little opportunity to reduce non-renewable energy carbon emissions or energy costs by shifting workloads over time, as the data center is continuously highly utilized, throughout the year and during all hours of the day. Servers in the data center are organized within racks, where each server is connected to two Power Distribution Units (PDU), for redundancy purposes. Each rack typically hosts three PDUs. We collect a dataset of power measurement reports by the PDUs over two months. The measurement resolution is 6 hours, and the start and stop time of all PDUs are identical. Data is collected from 335 PDUs, but only 315 contain non-NULL values and are used for the analysis. The power consumed by PDUs varies significantly, from 2.2W to 6634W, but per-PDU the variance over time is small, with the average standard deviation being 9.1%. While the initial goal of our analysis was to observe variability of power consumption at server level and identify trends in power consumption, this could not be achieved as available measurements include only the total power consumption per PDU, rather than per socket, and as servers within a rack have their pair of power connections spread between three or four PDUs in a mixed manner. For this reason, we consider rack-level power consumption, as an aggregation of the power consumed by all PDUs within a given rack, and therefore all the computing equipment within that rack. Once racks with partial PDU information (e.g., measurements for one PDU are missing) are omitted, we are left with 11 racks. The

Memory (GB) 160 192 192 192 512 512 1024 512 512 1024 512 512

Disk (GB) 2400 1863 1863 1863 3576 3576 7680 3576 3576 7680 3576 3576

Power per rack (W) Median 75% 3673 9747 18049 18369

Power per server (W) Median 75% 76.5 203.1 376 382.7

Power per core (W) Median 75% 1.91 5.08 5.88 5.98

17092

17980

356.1

374.59

5.93

6.24

10578 12578 12394 10669 12804 11390 30295 25037

10713 12712 12541 10753 12902 11700 30606 25444

528.9 524.1 1032.9 533.5 533.5 949.2 631.16 521.6

535.7 529.7 1045.1 537.6 537.6 975 637.65 530.1

4.13 4.09 4.03 4.17 4.17 3.71 4.93 4.08

4.19 4.14 4.08 4.2 4.2 3.81 4.98 4.14

specifications of the racks are provided in Table I. As can be seen, the dataset contains a mix of server specifications from different vendors, CPU generation, number of cores, and with a different number of servers per rack. Figure.3 shows the power consumption on rack level, for each of the 11 racks, based on the PDU measurements. We also capture the median and 75th percentile in Table I, with more information available in the accompanying repository [25]. As the figure shows, the power consumption of racks varies significantly between configurations: from 10kW to 30kW. Some equipment within the racks experiences failures from time to time, as demonstrated by significant power drops of certain racks. In particular, rack 113 had servers taken out of production between the middle of August 2022 and the end of September 2022. The power consumption of a rack is not representative of its computing power, as the number of servers and cores power varies between racks. We define server power = rack #servers rack power and core power = #servers×#cores per server . Note that we refer to the number of logical cores per server. The results are shown in Figure 3(b), Figure 3(c) and Table I. As can be expected, servers with more cores consume more power. A surprising result is that the difference in power per logical core is small when considering different generations of AMD processors and different number of cores. For example, based on their thermal design power (TDP), AMD EPYC 7763 is supposed to be 10% more power efficient per core than AMD EPYC 7452, while running at 4.2% higher frequency. However, our results show that racks with these two generations of servers can have differences in median and 75th percentile as small as 1.5%, e.g. racks 170 and 177. This is as the results take into account the entire power consumed by the rack, including a server’s power supply, memories, local storage and other overheads. While it is commonly assumed that moving to newer, denser generation of CPUs improves the power efficiency and should therefore be preferred, our insight is that it may be better to delay the decommission of systems, as the overall power saving per core will be small. This is reinforced when considering embodied carbon emissions of servers. IV. Modeling Energy Consumption Based on the previously discussed results, reducing the carbon emissions of the studied data center is not possible by a temporal workload shift. This leads us to explore carbon reduction through the reduction of energy consumption of workloads. Unlike a goal of purely reducing the power consumption of a server, our goal is to minimize the overall energy consumption of the data center over

5

20,000

10,000

0 Aug 09 2022

Sep 01 Sep 24 2022 2022 Timestamp

(a) Per-rack power consumption

800 600 400 200 0 Aug 09 2022

Oct 12 2022

7

1,100 1,000

Power consumption (W)

Power consumption (W)

32,000 30,000 Power consumption (W)

Rack 113 Rack 124 Rack 125 Rack 169 Rack 170 Rack 177 Rack 179 Rack 180 Rack 224 Rack 406 Rack 407

Sep 01 Sep 24 2022 2022 Timestamp

Oct 12 2022

(b) Per-server power consumption

6

4

2

0 Aug 09 2022

Sep 01 Sep 24 2022 2022 Timestamp

Oct 12 2022

(c) Per-core power consumption

Figure 3. Rack-level, server-level (normalized), and core-level (normalized) power consumption over time based on PDU measurements

time, while optimizing for performance. As discussed next, these two goals are not the same. To this end, this section outlines the modeling of energy usage by data center servers. A. Single server energy consumption model Baseline-Active (BA) model [17] is the most popular model of servers’ power consumption. The baseline power Pbaseline is the power consumption of an idle server, which includes the power usages of CPU, memory, disk, and any other device, like a Network Interface Card (NIC). The baseline power usually does not vary significantly over time. The active power Pactive indicates the additional power consumption when running one or several jobs on the server. Pactive contains elements such as the power consumed by memory and disk usage, network I/O as well as CPU utilization. Therefore, the overall power consumption P can be expressed as: P = Pbaseline + Pactive

(3)

In this paper, Pactive is extended to a non-linear model that considers two more parameters, the number of utilized logical cores Ncores (i.e., the concurrency level of the workload as described in [26]) and the number of running jobs Njobs on the server, the power consumption can be denoted by: P = Pbaseline + Punit · Ncores · Njobs

(4)

where Punit is the average power consumption (over the sampling period) of a single logical core executing one thread of a job, where the complete job is running over Ncores . Simply minimizing Punit (e.g., reducing CPU frequency) will reduce P , but may also increase execution time, leading to higher energy consumption. Therefore, a different method should be used to minimize energy consumption. Being P̄baseline the average idle power consumption during period T , by Equation 1, the total energy consumption is: E = P̄baseline · T + P̄unit · Ncores · Njobs · T = Ebaseline + Eunit · Ncores · Njobs

(5) (6)

Where Ebaseline = P̄baseline · T and Eunit = P̄unit · T . To minimize energy consumption both Ebaseline and Eunit should be minimized. However, this is not sufficient, as T depends on the number of cores used (Ncores ) and potentially also the number of jobs (Njobs ).

The above equation can be expressed as: ∗ E = Eunit · Ncores · Njobs

(7)

∗ where Eunit is the energy consumption per core per job including the baseline. This model is limited, as it assumes that just one type of workload is running. If different types of workloads (denoted by Kworkloads ) are running, the model would be:

E† =

Kworkloads ∑

∗ Eunit · Ncoresi · Njobsi i

(8)

i=1

The model above provides a level of abstraction above CPU power states, though their effect is captured in Section V-A. B. Data center energy consumption modeling In STFC data center, depending on rack and server dimensions, each rack contains 12-48 servers. A top-ofrack switch is deployed on the rack, so that the servers can connect to the data center network. The PDUs are also connected, allowing operators to remote monitor and control the power and energy consumption of servers. If only a single type of a job runs on all servers in a data center, and all servers are the same, the total IT energy consumption of data centers can be denoted by: Etot = E ·

N∑ rack

i Nserver

(9)

i=1

where E represents the total energy consumption of a i single server, Nserver is the number of servers in rack i, and Nrack is the number of racks in data center. However, the reality is that the data center runs multiple types of jobs in parallel, and different types of servers have different energy consumption. Therefore, a more realistic model would be: Etot =

N∑ rack Nserver ∑ i=1

† Eij

(10)

j=1

Considering that the average PUE of STFC RAL from 2020 to 2022 is 1.31 (see Figure 1), the overall energy consumption can be estimated as: Eoverall = 1.31Etot

(11)

that is, the energy consumption of non-IT equipment (e.g., cooling) is around 31% of that of IT equipment.

6

Energy consumption Etot = E1end + E2end −(E1start + E2start )

Power consumption P = P1 + P2

SNMP PDU 1

PDU 2

Table II Servers Information Name Type Motherboard CPU CPU Frequency CPU Frequency Range Num. of Sockets Cores per socket Threads per core Threads per server Memory SSD capacity NIC Accelerator

Container 1 Container 3 Container 2 Container 4 Intel Server

Container 1 Container 3 Container 2 Container 4 AMD Server

Figure 4. The physical testbed for power and energy measurements

It is crucial to understand that the goal of this paper is to reduce energy consumption of IT equipment (i.e., Etot ) with an optimized scheduled job execution strategy, meaning minimizing E † . Therefore, reducing the energy consumption of non-IT equipment is beyond the scope of our work. In the following sections, we limit the discussion to a single workload, with the objective function of minimizing E, and the discussion is extended to the general case in Section VI. V. Empirical results In this section, we will present our experimental results in terms of power consumption, total energy consumption, and job execution time in two different data center servers. In the following experiments, the comparison between the Intel and AMD servers should not be interpreted as a direct like-for-like comparison, since the setup of each server is quite different and their workload for the same job is not equal. Instead, it is useful to consider trends and the power/energy consumption per job when excluding the baseline. A. Experimental setup To better understand energy-to-performance trade-offs, a focused set of measurements is run on a small research and development cluster. This subsection describes the setup of our measurements. Figure 4 provides an overview of our physical testbed, focusing on the evaluated servers. 1) Physical setup: Two servers are used for the evaluation, one (referred to as Intel server for the sake of conciseness) is DELL PowerEdge R740 using Intel Xeon Gold 6234 CPU, running at 3.30GHz, and the other one (called AMD server) is ASUS ESC4000A-E10 server equipped with a AMD EPYC 7302P 16-Core Processor. As shown in Table II, both servers have a total of 16 cores and 32 threads (logical cores); the Intel server is dual sockets, while the AMD server has one CPU socket. Both servers are quipped with DDR4 memory, and use SSD NVMe disk during the experiments. Both servers are equipped with a dual-port NVIDIA Mellanox ConnectX 5 NIC (Network Interface Card). In addition, each server is equipped with HDD (idle), and one accelerator card: Alveo U280 in the Intel Server, and NVIDIA BlueField-2 in the AMD

Intel DELL PowerEdge R470 0WXD1Y A01 Intel Xeon Gold 6234 Base 3.3GHz, Max 4.0GHz 1.2GHz-4GHz 2 8 2 32 384GB DDR4-2933MT/s 1.6TB NVIDIA ConnectX-5 MCX516A-CCAT AMD Alveo U280 FPGA

AMD ASUS ESC4000A-E10 KRPG-U8 60SB08F0-SB0A04 AMD EPYC 7302P Base 3.0GHz, Max 3.3GHz 1.5GHz-3.3GHz 1 16 2 32 256GB DDR4-3200MT/s 2TB NVIDIA ConnectX-5 MCX516A-CDAT NVIDIA BlueField-2 DPU

servers. Both accelerator cards are idle, and represent the increasingly common case of acceleration presence. They are further discussed in Section V-B. Each server is connected to two smart PDUs (Eaton ePDU EMAH06), that collect real-time power and energy consumption. By default, DVFS and performance boosting technologies are enabled. Note that the PDUs have IEC Class 1 (±1%) billing grade accuracy. During our experiments, no other jobs are running in servers, and so no external interference. The read power is always stable. 2) Workload: ATLAS [27] is an experiment of CERN particle collider. Workloads related to ATLAS consume 37.94% of the Tier-1 batch farm, with simulations being about half of that. ATLAS simulations generate intensive computing loads with low concurrency, making it is a good use case for analyzing the energy efficiency of servers running HTC. More details are in [25]. ATLAS simulations are running on servers within Docker [28] containers. Each Docker container running anATLAS simulation is considered a job. CVMFS [29], the CERN VM File System is running on the server too. The number of logical cores used by a workload can be configured. In our experiments, each physical core has two threads (logical cores) running, i.e., if the number of logical cores is set to 32, it means that there are 16 cores running 32 threads. CPU utilization refers to per-core utilization. For example, when 8 cores are configured, the CPU utilization during an ATLAS simulation can reach 800%. 3) Power and energy measurements: The experimental setup uses SNMP (Simple Network Management Protocol) to read the power consumption from the two PDUs connected to the server. Given power read P1 from PDU 1 P2 from PDU 2, the sum of two read values, P1 + P2 , is the overall power consumption P of the server. Similarly, we read the cumulative energy consumption E from the start of a job. E1 and E2 are used to denote the energy consumption of PDU 1 and PDU 2, correspondingly. Therefore, the overall energy consumption of a job Etot is computed as the total energy consumption at the end E1end + E2end minus that at the beginning E1start + E2start . The PDUs sample at 50Hz frequency, and we collect all values every 5 seconds. The monitoring and data collection is done from a different server, so there is no additional overhead on the monitored servers. Throughout this paper, the power and energy consumption of an idle server is denoted as baseline. B. Baseline Power We begin by measuring the power and energy consumption of an idle server. In addition, we measure the

100

0

900

1,800 2,700 Time (SEC)

3,600

(a) Power consumption

200

0.15 0.1

150

0.05 0

900

1,800 2,700 Time (SEC)

3,600

(b) Energy consumption

Figure 5. Intel Server Baseline Measurements

power consumption contribution of each accelerator card and NICs. While we measure the idle power of a NIC, the power consumed by a NIC changes when a job is running, as data is fetched and sent. All cards used have a connected 100G Direct Attach Cable (DAC). Figure 5 shows the idle power and cumulative energy of the Intel server over an hour, and the effect of the cards. The baseline power of the server is 251W with both cards, and 235W without the FPGA card. The NIC consumes 15W, and is a mandatory component. As shown in Figure 6, the baseline power of the AMD server is 204W, including 58W contributed by the DPU and 38W by the NIC. The higher NIC contribution was checked multiple times, and is in part due to the server consuming more power when the NIC is present and in part due to higher transceivers draw. Overall, it is observed that the baseline energy consumption of the Intel server is 0.25kWh, and the baseline of the AMD server is 0.2kWh. To evaluate the maximum power consumption of each server, the Linux stress tool is used. The Intel server consumes up to 460W, while the maximum measured power consumption of the AMD server is 310W. This means that an idle Intel server consumes 54% of the maximum server power, and the idle AMD server consumes 66% of its maximum power. These idle-to-maximum power ratios match similar power measurements in STFC RAL for production servers of the same generation. Note that GPUs were not part of STFC data center servers. For information on energy consumption of GPUs, the readers can refer to the report of our UKRI Net-Zero partners [30]. Finding 1. The energy consumption of idle servers is considerable, and can be more than a half of a fully utilized server. C. Impact of DVFS This section explores the effect of DVFS on energy consumption. In all the experiments, we run a single job using its default configuration of 8 logical cores. We used cpupower [31] to force the CPU frequency of the Intel server to two different values, the highest 4GHz and the lowest 1.2GHz, and compared it to running with DVFS enabled. The results, shown in Figure 7(a) demonstrate that using DVFS and setting to peak frequency have almost the same power profile. This is because the CPU utilization in our experiments is always high and the CPU frequency is always at peak when running jobs.

100

0

900

1,800 2,700 Time (SEC)

3,600

Total energy consumption (kWh)

150

250

0.2

Baseline Baseline without NIC Baseline without DPU and NIC

0.3 0.25 0.2 0.15 0.1

Baseline 1.2GHz

0.05 0

900

1,800

2,700

3,600

0

2,000

4,000

6,000

8,000

DVFS 4GHz

Time (SEC) 0.65 0.6 (a) Power consumption (b) Energy consumption 0.55 Figure 6. AMD Server0.5 Baseline Measurements 0.45 Baseline DVFS 0.4 1.2GHz 4GHz Baseline DVFS 0.35 0.65 1.2GHz 4GHz 0.6 0.3 0.55 0.5 0.25 350 0.45 0.4 0.2 0.35 300 0.3 0.15 0.25 0.2 250 0.1 0.15 0.1 0.05 0.05 200 Total energy consumption (kWh)

200

0.25

Total energy consumption (kWh)

250

300

0.3

Power consumption (W)

Power consumption (W)

300

Baseline Baseline without NIC Baseline without DPU and NIC

Baseline Baseline without FPGA Baseline without FPGA and NIC

Power consumption (W)

Baseline Baseline without FPGA Baseline without FPGA and NIC

Total energy consumption (kWh)

7

Time (SEC) 0 2,000 4,000 6,000 8,000 Filled markers indicate the end of jobs Time (SEC) Filled markers indicate the end of (a) Power consumption (b) Energy consumption jobs

150

0

2,000 4,000 6,000 Time (SEC)

8,000

Figure 7. Intel server: The effect of CPU Frequency (1 job with 8 cores)

When the frequency is low (i.e., 1.2GHz), even though the server runs at full utilization, the power consumption decreases to 100W, and the execution time triples. The energy consumption shown in Figure 7(b) tells us that the lowest CPU frequency is much less energy efficient than DVFS: DVFS only needs 0.25kWh to finish the job, while 1.2GHz requires more than 0.5kWh to execute the same job. Note that in this and following energy consumption figures we use filled markers (e.g., ■) to indicate job completion time. A similar experiment is executed on the AMD server, adjusting the CPU frequency to the minimum 1.5 GHz and the peak 3GHz. Figure 8 shows a comparison between DVFS and fixed CPU frequencies. The power consumption in Figure 8(a) shows similar trends to the Intel server, that is, DVFS and CPU frequency of 3GHz have a similar performance, and 1.5GHz only consumes 10W more than the baseline. Figure 8(b) reveals running at a high frequency consumes about half the energy of running the same job at lower frequency. Finding 2. Running at low frequency reduces the momentary power consumption, but running at high frequency reduces the overall amounts of energy required to complete a job. Figure 9 shows a comparison of power and energy consumption when Turbo Boost is enabled or disabled in the Intel server. In Figure 9(a), shows that disabling Turbo Boost saves around 40W of power consumption, but it takes around 500 seconds longer to execute the job. When observing the energy consumption in Figure 9(b), it is evident that enabling boosting is more energy efficient, as for the same job execution, the total energy consumption

180

0

3GHz 8

1,200 2,400 3,600 4,800 6,000 Time (SEC)

Baseline 1.5GHz

Baseline Boosting disabled

DVFS 3GHz

0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05

250 230 200

150 0 1,200 2,400 3,600 4,800 6,000 Time (SEC) Filled markers indicate the end of jobs

0 1,200 2,400 3,600 4,800 6,000 Time (SEC) (a) Power consumption (b) Energy consumption Filled markers indicate the end of jobs Figure 8. AMD server: The effect of CPU Frequency (1 job with 8

Total energy consumption (kWh)

Power consumption (W)

Boosting enabled

350 300

900

1,800 2,700 Time (SEC)

3,600

Baseline Boosting disabled 0.25

Boosting enabled

0.2 0.15 0.1

250

0 900 1,800 2,700 3,600 Time (SEC) Filled markers indicate the end of jobs

200

(b) Energy consumption 0

900

1,800 2,700 Time (SEC)

3,600

Figure 10. AMD server: Impact of boosting (1 job with 8 cores)

(a) Power consumption Boosting enabled

0.3 0.25 0.2 0.15 0.1 0.05 0 900 1,800 2,700 3,600 Time (SEC) Filled markers indicate the end of jobs

(b) Energy consumption

Power consumption (W)

Baseline Boosting disabled 0.35

1 core 8 cores 16 cores (32 threads)

Baseline 4 cores 16 cores

450

1 core 8 cores 16 cores (32 threads)

0

1,800 2,700 Time (SEC)

350 300

400 350

250

300 250

200

200 150

Baseline 4 cores 16 cores Power consumption (W)

150

Total energy consumption (kWh)

0

(a) Power consumption

cores) Baseline Boosting disabled

Boosting enabled

Power consumption (W)

200

0.4 0.35 0.3 0.25 DVFS Baseline 1.5GHz 3GHz 0.2 0.15 0.1 0.05

Total energy consumption (kWh)

220

Total energy consumption (kWh)

Power consumption (W)

250

1.5GHz

0

900

1,800 2,700 Time (SEC)

(a) Intel

3,600

150

900

3,600

(b) AMD

Figure 11. Sensitivity analysis: Num. of cores (1 job) to power consumption

Figure 9. Intel server: Impact of boosting (1 job with 8 cores)

D. Impact of cores number and jobs number - Power with boosting enabled (■) is lower than that without boosting (▲). Figure 10 shows the equivalent experiment using the AMD server, when enabling or disabling Precision Boost. Figure 10(a) indicates that boosting has the same impact on AMD server as on Intel’s, that is, the power consumption with boosting disabled is lower than that with boosting enabled, but the execution time is longer. However, in AMD server, less than 15W are saved when boosting is disabled. This makes the differences in total energy consumption small: as shown in Figure 10(b), the curves of boosting enabled and boosting disabled almost overlap, but still with boosting enabled the server finishes the job earlier and saves energy. Finding 3. Enabling boosting can reduce job execution time with some additional power overhead, leading to lower total energy consumption. Note that the above observation holds mainly due to the significant contribution of baseline power.

In the following experiments, we explore how the number of jobs and the number of logical cores affect energy efficiency. Figure 11 reports the sensitivity of power consumption to the number of cores. When there are more cores assigned to a job, the execution time becomes lower, but the power consumption during execution is higher. As shown in Figure 11(a), the baseline of Intel server is around 250W. 4 cores execution requires additional 40W, while 8 cores execution needs 90W more than baseline. However, when the number of cores is 16, the power consumption reaches 460W, the maximum. Increasing the number of cores does not significantly increase the power consumption during execution, but the execution time is lower. Figure 11(b) shows the power consumption of AMD server: similar to Intel server, the power consumption of AMD server increases as the number of cores increases but requires shorter execution time. When we increase the number of threads from 16 to 32, unlike Intel server, the power consumption continues increasing. Figure 12 shows a comparison of power consumption when changing the number of jobs, while keeping the

2 jobs

4 jobs

1 job

2 jobs

4 jobs

Power consumption (W)

450

300

400 350

250

300 250

200

200 150

Baseline 350

0

1,000 2,000 3,000 4,000 5,000 Time (SEC)

150

0

(a) Intel

1,000 2,000 3,000 4,000 5,000 Time (SEC)

Baseline 0.65 0.6 0.55 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05

1 job

2 jobs

4 jobs

0 1,000 2,000 3,000 4,000 5,000 Time (SEC) Filled markers indicate the end of jobs

(b) AMD

Total energy consumption (kWh)

1 job

Power consumption (W)

Baseline 500

Total energy consumption (kWh)

9

Baseline 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05

0 900 1,800 2,700 3,600 Time (SEC) Filled markers indicate the end of jobs

(a) Intel

1 core 8 cores 16 cores (32 threads)

Figure 14. Sensitivity analysis: Num. of jobs (8 cores) to energy consumption Baseline Same socket

0.25 500

0.2

450

400

400

350

0.1

350

300

0.05

300

250

250

200

0 900 1,800 2,700 3,600 Time (SEC) Filled markers indicate the end of jobs

(b) AMD

150

Different sockets

500

450

0.15

Baseline Same socket

Different sockets Power consumption (W)

0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05

Baseline 4 cores 16 cores

4 jobs

(b) AMD

Power consumption (W)

1 core 8 cores 16 cores (32 threads)

Total energy consumption (kWh)

Total energy consumption (kWh)

Baseline 4 cores 16 cores

2 jobs

0 1,000 2,000 3,000 4,000 5,000 Time (SEC) Filled markers indicate the end of jobs

(a) Intel Figure 12. Sensitivity analysis: Num. of jobs (8 cores) to power consumption

1 job

200

0

900

1,800 2,700 Time (SEC)

(a) 1 job

3,600

150

0

1,000 2,000 3,000 4,000 5,000 Time (SEC)

(b) 2 jobs

Figure 15. Sensitivity analysis: Use of sockets to power consumption Figure 13. Sensitivity analysis: Num. of cores (1 job) to energy consumption

number of cores constant (8). From Figure 12(a) it can be seen that in Intel server, the power consumption of 2 jobs (∼200W more than baseline) is double of that of 1 job (∼100W), for almost same execution time. On the other hand, when running 4 jobs, the maximum power consumption does not increase relative to 2 jobs, but job execution is longer. This is due to resources bottleneck, as they are fully utilized. Figure 12(b) confirms that an increased number of jobs has the same effect in AMD server: the 1 job simulation requires additional 50W over the baseline, and 2 job simulations require twice of that. It is worth noticing that each job on the AMD server consumes only half of the energy consumption on Intel server. Once the power consumption of 4 jobs execution reaches around 310W, the AMD server is at peak utilization and requires more time for completing all 4 jobs. E. Impact of cores number and jobs number - Total energy Figure 13 shows a sensitivity analysis of energy consumption to the number of cores. In Intel server (see Figure 13(a)), during job execution, the slope of the graph increase with the number of cores (up to 16 physical cores). However, the overall energy consumption of jobs executed on a higher number of cores is lower than that of jobs using less cores. Figure 13(b) reveals that the number of cores has the same effect on energy consumption in AMD server. When more cores are used, less energy is consumed by the same job. Interestingly, using 16 cores (32 threads) less than 0.1kWh is required to complete a job. Figure 14 illustrates energy consumption sensitivity to the number of jobs. As shown in Figure 14(a), on Intel server, the energy consumption of 2 jobs is initially the

same as 4 jobs (until the 2 jobs are complete), and 1 job consumes about half the energy of 4 jobs. However, that means that running 4 jobs is twice as efficient as running a single job, in terms of energy per job. Figure 14(b) shows that similarly on AMD server, the total energy consumption increases as the number of jobs increases. These results are further analyzed for per-job energy efficiency in Section V-I. Finding 4. As long as CPU resources are not fully utilized, it is more energy efficient to increase the number of jobs running on a CPU. F. Impact of CPU sockets So far, running the simulation on Intel server with the default 8 cores per job, the cores were always distributed between two CPU sockets. In order to further understand the impact of CPU sockets usage on power and energy consumption, we used taskset, a Linux shell command, to force all cores to run on a specific socket. In this experiment, we only consider 1 job and 2 jobs, as each job uses 8 logical cores, and only 16 are available on a single socket. The difference in power consumption between using 1 or 2 sockets on Intel server is reported in Figure 15. For one job, Figure 15(a) shows that the curves of both cases almost overlap, meaning that the distribution between sockets does not affect the power consumption considerably, but the job execution time is slightly shorter on a single socket. However, as shown in Figure 15(b), when running two jobs and all 16 logical cores are forced on one socket, the power consumption is the same as that of one job, but the execution time is nearly as long. This also matches the power consumption results for 2 jobs and 4 jobs previously shown in Figure 12(a). Note that initially

Baseline Same socket

Different sockets

0 900 1,800 2,700 3,600 Time (SEC) Filled markers indicate the end of jobs

0.55 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05

250 200 150

450 400 350 300 250 200 150

0

1,200 2,400 3,600 Time (SEC)

4,800

(a) Power consumption

Total energy consumption (kWh)

1 job 4 jobs

Power consumption (W)

Baseline 3 jobs

4 jobs in parallel 2 jobs

Baseline 3 jobs

1 job 4 jobs

0.6 0.55 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0 1,200 2,400 3,600 4,800 Time (SEC) Filled markers indicate the end of jobs

0

1,000 2,000 3,000 4,000 5,000 Time (SEC)

(a) Power consumption

(b) 2 jobs

Figure 16. Sensitivity analysis: Use of sockets to energy consumption

4 jobs in parallel 2 jobs 500

1 job 4 jobs

300

0 1,000 2,000 3,000 4,000 5,000 Time (SEC) Filled markers indicate the end of jobs

(a) 1 job

Baseline 3 jobs

Power consumption (W)

0.55 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05

4 jobs in parallel 2 jobs 350

Different sockets

Total energy consumption (kWh)

Baseline Same socket

Total energy consumption (kWh)

Total energy consumption (kWh)

10

4 jobs in parallel 2 jobs

Baseline 3 jobs

1 job 4 jobs

0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0 1,000 2,000 3,000 4,000 5,000 Time (SEC) Filled markers indicate the end of jobs

(b) Total energy consumption Figure 18. Staggered jobs start on AMD server (Tint = 450s)

(b) Total energy consumption

Figure 17. Staggered jobs start on Intel server (Tint = 450s)

the power consumption is high for 2 jobs on a single socket as child processes can be pinned to cores only after the job is fully running and children processes are forked. The total energy consumption is analyzed in Figure 16. Figure 16(a) shows that for a single job the energy consumption of same socket is slightly lower than that of different sockets. Figure 16(b) shows however that for 2 jobs, the execution on different sockets is more energy efficient. This is because forcing 16 logical cores on the same socket lowers the performance and increases execution time, leading to overall higher execution time. Finding 5. If the total number of logical cores required by a set of jobs is less than the number of physical cores on a socket, it is preferred to use a single socket. G. Impact of job scheduling When starting a new job, there is a short period of initialization time, before children processes are forked. During this time, the power consumption of a job is low. In this subsection, we check the possibility of improving energy efficiency by taking advantage of this low-power period. Figure 17 shows the case of a staggered deployment of jobs on Intel server. At the beginning of the experiment, the first job is started. Afterwards, every 450 seconds we launch a new job until the number of jobs reaches 4. 450 seconds is chosen as it is the approximate initialization time for the default 8-cores job configuration. Figure 17(a), shows the difference between jobs starting at the same time or staggered: when 4 jobs are started in parallel, the power consumption quickly reaches 460W, and the jobs finish after 4100s, which is faster than staggered jobs (as can be expected due to the delayed starts). The power consumption of staggered deployment reaches 460W

when at least 2 jobs running. Figure 17(b) provides a view of the overall energy consumption. It shows that for jobs triggered in parallel, the overall energy consume is somewhat lower, even if we consider CPU idle time at the end of a run. Figure 18 performs the same comparison on the AMD server. As shown in Figure 18(a), given the same time interval of 450 seconds, the power consumption increases as the number of jobs increases, unlike on the first server where peak power was reaches with 2 jobs. 4 jobs triggered in parallel can finish execution 1350 seconds earlier than jobs deployed in a staggered manner. In terms of total energy consumption, the curves in Figure 18(b) reveal that the consumed energy of both scenarios is quite close, and the energy consumption of parallel start is slightly lower at the end of the experiment (including idle time). Finding 6. A parallel triggered start of jobs is slightly more energy efficient than a staggered start.

H. Execution time comparison Figure 19 presents a comparison of job execution time between the two server. The execution time is measured with a terminal application called prmon binding with the process ID of jobs. As shown in Figure 19(a), as the number of logical cores (threads) assigned for a single job increases, the execution time decreases. When varying the number of jobs, as shown in Figure 19(b), the execution time of 1 job and 2 jobs quite similar, while 4 jobs takes more time. However, one considers the per-job execution time, 4 jobs running in parallel require less time than the other cases. Finding 7. Given sufficient CPU resources, running multiple jobs in parallel can reduce per-job execution time.

11

Table III Summary of experiments Server

#Jobs

Intel (Baseline) Intel Intel Intel Intel

#Cores

Avg. power consumption per hour during (W) execution time (W)

Total energy consumption per hour during (kWh) execution time (kWh)

Per job per core Avg. power consumption during during per hour execution per hour execution ∗ (W) ∗ time P̄unit (W) † time P̄unit (W) ∗ (W) †

/

/

252.7

/

0.251

/

/

/

/

/

/

/

/

/

/

1 1 1 1

267.58 302.55 326.90 328.00

268.49 303.88 350.22 397.38

0.265 0.304 0.324 0.323

1.351 0.392 0.251 0.202

13.79 12.2 8.99 4.58

13.79 12.2 11.25 8.88

267.58 75.38 40.57 20.37

268.49 75.38 42.84 24.67

14.00 13.25 9.13 3.63

1351.00 98.00 31.38 12.63

80.22 16.62 9.45 4.63

1351 392.00 251.04 202.08

80.22 66.50 75.63 74.03

314.83

431.32

0.313

0.147

3.88

11.16

19.68

26.96

3.00

9.19

3.81

147.04

61.01

394.77 462.21

456.66 461.35

0.394 0.462

0.315 0.523

8.87 6.54

12.74 6.52

21.74 12.88

28.54 14.42

8.06 6.59

19.69 16.34

8.98 7.44

157.52 130.72

71.81 59.50

205.06

Intel

1

Intel Intel AMD (Baseline) AMD AMD AMD AMD

2 4

1 4 8 16 16 (32 threads) 8 8

/

/

1 211.39 4 223.93 8 234.63 16 235.33 16 (32 AMD 1 236.34 threads) AMD 2 8 264.49 AMD 4 8 310.76 ∗ Baseline excluded † Baseline included

1 1 1 1

/

0.205

/

/

/

/

/

/

/

/

/

/

211.06 223.33 242.95 274.87

0.210 0.225 0.234 0.235

1.241 0.354 0.189 0.118

6.66 4.21 3.70 1.89

6.39 4.21 4.74 4.36

211.39 55.48 29.33 14.71

211.06 55.48 30.37 17.18

4.00 5.00 3.63 1.875

1212.95 88.50 23.63 7.38

34.48 7.18 3.67 1.87

1212.95 351.24 189.04 118.08

34.48 28.73 29.39 29.86

290.48

0.236

0.104

2.00

5.34

14.77

18.16

1.94

6.56

1.92

104.90

30.08

280.37 311.02

0.266 0.297

0.222 0.310

3.71 2.86

2.35 2.86

16.53 9.27

17.52 9.27

3.81 2.88

13.88 9.69

3.72 3.56

111.04 77.52

29.75 28.50

·104

Intel AMD

1

4

8

4,000

Execution time (SEC)

Execution time (SEC)

2

0.6 0.4 0.2 0

Per job

Total energy consumption during during during during per hour execution execution execution execution ∗ ∗ −3 time E time Eunit time Eunit (×10 unit time Eunit (×10−3 (×10−3 (×10−3 (×10−3 kWh) ∗ kWh) † kWh) ∗ kWh) † kWh) ∗

16

Number of cores ∗: 32 threads

16∗

Intel AMD

3,000 2,000 1,000 0

1

2

Number of jobs

4

(a) Varying number of cores (1 (b) Varying number of jobs (8 job) cores) Figure 19. A comparison of job execution time

I. Collected statistics A partial statistics collection of this work’s experiments is provided in Table III. The table uses two computed metrics, average power consumption per hour, and average power consumption during execution time. Average power consumption per hour indicates the power efficiency including idle time at the end of a job, while average power consumption during execution time only considers the power consumption when the jobs are still running. The total energy consumption per hour and the total energy consumption during execution time, refers to the same two measurement scenarios applied to energy consumption. As the table shows, increasing the number of cores assigned to a job improves the energy efficiency. Moreover, enabling more threads on the same cores (e.g., from 16 threads to 32 threads) can efficiently reduce the energy consumption. Another observation is that for both, given the same number of active cores in an experiment, 2 jobs with 8 cores each consume less energy per job than 1 job with 16 cores. The next step is to subtract the baseline power from the results, and analyze per-core, per-job power consumption of each experiment. These two additional metrics are referred to as average power consumption per-job percore per-hour, and average power consumption per-job per-core during execution time. The results indicate that

when idle time exists, running a single job with more cores is preferred. Otherwise, running multiple jobs with the same overall number of cores is more power efficient. Still, the effect of baseline power cannot be neglected. With it, given the same number of cores, running multiple jobs in parallel is the best solution. Table III also presents the total energy consumption per-job, per-core per-hour and during execution time. Note that the baseline of energy consumption during execution time is the energy consumed until the end of each job’s execution, so the baseline is not the same for different cases. The result is noticeable: given the same total number of cores, running a single job is more energy efficient than running multiple jobs. However, per-job per-core energy consumption is not enough to evaluate a job’s energy efficiency. This is because this metric only indicates the energy consumption of each core, but without considering the impact of job execution time and per-job energy efficiency. Therefore, the table also presents the energy consumption per job, and this metric is evaluated together with per-job execution time in the next subsection. Finding 8. Regardless if servers baseline power is included (†) or excluded (∗), running a single job with more logical cores is better in terms of per-job per-core energy efficiency than multiple jobs using the same total number of logical cores for the ATLAS simulation workload. J. Energy-execution time trade-off In this subsection, we propose a key metric to evaluate energy efficiency for servers, that is, the energy-execution time trade-off. Intuitively, the best energy-execution time trade-off should present the lowest per-job execution time and total energy consumption. To explore the trade-off, we correlate per-job execution time and per-job total energy consumption as a linear regression model. The results of 1 job and 1 core are omitted from the figures of this subsection due to their extremely high execution time and energy consumption.

y = 7.03 · 10−2 · x + 65.51

300 1 job 4 cores 1 job 16 cores 2 jobs 8 cores

200 Total energy consumption per job (×10−3 kWh)

100

300 200 100 0

0

0

1 job 4 cores 1 job 16 cores 2 jobs 8 cores

1 job 8 cores 1 job 16* core (32 threads) 4 jobs 8 cores y = 7.62 · 10−4 · x + 66.45

80

Total energy consumption per job (×10−3 kWh)

1 job 4 cores 70 1 job 16 60cores 2 jobs 8 5040cores 400

1 job 8 cores 1 job 16* core (32 threads) 4 jobs 8 cores y = 5.72 · 10−2 · x + 28.09

30 20 10 0

1,000 2,000 3,000 4,000 5,000 6,000 Execution time per job (SEC)

300 (b) Baseline excluded Figure 20. Intel server: Energy-execution time trade-off comparison 1 job 4 cores

1 job 8 cores 1 job 16* core (32 threads) 4 jobs 8 cores y = 5.72 · 10−2 · x + 28.09

16 cores 200 12 job jobs 8 cores

Total energy consumption per job (×10−3 kWh)

400

100 1300job 4 cores 1 job 8 cores 1200job 16 cores 1 job 16* core (32 threads) 2 jobs 8 cores 4 jobs 8 cores 0 100 y =4,000 2.55 · 5,000 10−4 · x + 28.86 0 1,000 2,000 3,000 6,000 0 80 per 0Execution 1,000 2,000 3,000 time 4,000 5,000 6,000 job (SEC) Execution time per job (SEC) 70 (a) Baseline included 60

1 job 4 cores 1 job 16 cores 2 jobs 8 cores

Total energy consumption per job (×10−3 kWh)

Based on the experimental results, this section presents a series of lessons learned and recommendations, which suggest how jobs can be executed to improve energy efficiency for high-throughput workloads on data center servers. IT power is often referred to as critical power [32], and while it excludes power required for cooling and other overheads, it includes the power overheads of computing itself. One of the interesting lessons of this work, evident both from the study of power consumption at STFC RAL (Section III) and the focused measurements (Section V-A), is the significant effect of computing overheads. These take different forms, and the baseline power of a server only partially captures the effect. One clear recommendation is to focus efforts on minimizing the baseline power of servers. Servers from different vendors with similar CPU, memory and storage specifications may have different baseline power, and data center operator should be attentive to that. Note that over-

1,000 2,000 3,000 4,000 5,000 6,000

Execution time per job 3,000 (SEC) 1,000 2,000 4,000 5,000 6,000 time per job (SEC) (a)Execution Baseline included

1 job 8 cores 1 job 16* core (32 threads) 4 jobs 8 cores y = 2.55 · 10−4 · x + 28.86

50

80

VI. Lessons Learned and Recommendations

1 job 8 cores 1 job 16* core (32 threads) 4 jobs 8 cores y = 7.03 · 10−2 · x + 65.51

400

Total energy consumption per job (×10−3 kWh)

Finding 9. A linear regression model can accurately approximate the energy consumption of a job.

12

0

Total energy consumption per job (×10−3 kWh)

Figure 20 shows the energy-execution time trade-off on Intel server. As shown in Figure 20(a), when considering the idle power consumption, running 4 jobs in parallel with 8 cores has the best performance: it takes only around 1000 seconds and 130Wh to execute one job. Running 1 job with 16 cores (32 threads) has comparable performance as 2 jobs with 8 cores, but 1 job with 16 cores (32 threads) has a slightly lower energy consumption. Since all points are close to the linear regression line, it is also obvious that higher execution time leads to worse energy-time performance. On the other hand, when the baseline is excluded (see Figure 20(b)), the energy consumption of all job executions is mostly about 70Wh, with 10% relative error to the linear regression line. 4 jobs with 8 cores still outperform other options because of its low job execution time. Moreover, 1 job with 16 physical cores (32 threads) can save approximately 10Wh of energy with respect to that of 2 jobs with 8 cores. Figure 21 illustrates the energy-execution time tradeoff comparison for AMD server. Figure 21(a) presents energy-execution time trade-off when the baseline energy consumption is included. As can be expected, the energy consumption linearly increases with job execution time. 4jobs execution with 8 cores each outperforms the other 5 scenarios both in terms of execution time per job and total energy consumption per job. One observation is that the performance of (i.) 1 job with 16 cores (32 threads), (ii.) 1 job with 16 cores, and (iii.) 2 jobs with 8 cores is very similar: they have close energy consumption as well as execution time, but 1 job execution with 16 cores (32 threads) is slightly better than the other two scenarios. If the baseline is excluded, the results shown in Figure 21(b) provide an important observation, as the energy consumption of all scenarios is quite flat and close to 30Wh, and the difference is only the execution time per job. This is an indicator of a good resource usage by the AMD CPU, meaning that the energy consumed by similar processing jobs sufficiently resourced is the same.

Total energy consumption per job (×10−3 kWh)

400

40

70 60

30

50 40

20

30 20

10

10 0

1,000 2,000 3,000 4,000 5,000 6,000 Execution time per job (SEC)

0

1,000 2,000 3,000 4,000 5,000 6,000

(b) BaselineExecution excluded time per job (SEC) Figure 21. AMD server: Energy-execution time trade-off comparison

provisioning a server may increase its energy consumption over time. A related consideration, based on the analysis in Section III, is that upgrading to newer generations of CPUs may not (significantly) increase energy efficiency, as the normalized per-core power consumption (i.e., rack power divided by the number of cores in the rack) is not remarkably different. Given that significant carbonemissions are part of manufacturing new servers (embodied carbon emissions) and decommissioning servers, data center operators may opt to delay replacement of servers.

13

Note that this recommendation applies where CPUs have a mostly similar per-core computing performance, as faster processing time does reduce energy consumption per job. Still, this depends on the ratio of baseline server power and maximum server power. Our measurement results show that lower execution time improves the energy efficiency of jobs, due to the baseline power. Combined with DVFS-related measurements, we reinforce the value of using DVFS and boosting even in highly utilized servers, for the studied highthroughput workload. It may be assumed that an intuitive way to reduce the energy consumption of a data center is to shut down idle service. There are two limitations to this approach. First, frequently turning off servers reduces their reliability [33]. Second, data centers such as STFC RAL are highly utilized and may not have idle operational machines. One potential trajectory may be to introduce finer grain electronic power control mechanisms that allow to turn off an unused core or an unused socket, without affecting reliability. These may also be applied to other parts of the server, e.g., shutting down unused accelerator cards. A way for data center operators to reduce their energy consumption is to throttle the servers by reducing CPU speed. While this reduces the amount of energy taken from the grid, this approach has a negative effect on the energy consumption per job, meaning that over time the energy cost of running a fixed set of experiments will increase. This approach can still be used on short time scales, e.g., if there is a limit to the power that can be drawn from the grid, and the PUE is higher than usual due to hot weather. Another recommended method for data center operators to reduce energy consumption is by reducing utilization through policies and motivating users. For example, introducing spot jobs that run only when utilization is below a certain level or when spare resources are available (e.g., to fully populate CPU cores). Users can also be incentivized to run their jobs only at times that energy is cheap or carbon intensity is low, e.g., providing a designated queue or a class for jobs that are environmentally friendly. VII. Conclusion and future work This paper presented an empirical study on the energy efficiency of data center servers running high-throughput computing workloads. The data analysis of power measurements of a real data center, showed the similarity and differences between different server families. To further improve the energy efficiency of servers, we conduct intensive measurements, and obtain detailed statistics for power and energy usage of current servers. The findings reveal fundamental energy-performance trade-off in servers: minimizing both per-job energy consumption and per-job execution time can assist in choosing the best energy saving strategy. Last, this work discussed lessons learned and recommendations to help data center operators, administrators, and engineers to better design and execute jobs in an energy-efficient way for sustainable computing. Future work will explore effective mechanisms to reduce the idle power of servers. Moreover, we plan to explore power efficiency of other data center equipment.

Acknowledgement This work was funded by the UKRI Net-Zero Digital Research Infrastructure Scoping Project (NE/W007134/1). References [1] IEA, “Electricity 2024,” 2024, https://www.iea.org/reports/ electricity-2024. [2] A. Varasteh and M. Goudarzi, “Server consolidation techniques in virtualized data centers: A survey,” IEEE Systems Journal, vol. 11, no. 2, pp. 772–783, 2015. [3] D. Lo et al., “Towards energy proportionality for large-scale latency-critical workloads,” ACM SIGARCH Computer Architecture News, vol. 42, no. 3, pp. 301–312, 2014. [4] A. Capozzoli and G. Primiceri, “Cooling systems in data centers: state of art and emerging technologies,” Energy Procedia, vol. 83, 2015. [5] J. Shuja et al., “Survey of techniques and architectures for designing energy-efficient data centers,” IEEE Systems Journal, 2014. [6] A. Tsaregorodtsev, V. Garonne, and I. Stokes-Rees, “Dirac: A scalable lightweight architecture for high throughput computing,” in Fifth IEEE/ACM International Workshop on Grid Computing. IEEE, 2004. [7] J. A. Aroca et al., “A measurement-based characterization of the energy consumption in data center servers,” IEEE Journal on selected areas in communications, vol. 33, no. 12, pp. 2863– 2877, 2015. [8] S. Arslan, S. Renganathan, and B. Spang, “Green With Envy: Unfair Congestion Control Algorithms Can Be More Energy Efficient,” in Proceedings of the 22nd ACM Workshop on Hot Topics in Networks, 2023, pp. 220–228. [9] K. Ciko, M. Welzl, and P. Teymoori, “Going Dark: A Software “Light Switch” for Internet Servers,” in 2023 IEEE 29th International Symposium on Local and Metropolitan Area Networks (LANMAN). IEEE, 2023, pp. 1–6. [10] IEA, “Data centres and data transmission networks,” 2023, https://www.iea.org/energy-system/buildings/ data-centres-and-data-transmission-networks. [11] J. Malmodin et al., “Ict sector electricity consumption and greenhouse gas emissions–2020 outcome,” Telecommunications Policy, vol. 48, no. 3, p. 102701, 2024. [12] M. Dayarathna, Y. Wen, and R. Fan, “Data center energy consumption modeling: A survey,” IEEE Communications Surveys & Tutorials, vol. 18, no. 1, pp. 732–794, 2015. [13] G. A. Brady et al., “A case study and critical assessment in calculating power usage effectiveness for a data centre,” Energy Conversion and Management, vol. 76, pp. 155–161, 2013. [14] R. Krawczyk et al., “Ethernet for high-throughput computing at cern,” IEEE Transactions on Parallel and Distributed Systems, vol. 33, no. 12, pp. 3640–3650, 2022. [15] S. L. Song, K. Barker, and D. Kerbyson, “Unified performance and power modeling of scientific workloads,” in Proceedings of the 1st International Workshop on Energy Efficient Supercomputing, 2013. [16] V. Perumal and S. Subbiah, “Power-conservative server consolidation based resource management in cloud,” International Journal of Network Management, vol. 24, no. 6, pp. 415–432, 2014. [17] C. Jin et al., “A review of power consumption models of servers in data centers,” applied energy, vol. 265, p. 114806, 2020. [18] Y. Jin, Y. Wen, and Q. Chen, “Energy efficiency and server virtualization in data centers: An empirical investigation,” in 2012 Proceedings IEEE INFOCOM Workshops. IEEE, 2012, pp. 133–138. [19] X. Zhang et al., “Estimating power consumption of containers and virtual machines in data centers,” in 2020 IEEE International Conference on Cluster Computing (CLUSTER). IEEE, 2020, pp. 288–293. [20] K. Kaffes et al., “Leveraging application classes to save power in highly-utilized data centers,” in Proceedings of the 11th ACM Symposium on Cloud Computing, 2020, pp. 134–149. [21] D. Narayanan et al., “Migrating server storage to ssds: analysis of tradeoffs,” in Proceedings of the 4th ACM European conference on Computer systems, 2009, pp. 145–158. [22] B. Sprecher, R. Kleijn, and G. J. Kramer, “Recycling potential of neodymium: the case of computer hard disk drives,” Environmental science & technology, vol. 48, no. 16, pp. 9506–9513, 2014. [23] “Science and technology facilities council,” https://www.ukri. org/councils/stfc/.

14

[24] “Stfc sustainability and carbon management,” https://www. she.stfc.ac.uk/Pages/Sustainability-and-Carbon-Management. aspx. [25] “CQuanDRI scripts and results,” https://github.com/ ox-computing/project-cquandri. [26] W. Shi, S. Wang, and B. Luo, “Cpt: An energy-efficiency model for multi-core computer systems,” in Proc. 5th Workshop Energy-Efficient Des., 2013, pp. 1–6. [27] A. Rimoldi et al., “The simulation for the atlas experiment. present status and outlook,” in IEEE Symposium Conference Record Nuclear Science 2004., vol. 3. IEEE, 2004, pp. 1886– 1890. [28] “Docker,” https://www.docker.com/. [29] C. Aguado Sanchez et al., “Cvmfs-a file system for the cernvm

virtual appliance,” XII Advanced Computing and Analysis Techniques in Physics Research, p. 52, 2008. [30] M. K. Bane et al., “ENERGETIC: Final report,” 2023, https: //zenodo.org/records/7692272. [31] “cpupower(1) - linux man page,” https://linux.die.net/man/1/ cpupower. [32] L. A. Barroso, U. Hölzle, and P. Ranganathan, “The datacenter as a computer: Designing warehouse-scale machines,” Synthesis Lectures on Computer Architecture, vol. 13, no. 3, pp. i–189, 2018. [33] B. Guenter, N. Jain, and C. Williams, “Managing cost, performance, and reliability tradeoffs for energy-aware server provisioning,” in 2011 Proceedings IEEE INFOCOM. IEEE, 2011, pp. 1332–1340.

Related documents

• Large-scale workflow placement in serverless computing using integer nonlinear programming#458253
• The Energy Cost of Execution-Idle in GPU Clusters#2574
• Trade-off between complexity and energy in quantum phase estimation#986171
• Performance Study of Serverless Workloads in Confidential Virtual Machines#660790
• Redundancy Protects Human Collaborations from Failure#1159828