A Technique for Load Shifting Low-latency Applications in Multi-Region Renewables Harvesting via SMT Core Pooling

1

A Technique for Load Shifting Low-latency Applications in Multi-Region Renewables Harvesting via SMT Core Pooling

arXiv:2609.03297v1 [cs.DC] 3 Sep 2026

Tharindu B. Hewage, Shashikant Ilager, Maria A. Rodriguez, and Rajkumar Buyya Abstract—Load shifting across geographic regions to chase intermittent renewable energy availability is commonly used in reducing cloud infrastructure carbon footprint. However, it often omits low-latency applications due to high latency variances of wide area networks (WAN) that interconnect regions. This paper addresses accommodating low-latency applications into load shifting by minimizing their shifting across the WAN. We propose a technique using a hardware-software co-design approach. At the hardware level, we conduct server load matching over renewables supply peaks and valleys by deep idling physical cores in two otherwise identical server pools, with one enabling simultaneous multi-threading (SMT) in CPUs. In return, we achieve a static set of logical cores amidst energy supply dynamics, reducing the probability of workload shifting. At the software level, we efficiently chase the static set of cores for lowlatency applications within regions while prioritizing best-effort applications to accommodate shifting requirements across WANs. Our approach exploits the lower performance compromise of SMT cores due to their hardware multi-threading. We implement the proposed technique with OpenStack and CPU idle states and evaluate its performance on a real experimental testbed with Azure VM arrival traces. Results show an 80% reduction in offloading low-latency VMs and a 43.81% reduction in coefficient of variation of p90 end-user latency while having a worst-case latency compromise of 11.97% due to SMT cores. Index Terms—Carbon-aware computing, low-latency applications, cloud computing, load shifting, renewable energy.

I. I NTRODUCTION The data center industry is expected to reach nearly 13% of global electricity demand by 2030 [1]. With electricity generation being one of the most significant contributions to global CO2 emissions [2], data center operators are presented with the challenge of grid decarbonization. Cloud data center fleets are often spatially distributed and powered by sources integrated with intermittent renewables, emitting varying amounts of CO2 per kilowatt-hour (kWh) [3]. As a result, carbon optimization in cloud computing infrastructures predominantly relies on load shifting−shifting workloads across data center locations via cloud Virtual Machine (VM) scheduling [3]–[7]−to chase availability of renewables for workload execution. Adversely, network traffic among spatially distributed data centers can hop through Wide Area Networks (WAN) with high latency variances [8], resulting in unpredictable end-user latency performances in VMs. For instance, when aggregated network traffic of VMs reaches edge routers of WAN, traffic management at the IP layer that is unaware of the aggregation can split them into tunnels with different latency performances [8]. To avoid potential application performance degradation from such unpredictable latency performances, cloud providers often limit load shifting to best-effort workloads with flexible latency requirements, such as scientific simulations, batch processing, and machine learning training [3]. To this end, this work explores the opportunities in accommodating low-latency

workloads for cloud load shifting, emphasizing minimizing the high latency variance that VMs experience during load shifting. Use cases of cloud low-latency computing are emerging across various industries, including healthcare, factories, automotive, and aviation [9]. According to recent forecasts, they are predicted to accommodate nearly 30% of the world’s data in the near term [10]. Today, low-latency computing plays a vital role in content delivery networks, streaming applications, and applications of recent AI-boom, such as serving generative AI models. Figure 1 illustrates a spectrum of such applications available in the cloud according to their latency service level agreements (SLAs). Preserving latency SLAs is imperative to maintain application service quality, whereas accommodating the growing low-latency applications in load shifting is equally important for the sustainable growth of clouds. The key challenge is the workload impact from cloud network overheads. At renewables supply valleys, the data center’s workload must be migrated to match the server’s reduced resource capacity. As migrations often move workloads across WANs, workloads are then susceptible to WAN’s high latency variances. Existing techniques for load shifting either compromise latency performance with potential server performance throttling [11] or do not consider opportunities in limiting workload migration within the data center. To address these gaps, we propose a technique that aims to contain low-latency applications within the local cloud region. We leverage a hardware-software co-design approach. At the hardware level, we tackle the server resource capacity reduction problem in renewable supply valleys. We apply an application-independent CPU core-level power management mechanism with two heterogeneous server pools. CPUs in server pools are physically the same, yet one enables simultaneous multi-threading (SMT) to double the available server logical cores through hardware multi-threading. During supply valleys, half of the CPU cores are set to deep idle and restored at the peaks. In return, we maintain a static set of logical cores across the server pools. At supply valleys, servers of the SMT pool exhibit the static set, whereas the servers of the non-SMT pool exhibit the static set at supply peaks. At the software level, we dynamically chase the static set of logical cores across the server pools for low-latency workloads. In our technique, load shifting for low-latency workloads is mostly conducted within the local cloud region using its fast network fabric, eliminating the communication overheads of WAN. Further, executing low-latency workloads in the SMT pool incurs only a minimum performance overhead since SMT cores use hardware multi-threading, which provides better performance. We implement our technique with OpenStack and core-level power management with CPU idle states. To evaluate, we

2

Fig. 1. Low-latency applications in cloud data centers [12].

Fig. 3. Traffic management at IP layer splitting aggregated traffic flows of cloud VMs when communicating over WAN [8].

Fig. 2. Average operational gross carbon emissions per unit of energy from the power grid across Google cloud regions [13].

use an experimental cloud region with SMT server pooling and a local network fabric. We use an HP ProLiant server for each pool with a 12-core Intel Xeon silver CPU. We enable SMT through Intel Hyper-threading technology. We use VM arrival data from Azure’s workload traces and renewable dynamics from ELIA solar data. We measure low-latency performance inside VMs by running the Cyclictest tool. The key contributions of our work are as follows: (1) Propose a new technique for localized load shifting of lowlatency workloads to integrate renewable energy, avoiding the latency compromises of workload shifting over WANs across geographical regions. (2) Implement the proposed technique in real cloud settings and conduct detailed experiments using production VM and renewable energy data. (3) Evaluate the proposed technique against state-of-the-art baselines, focusing on maintaining low-latency performance. Our results show an 80% reduction in offloading low-latency VMs, a 43.81% reduction in coefficient of variation of p90 end-user latency, and an 11.97% performance compromise due to SMT cores. II. BACKGROUND AND M OTIVATION In this section, we provide background on load shifting for the utilization of renewable energy across geographically distributed cloud regions, introducing its dynamic server resource scaling problem and challenges in accommodating low-latency applications. We then detail our motivations for solving those. Dynamic server resource scaling problem in utilizing intermittent renewable energy availability across cloud regions: Cloud providers provision data centers across geographical regions around the globe to cater to various requirements,

such as end-user latency and data privacy. [13]. Those regions are powered by electricity grids with varying carbon intensities in their supply, primarily due to the integration of intermittent renewable energy sources such as solar and wind [14]. Recently, driven by 24/7 zero emission goals [1], hyper-scale cloud providers increasingly utilize low-carbon computing opportunities across their cloud regions [3]. Figure 2 illustrates grid carbon intensity per unit of energy across Google cloud regions. In that, executing workloads with lower carbon intensity is beneficial in reducing cloud providers’ carbon footprint. In doing so, cloud platforms model carbon intensity as a resource for optimization. Since the carbon intensity of cloud regions predominately depends on the supply dynamics of renewable energy sources, it can be modeled as a dynamic server resource scaling problem, where server resource capacity scales up or down depending on the supply dynamics of clean energy. Recent examples of such can be seen in virtual capacity caurves [3] and workload scaling for carbon-optimization [14]. The key approach in addressing dynamic server resource scaling is load shifting, where the execution of workloads is either shifted in time or space across cloud regions. Challenges in accommodating low-latency applications for load shifting: Low-latency applications require maintaining an upper bound in their end-user communication latency. Figure 1 illustrates a spectrum of end-user latency for various low-latency applications in the cloud. Such applications are increasingly prevalent [10]; thus, their impact on carbonaware cloud computing is growing. However, low-latency applications are not flexible in load shifting, primarily due to the complexities in maintaining their rigid performance constraints amidst server resource down-scaling of renewable energy supply valleys. These applications cannot tolerate longer service unavailabilities in shifting across time, and more importantly, they cannot tolerate shifting across space (i.e., shifting among cloud regions) due to unpredictability in network performance connecting the cloud regions. Figure 3 illustrates an example of that. Geographically distributed cloud regions are connected using wide area networks (WAN). Each cloud region connects to the WAN using an edge router and passes its network traffic flows to the WAN’s traffic management. Recent studies show that WAN’s traffic management at the IP layer is unable to distinguish aggregated traffic flows of

3

Fig. 4. System architecture of the proposed load shifting technique.

a cloud virtual machine; thus, an aggregated traffic flow can get split at the edge router, routed through different paths with various latency performances before arriving at the destination [8]. The resulting application end-user latency is unpredictable and unfavorable to application users [15]. Further, increased latency performance may violate application service level agreements (SLAs), which can incur costly penalties to the cloud provider. Motivations: Therefore, to accommodate low-latency applications in load shifting, we identify server resource down-scaling at renewable energy supply valleys as its major bottleneck. Since load shifting must offload workload outside the local cloud region to match scaled-down server resource capacity, we hypothesize that maintaining a static resource capacity amidst renewable supply valleys for low-latency applications would enable load shifting to retain those within the local cloud regions, avoiding costly communication overheads of WANs. In this context, we outline the following research questions to draw our motivations for this paper. o1: Can we manage static server resource capacity amidst renewables valleys to yield an infrastructure favorable for lowlatency applications? o2: For a technique designed to achieve o1, what compromises are made, and more importantly, can the compromises operate within the service level agreements (SLAs) of low-latency applications? In addressing o1 and o2, we propose a load-shifting technique to accommodate low-latency applications in carbonaware cloud computing. Section IV details its inner workings, Section V describes our implementation of the proposed technique, and Section VI evaluates and discusses its performance over state-of-the-art baselines.

resource availability via load shifting both within the data center and data centers across regions via VM offloading over a Wide Area Network (WAN). Regions receive workloads as VM arrivals, which are either VMs of low-latency applications or VMs of best-effort applications. The WAN that connects regions offer network routes with varying latency performance, which we model with the latency performance model. Using the derived models, we then formulate our problem. Server dynamic resource scaling model: We model server resource scaling from the dynamic availability of renewable energy capacity through availability of CPU cores. Given the energy capacity, server power is managed by enabling a sufficient amount of CPU cores. The same technique has been commonly used as a core-level server power management mechanism [16], [17]. A data center in our model integrates a mixed power delivery of both carbon-intensive stable power sources, such as fossil fuel-based energy generation, and low-carbon-intensive intermittent renewable energy sources, such as solar and wind. Stable sources provide baseline energy capacity for the data center, whereas intermittent sources provide peaks and valleys of low-carbon-intensive energy availability. Data center power consumption is typically a combination of information and technology components (IT) such as servers, cooling, and internal power conditioning systems [18]. Servers in our system model facilitate CPU-intensive low-latency and besteffort workloads. Therefore, majority of dynamic power draw variations corresponds to CPU power. For such systems, prior works show server power can be estimates as a linear function of CPU power with 90% accuracy [19]. Moreover, for multicore CPUs, the cumulative sum of core power becomes a close upper bound to the CPU power [20]. Combining both, we use the following model to estimate server power.

III. S YSTEM M ODEL AND P ROBLEM F ORMULATION Figure 4 illustrates our system model. We consider a multiregion cloud deployment. Each region integrates intermittent renewable energy in the data center power delivery. As a result, data center servers in each cloud region undergo dynamic energy capacity availability. In return, server compute capacity dynamically scales up and down. We model that with the dynamic server resource scaling model. Workload schedulers in cloud regions match their workload execution with the server

PS (t) = f (

N X

PCOREi (t))

(1)

i=1

where at time t = t, PS (t) is the server power, N is the number of homogeneous cores, PCOREi (t) is the power consumption of the ith core, and f is a linear function. We set server power capacity of the stable grid energy sources (Pgrid ), such that grid capacity is sufficient to support both non-IT power load and half of the peak IT power load.

4

Pgrid = PSpeak /2 We set power capacity from the renewable sources (Prnw (t)) as a two level power signal, where once sufficient capacity is available, renewable sources are able to provide power to support peak server load. ( PSpeak /2, t = High renewables Prnw (t) = 0, t = Low renewables With that, we define our dynamic server capacity scaling model (Pcap (t)) as, Pcap (t) = Pgrid + Prnw (t) WAN latency performance model: We model WAN latency performance based on the traffic management characteristics of WANs connecting multi-region cloud deployments [8]. In that, communication latency across WAN is unpredictable, thus changes overtime. With that, we model WAN latency performance as follows. For the duration of ∆T , the set of executed low-latency VMs (SV M ) is, SV M = {V M1 , ..., V Mm } For SV M , we calculate WAN latency performance (Lat(SV M )) as,

set, lowering their possibilities of shifting over the WAN. Our approach involve three steps: 1) Core-level power management to meet server resource scaling, 2) SMT pooling to yield static resource availability, and 3) novel VM scheduling algorithm to utilize SMT pooling for low-latency VMs. Collectively, we maximize retaining low-latency VMs within the local region by exploiting the hardware multi-threading of SMT. A. Hardware-level logical core management Server power management: To cater to dynamic server resource scaling for the supply dynamics of renewable energy, we employ per-core deep idling [21] to manage the server power draw. At t = Low Renewables, we set half of the server cores into the deep sleep, such that the peak power draw of the server meets Pcap (t). N/2

PS (t = Low Renewables) ≤ f (

X

PCOREpeak )

i=1

where PCOREpeak is the peak power draw of each homogeneous core. Opposed to that, we awake all cores to better utilize Pcap (t) at t = High Renewables. N X PS (t = High Renewables) ≤ f ( PCOREpeak ) i=1

We then leverage CPU Simultaneous Multi-threading (SMT) to exploit our server power management to yield a 1 Lat(SV M ) = Pm R ∆T static set of CPU resources. Li (t) dt i t SMT pooling: Simultaneous Multi-threading (SMT) in CPUs where Li (t) is the end-user latency of the ith VM at time t. doubles the available number of logical cores [22] while proProblem formulation: We formulate our problem as follows. viding better performance due to its hardware multi-threading Given an arbitrary time period ∆T , maximize end-user latency nature. To exploit our server power management with SMT, performance of VMs while meeting the dynamic capacity we maintain two server pools where the servers in one of the scaling of servers. pools enable SMT (i.e., SMT pool). The number of logical cores in servers of the SMT pool is double the amount of its physical CPU cores. In combination with our server power Maximize Lat(SV M ) and ∀S ∈ Servers, PS (t) ≤ Pcap (t) management, which deep idles half of the physical cores at renewable supply valleys, we realize a static set of logical IV. L OAD S HIFTING FOR L OW- LATENCY A PPLICATIONS cores across the server pools, regardless of the renewable WITH SMT C ORE P OOLING supply state. At supply valleys, servers in the SMT pool In this section, we detail the design of our load shifting provide the static set of logical cores, whereas, at supply technique. Our technique addresses the research questions peaks, servers in the non-SMT pool provide that instead. An identified in Section II to accommodate low-latency appli- example of that is illustrated in Figure 4. In that example, N cations in cloud load shifting. In Section VI we showcase is set to 4. At supply peaks, the non-SMT pool exhibits four its superiority in delivering improved latency performance for logical cores, and the SMT pool exhibits eight logical cores. low-latency applications. At supply valleys, the SMT pool exhibits four logical cores. Figure 4 illustrates the system architecture of the hardware- Collectively, a static set of four logical cores is present in the software co-design of the proposed technique. At the hardware local region at all times, which is a significant advantage over level, we leverage heterogeneous server pools of SMT cores traditional server power management, where the renewable to maintain a static CPU capacity amidst renewable supply energy integration often down scale resource capacity of each dynamics. We employ core-level power management to meet server. the dynamic resource scaling of servers for renewable supply The primary reason for not enabling SMT in both pools variations. In return, we maintain a static set of logical cores is that core oversubscription methods such as SMT can still cores across the server pools. We then exploit the static set at impact the low-latency performance of VMs [23]. In our the software layer. Using a novel VM scheduling algorithm, we system architecture, the static set of logical cores only rely efficiently manage low-latency application VMs for the static on SMT cores during energy supply valleys, thus the latency

5

performance impact is further reduced. In the next subsection, we efficiently utilize the static set of logical cores for lowlatency applications via a novel VM scheduling algorithm. B. Software-level VM scheduling algorithm Algorithm 1 Proposed VM Scheduling Algorithm for SMT Pooling. Input: • v: Incoming VM request. • Ssmt : Set of servers in the SMT pool. • Snon-smt : Set of servers in the non-SMT pool. • P (t) ∈ {peak , valley}: Renewable supply state at time t. − • P (t ) ∈ {peak , valley}: Renewable supply state before time t. Output: Scheduling decisions for incoming VM placement and server pool management. 1: Function H ANDLE VMP LACEMENT(v, P (t)): 2: if I S L OW L ATENCY (v) then 3: if P (t) = peak then 4: P LACE (v, SNON - SMT ) 5: else if P (t) = valley then 6: N OTA DMIT (v) 7: end if 8: else 9: if P (t) = peak then 10: P LACE (v, SSMT ) 11: else if P (t) = valley then 12: P LACE (v, SSMT ∪ SNON - SMT ) 13: end if 14: end function 15: Function H ANDLE P OWERT RANSITION(P (t− ), P (t)): 16: if P (t− ) = peak and P (t) = valley then 17: O FFLOAD B EST E FFORT VM S (SSMT ) 18: L IVE M IGRATE (v ∈ L OW- LATENCY , SNON - SMT , SSMT ) 19: else if P (t− ) = valley and P (t) = peak then 20: O FFLOAD B EST E FFORT VM S (SNON - SMT ) 21: L IVE M IGRATE (v ∈ L OW- LATENCY , SSMT , SNON - SMT ) 22: end if 23: end function 24: Main Procedure: 25: while S YSTEM I S RUNNING () do 26: if I S P OWERT RANSITION(t− , t) then 27: H ANDLE P OWERT RANSITION(P (t− ), P (t)) 28: end if 29: for each v do 30: H ANDLE VMP LACEMENT(v, P (t)) 31: end for 32: end while In this section, we propose a software-level VM scheduling algorithm to utilize the hardware-level logical cores provided by SMT pooling. Algorithm 1 outlines our VM scheduling algorithm. It takes incoming VM placement requests and the state of the renewable energy supply (i.e., peak or valley) as

inputs. It then determines VM placement decisions and handles server pool management for renewable energy dynamics. The pseudo-code for VM placement is outlined in the subroutine H ANDLE VMP LACEMENT (line 1). It first identifies the criticality of the VM as either low-latency or best-effort (line 2). For low-latency VM placement requests at peak renewable energy supply, requests are admitted to the cloud region for placement and deployed in the non-SMT pool. Since CPU cores of the non-SMT pool do not have the performance overhead of having hardware multi-threading, the placement decision aims for maximum CPU performance. Conversely, requests will not be admitted to the cloud region if the renewable energy supply is at a valley (line 6). At that stage, available cores in the non-SMT pool are reserved to restore the performance of already deployed low-latency VMs at renewable energy peaks. We provide inner details of the server pool management in the next paragraph. In the case of best-effort VM placement requests (line 8), we admit and deploy those regardless of the renewable energy supply state, with the exception of limiting their deployment to the SMT pool during supply peaks (line 10). The pseudo-code for server pool management is outlined in the subroutine H ANDLE P OWERT RANSITION (line 15). It takes two input parameters: the state of the renewable energy supply at times t and immediately before (t− ). In case of a transition from supply peak to a valley, we first offload besteffort VMs in the SMT pool from the cloud region (line 17) and internally live migrate low-latency VMs to the SMT pool from the non-SMT pool. Here, offloaded VMs are handled similarly to standard load shifting across cloud regions, which places those VMs outside the local cloud region. Our approach exploits the best-effort nature of VMs to relax constraints in offloading. In the case of a transition from supply valley to a peak, we first offload best-effort VMs from the non-SMT pool and live migrate VMs from the SMT pool to the nonSMT pool (line 19). Overall, through live migrations, server pool management maintains deployed low-latency VMs inside the data center. Migrations cause minimum disruption to the application execution, and most importantly, end-user latency performance is kept intact by keeping VMs in the same cloud region throughout. We make room for the low-latency VMs via offloading best-effort VMs, aiming for a minimum impact from geographical load shifting towards the application service quality. Our algorithm’s collective management of VM placement and server pool management is designed to improve the service quality of the cloud region’s already admitted low-latency VMs. It improves the utilization of SMT pooling and the server power management’s approach of halving the number of available physical cores, guaranteeing a fixed resource capacity for low-latency VMs through their live migrations and offloading best-effort VMs. V. I MPLEMENTATION We implement our proposed technique in a real experimental cloud environment. We implement core-level power management of servers via the CPU idle states feature [24]

6

and SMT pooling using the same feature present in server CPUs [25], both for Intel CPUs. We implement our proposed VM scheduling algorithm at the cloud resource management layer using OpenStack [26]. First, we implement core-level power management of the server by deploying a daemon service in each server [27]. The daemon service wraps the Intel power optimization library that provides low-level API control of the CPU c-states in each CPU core. We then expose high-level RESTful APIs to the deep idle half of the CPU cores. As a result, upon receiving a core deep idle request, the daemon service overrides the default kernel’s behavior of CPU idle states and maintains half of the CPU cores at the deepest c-state. Conversely, a wake request will restore the deep idle state. We write the daemon service in golang. Secondly, we select the set of servers for the SMT pool and enable CPU hyper-threading through their BIOS settings. Finally, we deploy OpenStack to manage the servers. We modify its default server filter in the VM scheduling workflow to omit specific server pools for certain VM types (see Algorithm 1). We assume VM placement requests that are unsuccessful in finding a scheduling decision will be offloaded from the cloud region. Further, we implement a Golang controller in the OpenStack control plane for server pool management. It is driven by the events from the renewable energy supply, for which we expose a high-level API. Upon triggering, it conducts live migrations between servers and emulates VM offloading by conducting VM evictions through OpenStack APIs (see Algorithm 1). VI. P ERFORMANCE E VALUATION In this section, we evaluate the performance of our proposed technique. We outline our experimental design and setup, compare our results with the state-of-the-art baseline, and analyze them in detail. A. Experimental design and setup We conduct evaluation experiments in a real prototype multi-node cloud region. We model the multi-region characteristics for our experiments using real measured data from production cloud regions [8]. In our prototype cloud, we allocate an identical HP ProLiant server to each SMT and non-SMT pool. Each server has an Intel Xeon CPU with 12 physical cores. For the SMT pool, we enable Intel Hyperthreading through its bios settings. Both servers are part of a research server cluster in a private network and share a fast network fabric. We install OpenStack in both servers, marking one as the control plane. We deploy our Golang daemon service on both servers and the Golang controller on the server marked for OpenStack’s control plane. 1) Baselines:: We compare our proposed technique with state-of-the-art Space-Shifting. Space-shifting is the predominantly used load shifting technique [3], [28]. It aims to manage the cloud region’s power draw by shifting its flexible workloads across space to other cloud regions based on energy availability. We use space-shifting to compare the superiority of our proposed technique in accommodating low-latency workloads.

Fig. 5. Comparison of mean application latency performance of heterogeneous server pools as resource contention increases.

2) Workloads:: We use Microsoft Azure’s VM packing data [29] to expose our testbed to realistic VM requests in production clouds. We scale Azure’s data to match our experimental deployment by sampling and synthesizing VM arrival traces using the data. For renewable energy supply dynamics, we use solar dynamics data from the ELIA dataset [30]. Further, we use real latency variation data measured for cloud regions [8] in our experimental setup to evaluate VM latency impact from shifting. 3) Metrics:: To measure the latency performance, we use two latency metrics. We use application latency performance to measure the performance impact inside the low-latency VM. We use end-user latency to measure the impact of shifting VMs across the WAN. To measure application latency performance, rather than measuring the latency performance of specific cloud applications, we monitor the latency performance of the VM’s guest operating system. For that, we use the cyclictest tool [31]. For end-user latency, we use WAN latency data. For that, we measure latency for the duration of the VM’s lifetime provided by the trace data. Using a statistical model we build from WAN latency data, we sample a latency value for each time step and get the aggregated value. To measure the scheduling performance, we monitor the number of Retained, Offloaded, and Not Admitted VMs in the cloud region. Retained VMs complete their lifetime inside the cloud region, offloaded VMs are interrupted at mid-life to get shifted over WAN, and not-admitted VMs are diverted to a different cloud region at admittance. B. Results and Analysis We evaluate the performance of our proposed technique using a two-fold approach. Firstly, we evaluate the performance impact of executing VMs on the heterogeneous server pools. Secondly, we evaluate the performance impact of various placement decisions made due to the load-shifting approach. Application latency performance with SMT pooling: Due to SMT pooling, VMs in our technique can be deployed on either SMT or non-SMT cores. To evaluate application latency performance, we measure the latency performance of the guest operating system for various core allocation configurations. Since SMT cores share CPU components, our experiment is designed to measure the performance impact as resource contention in the CPU increases. We first allocate VMs to occupy a portion of the CPU cores available in a server of each

7

(a) Comparison for Best-effort VMs

(b) Comparison for Low-latency VMs

Fig. 6. Comparison of VM-scheduling performance in the experimental cloud region.

pool. Then, we execute a system load in the VM to emulate a utilized application and measure the latency performance of the guest operating system. We then repeat the experiment for different allocated CPU core portions until it reaches 100%. Our experiment measures the low-latency application impact of deploying a VM in a server for different packing levels across the server pools. Figure 5 illustrates the results. It shows the mean latency value measured for the guest operating system over the utilization of the server’s virtual CPU cores (vCPU) across both SMT and non-SMT pools. The results show that the impact of application latency from deploying a VM on a specific server pool significantly depends on CPU utilization. For both 33% and 67% of the vCPU utilization, the mean latency performance of VMs on SMT and Non-SMT pools is around 8 microseconds. However, as utilization increases to 100%, there is a significant increase in the VM mean latency, where packing on SMT cores increases from 8 to 10 microseconds. Packing on non-SMT cores increases from 8 to only about 9 microseconds. In comparison, packing on SMT cores increases the mean latency by 11.97%. In this context, the application latency impact of placing a VM across heterogeneous server pools depends on the service quality of the application. For instance, server pooling will primarily impact an application sensitive to 1microsecond performance degradation. In contrast, low-latency applications that can tolerate larger latency penalties will not exhibit degradation in their service quality. Scheduling impact of low-latency VMs: In this experiment, we measure the scheduling impact of retaining deployed VMs inside the cloud region during its lifetime, offloading VMs in mid-life to other cloud regions, and VM admittance to the cloud region. We replay VM request arrivals of the Azure trace and 24-hour renewable supply change dynamics. We define a threshold in the renewable energy supply to define peaks and valleys. Afterward, we conduct the same experiment for our proposed technique and the baseline. Figure 6a and 6b illustrates the results for each best-effort and low-latency VM types. It shows the number of VMs in each category of the x-axis. The results show that the proposed technique significantly surpasses Space-Shift in reduced offloaded events for lowlatency VMs with an 80% reduction. In comparison, it

Fig. 7. Comparison of p90 end-user latency distribution of Low-latency VMs.

increases the best-effort VM offloading. In admitting VM requests for deployment, the proposed technique is closely the same, with a slightly decreasing number of not-admitting events for both best-effort and low-latency VM types. The Space-Shift approach performs better in retaining both lowlatency and best-effort VM types. Collectively, the behavior of the proposed technique shows its superiority in managing lowlatency VMs. This is because prior studies show that even with relatively degraded performance, users favor a deterministic application performance [15]. In that regard, offloading events incur the most disruption to low-latency applications since offloading can introduce latency overheads from the wide area network’s performance. The proposed technique shows the minimum offloading events and compromises in not admitting VMs. Although not admitting does shift the VM across WAN, end-user service quality would indicate the degraded performance since deployment rather than showing mid-life. End-user latency impact of low-latency VMs: We measure the impact of end-user latency in our scheduling impact experiments to evaluate the impact of shifting VMs across WAN. Using probabilistic models, we generate data from real inter-cloud-region latency over WAN. We sample a distribution of latency values for each VM for the lifetime and calculate its p90 value. WAN latency does not impact retained VMs, which complete their entire lifetime inside the local cloud region.

8

TABLE I C OMPARISON OF RELEVANT WORKS WITH OUR PROPOSED TECHNIQUE . Work Radovanovic’23 [3] Carbonscaler’23 [14] Zheng’20 [6] CDN-Shifter’24 [28] Our Proposed

Multi-cloud Carbon Optimization ✓ ✓ ✓ ✓ ✓

Load Shifting ✓ ✓ ✓ ✓ ✓

However, WAN latency does impact the not-admitted VMs and has a partial impact on offloaded VMs. Figure 7 illustrates the results. It compares the distributions of p90 latencies of VMs. The results show the superiority of the proposed technique in reducing the p90 latency variance. Although there are several outliers, the latency performance of VMs with the proposed technique primarily concentrates around 40 milliseconds with a coefficient of variation of 0.62. In contrast, with SpaceShift, the p90 latency value disperses around 20 milliseconds with a coefficient of variation of 1.10. The proposed technique reduces the coefficient of variation of p90 latency by 43.81%. Therefore, the results indicate that VMs managed with the Space-Shift technique will most likely deliver unpredictable end-user performance compared to the proposed approach, which is unfavorable for the service quality [15].

Uninterruptible Execution

Low-latency Applications

Static Resource Capacity

✓ ✓ ✓ ✓

✓ ✓

✓

resources are adjusted to tolerate the changes in the data center power delivery. For instance, Microsoft increase utilization of their clouds via power oversubscription, where server are provisioned to oversubscribe the power capacity [15] and power overdraw events are managed by throttling the server compute capabilities. They leverage a dynamic VM scheduling technique to reduce the workload impact, considering VM priority levels. However, workload throttling in their approach can lead to non-deterministic application performances. In our work, we employ a dynamic inventory to absorb renewable intermittencies and conduct VM scheduling to leverage the idea of chasing static resources within the datacenter for lowlatency applications, providing better stability in application performance.

VIII. C ONCLUSIONS AND F UTURE W ORK VII. R ELATED W ORK Integrating renewables in clouds: Many previous studies have explored integrating renewable energy into cloud data center operation. They commonly exploit load shifting techniques in both space and time, such as geographical load balancing [3], [14], [28], VM migrations [3]–[7], and workload admissions and capacity planing [3]. Most techniques are intended for flexible workloads that can tolerate delayed execution and responses. For instance, carbon optimization at Google [3] leverage internal workloads that can withstand delays as long as 24 hours, such as machine learning, data processing, and simulation. They employ virtual capacity curves across their geographically dispersed data centers and employ temporal shifting to perform carbon-optimized computing. Some works explore shifting workloads across locations over WAN, considering the impact of WAN’s dynamic traffic congestion towards the workloads [32]. However, they do not focus on low-latency applications and only focus migration cost calculations for the energy cost of the transfer. In contrast, we explore geographical load shifting to accommodate lowlatency applications and provide a technique to better manage server power over intermittent renewable energy while maintaining service quality of low latency applications. VM scheduling of dynamic inventories: Cloud VM scheduling for energy management is a widely researched problem [33], [34]. Many aim to optimize VM scheduling to increase energy efficiency for static inventories, where server resources remain static and server power is managed through the workload management [35]. However, recent works explore VM scheduling for dynamic inventories where available server

Although load shifting is commonly used in carbonoptimized clouds to utilize intermittent renewable energy availability across regions, it often overlooks low-latency applications due to dynamic latency overheads of wide area networks (WAN) connecting regions. To this end, our work proposes a load-shifting technique to accommodate low-latency applications with minimum impact. The proposed technique employs heterogeneous server pools differentiated by simultaneous multi-threading (SMT) feature. It exploits SMT pools with core-level power management to maintain a static set of logical cores across server pools, which then is utilized with a VM scheduling algorithm to prevent offloading low-latency VMs from the local cloud regions, avoiding costly communication overheads of WAN. We implement the proposed technique with OpenStack and CPU idle states and evaluate its performance using Azure VM arrival traces on an experimental cloud region with an HP ProLiant server allocated to each SMT and non-SMT pool. In comparison to state-of-the-art spaceshifting, our empirical results demonstrate the superiority of the proposed technique with an 80% reduction in offloading low-latency VMs, 43.81% reduction in coefficient of variation of end-user latency, and worst-case latency compromise of 11.97% due to SMT cores. In future work, we plan to advance the VM scheduling algorithm in the proposed technique to utilize fine-grain allocations of non-SMT cores during renewable energy valleys. Further, tuning the proposed technique for application-specific requirements can enable relaxed constraints of VM scheduling, which can further present opportunities to improve CPU utilization.

9

R EFERENCES [1] Google, “24/7 by 2030: Realizing a carbon-free future,” 2020. [Online]. Available: https://www.gstatic.com/gumdrop/sustainability/ 247-carbon-free-energy.pdf [2] P. Hawken, Drawdown: The Most Comprehensive Plan Ever Proposed to Reverse Global Warming. penguin, 2017. [3] A. Radovanović, R. Koningstein, I. Schneider, B. Chen, A. Duarte, B. Roy, D. Xiao, M. Haridasan, P. Hung, N. Care, S. Talukdar, E. Mullen, K. Smith, M. Cottman, and W. Cirne, “Carbon-aware computing for datacenters,” IEEE Transactions on Power Systems, vol. 38, pp. 1270– 1280, 2023. [4] E. Masanet, A. Shehabi, N. Lei, S. Smith, and J. Koomey, “Recalibrating global data center energy-use estimates,” Science, pp. 984–986, 2020. [5] Z. Liu, M. Lin, A. Wierman, S. H. Low, and L. L. Andrew, “Greening geographical load balancing,” SIGMETRICS Perform. Eval. Rev., vol. 39, p. 193–204, 2011. [6] J. Zheng, A. A. Chien, and S. Suh, “Mitigating curtailment and carbon emissions through load migration between data centers,” Joule, vol. 4, pp. 2208–2222, 2020. [7] A. James and D. Schien, “A low carbon kubernetes scheduler,” in Proceedings of the ICT for Sustainability, 2019, p. 28. [8] C. Miao, Z. Zhong, Y. Xiao, F. Yang, S. Zhang, Y. Jiang, Z. Bai, C. Lu, J. Geng, Z. He, Y. Wang, X. Zou, and C. Yang, “Megate: Extending wan traffic engineering to millions of endpoints in virtualized cloud,” in In Proceedings of the ACM SIGCOMM 2024 Conference, 2024, p. 103–116. [9] Ubuntu, “A cto’s guide to real-time linux,” 2023. [Online]. Available: https://ubuntu.com/engage/cto-guide-real-time-kernel [10] J. R. David Reinsel, John Gantz, “The digitization of the world from edge to core,” 2018. [Online]. Available: https://www.readkong.com/ page/the-digitization-of-the-world-from-edge-to-core-8666239 [11] A. Jahanshahi, N. Yu, and D. Wong, “Powermorph: Qos-aware server power reshaping for data center regulation service,” ACM Trans. Archit. Code Optim., 2022. [12] T. Anderson, A. Belay, M. Chowdhury, A. Cidon, and I. Zhang, “Treehouse: A case for carbon-aware datacenter software,” SIGENERGY Energy Inform. Rev., p. 64–70, 2023. [13] Google, “Carbon free energy for google cloud regions,” 2025. [Online]. Available: https://cloud.google.com/sustainability/region-carbon [14] W. A. Hanafy, Q. Liang, N. Bashir, D. Irwin, and P. Shenoy, “Carbonscaler: Leveraging cloud workload elasticity for optimizing carbonefficiency,” Proc. ACM Meas. Anal. Comput. Syst., 2023. [15] A. G. Kumbhare, R. Azimi, I. Manousakis, A. Bonde, F. Frujeri, N. Mahalingam, P. A. Misra, S. A. Javadi, B. Schroeder, M. Fontoura, and R. Bianchini, “Prediction-Based power oversubscription in cloud platforms,” in Proceedings of the 2021 USENIX Annual Technical Conference (USENIX ATC 21), 2021, pp. 473–487. [16] C. Li, W. Zhang, C.-B. Cho, and T. Li, “Solarcore: Solar energy driven multi-core architecture power management,” in Proceedings of the 17th IEEE International Symposium on High Performance Computer Architecture, 2011, pp. 205–216. [17] A. Agarwal, S. Noghabi, I. Goiri, S. Seshan, and A. Badam, “Unlocking unallocated cloud capacity for long, uninterruptible workloads,” in Proceedings of the 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), 2023, pp. 457–478. [18] K. M. U. Ahmed, M. H. J. Bollen, and M. Alvarez, “A review of data centers energy consumption and reliability modeling,” IEEE Access, pp. 152 536–152 563, 2021. [19] A. Radovanovic, B. Chen, S. Talukdar, B. Roy, A. Duarte, and M. Shahbazi, “Power modeling for effective datacenter planning and compute management,” IEEE Transactions on Smart Grid, vol. 13, pp. 1611– 1621, 2022. [20] R. Basmadjian and H. de Meer, “Evaluating and modeling power consumption of multi-core processors,” in Proceedings of the 3rd International Conference on Future Energy Systems: Where Energy, Computing and Communication Meet, 2012. [21] R. J. Wysocki, “Cpu idle time management,” 2018. [Online]. Available: https://docs.kernel.org/admin-guide/pm/cpuidle.html [22] Openstack, “Cpu topologies,” 2023. [Online]. Available: https: //docs.openstack.org/nova/2023.2/admin/cpu-topologies.html [23] O. Foundation, “Real time,” 2021. [Online]. Available: https: //docs.openstack.org/nova/wallaby/admin/real-time.html [24] I. Corporation, “Cpu idle time management,” 2018. [Online]. Available: https://www.kernel.org/doc/html/v5.4/admin-guide/pm/cpuidle.html [25] ——, “Intel hyper-threading technology,” 2003. [Online]. Available: https://read.seas.harvard.edu/cs161/2022/pdf/intel-hyperthreading.pdf

[26] OpenStack, “Compute schedulers,” 2023. [Online]. Available: https: //docs.openstack.org/nova/2023.2/admin/scheduling.html [27] T. B. Hewage, “core-power-mgt,” 2025. [Online]. Available: https: //github.com/tharindu-b-hewage/core-power-mgt [28] J. Murillo, W. A. Hanafy, D. Irwin, R. Sitaraman, and P. Shenoy, “Cdnshifter: Leveraging spatial workload shifting to decarbonize content delivery networks,” in Proceedings of the 2024 ACM Symposium on Cloud Computing, 2024, p. 505–521. [29] Azure, “Azure trace for packing 2020,” 2020. [Online]. Available: https://github.com/Azure/AzurePublicDataset/blob/master/ AzureTracesForPacking2020.md [30] ELIA, “Elia group open data platform,” 2024. [Online]. Available: https://www.elia.be/en/grid-data/open-data [31] Linux, “Linux foundation projects: Real-time linux tools: Cyclictest,” 2023. [Online]. Available: https://wiki.linuxfoundation.org/realtime/ documentation/howto/tools/cyclictest/start [32] Y. Guo and G. Porter, “Carbon-aware inter-datacenter workload scheduling and placement,” in Poster Session of the 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI’23), 2023. [33] O. Hadary, L. Marshall, I. Menache, A. Pan, E. E. Greeff, D. Dion, S. Dorminey, S. Joshi, Y. Chen, M. Russinovich, and T. Moscibroda, “Protean: VM allocation service at scale,” in Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), 2020, pp. 845–861. [34] T. Baker, B. Aldawsari, M. Asim, H. Tawfik, Z. Maamar, and R. Buyya, “Cloud-senergy: A bin-packing based multi-cloud service broker for energy efficient composition and execution of data-intensive applications,” Sustainable Computing: informatics and systems, pp. 242–252, 2018. [35] M. Liu and X. Tang, “Dynamic bin packing with predictions,” Proc. ACM Meas. Anal. Comput. Syst., vol. 6, 2022.

Related documents

• A Technique for Load Shifting Low-latency Applications in Multi-Region Renewables Harvesting via SMT Core Pooling#925642
• System Strength-Constrained Scheduling with Switchable Grid-Forming and Grid-Following Generation Resources#1077341
• 42 U.S.C. § 18715b — Interregional and offshore wind electricity transmission planning, modeling, and analysis#588462
• Multi-source technical assessment and turbine-based energy yield for offshore wind farms: A case study of the Algerian coast.#189977
• Source Code for "A multi-stage engineering framework for feasible scheduling of large-scale energy batch transport systems"#195183