THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
1
K8S Power Irrigation: Deep Reinforcement Learning for Performance-Aware Power Efficiency of Kubernetes Cloud-Native arXiv:2605.25218v1 [cs.NI] 24 May 2026
Microservices Zouhir Bellal, Laaziz Lahlou, Nadjia Kara, Timothy Murphy, Tan Phat Nguyen
Abstract Modern cloud platforms are facing a sharp increase in power demand driven by the rapid adoption of AI-powered applications, making power optimization urgent under net-zero commitments and sustainability goals. Yet, reducing power in production remains challenging for latency-sensitive microservices (e.g., backend pipelines for autonomous driving, real-time AI inference, and AI/IoT workloads), where performance violations directly impact user experience and operational risk. These microservices exhibit heterogeneous workload characteristics (CPU-bound, memory-bound, and mixed) and diverse load patterns. In multi-tenant environments, especially for memory-intensive workloads, contention on shared uncore resources (e.g., last-level cache and memory bandwidth) can degrade performance and trigger violations of performance requirements. As a conservative safeguard, providers often pin servers to performance mode (maximum core and uncore frequencies). This occurs because existing power governors largely disregard application-level performance requirements and ignore uncore interference under contention, maintaining reliability by sacrificing energy efficiency and causing systematic power This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible. This work is supported by NSERC and Ericsson Canada, Grant # 561771 Z. Bellal is with École de technologie supérieure (ÉTS), Montréal, Canada ([email protected]) L. Lahlou is with École de technologie supérieure (ÉTS), Montréal, Canada ([email protected]) N. Kara is with École de technologie supérieure (ÉTS), Montréal, Canada ([email protected]) T. Murphy is with Ericsson, Canada ([email protected]) T. Phat Nguyen is with Ericsson, Canada ([email protected])
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
2
over-provisioning. To address this, we introduce K8SPI (Kubernetes Power Irrigation), a hierarchical reinforcement learning (HRL) controller that jointly optimizes CPU core and uncore frequencies for cloud-native deployments. K8SPI uses a two-stage control architecture: a coarse-grained agent rapidly mitigates performance violations, while a fine-grained agent iteratively minimizes power consumption once performance requirements are met. Guided by multi-level telemetry spanning hardware, Kubernetes, and application performance signals, K8SPI adapts to workload heterogeneity and cross-microservice interference to meet performance requirements with minimal node power. We evaluate K8SPI on a Kubernetes testbed across multiple scenarios. Experimental results show that K8SPI reduces node-level power consumption by 23–30% relative to the Linux performance governor while keeping performance requirement violations below 2–3%, even under severe uncore contention and dynamic load fluctuations.
Index Terms Power monitoring, Cloud power observability, Container-level power monitoring, Power accuracy validation framework, Kepler accuracy validation,
I. I NTRODUCTION By 2030, the electricity demand of the IT sector is projected to reach approximately 3,200 TWh [1]. These projections likely underestimate future consumption due to the rapid proliferation of energy-intensive Artificial Intelligence (AI) workloads, particularly generative AI. Reports indicate that a single ChatGPT query may consume significantly more energy than a conventional web search [2], [3], with the additional global power capacity required for AI projected to reach 327 GW by 2030 [4]. Even with innovations such as DeepSeek that reduce the cost of developing large language models (LLMs), overall energy demand is unlikely to slow down due to largescale deployment and workload growth [5]. This surge, combined with regulatory pressures such as carbon pricing schemes [6], prompts energy efficiency to be a primary operational objective for cloud providers. However, achieving efficiency in cloud-native environments, characterized by multi-tenancy, heterogeneous microservices, and dynamic load, remains a complex control problem. Dynamic Voltage and Frequency Scaling (DVFS) is the standard mechanism for managing processor power. Modern architectures support scaling for both processing cores and the “uncore” (interconnects and Last Level Cache). While core frequency dictates computational throughput, uncore frequency governs memory subsystem performance, including L3 cache latency and
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
3
bandwidth availability. Existing power governors (e.g., intel_pstate, lunix schedutil) rely on hardware-derived utilization metrics (CPU usage, instruction by cycle) to adjust these frequencies. However, these governors optimize system-level utilization rather than enforcing microservicelevel performance requirements (e.g., latency constraints). Consequently, they may over-provision resources—leading to unnecessary power waste—or under-provision them, resulting in Performance requirement violations. This limitation becomes particularly critical in performancesensitive microservices deployed in cloud environments. The problem is exacerbated in multi-tenant nodes where microservices exhibit diverse power– performance sensitivities. CPU-bound services rely on core frequency, while memory-bound services depend heavily on uncore frequency. Furthermore, strictly enforcing CPU core isolation (e.g., via Kubernetes QoS classes) does not isolate shared uncore resources. Consequently, uncore interference—where co-located workloads contend for memory bandwidth—can degrade performance unpredictably. In practice, to mitigate this risk, providers often default to maximum frequency (performance mode), sacrificing energy efficiency for reliability. Several solutions have been proposed in the literature to address power–performance tradeoffs, where Reinforcement Learning (RL) has emerged as a promising approach for adaptive power control. Existing solutions often lack the granularity required for modern cloud environments. Most state-of-the-art RL approaches either control frequencies in isolation, optimize for utilization rather than Performance requirements, or neglect the impact of uncore interference. To bridge this gap, we propose K8SPI (Kubernetes Power Irrigation), a Hierarchical Reinforcement Learning (HRL) controller that jointly optimizes core and uncore frequencies. K8SPI employs a two-stage control architecture to balance rapid performance requirement enforcement with energy minimization: •
Coarse-grained Restoration: An agent that reacts to performance requirement violations by performing significant frequency adjustments to rapidly restore performance.
•
Fine-grained Refinement: Once performance constraints are met, a second agent iteratively refines the configuration to minimize power consumption without triggering new violations.
To support this decision-making, K8SPI utilizes a multi-level telemetry stack: •
Low-level: Per-microservice hardware performance counters capturing instruction execution and stall cycles.
•
Application-level: Real-time microservice performance metrics (e.g., request latency).
•
Global-level: Socket-wide counters reflecting memory controller and interconnect activity.
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
4
By synthesizing these signals, K8SPI effectively manages workload heterogeneity, dynamic load variations, and uncore interference. We implement and evaluate K8SPI on a dedicated Kubernetes-based testbed that emulates cloud deployment conditions using benchmark-driven microservices. The evaluation covers three scenarios with increasing co-location and contention: (i) a single performance-sensitive microservice, (ii) a single performance-sensitive microservice co-located with best-effort workloads, and (iii) multiple performance-sensitive microservices sharing the same node under high contention. Across all scenarios, K8SPI achieves a 23%–30% reduction in node-level power compared to the Linux performance governor operating at maximum frequency, while keeping latency violations below 2%–3%, even under dynamic load fluctuations and uncore interference. These results indicate that K8SPI can reduce power waste without compromising performance requirements in multi-tenant, cloud-native deployments. In summary, the main contributions of this paper are as follows: •
We propose K8SPI (Kubernetes Power Irrigation), a novel Hierarchical Reinforcement Learning (HRL) controller that jointly optimizes CPU core and uncore frequencies to reduce power consumption in cloud-native environments without violating microservice performance requirements.
•
We develop an end-to-end framework for the rapid prototyping of RL-based power optimization policies, specifically tailored for latency-sensitive microservices.
•
We implement K8SPI within a Kubernetes testbed, evaluating its robustness across increasingly complex scenarios, including isolated execution, best-effort co-location, and highcontention multi-instance deployments of latency-sensitive microservices.
•
We demonstrate that K8SPI achieves a 23%–30% reduction in node-level power consumption compared to the default Linux performance governor operating at maximum frequency, while strictly maintaining performance requirement violations below 2%–3% under severe uncore interference and dynamic load fluctuations.
A. Background & Motivation 1) Latency-Sensitive Microservices: Latency-sensitive microservices (e.g., real-time IoT analytics and interactive online services) operate under strict response-time Service Level Agreements (SLAs). In this work, latency is defined as the time elapsed between request arrival at
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
5
the service and generation of the corresponding response. These constraints are strictly enforced because they directly impact user experience and contractual compliance. Cloud services consume compute/memory resources (CPU, cache, memory bandwidth) in addition to storage/network; this work focuses on compute/memory only. Microservices exhibit heterogeneous workload characteristics, which we categorize into three classes: •
CPU-intensive (compute-dominated and sensitive to core frequency),
•
memory-intensive (memory/bandwidth-dominated and typically more sensitive to uncore frequency), and
•
mixed (joint compute and memory demands, sensitive to both core and uncore frequencies).
2) Understanding Uncore Resource Contention: Uncore resources, including the shared lastlevel cache (LLC) and memory bandwidth, are shared across CPU cores within a socket. This shared design introduces contention and can degrade performance when multiple memoryintensive microservices are co-located on the same node. Uncore frequency plays a central role in this interference because it influences the operating point of the memory subsystem and, consequently, the achievable memory bandwidth at both per-core and socket-wide levels. Since the uncore is shared, bandwidth utilization is inherently interdependent: a core may achieve a high bandwidth in isolation at a given uncore frequency, but its effective throughput is bounded by the socket’s aggregate capacity and the concurrent demand from other cores. For illustration, consider a 16-core server at a baseline uncore frequency with an aggregate socket bandwidth limit of 120 GB/s and an achievable single-core bandwidth of 15 GB/s. A single memory-intensive microservice pinned to one core can reach close to 15 GB/s. Under full co-location, if all 16 cores execute memory-bound workloads simultaneously, the aggregate demand exceeds 120 GB/s, and the effective bandwidth per core is reduced (e.g., to approximately 7.5 GB/s). This reduction increases memory stall time and can cause noticeable latency degradation for memory bandwidth-intensive microservices. Fig. 1 shows a 21% per-core bandwidth drop under co-location. An efficient uncore frequency scaling policy that can scale up frequency under contention to improve bandwidth, and scale down when demand is low to avoid over-provisioning and reduce power.
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
6
Fig. 1: Per-core memory bandwidth of a memory-intensive microservice running on a single CPU core in isolation versus co-located with 13 memory-intensive co-runners on a 14-core socket (core/uncore fixed at 2.6/2.4 GHz). Co-runners reduce per-core bandwidth by 21%.
3) Frequency Management in Cloud Servers: Cloud servers manage the performance–power trade-off using DVFS policies, commonly exposed as CPU frequency governors. These policies are typically categorized as static or dynamic. Static governors (e.g., performance and powersave) operate at fixed operating points. The performance governor keeps frequencies at (or near) their maximum to prioritize performance, which helps performance-sensitive services satisfy latency requirements but often causes power waste through over-provisioning. In contrast, powersave prioritizes low frequencies to reduce power, which can degrade performance and increase the risk of performance requirement violations. Dynamic governors (e.g., Intel P-state and Linux ondemand/schedutil) adapt frequencies online using utilization-driven heuristics derived from runtime signals. In general, core frequency is increased under higher compute demand and reduced when demand subsides. 4) Core/Uncore Adjustment Policies: a) Core frequency control: Core DVFS policies typically adjust frequency using systemlevel indicators of compute demand, such as: CPU load or IPC (or related hardware indicators), where higher instruction throughput can indicate compute-intensive execution and motivate higher frequency.
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
7
b) Uncore frequency governor: While our solution targets Intel processors, the proposed methodology is architecture-agnostic and adaptable to other multi-core systems possessing, such as the AMD Zen architecture. The uncore is a substantial resource consumer, occupying roughly 30% of the die area and driving approximately 20% of total power consumption. Uncore frequency policies can be grouped into: 1) CPU load-based governors: On some architectures (e.g., Intel Skylake and early Sapphire Rapids generations), uncore frequency is pushed to its maximum whenever any core reaches its base frequency. This approach ignores actual memory or cache demand. Such a strategy can lead to significant energy waste for CPU-bound applications where the application performance is insensitive to the memory latency. 2) Memory operation-based governors: More recent architectures (e.g., Intel Sierra Forest) introduce activity-driven uncore scaling, adjusting uncore frequency based on memory bandwidth usage and cache access patterns. c) Limitation: lack of application awareness: Despite these advances, core and uncore governors are primarily driven by system-level signals and do not explicitly incorporate microservice performance requirements (e.g., latency). Consequently, they cannot reliably distinguish between (i) conservative scaling that preserves latency targets while reducing over-provisioning and (ii) aggressive downscaling that saves power but induces performance requirement violations, especially under heterogeneous workloads and multi-tenant interference. B. Reinforcement Learning Our work employs RL to develop a dynamic, performance-aware power optimization scheme. RL works by obtaining strategy improvements through continuous interactions with the changing environment in discrete time steps. At each step, an agent receives the current state and a reward (e.g., a function of the measured energy consumption and application-layer performance). It then chooses an action from the set of available actions (e.g., core and uncore configuration) and uses the action to configure the environment (e.g., the hardware). The environment will then move to a new state, and the reward associated with the transition is determined. The goal of the RL agent is to learn a policy that maximizes the expected cumulative reward, i.e., the overall power saving with guaranteed application performance. In this work, we choose to use a policy network optimized via Proximal Policy Optimization (PPO). Given the discrete nature of the action space and the need for fast online adaptation, PPO directly optimizes the policy and gracefully handles
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
8
noisy environments. Unlike Q-learning-based methods, PPO utilizes incremental, continuous policy updates that ensure smooth adaptation to the evolving workloads and resource contention typical in dynamic cloud environments. This property makes the controller highly robust against runtime anomalies and short-term data drifts. Furthermore, PPO is generally less sensitive to hyperparameter tuning, providing the additional stability required for reliable online deployment. C. Related Work Prior work manages the power/energy–performance trade-off using DVFS and UFS. Early controllers are profile- or rule-based. RL has recently become a promising runtime alternative because it can adapt online under a dynamic environment. 1) Non-RL Runtime Control: A large body of prior work manages the power/energy–performance trade-off using DVFS and uncore frequency scaling (UFS) via profiling and rule-based runtime control. These methods are effective in structured settings (often HPC) where objectives are typically expressed as energy reduction under a bounded slowdown budget. Cuttlefish [7] profiles MSRs to compute TIPI (memory requests per retired instruction), classifies workloads as compute- vs memory-bound, and triggers an exploration-based linear search to select core/uncore frequencies (CF/UF) minimizing joules per instruction (JPI). On a 20-core Intel Xeon Haswell E5-2650 with ten HPC benchmarks, it reports 19.4% geometricmean energy savings with 3.6% slowdown versus the performance governor, but its deterministic classification policy can be brittle under phase changes and co-runner interference. DUFP [8] combines package power capping with UFS using runtime FLOPS/s and bandwidth signals to detect phases and adjust RAPL caps within a tolerated slowdown (0–20%), while tuning uncore using the same indicators; on NAS, HPL, and LAMMPS it reports improved energy efficiency, with strong results near a 10% slowdown budget. Complementary UFS-focused runtimes further validate uncore as a first-order control knob. DUF [9] adapts uncore frequency online to reduce socket power/energy under bounded slowdown, and UPSCavenger [10] selects uncore frequency in a phase-aware manner for HPC workloads. Collectively, these efforts show that coordinated core–uncore tuning expands the energy–performance trade-off, but the objective is typically slowdown-centric rather than latencySLO-centric. For latency-critical services, non-RL controllers shift the objective from slowdown to explicit QoS preservation. PEGASUS [11] uses feedback control for energy proportionality under service-
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
9
level constraints; SleepScale [12] jointly selects DVFS and sleep states under QoS constraints; TimeTrader [13] leverages tail-latency slack by slowing non-critical requests; and Gemini [14] uses learning-based prediction for per-query power/frequency decisions under tight latency targets. 2) RL-Based DVFS/UFS Control Under QoS Constraints: RL has emerged as a promising alternative for DVFS/UFS control under workload non-stationarity and co-location dynamics, because it can learn adaptive policies from telemetry rather than relying on fixed heuristics. In latency-sensitive, multi-workload environments, Twig [15] applies deep RL with hardware counters to model QoS behavior and drive energy-efficient control under co-location. Hipster [16] combines RL with heuristic structure to manage latency-critical workloads alongside batch corunners, using DVFS and runtime resource control to reduce energy while preserving QoS and improving throughput. Greeniac [17] extends QoS-aware control to fog/edge clusters, framing frequency selection as a bandit problem under response-time constraints. In service-centric networking, RL is also applied under explicit performance constraints. RL-ADR [18] selects frequency for a 5G user-plane function to minimize power under nearhard packet-drop constraints, using a policy library and conservative fallback to handle traffic distribution shifts; the authors report zero packet drops on long unseen traces while reducing power versus static baselines. GreenNFV [19] uses actor–critic learning (DDPG) for NFV scheduling and tuning, including CPU frequency among multiple actuators to satisfy performance requirement/throughput targets while improving energy efficiency. A second cluster shows the benefit of joint core–uncore control. Juan and Marculescu [20] study reinforcement-based policies over core and uncore DVFS under power constraints, showing coordinated policies outperform core-only or uncore-only scaling under iso-power conditions. PManager [21] co-optimizes power caps and UFS across phases using RL, and Gocht et al. [22] explore the core×uncore space using Q-learning-style exploration with region awareness and direct energy feedback. A hierarchical RL design is also proposed in patent literature [23], but it lacks peer-reviewed evaluation. Finally, embedded platforms study RL-based DVFS under deadline/performance constraints (e.g., Q-learning in [24] and DDQN with Linux CPUFreq in [25]). Despite this progress, many studies still assume isolation or whole-socket occupancy. This abstraction does not match cloud nodes where multiple services share a socket and compete for uncore resources. Our objective is to satisfy per-service latency SLAs while minimizing power by dynamically scaling core and uncore frequencies under co-location, heterogeneous workload
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
10
Fig. 2: K8SPI architecture overview.
types, and time-varying load. II. K8SPI A RCHITECTURE A. System Overview In this paper, we propose K8SPI, an online, hierarchical reinforcement learning (HRL) based power governor tailored for multi-tenant cloud environments. Specifically, K8SPI targets heterogeneous microservices deployed via Kubernetes (K8s) under the Guaranteed Quality of Service (QoS) class, a configuration that allocates dedicated CPU cores to individual microservices. The primary objective of the K8SPI framework is to minimize the CPU socket power consumption without incurring performance degradation that violates the performance requirements of the hosted microservices. As illustrated in Fig. 2, the K8SPI control loop consists of two primary modules: a multilayered Metrics Monitoring module and an HRL-based Power Governor. B. Multi-Layered Telemetry and Monitoring To accurately capture the dynamic behavior of the system, the continuous monitoring module aggregates telemetry data across three distinct abstraction layers, providing both a global system view (i.e., socket-level view) and per-microservice visibility using: •
Application Layer: Captures high-level performance metrics (e.g., target latency) and dynamic microservice loads, measured in requests per second (RPS) for each microservice.
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
•
11
Orchestration (K8s) Layer: Gathers container- and pod-level metrics from the Kubernetes control plane to monitor deployment constraints and accurately map CPU cores to their respective microservices.
•
Hardware (Socket) Layer: Collects lightweight, low-level hardware performance counter measurements (e.g., instructions per cycle, cache miss rates, and memory bandwidth utilization) at both the socket and core levels.
C. Event-Triggered Hierarchical Power Governor K8SPI employs an event-driven control loop that reacts dynamically to the evaluated system state. The control loop actively evaluates the aggregated telemetry against current performance thresholds. As depicted by the decision logic in Fig. X, the RL-based Power Governor is invoked exclusively when the monitoring subsystem detects an imminent performance requirement violation or a state of significant resource overprovisioning. Upon identifying such conditions, the HRL-based Power Governor intervenes to concurrently modulate the dedicated CPU core frequency of the target microservice and the shared uncore frequency of the processor socket. To efficiently navigate this expansive, multi-dimensional configuration space, the K8SPI governor relies on a two-stage hierarchical RL architecture: •
Stage 1 - Coarse Agent: This agent makes large, rapid adjustments to the frequency settings. Its primary goal is to quickly mitigate performance violations or eliminate massive overprovisioning, bringing the system into a safe operational region. In doing so, it creates an optimal starting point for the subsequent tuning phase.
•
Stage 2 - Fine Agent: Once the system is stabilized near the safe threshold, the fine agent takes over. It performs smaller, granular frequency refinements to minimize power consumption while ensuring that the performance requirement remains strictly satisfied. III. S YSTEM M ODEL AND P ROBLEM F ORMULATION
A. Socket-Level Orchestration Architecture We consider a multi-tenant cloud compute node comprising multiple physical processor sockets; K8SPI acts at the socket level. The socket encompasses a set of physical CPU cores, denoted by C, and hosts a set of heterogeneous, latency-critical microservices, M. To guarantee strict compute isolation and satisfy Kubernetes Guaranteed Quality of Service (QoS) requirements,
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
12
we assume an exclusive deployment mapping where each microservice mi ∈ M is pinned to a dedicated core ci ∈ C. The hardware exposes two primary control domains for dynamic voltage and frequency scaling (DVFS), and uncore frequency scaling (UFS). First, each core ci operates at an independent core min max frequency, fcore,i , bounded by [fcore , fcore ]. Second, all cores within the socket share a common
uncore subsystem, which includes the Last Level Cache (LLC) and memory controllers. This min max shared domain operates at a global uncore frequency, funcore , bounded by [funcore , funcore ].
B. Workload and Performance Model The system operates under an event-triggered control paradigm. Let tk denote the timestamp of the k-th control invocation epoch, triggered by either an imminent performance violation or a state of massive resource overprovisioning. Each microservice mi is characterized by an architectural workload profile, ωi (e.g., computebound or memory-bound), and experiences a dynamic incoming request load, λi (tk ) ∈ [0, 1]. The performance of mi is strictly governed by a target latency constraint, Ltarget,i . The measured tail latency at epoch tk , denoted as Lcurrent,i (tk ), is modeled as the sum of its isolated execution latency and a contention time penalty:
Lcurrent,i (tk ) = Φ (fcore,i (tk ), funcore (tk ), λi (tk ), ωi ) + ∆Linterf,i (tk )
(1)
where Φ(·) represents the baseline latency function of the microservice executed without noisy neighbors. The term ∆Linterf,i (tk ) ≥ 0 encapsulates the latency time penalty induced by uncore interference. Because latency measures execution delay, interference strictly adds to this delay. This degradation occurs because co-located microservices aggressively compete for the shared memory bandwidth. While increasing funcore expands the aggregate socket bandwidth, the effective throughput available to mi decreases as co-runner demand increases, causing memory stall cycles that strictly inflate Lcurrent,i . Crucially, because Φ(·) and ∆Linterf,i depend on highly dynamic runtime behaviors, opaque microarchitectural state interactions (e.g., core pipeline stalls and cache hierarchy thrashing), and complex, non-linear resource contention across the entire socket, deriving an exact analytical model is computationally intractable. Instead, these highly dynamic and coupled interactions must be approximated and learned iteratively at runtime.
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
13
C. Power Model The total power consumption of the socket, Psocket (tk ), is decomposed into the dynamic power of the active cores, the dynamic power of the shared uncore domain, and the baseline static leakage power (Pstatic ):
Psocket (tk ) =
X
uncore core (funcore (tk ), ⃗λ(tk )) + Pstatic (fcore,i (tk ), λi (tk )) + Pdyn Pdyn
(2)
i∈C
Because dynamic power scales proportionally with the switching frequency and the square of the supply voltage (P ∝ V 2 f ), aggressive scaling of both core and uncore frequencies provides the primary lever for power minimization. Crucially, in modern high-density processors, the uncore uncore subsystem represents a massive physical domain. Consequently, Pdyn consumes a
disproportionately dominant fraction of the total socket power compared to any individual core’s core dynamic power (Pdyn ). Therefore, the K8SPI controller treats the uncore frequency not just as
a performance knob, but as the most critical variable for achieving node-wide energy efficiency, albeit one highly coupled to the aggregate memory transaction rate ⃗λ(tk ) generated by all cores on the socket. Similar to the latency model, explicitly formalizing the complex, non-linear coupling between highly dynamic multi-core workloads, joint frequency scaling, and holistic socket power consumption across diverse hardware is practically infeasible at runtime. Therefore, the exact powerperformance manifold is treated as a highly dynamic, black-box environment whose underlying transition dynamics will be implicitly learned via the Deep Reinforcement Learning policy. D. Problem Definition The objective of the proposed K8SPI framework is to determine the optimal frequency scaling actions at each event epoch tk to minimize the total socket power consumption, while strictly guaranteeing that no microservice violates its performance requirement. The constrained optimization problem is formulated as:
min
f⃗core ,funcore
subject to
K X
Psocket (tk )
(3)
k=0
Lcurrent,i (tk ) ≤ Ltarget,i
∀mi ∈ M, ∀k
(4)
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
14
min max fcore ≤ fcore,i (tk ) ≤ fcore ,
(5)
min max funcore ≤ funcore (tk ) ≤ funcore
This formulation highlights the core control challenge: minimizing Equation 2 by lowering frequencies directly penalizes the latency function in Equation 1, thereby risking violations of the strict constraint in Equation 4 amid unpredictable uncore interference. Because the functions governing power (Eq. 2) and latency (Eq. 1) exhibit complex, non-linear, and highly dynamic behaviors at runtime across both core and uncore domains, traditional convex optimization and static heuristic control are inadequate. Consequently, we frame this optimization as a Markov Decision Process (MDP). K8SPI leverages a Deep Reinforcement Learning agent to map real-time telemetry states to optimal frequency actions, inherently learning a robust control policy capable of navigating the complex trade-offs between strict performance requirement enforcement and maximum energy efficiency. IV. H IERARCHICAL R EINFORCEMENT L EARNING C ONTROLLER To navigate the high-dimensional joint frequency space efficiently, we propose a Hierarchical Reinforcement Learning (HRL) architecture. The hierarchical approach decomposes the DVFS adjustment into two sequential decisions. By splitting the action into coarse and fine stages, the hierarchical controller reduces the search space and explicitly separates the objectives of rapid SLA stabilization and fine-grained energy minimization. Triggered whenever a performance requirement violation or massive resource overprovisioning crosses a defined threshold, the control loop operates over a fixed two-step episode: Agent A1 (coarse) selects a large frequency delta to immediately exit the performance requirement violation or reduce gross overprovisioning, providing a safe and feasible starting point. Agent A2 (fine) then refines A1’s action by taking smaller delta steps to safely maximize energy savings before the episode terminates. A. State Representation and Observations At each decision step, the active agent receives a structured observation representing the system’s telemetry. The state space is composed of three primary components: •
Service-level vector (xs ): Contains microservice-specific performance counters at the hardware level (e.g., IPC, bandwidth per core) and at the application level (e.g., current latency). This vector is primarily used to guide the core frequency scaling decisions.
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
•
15
Socket-level vector (xu ): Contains aggregated socket-wide counters, reflecting the "outsidethe-core" view and the resource pressure exerted by concurrent, co-located microservices. This vector is primarily used to guide uncore frequency scaling.
•
Action Mask (m): A mask that marks infeasible actions as invalid to prevent the agent from choosing frequencies outside the allowable hardware ranges.
The specific metrics tracked in the observation space are detailed in Table I. These metrics are highly curated as they are the primary factors determining the hardware-level performance of the microservice. Crucially, they allow the RL agent to identify the architectural bottleneck (e.g., compute-bound vs. memory-bound) at runtime. 1) Frequency-Conditioned State Normalization: Standard global normalization fails to capture how physical hardware boundaries shift under Dynamic and Frequency Scaling. To ensure stable convergence and explicitly embed domain knowledge, we introduce a frequency-conditioned online normalization strategy. The environment maintains independent running statistics (mean µ, standard deviation σ) for each unique core and uncore frequency pair (fcore , funcore ). The telemetry vector x is normalized via an online z-score strictly within its active frequency bucket: x(norm) =
x − µ(fcore ,funcore ) σ(fcore ,funcore ) + ξ
(6)
This bucketed approach enables the RL agent to intelligently isolate architectural bottlenecks. If compute-related metrics (e.g., KIPS) yield high positive z-scores during an performance requirement violation, the agent infers core saturation and scales up fcore . Conversely, saturating memory metrics (e.g., bandwidth) prompts the agent to scale funcore . While this accurately captures regime-specific hardware limits, it introduces a minor cold-start phase where newly visited frequency pairs require a brief initialization window to stabilize their statistics. B. Hierarchical Action Space Rather than exploring a massive flat joint action space (e.g., 289 actions), the two-agent design drastically reduces the dimensionality. •
Coarse Stage (Agent A1): Selects broad adjustments with ∆f ∈ {−0.8, 0, +0.8} GHz for both the core and uncore knobs, yielding a highly focused space of 3 × 3 = 9 actions.
•
Fine Stage (Agent A2): Refines the coarse adjustment by selecting smaller deltas with ∆f ∈ {−0.4, −0.2, 0, +0.2, +0.4} GHz, yielding 25 possible actions.
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
16
TABLE I: RL Agent Observation Space Metrics Metric Name
Description Service-Level Metrics (Used to guide core)
Core_L3_Misses
L3 cache misses on the service core
Core_L3_Hits
L3 cache hits on the service core
Core_Mem_Bandwidth
Local memory bandwidth used by the service
Service_Avg_Latency
Average latency of service requests
Service_Target_Latency
Target latency of service requests
Core_MPKC
Misses per thousand instructions
Core_IPC
Instructions per cycle
Core_KIPS
Kilo Instructions per Second
CPU_Time_System
Time spent in system/kernel mode
CPU_Time_User
Time spent in user mode
CPU_Utilization
Container’s total CPU usage (%)
Core_Frequency
Core frequency assigned to the service
Core_L3_Occupancy
L3 cache usage by the service core
Socket-Level Metrics (Used to guide uncore) Socket_L3_Misses
Aggregate L3 cache misses on the socket
Socket_L3_Hits
Aggregate L3 cache hits on the socket
Socket_Mem_Bandwidth
Total memory bandwidth on the socket
Socket_L3_Occupancy
Overall L3 cache occupancy
Socket_Power_CPU
Package-level CPU power consumption (W)
Socket_Power_DRAM
DRAM power consumption (W)
Socket_Uncore_Frequency
Current uncore frequency on the socket
For both domains, the selected action is a discrete frequency delta applied to the current operating state: (new) fcore = fcore + ∆fcore ,
(new) funcore = funcore + ∆funcore
(7)
To strictly prevent the agent from selecting out-of-bound frequencies, the environment applies
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
17
a dynamic binary mask M that filters infeasible deltas prior to action selection: 1, if f min ≤ fcore + ∆fcore ≤ f max and f min ≤ funcore + ∆funcore ≤ f max core core uncore uncore M(∆fcore , ∆funcore ) = 0, otherwise (8) C. performance Gap Quantization To stabilize the learning process and prevent control jitter, the performance state of each microservice is evaluated via a quantized latency gap. First, the raw relative latency gap for a given microservice is defined as: graw =
L − L∗ L∗
(9)
where L is the current measured average latency and L∗ is the strict target latency constraint. Under this formulation, a positive value indicates an performance requirement violation, whereas a negative value indicates resource overprovisioning. Crucially, the magnitude of this raw gap serves as the primary trigger for corrective action. The control loop is only invoked when the performance deviates beyond an acceptable boundary—for instance, an acceptable latency degradation threshold (e.g., graw > +15%) or a severe overprovisioning threshold (e.g., graw < −15%). Rather than feeding the continuous, noisy graw directly to the RL agent, the environment maps it to a discrete, bucketed gap g(L, fc , fu ) using a three-tiered logic: 0, if hardware-saturated g(L, fc , fu ) = 0, if |graw | < ϵ sgn(g )δ⌈|g |/δ⌉, otherwise raw raw
(10)
This quantization strictly enforces three operational rules to ensure predictable power transitions: 1) Deadzone (ϵ): A tight tolerance band (e.g., τ = 5%) masks small, transient latency fluctuations. This prevents jitter and unnecessary frequency toggling when the system is operating near the target. 2) Step-Based Scaling (δ): The gap is discretized into fixed intervals (e.g., δ = 10%) to group state observations into distinct buckets, rendering the optimization landscape more stable for the agent.
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
18
3) Hardware-Aware Safety Cap: The gap is forced to zero if the system is hardware-saturated. Specifically, this occurs if the performance requirement is satisfied but frequencies are already at their lowest supported bounds (L ≤ L∗ ∧ fc = fc,min ∧ fu = fu,min ), or if the performance requirement is violated but frequencies are already maxed out (L ≥ L∗ ∧ fc = fc,max ∧ fu = fu,max ). This hardware awareness stops the controller from attempting physically impossible adjustments. D. Hierarchical Reward and Credit Assignment The reward signals are engineered to encode both strict performance requirement constraint satisfaction and power efficiency. The hierarchical controller operates over a fixed two-step episode: A1 (coarse)
A2 (fine)
s0 −−−−−−−→ s1 −−−−−−−→ s2 (1)
(1)
∆fc ,∆fu
(2)
(11)
(2)
∆fc ,∆fu
where Agent A1 performs a coarse move transitioning the system to intermediate state s1 , and Agent A2 refines it to reach the terminal state s2 . Let g0 , g1 , and g2 represent the quantized performance gaps at the pre-A1, post-A1, and terminal stages, respectively. a) Universal Performance Potential: To evaluate performance requirement compliance consistently across both hierarchical stages, we define a universal performance potential function Φ(g) based on the quantized gap:
Φ(g) =
−(2 + g),
g>ϵ
1 − |g|,
g≤ϵ
(violation) (12) (safe)
where ϵ represents the SLA safety threshold (e.g., ϵ = 0.05). This piecewise function heavily penalizes performance requirement relatively to variation magnitude (g) while rewarding proximity to the target latency when operating in the safe zone. b) Etiquette Penalty: To embed domain knowledge and prevent unsafe throttling, an etiquette penalty applies at both steps t ∈ {1, 2}. It strictly penalizes the active agent for decreasing both frequencies if the system is already in a state of performance requirement violation: ψt = ηboth · I gt−1 > 0 ∧ ∆fc(t) < 0 ∧ ∆fu(t) < 0
(with ηboth = 0.5)
(13)
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
19
E. Agent A2 (Fine) Reward Function The goal of the fine agent is to refine the frequencies to minimize power once the system is near or within the safe region. Its total reward, RA2 , evaluates the terminal performance state g2 and adds an efficiency bonus R∗ if the SLA is satisfied, minus its step-specific etiquette penalty: RA2 = Φ(g2 ) + I[g2 ≤ ϵ] · R∗ − ψ2
(14)
To explicitly favor low-power operating points, each discrete hardware frequency is mapped to a normalized index ax ∈ [0, 2], where 0 is the minimum supported frequency and 2 is the maximum. The efficiency bonus is defined as: R∗ = wc (1 − ac,2 ) + wu (1 − au,2 )
(15)
When the performance requirement is met (g2 ≤ ϵ), the agent balances two competing objectives. The performance term Φ(g2 ) = 1 − |g2 | encourages the agent to operate closely to the target latency, minimizing unnecessary performance slack and overproviding. Concurrently, the efficiency bonus R∗ drives power reduction. Because dynamic power scales directly with operating frequency, core and uncore frequencies serve as reliable proxies for power consumption. The normalization mapping to ax ∈ [0, 2] ensures that selecting the minimum supported frequency yields a maximum positive reward (e.g., 1−0 = 1), whereas operating at the maximum frequency introduces a penalty (e.g., 1 − 2 = −1). Furthermore, we configure the weights such that wc < wu ; consequently, raising the core frequency is less penalized than raising the uncore frequency, accurately reflecting the higher power cost of the socket-wide uncore domain. F. Agent A1 (Coarse) Reward Function The primary goal of the coarse agent is to rapidly reduce the latency gap and create an optimal, safe starting point for A2’s fine-tuning. Because A1 transitions the system to the intermediate state s1 , its immediate performance reward is Φ(g1 ). To ensure cohesive teamwork and prevent A1 from acting greedily, A1’s total reward directly incorporates A2’s terminal outcome, while subtracting its own step-specific etiquette penalty (ψ1 ): RA1 = Φ(g1 ) − ψ1 + RA2
(16)
This elegantly coupled design forces A1 to select coarse actions that not only immediately improve the SLA state (Φ(g1 )) but also position the intermediate state such that A2 can successfully minimize energy without triggering a terminal violation (RA2 ).
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
20
G. Runtime Orchestration and Action Aggregation In a live production environment, K8SPI operates as an event-triggered control loop monitoring the performance gaps of all co-located microservices on a shared socket. When a specific microservice mi ∈ M actively violates its performance requirement/severely over-provisioned (gi > +/ − 20%), the RL controller proposes a corrective joint frequency action (fcore,i , funcore,i ) per microservice. However, the physical hardware imposes distinct granularity constraints for applying these independent per-service actions. When come to the proposed core frequency is applied directly and independently to the specific core hosting mi (as microservices are pinned to a single cpu core): fcore (ci ) = fcore,i
(17)
Conversely, the uncore subsystem is a shared, socket-wide resource. To resolve conflicting uncore frequency proposals among co-located services while ensuring the performance require∗ ments of all microservices are met, the final uncore frequency funcore applied to the socket is the
maximum of all individual proposals: ∗ funcore = max {funcore,i } mi ∈M
(18)
This pessimistic aggregation guarantees that the most memory-constrained service is strictly protected against uncore throttling, while node-wide power savings are successfully captured only when all co-located services mutually tolerate a scaled-down uncore frequency. V. S CALABLE I MPLEMENTATION AND T RAINING M ETHODOLOGY Hardware-in-the-loop RL is slowed by multi-second feedback: each frequency action must be applied and then observed through the monitoring stack (e.g., Prometheus) before the next
Fig. 3: Fast RL Prototyping Framework Architecture Overview.
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
21
Fig. 4: Testbed Architecture Overview.
step. Since RL needs thousands of trials, this makes training and iterative tuning (reward design, hyperparameters) prohibitively slow on a live Kubernetes node. A. Distributed Training and Snapshot Caching To accelerate learning, K8SPI uses Ray RLlib to enable distributed training with a centralized learner coordinating multiple concurrent environment runners. However, running many environments is costly because each environment ultimately requires execution on the physical node (frequency actions are hardware operations and cannot be virtualized). K8SPI resolves this by inserting an offline Config–Action–Observation Snapshot Cache. Fig. 3 shows the architecture of the fast RL prototyping framework. When an RL agent instance encounters a new ⟨state, action⟩ pair, the framework performs a one-time hardware profiling run: it deploys the state (the specific microservice executing its corresponding benchmark under a defined uncore stress level), applies the selected frequency configuration on the testbed, and collects telemetry over a sustained 5minute window to average out transient effects. The resulting high-fidelity observation vector is stored in a Parquet-backed datastore. Future visits to the same ⟨state, action⟩ pair are served instantly from the cache, enabling safe parallel RLlib training without live-hardware latency. VI. E XPERIMENTAL M ETHODOLOGY A. Testbed Infrastructure and Software Stack To ensure stable, reproducible, and noise-free Reinforcement Learning observations and power measurements, we constructed a strictly controlled, dual-socket Kubernetes testbed wchich it congration illustrated in Table II and in Fig 4. To prevent background OS noise from corrupting the RL agent’s telemetry or power measurements, we enforced strict hardware-level isolation. Socket 1 was dedicated to executing
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
22
TABLE II: Testbed Hardware and Software Configuration Attribute
Specification
CPU Model
Intel Xeon Gold 6132 Threads/Core
1 (HT Disabled)
Cores/Socket
14
Sockets
2
NUMA Nodes
2
Core Freq (GHz)
1.0 – 2.6
L3 Cache
38.5 MB/socket
Uncore Freq (GHz) 1.2 – 2.4
OS
Ubuntu 22.04.5
Kubernetes QoS
Guaranteed
Turbo & C-States Disabled
Memory Swap
Disabled
Core Governor
Uncore Governor
userspace
userspace
Attribute
Specification
the experimental microservices and the telemetry stack. Socket 2 was reserved entirely for the Kubernetes control plane and operating system background processes. Moreover, to eliminate unpredictable power fluctuations, several BIOS and kernel-level features were explicitly disabled (see Table II). The telemetry stack utilized Intel PCM for hardware performance counters, Intel PowerStat (i.e., Intel RAPL) for socket-level ground-truth power measurements, and cAdvisor alongside Kubelet metrics for per-microservice resource usage and core-pinning resolution. Finally, we employed custom scripts to measure the end-to-end latency per HTTP request for each hosted service, establishing the application-level performance of each microservice. Data was scraped and aggregated using a telemetry stack comprising Telegraf and Prometheus. To prevent load-generation overhead from interfering with the testbed, the traffic generators were hosted remotely and triggered the microservices via HTTP. To control the core and uncore frequencies during the experiments, we utilized the open-source Intel pepc library, which provides a sophisticated power control toolkit. B. Benchmarks and Workloads We construct a benchmark-driven microservice corpus using two micro-benchmark suites: Stress-NG (primarily CPU-bound and mixed behaviors) and Parallel Memory Bandwidth (PMBW) (memory-bound behaviors with strong sensitivity to the shared memory subsystem and uncore frequency). Each selected benchmark is containerized and deployed as a stateless HTTP microservice with fixed resource allocations (1 CPU and 2 GB memory). Each request triggers a bounded unit of benchmark work executed at a controlled load level. For each benchmark, we define 4–5 discrete load settings (e.g., a fixed number of memory-copy operations) to expose
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
23
Fig. 5: Micro-benchmark Dataset Split for Model Development and Evaluation.
different demand workload levels. To separate learning and evaluation from final testing, we create two disjoint datasets. Dataset 1 contains 20 services, split into 16 for training and 4 held out for RL model evaluation. Dataset 2 contains 8 additional services reserved exclusively for final testing and validation. Fig. 5 illustrates the dataset partitioning across the training, evaluation, and test phases. Crucially, each dataset incorporates a diverse mix of CPU-bound, memory-bound, and mixedworkload microservices. Each workload class exhibits a unique power and performance profile in response to independent core and uncore frequency scaling. We illustrate the impact of core and uncore frequency scaling on performance (microservice latency) and power consumption (package and DRAM power) across three representative benchmarks: a CPU-bound (BG_CPUAckermann) in Fig. 6, a memory-bound (BG_MemrateFlush) in Fig. 7, and a mixed (BG_Memthrash) microservice in Fig. 8. To isolate the specific impact of core frequency scaling, we fix the uncore frequency to its minimum. Similarly, we lock the core frequency at its minimum when varying the uncore frequency. To establish a performance reference for each service, we sweep joint core/uncore frequency configurations and execute the service in isolation on the socket across the defined load levels. We record end-to-end response times and define the target latency for each microservice at each configuration as the 70th percentile of the measured samples. 1) Uncore Interference Generation: To evaluate the controller’s robustness against noisy neighbors, we generate controlled uncore interference using the PMBW benchmark. We employ two distinct memory-bandwidth stressor profiles: ScanRead (memory read pressure) and ScanWrite (memory write pressure).
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
(a) Latency vs. Core Freq. (b) Latency vs. Uncore
(c) Power vs. Core Freq.
24
(d) Power vs. Uncore Freq.
Freq.
Fig. 6: Impact of core and uncore frequency scaling on the CPU-bound BG_CPUAckermann microservice.
(a) Latency vs. Core Freq. (b) Latency vs. Uncore
(c) Power vs. Core Freq.
(d) Power vs. Uncore Freq.
Freq.
Fig. 7: Impact of core and uncore frequency scaling on the memory-bound BG_MemrateFlush microservice.
As illustrated in Fig. 9, on the 14-core test socket, one physical core is strictly dedicated to the target latency-sensitive microservice deployed within the Kubernetes environment. The remaining 13 cores are utilized to host uncore stressor instances (USI) to generate varying levels of interference. These uncore stressors are deployed outside the Kubernetes boundary as standalone, non-containerized processes. To ensure precise and consistent interference scaling, each stressor instance is pinned to a distinct physical core via taskset, allocated 2 GB of memory, and its core frequency is statically locked at the maximum 2.6 GHz. By dynamically scaling the number of active stressor instances from 1 to 13, we accurately emulate escalating levels of background uncore contention. This process generates a detailed dataset for every benchmark across our training, evaluation, and test dataset splits. Specifically, for every core and uncore frequency combination, the dataset records both the microservice’s performance and the resulting socket-level metrics under varying levels of uncore stress.
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
(a) Latency vs. Core Freq. (b) Latency vs. Uncore
(c) Power vs. Core Freq.
25
(d) Power vs. Uncore Freq.
Freq.
Fig. 8: Impact of core and uncore frequency scaling on the mixed-workload BG_Memthrash microservice.
Fig. 10a, Fig. 10b, and Fig. 10c show the impact of escalating uncore stress levels (0, 6, 9, and 13 instances) on CPU-bound, memory-bound, and mixed workloads, respectively. For these experiments, we fix the core frequency at 1.8 GHz and the offered load at its maximum. The data reveals that CPU-bound workloads are largely immune to background uncore contention, as their latency is strictly compute-dominated. Conversely, memory-bound workloads suffer severe, yet approximately linear, performance degradation as the interference scales. Mixed workloads exhibit a moderate, hybrid impact that depends heavily on their active memory bandwidth reliance. Crucially, the latency degradation footprint is dictated not only by the intensity of the interference (i.e., the uncore stress level) but also by the specific workload nature of the uncore stressor (e.g., ScanRead vs. ScanWrite) and the target microservice workload. This demonstrates the complex landscape of real-world cloud environments, where diverse workloads
Fig. 9: Uncore Pressure Generation and Its Real-World Impact: Trace Collection Setup
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
26
(a) impact on CPU-bound microser- (b) impact on Memory-bound mi- (c) impact on Mixed workload mivice performance.
croservice performance..
croservice performance.
Fig. 10: Impact of escalating uncore stress levels on different microservice workload profiles with core frequency fixed at 1.8 GHz and maximum load.
running on the same socket generate highly variable patterns of uncore interference. Crucially, the RL agent is completely blind to the explicit stressor configuration and active instance count. It must infer the presence and magnitude of external interference indirectly through socket-level telemetry (e.g., shared uncore counters and memory bandwidth saturation). This design forces the agent to dynamically detect and mitigate performance degradation caused by unseen, co-located workloads using strictly observable hardware signals. C. Offline reward shaping and weight selection.
(a) BG_CPUAckermann microbenchmark
(b) BG_Memcpy microbenchmark
Fig. 11: Reward function behavior under different core and uncore frequency configurations.
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
27
We use historical measurements that jointly record latency outcomes and power consumption over a dense set of core/uncore frequency configurations, spanning different microservices and load levels. For each (microservice, load) condition, we define the optimal point as the lowestpower frequency configuration that satisfies the performance requirement (i.e., the minimumpower setting with non-positive gap, g ≤ ϵ). Reward shaping is calibrated to make this reference point uniquely desirable: the reward attains its maximum at the optimal point, drops sharply under performance requirement violations (g > ϵ), and decreases smoothly (yet perceptibly) in the SLO-safe region when frequencies are unnecessarily high, thereby penalizing wasted power due to over-provisioning. Because core and uncore frequency scaling have different marginal impacts on package power, we control their relative penalties using weights (wc , wu ) in R. Since the uncore domain is shared and strongly influences package power, we set wu = 2wc (a ratio commonly adopted in vendor and guidance for uncore power management [20]) and evaluate three configurations: (wc , wu ) ∈ {(0.1, 0.2), (0.2, 0.4), (1, 2)}. As shown for BG_CPUAckermann (see Fig 11a) and BG_Memcpy (see Fig 11b) across multiple loads, (wc , wu ) = (1, 2) yields the clearest shaping: R peaks at the circled global optimum and then decreases approximately linearly as core frequency increases. The smaller-weight settings produce flatter reward curves after reaching the SLO-safe region, which weakens the incentive to downscale. We therefore adopt (wc , wu ) = (1, 2) in the remainder of the study. D. Impact of State Normalization We evaluate four normalization strategies: •
No normalization. The agent receives raw metrics directly. This introduces zero preprocessing overhead but is prone to severe scale imbalance, where high-magnitude features dominate learning.
•
Global Z-score. A single Z-score transform is applied to all metrics. Although it standardizes scales, it can distort the physical meaning of key control and target variables (e.g., frequencies and latency).
•
Selective Z-score. Only background/system metrics are normalized, while control fields are kept in their raw units. This mitigates scale mismatch, but remains frequency-blind by mixing statistics across heterogeneous operating points.
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
28
TABLE III: Impact of normalization methods on training stability and convergence. Method
•
Convergence (iters)
Final performance
Post-conv. CoV
Custom (freq-conditioned)
15–20
0.892
10.1%
Z-score (selective)
25–30
0.850
14.7%
Z-score (global)
30–40
0.820
18.9%
No normalization
50+ (unstable)
0.650
43.1%
Custom (frequency-conditioned). Separate normalization statistics are maintained for each (core, uncore) frequency pair. This captures regime-specific behavior without cross-contamination, at the cost of a brief cold-start period when a new pair is encountered.
The quantitative impact on training stability is summarized in Table III. The performance is defined as the mean normalized evaluation reward after convergence. Figure 12 shows the normalized reward during evaluation under different normalization strategies.
Fig. 12: Normalized reward during evaluation under different normalization strategies.
Overall, the Custom normalization yields the best combination of sample efficiency and robustness. By isolating statistical baselines per hardware operating regime, it stabilizes learning and evaluation behavior, converging within 15–20 iterations while achieving the highest final performance (0.892). It also exhibits the lowest post-convergence variability (CoV 10.1%), compared to the highly unstable no-normalization baseline (CoV 43.1%).
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
29
TABLE IV: Minimal hyperparameter summary. We keep the same training budget (Nsteps ) and shared settings across all methods; only algorithm-specific knobs differ. Parameter
HL-PPO
HL-DQN Nsteps =100
Training iteraction Optimizer / lr
Adam; 3 × 10−4
Discount γ
0.99
Train batch size
4096
Policy structure
Hierarchical (coarse+fine)
Action space size
(|Acoarse |, |Afine |) = (9, 25)
PPO: clip / GAE / entropy
0.2 / 0.95 / 0.01
–
DQN: buffer / target update
–
65,536 / 500
DQN variants (if used)
–
Double, Dueling, Rainbow (toggles)
E. RL Configuration and Training Details We implement the hierarchical controller in Ray RLlib and adopt Proximal Policy Optimization (HL-PPO) as the default optimizer; hyperparameters and network settings are summarized in Table IV. We select PPO because its clipped on-policy updates are empirically more stable under the non-stationarity induced by dynamic load and co-runner interference, and it typically requires less sensitive hyperparameter tuning than value-based methods in continuous-control settings.
(a) Reward function trajectories during training.
(b) Reward function trajectories during evaluation
Fig. 13: Reward function trajectories for different RL algorithms
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
30
We validate this choice by comparing HL-PPO with standard DQN, Double DQN, and Rainbow DQN under an identical reward definition and training. Figure 13a and Figure 13b report the reward trajectories during training and evaluation, respectively, across the different RL algorithms. After convergence (approximately iteration > 20), all methods reach comparable average reward; however, HL-PPO exhibits substantially lower post-convergence reward volatility (about ±0.2) than Double DQN and DQN (about ±0.5, ≈ 2.5× higher) and Rainbow DQN (about ±0.6, ≈ 3× higher). This stability is important for deployment because it translates into more predictable frequency decisions and reduced control jitter. A further reason for adopting PPO is its gradual on-policy adaptation under workload nonstationarity: in cloud deployments, K8SPI can optionally perform periodic online calibration using newly collected runtime samples to refine the policy. PPO’s clipped updates help limit abrupt policy changes during such fine-tuning, reducing the risk of transient control spikes under distribution shift. F. Baseline Governors We benchmark K8SPI against the following baselines: 1) performance Governor: Default used power Governor that pinning core and uncore frequencies to maximum hardware limits, establishing upper bounds for both power and performance. 2) Flat Single-Agent PPO (Ablation): A non-hierarchical PPO agent managing the joint coreuncore action space. For fair algorithmic comparison, it uses K8SPI’s exact observation space and a mathematically equivalent single-step reward: −(2 + g) − ψ, g>ϵ Rflat = (1 − |g|) + wc (1 − acore ) + wu (1 − auncore ), g ≤ ϵ
(19)
3) Offline Optimal (Oracle): Computed offline by scanning all frequency permutations per opt opt (microservice, load) pair. It selects the tuple (fcore , funcore , P opt , g opt ) that strictly satisfies
the performance requirement (g ≤ 0) while minimizing total power, providing an absolute reference to quantify our policy’s optimality gap. G. Performance Metrics During the training/evaluation phase, the agent is evaluated over entire episodes of length T . We define four primary metrics averaged over the episode steps t ∈ {1, . . . , T }:
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
31
1) Gap Percentage (g̃): The mean relative deviation of the resulting latency from the target latency over the episode. Let gt be the latency gap at step t. The mean gap is calculated as: T
g̃ =
1X gt T t=1
(20)
A positive value (g̃ > 0) strictly indicates an performance requirement violation, whereas g̃ ≤ 0 signifies compliance. 2) 95th Percentile Gap (g̃p95 ): Because the mean gap can obscure transient performance spikes, we also evaluate the 95th percentile gap. This represents the value that 95% of the observed latency gaps (gt ) fall below. It serves as a strict measure of tail latency, ensuring the system remains reliable even under worst-case interference. 3) Power Distance Percentage (DP ): The mean relative deviation of the controller’s power consumption (Pt ) from the offline optimal oracle (Ptopt ) across the episode. It is calculated as:
T
1X DP = T t=1
Pt − Ptopt Ptopt
× 100
(21)
A lower value means better power efficiency. 4) Normalized Reward: The episodic reward scaled within each testing run to enable fair comparison across different algorithms and configurations. 5) Strict Efficiency Score (SES): A single metric that combines power efficiency and performance requirement compliance, with strong penalties for latency violations. It is defined as: SES = −DP − Penalty(g̃) where Penalty(g̃) applies a regime-specific cost based on the gap: 0, g̃ ≤ −ϵ Penalty(g̃) = g̃+ϵ , −ϵ < g̃ ≤ 0 ϵ g̃ e ϵ , g̃ > 0
(22)
(23)
H. Testing Environment We evaluate the trained policy using a replay-driven simulator (see Figure14 illustrates the testing environment simulator) in closed loop. after the offline phase that builds the test dataset as described in VI-B. The simulator is driven by (i) a traffic generator that determines the timevarying load profile and (ii) a simulator component that replays the system state (online phase).
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
32
Fig. 14: Normalized reward during evaluation under different normalization strategies.
It emulates a non-stationary cloud node by executing microservices in fixed time windows while varying offered load and uncore interference at a fixed interval, we refer to this fixed interval as a workload epoch (Tepoch = 60 s) to emulate time-varying demand in real cloud settings. At each workload epoch, the PPO agent receives the same observation representation used in the live system and outputs a joint core/uncore frequency adjustment. The simulator resolves each action by replaying the corresponding measured outcome from the dataset, conditioned on the active microservice, load level, interference setting, and selected core/uncore configuration. This design enables reproducible closed-loop testing while avoiding perturbations that would be introduced by exploratory actions on the physical node. VII. E VALUATION AND R ESULTS To thoroughly validate K8SPI’s effectiveness, we evaluate the framework across three complex, gradual, real-world scenarios. A. Scenario 1: Isolated Latency-Critical Microservice Setup. In this scenario, we evaluate our solution in an interference-free environment. A single latency-critical microservice is deployed on the test socket at a time. For each microservice in the test dataset, we run a 300 s trace under continuous sequential HTTP requests. The load level is updated every 60 s; (Tepoch = 60 s). K8SPI employs event-triggered actuation and intervenes only when the performance gap leaves the safety band, i.e., g > +20% or g < −20%. To evaluate the efficiency of our RL design in reaching optimal hardware configurations with minimal action, we
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
33
impose strict action budgets. Per load period, the hierarchical controller is allocated a maximum of 6 steps (equivalent to three full coarse→fine cycles), while the flat single-agent PPO baseline is strictly limited to 3 actions. B. Results and Trace Analysis
Fig. 15: Total Power Consumption per Benchmark Under Different Power Governance Policies (Scenario 1)
Fig. 16: Average Performance Gap per Benchmark Under Different Power Governance Policies (Scenario 1).
Across isolated microbenchmarks, the hierarchical controller (HRL) consistently selects safer and more power-efficient operating points than the flat single-agent PPO: as summarized in Fig. 15 and Fig. 16, HRL maintains stable performace behavior (gap g̃ near zero or negative) while delivering high power efficiency (typically ≈30% savings) and closely tracking the offline oracle, whereas the single agent achieves only ≈20% savings on average and exhibits instability on sensitive workloads. The clearest failure case is STRESS_CPU_Longdouble, where the
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
34
Fig. 17: 95th percentile Performance Gap per Benchmark Under Different Power Governance Policies (Scenario 1).
single agent aggressively downscales and violates the performance requirement (g̃ = +0.58 with 100% violation rate), while HRL preserves compliance (g̃ = −0.03) and still attains ≈30% power savings. This stability is further validated by analyzing the 95th percentile gap (g̃p95 ), which exposes transient tail latency spikes that the mean gap might obscure. As depicted in the Fig 17, the flat single-agent PPO frequently breaches the 20% performance requirement violation threshold (g̃p95 > 0.2). It exhibits severe tail latency spikes on workloads such as STRESS_CPU_Longdouble (g̃p95 > 1.0), STRESS_CPU_Float (g̃p95 ≈ 0.28), and PMBW_ScanWrite64IndexUnroll (g̃p95 ≈ 0.25). Conversely, HRL strictly suppresses these worst-case transient spikes, maintaining g̃p95 well below the violation threshold for nearly all evaluated microbenchmarks. While HRL experiences a singular transient spike on STRESS_CPU_Div64, it overwhelmingly outperforms the single agent in bounding worst-case latency variations across the dataset. While aggregate metrics demonstrate HRL’s overall efficiency, a trace-level analysis further highlights the performance of HRL compared to the baselines. By examining temporal decisions across distinct microbenchmarks, we identify how HRL successfully navigates the joint core/uncore frequency state-space to prevent destructive local optima. For the STRESS_CPU_Float microservice (See the Fig18a), the offered load undergoes discrete step upgrades at t = 100 s and t = 200 s. During the initial phase, the flat single agent maximizes the uncore frequency to 2.4 GHz while starving the core frequency at its minimum of 1.0 GHz. Because this workload is heavily compute-bound, scaling the uncore provides zero latency benefit. Consequently, the single agent consumes 76 W while continuously violating the
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
(a) STRESS_CPU_Float
35
(b) STRESS_CPU_Longdouble
Fig. 18: Trace analysis.
+0.2 performance requirement threshold (gap ≈ +0.3). In contrast, HRL correctly identifies the compute sensitivity, stabilizing at a core frequency of 1.2 GHz and an uncore of 1.8 GHz. This allows HRL to safely maintain compliance (gap ≈ +0.08) while saving 15 W compared to the single agent. As load increases, HRL dynamically readjusts to perfectly overlap the offline oracle’s trajectory, pinning the uncore to its minimum and scaling the core to maintain a nearzero gap at optimal power (∼54 W) without massive oscillations. Meanwhile, at maximum load (200–300 s), the single agent fails to adapt its actions to the workload’s nature, unnecessarily increasing the uncore frequency from 1.0 GHz to 1.2 GHz, which needlessly increases total power consumption.
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
36
The trace of STRESS_CPU_Longdouble (see the Fig18b) further exposes the outperformance of the HRL agent compared to the baselines. HRL perfectly mimics the oracle policy throughout the execution (core 2.2 GHz, uncore 1.0 GHz), ensuring strict performance requirement compliance (gap never exceeding +0.02) at 59 W. The single agent, however, fundamentally fails to discover an adequate hardware configuration. It operates with a sustained +0.35 gap violation initially (0 to 200 s) and then suffers a catastrophic performance collapse at peak load (gap > 1.0) because it inexplicably bottoms out the core frequency to 1.0 GHz. Ultimately, these traces demonstrate that HRL provides a highly generalizable and robust solution for joint core and uncore frequency scaling. In memory-intensive regimes where core frequency yields diminishing latency benefit, HRL improves energy proportionality by confidently bottoming out compute resources without sacrificing performance: on STRESS_MEM_Vm, HRL reaches 51 W (31.9% savings) versus 58 W for the single agent (22.7% savings), effectively matching the oracle (50.9 W), and on PMBW_ScanRead64 HRL achieves 29.2% savings versus 21.2% for the single agent; more broadly, HRL remains within 1–2% of the oracle across nearly all memory-bound cases. Overall, under interference-free conditions, HRL exhibits oracle-level efficiency while maintaining performance-stable behavior, whereas the flat single-agent PPO is more prone to conservative local optima and occasional under-provisioning. C. Scenario 2: Single Microservice with Best-Effort Interference 1) Step: To evaluate the controller under resource contention, we design a memory-bandwidth interference sweep. The target service is a memory-bandwidth-bounded application (BG_Memthrash) operating at a fixed offered load of 67%. This service is co-located with memory-intensive besteffort (BE) services (no performance requirement is considered for the BE). We progressively increase the number of BE co-runners from 1 to 13 to emulate rising memory bandwidth contention, evaluating two BE workload profiles: ScanRead and ScanWrite. As contention intensifies, the target microservice’s performance degrades, which manifests as an increased performance gap. Fig 19 illustrates the interference sweep setup and the resulting trace behavior as the number of BE co-runners increases. The RL controller is expected to minimize power consumption while strictly maintaining g̃ below the performance requirement constraint. Driven by observed interference intensity, socket-level metrics, and the real-time performance
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
37
(a) Power reduction summary.
(b) Mean gap summary.
(c) P95 gap summary.
Fig. 19: Trace analysis: BG_Memthrash.
Fig. 20: Performance summary (power reduction, mean performance gap, and P95 performance gap).
gap, the controller must dynamically adapt core and uncore frequencies to counteract the BE workload interference and its nature (scan read, scan write).
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
38
2) Results and Trace Analysis: As illustrate on the Figure 19, under low to moderate memory contention (Stress Levels 1–8), the performance gap (g̃) remains well within the safe boundary (-0.1 to 0.1). In this regime, both the hierarchical reinforcement learning (HRL) and single-agent controllers exhibit comparable baseline performance, maintaining g̃ between -4% and +3% while consuming 84 to 87 W. However, even in this low-stress environment, HRL demonstrates superior action-space efficiency by reliably pinning the uncore frequency to 1.0 GHz (note: a slight visual margin is added in the trace plots to prevent the HRL line from obscuring the optimal oracle). This strategy prevents unnecessary DRAM power consumption, whereas the single agent exhibits exploratory and inefficient scaling of uncore frequencies. As best-effort interference amplifies (Stress Levels 9–13) and threatens the 0.10 (10%) performance gap threshold. The single agent adopts a reactive, brute-force approach, aggressively boosting core and uncore frequencies to mitigate the rising gap. This leads to severe over-provisioning; for instance, at Stress Level 10, the single agent consumes 108.9 W—wasting approximately 18 W compared to HRL—to achieve an excessive negative gap of -21.84%. Throughout these high-interference phases, the single agent exhibits spiky frequency oscillations and power bursts, indicating unstable control and the creation of an energy-expensive, conservative buffer. Conversely, HRL demonstrates efficiency-aware control that closely aligns with the optimal oracle, avoiding the panic-driven over-provisioning of the single agent. Instead, it accepts a controlled and acceptable performance degradation (+5.01% gap at peak Stress Level 13, remaining well below the 10% threshold) to preserve energy proportionality. align with the optimal policy that accepts a marginal gap increase (≈+0.6%). Overall summary metrics for the BG_Memthrash microservice confirm this efficiency. HRL achieves an average power reduction of 28.0% relative to the maximum performance baseline, nearly matching the optimal oracle’s 28.9% savings, while the single agent achieves only a 23.8% reduction as depicted on Fig 19a. Furthermore, HRL maintains a mean gap almost perfectly at zero (see Fig 19b). While HRL’s 95th percentile (P95) gap experiences brief transient spikes slightly above the 0.10 threshold (≈0.13) as shown in Fig 19c, the single agent aggressively suppresses these variations at a significant power cost. By reducing overall gap variance (σgap ) despite stochastic memory stress, the hierarchical architecture successfully navigates the complex power-performance trade-offs inherent to memory subsystem contention. Overall, for the BG_Memthrash microservice, HRL achieves a 28.0% power reduction, nearly matching the oracle’s 28.9% and outperforming the single agent’s 23.8%. HRL maintains
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
39
an optimal mean gap near zero, whereas the single agent over-provisions to force a negative mean gap of ≈-0.09. While HRL’s 95th percentile (P95) gap briefly exceeds the 0.10 threshold (≈0.13), the single agent strictly suppresses its P95 gap to ≈0.05 at a significant power cost (4.2% more power). By reducing the gap under stochastic stress, the hierarchical architecture successfully navigates non-linear power-performance trade-offs in the memory subsystem. D. Scenario 3: Co-located Latency-Sensitive Microservices 1) Step: To evaluate the controller under complex, multi-tenant conditions, we deploy 14 latency-sensitive microservice instances co-located on the same CPU socket, as detailed in Table V. These instances are instantiated exclusively from the memory-bound and mixed-workload test benchmarks. The experiment is conducted over a fixed duration of 3000 s, To emulate a highly dynamic cloud environment, the offered load for each service changes randomly at the onset of each workload epoch (Tepoch = 60 s). Throughout the execution, the colocated services naturally interfere with one another by competing for shared memory bandwidth and uncore resources. Consequently, the performance requirement compliance of each individual instance is simultaneously challenged by its own stochastic load variations and the compounding effects of cross-service interference. TABLE V: Scenario 3 Co-located Microservices Configuration Service Type
Instances
Workload Characteristic
PMBW_ScanWrite256IndexUnroll
5
Memory-bounded (write-intensive)
STRESS_MEM_Mmap
5
Memory-bounded (mmap operations)
PMBW_PermRead64SimpleLoop
4
Memory-bounded (read-intensive)
2) Results and Trace Analysis: Under realistic, multi-tenant cloud conditions where microservices demonstrate stochastic execution behavior, the HRL controller proves exceptionally resilient. The evaluation spans a 3000-second duration. The primary objectives are to enforce a strict performance requirement gap bound of +0.1 while maximizing energy savings. Acting entirely autonomously, the HRL agent explores a vast combinatorial space, concurrently managing independent core frequencies across 14 instances alongside the shared uncore frequency. Fig. 25 illustrates the total power consumption reduction achieved by the proposed approach. Overall, the HRL agent achieves a 22.6% reduction in total power consumption compared to
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
40
Fig. 21: Detailed breakdown of performance requirement violation percentages across the diffrent microservices
Fig. 22: Dynamic trace of the independent adjustments made to the 14 CPU core frequencies over the 3000-second evaluation.
the baseline. Despite stochastic load variations and severe cross-microservice interference at the uncore level, the HRL agent successfully meets the varying performance requirements across the diverse deployed microservices. It achieves an aggregate compliance rate of 97.59%, suffering only 2.41% total violations across the entire 3000-second window. Fig. 22 and Fig. 24 show the dynamic trace of the independent CPU core and shared uncore frequencies over time. Trace analysis reveals the agent’s highly dynamic learned policies: it rapidly and continuously adjusts all 14 independent CPU core frequencies jointly with the shared uncore frequency to maintain strict compliance under highly volatile contention. Fig. 21 details the violation percentages across the deployed microservices. A detailed break-
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
41
Fig. 23: The 95th percentile (P95) performance gaps evaluated across the deployed microservices.
Fig. 24: Trace of shared uncore frequency modulations Fig. 25: Total power consumption executed by the HRL agent to manage cross-microservice of the proposed HRL agent VS the interference.
default performance mode.
down of the violation percentages confirms the policy’s robustness. Half of the evaluated microservices— such as PMBW_ScanWrite256IndexUnroll_ins0, PMBW_ScanWrite256IndexUnroll_ins3, and STRESS_MEM_Mmap_ins7—achieved near-zero (≈ 1%–2%) violation rates over the test-
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
42
ing period. Even the most sensitive and demanding microservice, STRESS_MEM_Mmap_ins4, peaked at a mere ≤ 6% violation percentage under maximal contention scenarios. This represents a highly acceptable worst-case operational bound, ensuring no single noisy neighbor leads to the system’s full failure. Fig. 23 illustrates the P95 performance gaps for the evaluated microservices. Operating exactly on the edge of a performance requirement threshold is risky under stochastic load. The proposed HRL avoids dangerous limit-riding via conservative optimization boundaries. The 95th percentile (P95) gaps remain in negative or near-zero alignment for the vast multitude of services. Even for the most vulnerable tasks, such as STRESS_MEM_Mmap_ins4 and PMBW_ScanWrite256IndexUnroll_ins8, the 95th percentile spikes stabilize right at the +0.2 boundary. This highlights that any latency spikes above the target are rapidly mitigated transients rather than sustained degradations. VIII. D ISCUSSION The evaluation reveals a consistent control pattern: K8SPI’s hierarchical reinforcement learning (HRL) separates rapid mitigation of performance requirement violations from fine-grained power minimization, which prevents conservative “always-max” operating points under both isolated execution and multi-tenant contention. This section explains the mechanisms behind this behavior and the implications for cloud-native power governance with core/uncore actuation. A. Decoupling Safety from Optimization The failure of the single-agent baseline underscores a critical flaw in flat RL architectures when applied to latency-sensitive environments: the confounding of immediate constraint satisfaction with long-term efficiency. The single agent frequently falls into conservative local optima, aggressively over-provisioning resources at the first sign of performance degradation beyond the requirement. This behavior mimics traditional utilization-driven governors that conflate high resource allocation with system stability. By decoupling these objectives, K8SPI’s hierarchical design ensures that the coarse-grained agent strictly handles rapid performance requirement mitigation, freeing the fine-grained agent to safely explore the lower bounds of power consumption. B. Navigating the Memory Wall A primary insight from the memory-contention scenarios is the physical limitation of independent frequency scaling. While CPU core frequency generally exhibits a linear relationship with
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
43
computational throughput for CPU-bound tasks, the memory subsystem (governed by the uncore frequency) presents a strict architectural bottleneck. When this "memory wall" is reached under high contention, further increases in core frequency result in diminishing latency benefits while exponentially increasing power consumption. The HRL agent successfully learns this non-linear boundary. It jointly scales core and uncore frequencies to adapt to a saturated memory bus. This indicates that the HRL model implicitly learns the hardware’s microarchitectural constraints, avoiding the power-expensive states that trap default governor strategies. C. Implications for Cloud Power Governance and Limitations Two broader implications follow from the evaluation. First, application-level performance feedback (via the performance gap) is critical: without it, node-level governors cannot distinguish safe power reductions from under-provisioning, especially under interference. Second, uncore scaling governors must be integrated with core DVFS into a single policy. Because each workload has a unique footprint, joint core and uncore governance leverages multi-dimensional optimization to efficiently manage shared resources. Despite these advantages, the current K8SPI framework has notable limitations. The system’s responsiveness is bottlenecked by the observation latency of the monitoring stack. In our evaluation, telemetry aggregation via Prometheus introduces a default observation delay of up to 30 s. Consequently, making immediate reactions to highly dynamic load shifts and uncore interference remains challenging. To accommodate this, we fixed the load periods to 60 s (Tepoch = 60 s) to allow the observation stack to reliably capture the necessary telemetry across all levels (application latency and hardware performance counters). Deploying K8SPI in highly volatile production environments will require integrating a low-latency, real-time observation system. Furthermore, the current design is tailored for strict CPU pinning scenarios, utilizing a oneto-one mapping between microservice instances and physical CPU cores. While this isolates core-level compute, future work must extend the framework to cover broader, unpinned CPU execution scenarios and multi-core allocations to fully address diverse cloud-native deployments. IX. C ONCLUSION AND F UTURE W ORK This paper presented K8SPI (Kubernetes Power Irrigation), an event-triggered hierarchical reinforcement learning (HRL) controller for node-level power optimization of latency-sensitive
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
44
microservices in Kubernetes. K8SPI adopts a two-stage architecture that separates rapid mitigation of performance requirement violations from fine-grained power minimization, enabling stable control over complex power–performance trade-offs. This separation is especially important in consolidated multi-tenant deployments, where co-located microservices have heterogeneous workload footprints and distinct performance requirements while contending for shared uncore resources. To make RL-driven control practical for real systems, we also developed a rapid prototyping and training workflow. By combining RLlib’s parallel training with system-state snapshots and trace-driven replay, the workflow supports efficient multi-environment training using real execution traces, reducing iteration time when reward shaping and policy design are refined. Evaluation on a Kubernetes testbed shows that K8SPI reduces node-level power by 23%–30% relative to the Linux performance governor while keeping performance requirement violations below 2%–3%, even under dynamic load fluctuations and severe multi-tenant uncore contention. These results indicate that hierarchical RL with joint core/uncore actuation can avoid conservative performance-mode provisioning and achieve substantial power savings without compromising required performance. Future work will proceed along two directions. First, we will extend K8SPI beyond strict CPU pinning to support unpinned execution and dynamic multi-core allocations, improving generality across cloud-native scheduling regimes. Second, we will address observability latency by integrating a low-latency telemetry path for critical signals, enabling faster reaction to microbursts and short-lived interference events. R EFERENCES [1] i scoop, “The impact of growing it sector electricity demand.” https://tinyurl.com/3dtm23f4, accessed: 03-07-2024. [2] K. Saenko, “Chatgpt’s energy consumption compared to google,” https://tinyurl.com/3t46khaj, accessed: 12-05-2024. [3] K. Saenko, “Generative ai’s environmental impact and carbon footprint,” https://shorturl.at/iYdoj, accessed: 03-07-2024. [4] K. F. Pilz et al., “AI’s Power Requirements Under Exponential Growth,” RAND Corporation, Tech. Rep. RR-A3572-1, 2025. [5] P. de Kruijff, “Deepseek efficiencies make ai cheaper,” ABC News, February 2025. [6] K. Ahuja, “Legal framework for carbon credits and emission trading,” December 2024. [7] R. Nishtala et al., “Cuttlefish: Dynamic optimization of core and uncore frequency scaling,” Unpublished technical report, 2021, preprint or unpublished work; no DOI available at the time of writing. [8] A. Guermouche et al., “Combining power capping and uncore frequency scaling for energy efficiency,” In Proceedings of an IPDPS Workshop, 2022, no DOI available; workshop paper.
THIS WORK HAS BEEN SUBMITTED TO THE IEEE FOR POSSIBLE PUBLICATION
45
[9] Étienne André et al., “duf: Dynamic uncore frequency scaling to reduce power consumption,” Concurrency and Computation: Practice and Experience, vol. 34, no. 3, 2022. [10] N. Gholkar et al., “Uncore power scavenger,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC 2019).
ACM, 2019, pp. 1–23.
[11] D. Lo et al., “Towards energy proportionality for large-scale latency-critical workloads,” in Proceedings of the 41st Annual International Symposium on Computer Architecture (ISCA).
IEEE, 2014, pp. 301–312.
[12] Y. Liu et al., “Sleepscale: Runtime joint speed scaling and sleep states management for power efficient data centers,” in Proceedings of the 41st Annual International Symposium on Computer Architecture (ISCA).
IEEE, 2014, pp. 313–324.
[13] S. U. Vamanan et al., “Timetrader: Exploiting latency slack to save energy in online data-intensive applications,” Technical report, 2014, extended technical report; DOI not assigned. [14] J. Jeong et al., “Gemini: Learning-based per-query dvfs for latency-critical workloads,” In Proceedings of the 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), 2020, no DOI available; conference paper. [15] C. Li et al., “Twig: Energy-efficient qos-aware co-location via deep reinforcement learning,” in Proceedings of the 26th IEEE IEEE, 2020, pp. 259–272, conference
International Symposium on High Performance Computer Architecture (HPCA). held in San Diego, CA, USA.
[16] D. Lo et al., “Hipster: Hybrid power management for latency-critical cloud workloads,” in Proceedings of the 23rd IEEE International Symposium on High Performance Computer Architecture (HPCA).
IEEE, 2017, pp. 409–421.
[17] L. Labassi et al., “Co-optimizing latency and energy for iot services using heterogeneous multicore processors in fog clusters,” in 2019 Fourth International Conference on Fog and Mobile Edge Computing (FMEC).
IEEE, 2019, pp.
121–128, this paper introduces Greeniac, an RL-based approach for fog energy optimisation. [18] Y. Wang et al., “Scalable dynamic resource allocation via domain randomized reinforcement learning,” arXiv preprint, 2024, the RL-ADR approach for 5G UPF frequency scaling and power reduction; no DOI assigned at the time of writing. [19] M. S. Q. Z. Nine et al., “Greennfv: Energy-efficient network function virtualization with service level agreement constraints,” in Proceedings of the 2023 ACM/IEEE Supercomputing Conference (SC).
ACM, 2023, available via ACM
Digital Library. [20] J. L. Juan et al., “Power-aware performance increase via core/uncore reinforcement control for chip multiprocessors,” in Proceedings of the 17th ACM/IEEE International Symposium on Low-Power Electronics and Design (ISLPED).
ACM,
2012, pp. 99–104. [21] Y. Wang et al., “Online power management for multi-cores: a reinforcement learning based approach,” IEEE Transactions on Parallel and Distributed Systems, vol. 33, no. 4, pp. 751–764, 2022. [22] A. Gocht et al., “Q-learning inspired self-tuning for energy efficiency in hpc,” in 2019 IEEE 17th International Conference on High Performance Computing, Data and Analytics (HiPC) or HPCS.
IEEE, 2019, pp. 344–347.
[23] Y. Zhu et al., “Hierarchical reinforcement learning for nfv server power management,” Patent, 2024, patent application describing a hierarchical RL controller for core and uncore frequency; no peer-reviewed publication. [24] D. Biswas et al., “Machine learning for run-time energy optimisation in many-core systems,” in 2017 Design, Automation & Test in Europe Conference & Exhibition (DATE).
IEEE, 2017, pp. 1588–1592.
[25] Y. Li et al., “Cpu frequency scheduling using reinforcement learning for periodic real-time applications,” Journal of Systems Architecture, vol. 142, p. 102923, 2023.