VERA: Reinforcement Learning for Dynamic Memory Scaling of HPC Workloads in Kubernetes

arXiv:2609.19936v1 [cs.DC] 17 Sep 2026

VERA: Reinforcement Learning for Dynamic Memory Scaling of HPC Workloads in Kubernetes Ade Pramono

Jie Ren

Ivy Peng

KTH Royal Institute of Technology Sweden [email protected]

William & Mary USA [email protected]

KTH Royal Institute of Technology Sweden [email protected]

Abstract—Memory over-provisioning results in resource underutilization when HPC workloads run on Kubernetes. The default Vertical Pod Autoscaler (VPA) cannot anticipate phasedriven memory spikes for first-run HPC jobs. In this work, we present a reinforcement learning (RL) recommender called VERA that formulates vertical memory scaling as a Markov Decision Process and trains an agent on 3,353 real Prometheus traces. Evaluated on a live Google Kubernetes Engine (GKE) cluster using LAMMPS, graph analytics, in-memory analytics, and MLPerf 3D-UNet, the RL agent reclaims 31.6 % of the available memory headroom and incurs at most one OOM event while VPA reclaims −7.9 % over the same runs, raising memory provisioning, and its recommendation would have been insufficient to avoid OOM in 30 runs. The results demonstrate that an observation-driven RL recommender could outperform retrospective heuristics for dynamic memory scaling.

I. I NTRODUCTION Memory utilization is a critical factor in the performance and cost-efficiency of HPC workloads. Under-provisioning memory leads to OOM events that terminate jobs prematurely, while over-provisioning wastes resources and increases cost, and motivate for emerging memory systems, such as hybrid memory and disaggregated memory systems in recent years [1]–[4]. Kubernetes is increasingly the common control plane along the edge-cloud-HPC continuum, however, memory management primitives designed for cloud-native microservices translate poorly to the characteristics of HPC workloads, which are often tightly coupled and exhibit diverse memory usage patterns [5], [6]. The Kubernetes Vertical Pod Autoscaler (VPA) is designed to adjust container memory limits based on historical usage patterns, but its default policy is ill-suited for the dynamic and diverse memory profiles of HPC workloads. VPA’s recommender observes a rolling histogram of memory consumption and recommends the 90th percentile (P90) plus a 15 % static safety margin [7], [8]. While sufficient for stateless web microservices, this design exposes two critical limitations on HPC workloads. The first limitation is a workload-agnostic policy. A fixed percentile threshold is appropriate for stationary workloads but fails across diverse HPC memory archetypes: workloads with monotonically growing memory usage require a continuously rising limit; workloads with a sawtooth pattern require a limit set to the peak of each cycle. No single static policy handles all archetypes simultaneously. The second limitation

is the inability to adapt to quick phase transitions. HPC jobs frequently exhibit initialization bursts (e.g., graph building, data pre-processing) that complete in seconds and are therefore underrepresented or absent in multi-day histograms [9], [10]. VPA’s recommender targets historical averages rather than worst-case future states, making it structurally incapable of anticipating phase-driven memory spikes. The consequences are severe in both directions. Under-provisioning triggers OOM terminating all processes and discarding the entire job run. Over-provisioning consumes cloud memory budget that could otherwise serve additional jobs. In this work, we formalize memory scaling as a Markov Decision Process with a continuous, proportional action space that rescales the memory limit by a learned factor. We extract a 14-dimensional observation vector from Prometheus metrics. We also design a six-component reward structure that penalizes OOM events while rewarding waste reduction and aligning the scaling direction with observed memory trends. Moreover, the reward applies an init-state guard during the initialization phase, when the metrics are not yet informative. We use PPO to train the RL agent called VERA for VErtical Reinforcement Autoscaler. We evaluate VERA on a live Google Kubernetes Engine (GKE) cluster. We collect 4,790 execution traces from HPC workloads, including LAMMPS, Graph Analytics kernels (triangle counting, PageRank, and connected components), Spark-based In-Memory Analytics, and the machine learning application MLPerf 3D-UNet, running in a real cloud environment. The traces cover a range of memory usage patterns from representative workloads in scientific, graph, and machine learning domains. We compare the RL agent with the default VPA recommender on three metrics: OOM avoidance, memory waste reduction, and utilization. Across 100 runs per agent version, the RL agent reclaims 31.6% of the available headroom against VPA’s −7.9%, which raises the provisioned limit above its original value on average. The VPA recommendation would have been insufficient to avoid OOM in 30 runs, while only at most one OOM event occurs with the RL agent. These results demonstrate that an observation-driven RL recommender can outperform retrospective heuristics for dynamic memory scaling of HPC workloads. In summary, our contributions are as follows: • We formalize memory scaling as an MDP and design an

RL agent called VERA that learns to scale memory limits autonomously. • We design a six-component reward structure that penalizes OOM events while rewarding waste reduction aligned with the observed memory trends. • We collect and analyze 4,790 execution traces from diverse HPC workloads, covering a range of memory usage patterns. • We evaluate the RL agent against the default VPA recommender on a live GKE cluster, demonstrating significant improvements in memory waste reduction 31.6% against VPA’s −7.9%,. II. BACKGROUND AND M OTIVATION A. Memory Characteristics of HPC Workloads The memory consumption of HPC workloads is often phase-structured. Autoscaling tools designed for stateless microservices in the cloud face three difficulties when applied to HPC batch jobs. First, HPC workloads are tightly coupled through MPI communication and synchronization, whereas cloud-native workloads are loosely coupled stateless replicas. Second, HPC jobs rely on application-level checkpoint/restart for fault tolerance and otherwise fail as a unit and discard all progress, whereas cloud workloads tolerate stateless restart and degrade gracefully. Third, an OOM event terminates an entire HPC job and loses all ranks, whereas a cloud-native deployment loses a single replica while the others continue serving. B. Kubernetes Vertical Pod Autoscaler Every container in Kubernetes declares two memory fields: a request and a limit. The request is the minimum guaranteed allocation, used by the scheduler to decide which node can host the pod. The limit is the maximum allowed allocation, enforced at runtime by the Linux cgroup subsystem. When a container’s total memory consumption exceeds its limit, the Linux OOM killer terminates the container. The Kubernetes VPA attempts to automate the selection of request and limit values. It comprises three components. The Recommender monitors historical pod resource usage via Prometheus, builds a histogram over an eight-day observation window, and targets the 90th percentile of observed memory with an additional 15% safety margin. The Updater periodically checks whether running pods deviate significantly from the current recommendation and, if so, evicts them so that updated values can be applied. The Admission Controller intercepts pod creation requests and injects the latest recommended values into the pod spec before scheduling. The current VPA has two fundamental limitations for HPC workloads. First, its recommendations are derived from historical usage, but single-run HPC jobs have no prior history. VPA can therefore produce a useful recommendation only after a job has already run, making it retrospective rather than adaptive to the current execution. Second, applying a new recommendation requires evicting and restarting the pod; for HPC jobs that run for hours without checkpointing, this

Fig. 1. The overall architecture of integrating an RL recommender for vertical memory scaling in Kubernetes’s VPA.

discards all completed computation. Kubernetes Enhancement Proposal 1287 (KEP-1287) [11] introduces the ability to update a container’s resource limits at runtime without restarting the pod. C. Reinforcement Learning for Autoscaling Traditional autoscaling mechanisms, including Kubernetes’ default VPA and HPA, rely on threshold-based heuristics that react only after resource utilization crosses a predefined limit. These approaches require application-specific expertise and manual threshold tuning, and they adapt poorly to dynamic workload behavior. Resource allocation is, however, fundamentally a sequential decision-making problem in which each action influences future system states, which makes it a natural fit for reinforcement learning (RL). Consequently, there is growing interest in RL agents that learn scaling policies directly from interaction with the environment, optimizing cumulative long-term objectives that are often needed to combine conflicting goals, such as sustained performance and cost efficiency. Proximal Policy Optimization (PPO) is a policy gradient method that optimizes a surrogate objective with a clipped probability ratio to ensure stable updates [12]. PPO has emerged as the preferred algorithm for autoscaling problems [13]–[15]. The combination of stability, sample efficiency, and implementation simplicity makes PPO well suited to the noisy state spaces in Kubernetes environments. III. D ESIGN The overall architecture for integrating an RL agent into the Kubernetes autoscaler is illustrated in Fig. 1. The Deployment Controller requests pod creation from the Kubernetes API Server, which holds all cluster state including the VerticalPodAutoscaler CRD. The Kubelet receives the pod assignment and deploys the workload containers, which write their memory consumption directly to cgroups. cAdvisor reads from the cgroups and collects container metrics. The default VPA Recommender queries these metrics through the Metrics Server to produce its memory recommendation, whereas the RL Agent queries Prometheus directly. Because Prometheus

scrapes cAdvisor at a finer granularity than the Metrics Server exposes, the agent obtains a richer and more timely observation signal. To train and deploy the RL agents, we adopt a simto-real pipeline [14]. Finally, we propose a composite metric combining waste reduction and OOM avoidance to guide the model selection from a grid of trained RL agents. A. Reinforcement Learning Environment We model the vertical memory scaling task in Kubernetes pods as a discrete-time Markov Decision Process (MDP). At each decision step t, an RL agent observes the current state st ∈ S, selects a scaling action at ∈ A, receives a scalar reward rt ∈ R, and transitions to the next state st+1 . The episode advances at 2-second intervals, matching Prometheus scrape resolution. A coarser frequency (e.g., 10 s) would miss sub-10-second memory spikes while a finer frequency cannot be supported by cAdvisor, whose refresh cycle is 10–15 s. An episode begins at the first Prometheus reading and ends when the pod either completes naturally or is killed by an OOM event. This maps well to HPC jobs and ensures the agent is rewarded across the full memory lifecycle. 1) Action Space: A discrete action set (e.g., {−10%; +10%}) cannot express fine-grained responses. We model action at ∈ [−1, +1] as a continuous adjustment factor applied to the current memory   limit Lt : L̂t+1 = clamp L̂t + at · α · L̂t , Lmin , 2 · L0 , where α = 0.10 is the maximum adjustment fraction. Lmin = 64 MB provides a hard floor below which OOM is certain for most realistic workloads. The ceiling 2 · L0 prevents the agent from expanding too much over the operator-set limit, which would waste cluster capacity without benefit. Proportionality (scaling by L̂t , not L0 ) provides natural deceleration. When the limit approaches the floor, each step adjusts by a smaller absolute amount, preventing oscillation around the floor. For instance, a pod approaching OOM at 5 MB/s needs a small positive adjustment, not a maximum expansion. Continuous at ∈ [−1, +1] lets the agent express both direction and magnitude of adjustment simultaneously. A value of α = 0.10 means the agent can move the limit by up to 10 % of its current value per 2-second step. Three consecutive maximum-trim steps bring the limit to 0.853 ≈ 61.4 % of its current value, reclaiming meaningful memory within six seconds of a usage drop. Conversely, three consecutive maximum-expand steps can absorb a sudden 52 % memory spike (1.153 ≈ 1.52), covering the spike magnitudes observed in many workloads. In contrast, VPA applies its recommendation in a single step on pod restart and cannot make in-episode adjustments. 2) State and Observation Space: In the memory scaling problem, a single measurement of current memory usage does not reveal whether memory need is growing, stable, or declining, which is critical for proactively adjusting the memory limit. To address this partial observability problem, we leverage carefully selected feature engineering in the observation vector. We select features into the observation vector based on three criteria. First, Memory relevance, a feature

must directly reflect the pod’s memory state or pressure. Features that only loosely correlate with memory behaviour are excluded. Second, Markov sufficiency, a feature must contribute information about the current memory phase that cannot be inferred from the remaining features alone. Third, deployment symmetry, a feature must be observable at inference time from the same Prometheus/cAdvisor pipeline used during training. Based on the criteria, we select 14 features. All features are normalized by the pod’s original Kubernetes memory limit L0 (the limit set by the cluster operator), ensuring the policy generalizes across pods of different sizes. We categorize these features into five signal types: OOM-critical signal (f0 ), OOM precursor signal(f6 ), trend signals(f7 –f9 , f12 –f13 ), limit signals(f1 –f5 ), and CPU context signals(f1 0–f1 1). • f0 : memory_working_set measures working-set bytes (RSS + active file-backed pages). This is the quantity enforced by the Linux cgroup OOM killer, and thus, the primary safety signal. • f1 : memory_rss measures non-reclaimable pressure (resident set size) and distinguishes from reclaimable size. • f2 : memory_usage is the total memory footprint including page cache. • f3 : memory_cache_ratio is the fraction of reclaimable page cache over the memory footprint. • f4 : memory_limit_kube is a cgroup-enforced limit from kube-state-metrics. It represents the hard upper limit which the kernel will OOM-kill the container. • f5 : memory_request_kube is the minimum memory guaranteed to the pod by the Kubernetes scheduler. • f6 :memory_failures_rate measures the rate of denied kernel memory allocation requests. This fires before the container working set (Wt ) breaches Lt , giving the agent an early-warning signal to act before OOM is triggered. • f7 : agent_limit is the agent’s last recommended limit, providing self-awareness of prior action. • f8 : usage_trend is the rolling slope of working-set = (Wt − Wt−1 )/Lorig . It is initialized to −1.0. This feature addresses Markov insufficiency by encoding the direction of memory demand. • f9 : utilization_ratio = Wt /Lt is the primary efficiency signal. • f10 : cpu_limit_kube provides CPU/memory coupling context. • f11 : cpu_usage_rate is the current CPU utilization. It distinguishes CPU-bound from memory-bound phase and acts as a phase transition signal. • f12 : idle_steps_norm measures the consecutive steps where Wt < 0.15Lorig . It enables trimming during postcomputation idle phases. • f13 : episode_peak_norm is the maximum Wt observed so far in the episode, used for worst-case planning. In this work, we engineered features over raw time series, instead of feeding into a recurrent policy such as an LSTM [16]. LSTM training is substantially less stable than MLP training at the 1–5M timestep budgets used in training. We also

TABLE I P HASE -AWARE R EWARD S TATE -M ACHINE T RANSITIONS Transition

Fig. 2. Reward function viewed as a state machine. Green transitions carry positive reward; red transitions carry penalties and represent unsafe or wasteful actions.

explored frame stacking [17] as a lightweight alternative for adding temporal context, but its results did not consistently outperform the Markov baseline. A few engineered features are critical for solving ambiguity. For instance, (idle_steps_norm) and (episode_peak_norm) were added to solve the ambiguity where memory usage near zero cannot distinguish between “workload not yet started” and “agent successfully trimmed the limit”. Without these features, the reward function cannot differentiate the two states, causing the efficiency signal to misfire during the init phase. Another example is (cpu_usage_rate), which provides the primary init-phase signal where a CPU utilization below a threshold τcpu = 0.20 of the container’s CPU limit indicates the main process is not yet executing. The threshold τcpu = 0.20 was selected empirically and is a configurable hyperparameter. A lower value risks missing the init phase for slow-starting workloads, while a higher value risks exiting the init phase prematurely for workloads with a CPU-active setup stage. 3) Reward: We design the reward function to simultaneously encode two objectives that are in direct tension [18]: Safety, where OOM kills must be heavily penalized as an OOM event crashes the HPC job and wastes all compute time already invested; Efficiency, where the agent must be incentivized to trim the limit toward actual usage to reduce over-provisioned memory waste that could be used to serve other pods. Fig. 2 illustrates in a finite-state machine that sees the reward function as a whole through the lens of the operational states a pod can occupy. The agent must navigate four states: the initialization state, where memory usage is near zero and the computation has not yet started; the active efficient state, where utilization is close to the target (85 %) and the agent earns its highest reward; the active over-provisioned state, where the limit is set higher than demand and memory is being wasted; and the active dangerous state, where utilization is approaching 100 % and an OOM kill is imminent. The target zone corresponds to a memory limit set 10–15% above the current working-set: tight enough to recover wasted memory but with sufficient memory headroom to absorb short-term

Reward

Init: Expand with strong rising trend Init: Expand (trend 0.1–0.2, large) Init: Expand (trend 0.1–0.2, small) Init: Expand (no trend detected) Init: Trim (any) |Y | Init: Trim + allocation failures

[+1.0, +1.3] +0.30 +0.10 −0.15 [−0.15, −0.25, −0.50, −0.75] −0.35

Efficient: Hold / trim at peak Efficient → Over-provisioned Over-provisioned: Trim (falling usage) Efficient: Over-trim Efficient → Dangerous: trim+fail Dangerous / Efficient: Trim too hard

+1.0 < 1.0 [0, 1] + 0.05 [0, −0.5] −0.2 [−4.0, −2.0]

OOM event (Wt > Lt )

≪ 0 (terminal)

spikes before the OOM killer fires. The goal of the reward function is to guide the agent toward the active efficient state and keep it there. From the initialization state, the agent is discouraged from trimming prematurely and rewarded for expanding ahead of a detected memory spike, transitioning cleanly into the efficient state when the workload starts. From the over-provisioned state, small trim bonuses nudge the agent back toward the efficient target. From the dangerous state, the failure pressure penalty and the OOM penalty together create a strong barrier against further trimming. Crossing into the OOM state terminates the episode, making it the absorbing failure state the agent must avoid at all costs. This structure motivates us to design the reward function into six components, each of which governs one or more transitions in the diagram. Table I summarizes the reward values assigned to each transition. Strong positive rewards reinforce maintaining the efficient zone; large negative penalties escalate as the policy approaches or triggers OOM, calibrated to the asymmetric cost of OOM events (complete job loss) versus over-provisioning (wasted budget). The six components are evaluated in priority order: higherpriority components short-circuit lower ones. The total reward is clipped to [−9.0, +1.5] to avoid unstable policy reward. The first priority component is OOM Penalty. When the agent’s limit L̂t < Wt (working set exceeds the limit): ! Wt − L̂t Room = −2.0 − 2.0 · min , 1.0 ∈ [−4.0, −2.0]. Wt (1) This component returns immediately, short-circuiting all other components so that no efficiency or trend signal can soften an OOM step. We chose a graduated penalty rather than a fixed value because OOM severity varies across workloads and a constant signal would produce uninformative policy gradients. A slight OOM yields −2.0, while a catastrophic OOM where the deficit reaches 100 % of usage yields −4.0. The environment terminates the episode when an OOM occurs during the active phase (episode_peak_norm > 0). Without termination, the agent could exploit the trajectory “trim aggressively → trigger OOM → recover slowly → earn

sustained efficiency reward”, which is net profitable under the Bellman equation despite the OOM penalty [18]. Termination closes this exploit by setting all future rewards to zero after an OOM kill, making aggressive trimming a dead end. The second highest priority component is Cold-Start Guard. During the init phase (idle_steps_norm > 0 ∧ cpu_usage_rate < 0.2): Rinit = 0.0

(neutral; no incentive to trim or expand). (2)

Without this guard, the efficiency reward in Component 3 would observe near-zero utilization in the initialization state and incorrectly penalize the agent for over-provisioning, incentivizing aggressive trimming before the job has even started its active computation phase. When memory demand then spikes at the transition to the active state, the limit would already be too low, causing an immediate OOM. The neutral signal means holding the limit steady is the correct action during init. The third component is Efficiency Reward that applies to active phases.  ∗ 1.0 + ρt − ρ if ρt ≤ ρ∗ ρ∗ ∈ [0, 1], Reff =  max(0, 1.0 − 10(ρt − ρ∗ )) if ρt > ρ∗ (3) 1 ∗ t , and δ = 0.15 is the target safety , ρ = where ρt = W 1+δ L̂t margin, giving ρ∗ = 1/1.15 ≈ 0.870. The asymmetric slope reflects the asymmetric cost structure we observe in practice. On the safe side of the target, the agent has 0.87 utilization units of headroom before reaching complete over-provisioning. On the dangerous side, it has only 0.13 units before hitting the OOM boundary. We therefore apply a 10× steeper slope on the right side of ρ∗ , so that a small drift toward OOM costs the agent far more reward than an equivalent drift toward overprovisioning, naturally biasing the policy toward safety. We set the target safety margin to δ = 0.15, which aligns with the default recommendation-margin-fraction of 15 % used by the Kubernetes VPA [7] and corroborated by industry practice reported in SelfTune [8]. This margin is large enough to absorb cAdvisor polling jitter across a 2second scrape interval and small Spark allocation fluctuations, while remaining tight enough to produce a meaningful waste reduction over VPA’s conservative P90 baseline. The fourth component is Trend Alignment Bonus. We designed this component to reward the agent for moving in the same direction as memory demand. The two directions of memory demand (increase/decrease) carry very different urgency: failing to expand ahead of a rising spike leads directly to OOM, while failing to trim during a usage decline leads to over-provisioning. Both outcomes are undesirable, but OOM is immediately catastrophic whereas over-provisioning is merely wasteful. We therefore apply an asymmetric 6:1 ratio between the expand and trim bonuses, with a maximum of +0.30 for expanding into a rising spike and +0.05 for trimming during

a usage decline, so that the agent learns to prioritize safety over efficiency when the two objectives conflict.  +0.30 · min(dt , 1) dt > 0.2 ∧ at > 0    +0.10 · min(d , 1) dt > 0.1 ∧ at > 0 t Rtrend = (4)  +0.05 · min(−d , 1) d < −0.1 ∧ a < 0 t t t    0 otherwise, where dt = usage_trend = (Wt − Wt−1 )/L0 . The fifth component is Failure Pressure Penalty. If memory_failures_norm > 0.5 and at < 0 (agent is trimming under active OOM pressure), it receives Rfail = −0.20. During our experiments, we observed that the OOM penalty in Component 1 alone cannot always prevent the agent from trimming a pod that is already under memory pressure. A page-major-fault rate above a configurable threshold τfail = 0.5 indicates that the pod is actively struggling to meet its memory demand, and trimming further in this state risks triggering an OOM kill. We therefore introduce a second-level safety interlock that applies a penalty if the agent trims when memory_failures_norm > τfail . This component acts as a last line of defense that activates only when Component 1 has not yet fired, and τfail is a configurable hyperparameter. The sixth component is Init-Trim Penalty. During the init phase, if at < 0 (i.e., agent trims), Ritrim = Y < 0. This penalty discourages the agent from reclaiming memory when it is in init state, where the magnitude Y is tuned empirically via a sensitivity study in this work. If the initphase trimming is too costly, the agent enters the active state with the limit still close to L0 , at which point the weak trim signal of at most +0.05 from Component 4 is insufficient to bring the limit down to an efficient level before the episode ends. We find Y = −0.15 to strike the best balance. B. Reinforcement Learning Agent We adopt an actor-critic network architecture, where the actor (the policy network) proposes memory adjustment actions, and the critic (the value network Vπ ) estimates expected returns under the actor’s policy. 1) Asymmetric Actor-Critic Architecture: We keep actor and critic networks separate to ensure that value function updates do not interfere with policy updates through a shared backbone, a known source of instability in actor-critic training [19]. The policy network uses a small ((32, 32)) twolayer MLP to encourage generalization across diverse workload traces. Since the action space is a single scalar that requires minimal parametric capacity, a smaller network is less prone to overfitting across the diverse workload traces in the training set. The critic, on the other hand, needs to memorize a complex, phase-dependent value landscape to produce accurate long-horizon GAE estimates [20]. Therefore, the value network uses a larger ((256, 256)) two-layer MLP to accurately estimate long-horizon returns. We choose an MLP policy because the idle_steps_norm and usage_trend features already encode the most critical temporal information in the state

TABLE II PPO HYPERPARAMETER CONFIGURATION . Parameter

Value

Rationale

Learning rate η Rollout steps N Mini-batch size Gradient epochs K Discount γ GAE λ Clip range ϵ Entropy coefficient Value coefficient Max gradient norm Training timesteps

3 × 10−4 2048 1024 10 0.99 0.95 0.2 0.01 0.5 0.5 1 − 3 × 106

Adam default; stable for continuous control Captures full episode temporal dynamics 8 mini-batches per update Improves sample efficiency; clip prevents collapse Long episodes require high discount Standard bias-variance trade-off Standard PPO; prevents destructive updates Small exploration bonus Standard value loss weight Gradient clipping for stability Per workload; tuned by convergence

vector, removing the need for a recurrent architecture to infer temporal context from raw sequences. We use Tanh activation because they produce outputs bounded in [−1, +1] that aligns naturally with the action space range and avoids the saturation instabilities that ReLU activation can introduce in continuous control settings. The observation captures the current memory state well but tells the agent little about how that state has been evolving over recent steps. We address this by frame stacking [17] that concatenates the last N observations into a flat vector of dimension 14 × (N + 1), giving the policy a short memory of recent states. We train the RL agent using Proximal Policy Optimization (PPO) [12]. The clipped surrogate objective (ε = 0.2) bounds each policy update. Since the OOM reward signal fires infrequently during training, a single catastrophic policy update could destroy accumulated policy quality, and thus training stability from PPO is critical. Also, PPO with a Gaussian policy handles at ∈ [−1, +1] natively without discretization artifacts. Finally, PPO applies multiple gradient epochs to the same batch under its clipped surrogate objective, which improves learning efficiency without requiring additional data collection. This sample efficiency is important when real pod executions take minutes to hours. Table II lists the PPO hyperparameter configuration. The long rollout buffer (N = 2048 steps per environment) captures full-episode temporal dynamics. The discount factor γ = 0.99 is chosen because HPC jobs run for thousands of steps, and a low discount value would excessively discount late-episode efficiency gains that represent the majority of a job’s wall time. The agent is trained in a sim-to-real pipeline and deployed to the live GKE cluster for evaluation. We collect the dataset by sweeping workload-specific problem-size parameters across memory provisioning tiers.This stage produces 1,214,600 rows of Prometheus time-series samples and 4,790 labeled execution traces. The dataset is shuffled across workload types and split 70/30 into training and evaluation. 2) Training Hyperparameter Grid and Model Selection: A total of 96 PPO agents are trained by varying frame-stack depth and training budget. All configurations use the same PPO hyperparameters: γ = 0.99, λGAE = 0.95, clip ε = 0.2, K = 10 gradient epochs per rollout, η = 3 × 10−4 (Adam). The critic network is fixed at (256, 256) hidden units in all configurations. We observed that relying on a single scalar metric to select

the production agent model is insufficient. A fully trained model with the highest utilization ratio may operate dangerously close to the OOM boundary, while a model with the lowest OOM rate may over-provision excessively. Therefore, we evaluate each trained model using three metrics jointly, each capturing a dimension of the safety-efficiency trade-off: m1 : mean utilization ratio = Wt /L̂t ; m2 : Total OOM events per experiment; m3 : mean waste ratio = (L̂t − Wt )/L0 . To produce a single comparable score, we apply two complementary normalization schemes. Min-Max normalization rescales each metric relative to the best and worst values observed across all candidates; it assumes a uniform distribution of scores and is sensitive to outlier models. Z-score normalization instead centers each metric on its mean and scales by its standard deviation, which is more robust to outliers but sensitive to the population standard deviation. The final production model is the model on the Pareto front of (m2 , m3 ) with the shortest distance to the utopia point under both schemes, so that the selection is robust to either distributional assumption. IV. E XPERIMENTAL S ETUP We evaluate the RL agent on a live GKE cluster. Data collection is performed on GKE Cluster v1.35.1-gke.1396002 in region europe-north1-a. Training traces are collected on a single n2-highmem-8 node (8 vCPUs, 64 GB RAM), and evaluation traces on an n2-standard-16 node (16 vCPUs, 64 GB RAM). At each 5-second interval, the RL agent queries Prometheus, computes action at to obtain a new memory limit. Meanwhile, the VPA baseline runs with updateMode off, and its emitted recommendations are logged for comparison. Both recommenders run in shadow mode: VERA’s computed limit L̂t is logged rather than applied, so every pod executes under its original limit L0 for the full run, a run is counted as an OOM for a recommender if Wt exceeded that recommender’s Lrec at any point. The OOM counts are therefore first-crossing counts on a fixed trace comparable between the two recommenders, since both are evaluated against the identical execution. Enforcing the limit requires in-place limit updates via KEP-1287, and in particular whether the kubelet honours limit decreases on a live container, which is left to future work. We evaluate four agent versions trained with different inittrim penalties, Y ∈ {−0.15, −0.25, −0.50, −0.75}. Each version is evaluated over 100 workload runs spanning all four workloads: 50 runs for Graph Analytics (including BFS, PR, and CC applications), 15 for In-Memory Analytics, 18 for LAMMPS, and 17 for MLPerf 3D-UNet. These counts reflect the number of algorithm variants, input configurations, and memory tiers available for each workload. We use three metrics to evaluate the performance of memory scaling under RL agents and VPA: Memory waste reduction L−r̄ × 100 %, Utilization ratio Util = LWrect , and ∆W = L−ū the number of OOM events when Wt > Lrec where L is the original memory limit, r̄ is the recommended memory

TABLE III T HREE EVALUATION METRICS MEASURED IN ALL WORKLOADS . − MEANS THE RECOMMENDATION IS UNUSABLE . Workload

RL Agent

RL Model

VPA Baseline (mean)

∆W

util

#OOM

∆W

util

#OOM

CS Graph (n=50)

1 2 3 4

30.0% 28.6% 17.5% -0.3%

0.136 0.142 0.129 0.113

0 0 0 0

2.2%

0.017

7

CS In-Mem (n=15)

1 2 3 4

62.7% 67.2% 61.3% 54.1%

0.462 0.514 0.474 0.467

0 0 0 0

-

-

13

LAMMPS (n=18)

1 2 3 4

25.5% 17.9% 3.2% 4.6%

0.404 0.435 0.419 0.419

1 0 0 0

−51.5%

0.298

4

MLPerf (n=17)

1 2 3 4

15.4% 16.2% 4.8% 4.5%

0.225 0.245 0.232 0.277

0 1 0 0

1.5%

0.014

6

Overall (n=100)

1 2 3 4

31.6% 30.4% 19.3% 9.6%

0.248 0.268 0.250 0.249

1 1 0 0

−7.9%

0.065

30

limit, ū is the mean memory usage, Wt is the mean working set across the episode, and Lrec is the recommended limit. Util is particularly relevant because it reflects how tightly the recommended limit tracks the actual demand. V. GKE P RODUCTION E VALUATION In this section, we evaluate our RL agents against the default Kubernetes VPA recommender on real workloads, and analyze the agent’s behavior on each workload. A. Overall Performance Across all three evaluation metrics, the RL agent consistently outperforms VPA. Table III reports the mean value of each metric per agent version and workload. RL Model 1 (Y = −0.15) is the best-performing configuration. On OOM count, the RL agents incur at most one event out of 100 runs, whereas the VPA recommendation would have been insufficient in 30 runs. On waste reduction, VPA records −7.9 % overall, meaning that it raises the provisioned memory above the original limit on average rather than reclaiming headroom. The RL agent, in contrast, achieves an overall waste reduction of 31.6 %. As the init-trim penalty |Y | increases, the waste reduction achieved by the RL agent falls from 31.6 % to 9.6 % while the OOM count falls to zero, exposing a consistent tradeoff between init-phase conservatism and efficiency. B. Graph Analytic Applications A Graph Analytics job starts with a fast initialization ramp, after which memory grows steadily as algorithms traverse the graph and accumulate intermediate results. Fig. 3 presents a real trace. VPA cannot anticipate this growth because it derives its recommendation from historical usage and sets the memory limit to previously observed peaks. As traversal depth and working-set size increase throughout the execution, the actual demand often exceeds this historical ceiling. We evaluate the graph workload on real-world networks from the Stanford Large Network Dataset

Fig. 3. A real trace of a Graph Analytics Connected Components application (16 GiB tier, Model 1).

Fig. 4. A real trace of In-Memory Analytics (tier-3-big-5, Model 1).

Collection (SNAP) [21], using the temporalnetwork (sx-stackoverflow) and web (web-Google) graphs. Each algorithm (CC, PR, TC) is deployed across three memory tiers, 8, 16, and 32 GiB, with the pod’s memory request fixed to its limit. Across the 50 runs, the RL agent sets a tighter memory limit while maintaining a better safety record than VPA: the RL agents record no OOM events, whereas the VPA recommendation would have been insufficient in seven runs on average. Waste reduction under the RL agent is 30.0 % against VPA’s 2.2 %. The most severe VPA failures occur on the temporalnetwork dataset family. At the 8 GiB tier, CC, PR, and TC each record a VPA OOM; at the 16 GiB and 32 GiB tiers, PR and TC do so as well. The RL agent records zero OOM events across all of these executions. It avoids OOM by adjusting the limit upward continuously in response to the observed growth signal, rather than relying on a fixed historical estimate. C. In-Memory Analytics The In-Memory Analytics workload exhibits the sharpest contrast between RL and VPA. This workload has memory usage with no regular period or predictable ceiling. Across 15 runs, VPA records an average of 13 OOM events. The RL agent records zero OOM events in all four versions. Moreover, VPA’s cannot reduce any memory waste because it produced no usable recommendation for any in-memory configuration when a workload’s burst magnitude is uncapped and irregular. The RL agent achieves waste reductions of 62.7 % (Model 1), 67.2 % (Model 2), 61.3 % (Model 3), and 54.1 % (Model 4), with an average memory utilization in 0.462–0.514. We show a representative trace in Fig. 4. The working set oscillates throughout the whole run. The RL target begins conservatively near 49 GiB and contracts steadily as demand becomes observable. It settles between 17-24 GiB for the remainder of the run and consistently tracks tightly above each

Fig. 5. A real trace of LAMMPS MD simulation (30m-large-lj, Model 1).

Fig. 6. A real trace of MLPerf 3D-UNet workload (tier2, 24 GiB, Model 1).

burst peak without causing any OOM event. In contrast, VPA issues its first recommendation at only 2.8 GiB, a value already 3.8× below the observed working-set of 10.7 GiB. By setting the limit around 20 GiB against an original memory limit of 57 GiB, the RL agent recovers 37 GiB of memory resources, achieving a waste reduction of 65 %.

sor caching. One representative trace is shown in Fig. 6. Every burst peak is an OOM risk if the limit falls below it. Across the 17 MLPerf evaluation runs, the VPA recommendation would have been insufficient in six runs, whereas the RL agent records zero OOM events in Models 1, 3, and 4, and one in Model 2. The tier1, tier2, and tier3 labels denote 3D-UNet configurations with activation sizes of 1024, 4096, and 7680, respectively. Within tier2, VPA records OOM at the 12, 24, and 49 GiB memory limits while the RL agent records none; the same pattern holds within tier3 at 32 and 49 GiB. The RL agents achieve a modest waste reduction on this workload: 15.4 % (Model 1), 16.2 % (Model 2), 4.8 % (Model 3), and 4.5 % (Model 4), against VPA’s 1.5 %. The sawtooth peaks set a hard demand ceiling that the agent cannot predict in advance, so avoiding OOM requires holding a conservative buffer above every peak at all times, which bounds the headroom that can be reclaimed. As shown in Fig. 6, the RL agent maintains a buffer above each sawtooth peak, while VPA’s recommendation falls below peak demand and would have caused an OOM.

D. LAMMPS LAMMPS runs molecular dynamics simulations whose memory consumption scales with atom count, producing a monotonically rising demand trajectory. The most notable finding on this workload is VPA’s negative waste reduction of −51.5 %. A negative ∆W indicates that VPA’s recommended limit exceeds the originally provisioned limit, aggravating over-provisioning. For a monotonically increasing workload, the running peak always sits at the top of the memory histogram, so VPA continually recommends above the previous provisioning level. The RL agent achieves positive waste reduction in all four models: 25.5 %(Model 1), 17.9 % (Model 2), 3.2 % (Model 3), and 4.6 % (Model 4). Utilization under the RL agent rises to 0.404–0.435, against 0.298 for VPA. Fig. 5 shows a representative LAMMPS episode trace in which the RL agent closely tracks the linear growth curve while VPA’s recommendation remains constant throughout the run. On large Lennard-Jones simulations, Model 1 achieves the highest waste reduction of 81.1 % for 30m-large-lj, 75.4 % for 40m-large-lj, and 69.4 % for 30m-medium-lj. In these configurations the agent tracks the predictable linear growth and trims the limit closely above demand. The real and metal unit-system variants at smaller atom counts yield lower waste reduction, consistent with their smaller memory footprint, which leaves less headroom to reclaim. On small simulations (40m-small-lj), one OOM event is recorded under Model 1 at a 16 GiB limit. This configuration runs a simulation with an unusually steep early-phase memory growth rate, and the init-trim penalty Y = −0.15 is insufficient to hold the limit open during the init window on this trajectory. Models 2, 3, and 4, with stronger penalties, record zero OOM on the same sub-configuration, which confirms that the inittrim penalty is a critical design element for HPC workloads such as LAMMPS. E. MLPerf 3D-UNet MLPerf 3D-UNet exhibits a repeating sawtooth memory pattern driven by mini-batch loading and framework-level ten-

VI. R ELATED W ORK Rossi et al. [22] present RL-based scaling controllers for containerized applications, using DQN to jointly manage horizontal and vertical scaling with an objective balancing response time, cost, and adaptation frequency. Their work targets CPU and latency for web services rather than memory waste and OOM prevention. Gym-hpa [13] implements PPO, A2C, and RPPO agents for horizontal autoscaling of microservices in Kubernetes, with the objective of reducing pod counts relative to the default HPA while maintaining acceptable service latency. Gwydion [14] extends gym-hpa with a sim-to-real transfer interface and integrated statistical forecasting, showing that simulation-trained agents reach competitive performance at approximately one-tenth of the training cost of online-trained agents. KIS-S [15] applies PPO to GPU-aware horizontal autoscaling for inference on Kubernetes, reducing P95 latency by up to 6.7× relative to CPU-only baselines and showing that PPObased scaling policies generalize well across inference workload configurations. Different from these works on optimizing horizontal scaling, this work targets vertical scaling. AWARE [23] explored RL in workload autoscaling in production cloud platforms. They proposed an extensible

framework for deploying RL agents in production systems and can adapt a learned auto-scaling policy much faster than the existing transfer-learning-based approach for new workloads and significantly reduce SLO violations. In addition to RL-based methods, threshold-based reactive scaling approaches are also used to scales up or down a Kubernetes cluster when usage exceeds or falls below a threshold. ARC-V [24] implements a three-state heuristic controller (growing, stable, dynamic) with in-place memory limit adjustment. They evaluated on nine HPC applications, and confirms that workload-aware in-place resizing can outperform the default VPA heuristics for HPC workloads. However, their thresholds need to be hand-tuned per workload class and do not adapt from experience, so each new workload type requires manual re-engineering. VII. C ONCLUSION In this work, we presented an RL-based recommendation framework called VERA for dynamic vertical memory autoscaling on Kubernetes. We formulated the task as an MDP, trained PPO agents offline on 3,353 real traces collected from GKE clusters, and evaluated them against the default VPA recommender on a live GKE cluster in LAMMPS, MLPerf 3D-UNet, In-Memory Analytics, and Graph Analytics workloads. The results show that the RL agent prevents OOM more reliably than VPA while recovering substantial memory waste. It incurs at most one OOM event, whereas the VPA recommendation would have been insufficient in 30 runs, and it reduces memory waste by 31.6 % while VPA raises the provisioned memory limit by 7.9 % on average. Extending coverage to more HPC workloads with diverse phase patterns is the natural next step toward an RL-driven Recommender that complements the default VPA. ACKNOWLEDGMENT This research is supported by the European Commission under the Horizon project OpenCUBE (101092984). R EFERENCES [1] I. B. Peng, S. Markidis, E. Laure, G. Kestor, and R. Gioiosa, “Exploring application performance on emerging hybrid-memory supercomputers,” in 2016 IEEE 18th International Conference on High Performance Computing and Communications; IEEE 14th International Conference on Smart City; IEEE 2nd International Conference on Data Science and Systems (HPCC/SmartCity/DSS). IEEE, 2016, pp. 473–480. [2] I. B. Peng, R. Gioiosa, G. Kestor, J. S. Vetter, P. Cicotti, E. Laure, and S. Markidis, “Characterizing the performance benefit of hybrid memory system for hpc applications,” Parallel Computing, vol. 76, pp. 57–69, 2018. [3] I. Peng, K. Wu, J. Ren, D. Li, and M. Gokhale, “Demystifying the performance of hpc scientific applications on nvm-based memory systems,” in 2020 IEEE International Parallel and Distributed Processing Symposium (IPDPS). IEEE, 2020, pp. 916–925. [4] J. Wahlgren, G. Schieffer, M. Gokhale, and I. Peng, “A quantitative approach for adopting disaggregated memory in hpc systems,” in Proceedings of the international conference for high performance computing, networking, storage and analysis, 2023, pp. 1–14. [5] A. M. Beltre, P. Saha, M. Govindaraju, A. Younge, and R. E. Grant, “Enabling hpc workloads on cloud infrastructure using kubernetes container orchestration mechanisms,” in 2019 IEEE/ACM International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC). IEEE, 2019, pp. 11–20.

[6] D. Medeiros, J. Wahlgren, G. Schieffer, and I. Peng, “Kub: enabling elastic hpc workloads on containerized environments,” in 2023 IEEE 35th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD). IEEE, 2023, pp. 219–229. [7] Kubernetes Community, “Vertical pod autoscaler,” https://github. com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler, 2023, accessed: 2025. [8] A. Karthikeyan, N. Natarajan, G. Somashekar, L. Zhao, R. Bhagwan, R. Fonseca, T. Racheva, and Y. Bansal, “{SelfTune}: Tuning cluster managers,” in 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), 2023, pp. 1097–1114. [9] I. Peng, I. Karlin, M. Gokhale, K. Shoga, M. Legendre, and T. Gamblin, “A holistic view of memory utilization on HPC systems: Current and future trends,” in Proceedings of the International Symposium on Memory Systems, 2021, pp. 1–11. [10] J. Li, G. Michelogiannakis, B. Cook, D. Cooray, and Y. Chen, “Analyzing resource utilization in an hpc system: A case study of nersc’s perlmutter,” in International Conference on High Performance Computing. Springer, 2023, pp. 297–316. [11] Kubernetes Community, “KEP-1287: In-place update of pod resources,” 2023, kubernetes Enhancement Proposal. [Online]. Available: https: //github.com/kubernetes/enhancements/issues/1287 [12] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017. [13] J. Santos, T. Wauters, B. Volckaert, and F. De Turck, “gym-hpa: Efficient auto-scaling via reinforcement learning for complex microservice-based applications in Kubernetes,” 2023. [14] J. Santos, E. Reppas, T. Wauters, B. Volckaert, and F. De Turck, “Gwydion: Efficient auto-scaling for complex containerized applications in kubernetes through reinforcement learning,” Journal of Network and Computer Applications, vol. 234, p. 104067, 2025. [15] G. Zhang, W. Guo, Z. Tan, Q. Guan, and H. Jiang, “Kis-s: A gpuaware kubernetes inference simulator with rl-based auto-scaling,” in 2025 IEEE International Performance, Computing, and Communications Conference (IPCCC). IEEE, 2025, pp. 1–8. [16] M. J. Hausknecht and P. Stone, “Deep recurrent q-learning for partially observable MDPs,” in AAAI fall symposia, vol. 45, 2015, p. 141. [17] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, “Human-level control through deep reinforcement learning,” vol. 518, 2015, pp. 529–533. [18] R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed. MIT Press, 2018. [Online]. Available: http://incompleteideas.net/book/the-book-2nd.html [19] M. Andrychowicz, A. Raichuk, P. Stańczyk, M. Orsini, S. Girgin, R. Marinier, L. Hussenot, M. Geist, O. Pietquin, M. Michalski et al., “What matters for on-policy deep actor-critic methods? a large-scale study,” in International conference on learning representations, 2021. [20] J. Schulman, P. Moritz, S. Levine, M. I. Jordan, and P. Abbeel, “High-dimensional continuous control using generalized advantage estimation,” arXiv preprint arXiv:1506.02438, 2015. [Online]. Available: https://arxiv.org/abs/1506.02438 [21] J. Leskovec and A. Krevl, “SNAP Datasets: Stanford large network dataset collection,” http://snap.stanford.edu/data, Jun. 2014. [22] F. Rossi, M. Nardelli, and V. Cardellini, “Horizontal and vertical scaling of container-based applications using reinforcement learning,” in IEEE 12th International Conference on Cloud Computing (CLOUD), 2019, pp. 350–357. [23] H. Qiu, W. Mao, C. Wang, H. Franke, A. Youssef, Z. T. Kalbarczyk, T. Başar, and R. K. Iyer, “{AWARE}: Automate workload autoscaling with reinforcement learning in production cloud systems,” in 2023 USENIX Annual Technical Conference (USENIX ATC 23), 2023, pp. 387–402. [24] D. Medeiros, J. J. Williams, J. Wahlgren, L. S. M. Leite, and I. Peng, “ARC-V: Vertical resource adaptivity for hpc workloads in containerized environments,” in European Conference on Parallel Processing. Springer, 2025, pp. 175–189.