nvPAX: Constrained Optimization for Dynamic Power Allocation in Hierarchical and Multi-Tenant Systems
arXiv:2605.01837v1 [cs.DC] 3 May 2026
Hadar Sivan NVIDIA [email protected]
Gil Shabat NVIDIA [email protected]
Yoel Shkolnisky Tel Aviv University & NVIDIA [email protected]
Abstract Power oversubscription is increasingly central to datacenter operation as power density grows, making it necessary to dynamically allocate limited power budgets across devices based on real-time demand. Existing approaches typically assume flat power domains, whereas in practice power distribution is hierarchical and allocation decisions must additionally respect tenant-level contractual constraints. We present nvPAX, a constrained-optimization policy that computes feasible power allocations at every control step via a three-phase hybrid QP/LP procedure. Phase I allocates power with minimum deviation from each device’s power request, while respecting job priorities. Phase II fairly distributes excess power among active devices. Phase III fairly distributes any remaining power to idle devices. The rationale behind the three phases is to allow power oversubscription while maximizing datacenter utilization. On a trace-driven large-scale simulation using GPU power telemetry from a production datacenter, nvPAX runs with a mean wall-clock time of 264.69 ms per allocation interval and achieves a mean satisfaction ratio of 98.92%, outperforming static equal-share allocation and providing robustness beyond greedy proportional allocation in the presence of non-uniform hierarchical bottlenecks.
1
Introduction
Current and future datacenters are designed and built with power oversubscription: the aggregate peak power consumption of devices exceeds the power available from the utility feed or from intermediate distribution nodes [21]. Power oversubscription can increase effective cluster throughput by exploiting temporal diversity in demand, since not all jobs draw peak power simultaneously, so provisioning for the sum of per-device peaks is inefficient. However, oversubscription increases the risk of violating upstream power distribution network (PDN) constraints (e.g., rack, power distribution unit (PDU), or facility limits), which can trigger capping actions or outages. This motivates fast control policies that, at every decision point, enforce feasible power allocations close to the per-device power requests. Operators therefore require dynamic allocation policies that translate per-device requests into enforceable power limits while respecting PDN constraints and tenant-level contractual guarantees. A variety of approaches exist in the literature and in practice. Control-theoretic methods enforce power limits at multiple levels of the hierarchy (e.g., SHIP [29] at rack, PDU, and datacenter) but typically do not incorporate tenant-level SLAs or explicit fairness across devices. Linear programming has been used for energy-aware server provisioning and cluster-level performance [7], though without modeling a full hierarchical oversubscribed PDN or tenant domains. Prediction- and learningbased systems improve oversubscription or service level objective (SLO) under capping [12, 27, 25] but do not guarantee that every control decision satisfies all physical and contractual constraints. Reinforcement learning (RL) and related methods [10] generally lack deterministic feasibility at each step. Commercial solutions and disclosed patents often target a single level (e.g., rack or server) Preprint.
load monitoring and provisioning [16], rather than per-device allocation under a full PDN tree and tenant SLAs. Section 2 reviews and compares these approaches. A common pattern in production datacenters is a closed-loop power capping system that periodically collects power data, predicts demand, applies a policy to set power caps, and configures devices accordingly. PRS is one such framework [2]; it currently supports only a flat (non-hierarchical) power domain. The gap we address is that no existing method jointly handles a full hierarchical PDN and tenant SLA constraints, while providing a provably fair, optimal policy in a single formulation suitable for fast control loops. Hierarchical PDN and Physical Constraints. The actual PDN in datacenters is hierarchical: a tree from the utility feed down to devices. At each level, oversubscription can occur. Specifically, the utility feed supplies a finite budget to the datacenter, and each data hall’s PDU is allocated a share such that the sum across halls may exceed the utility budget. This structure continues through rack PDUs down to devices, where the combined nominal power of devices may exceed the capacity of their parent node. Power consumption at any level must not exceed the capacity of its parent. These hierarchical capacity constraints are vertical in that they apply to each subtree along the PDN tree. In addition, each device has minimum and maximum permissible power limits, and its allocated power must lie within this range. We refer to these as physical constraints. Tenant SLAs. In cloud environments, the provider assigns servers to tenants; each tenant domain is governed by a service level agreement (SLA) that may include, for example, a minimum power allocation guarantee over the tenant’s domain, per-device minimum guarantees, domains with “maxQ” or “max-P” constraints (GPU power profiles that cap a device at a reduced or full thermal design power, respectively) [18], and the option for tenants to opt in or out of dynamic power allocation. These form a set of service-level constraints that any power allocation must satisfy. Unlike the vertical PDN constraints, tenant SLAs are horizontal, namely, they can couple devices across different branches of the physical hierarchy. Priorities and Active Versus Idle Devices. Power allocation should respect job priorities, where higher-priority jobs receive precedence when fulfilling requests. It should also distinguish active devices (assigned to running jobs) from idle ones, prioritizing active devices and allocating power to idle devices only when headroom remains. Fairness. In distributed multi-device workloads (e.g., multi-GPU training), end-to-end progress is often determined by the slowest device, so uneven allocations among devices can create stragglers and bottlenecks even when aggregate allocated power is high. A practical policy should therefore distribute shortages and surplus so as to avoid disproportionate degradation of any single device within a job or priority class. Figure 1 illustrates a datacenter’s power and tenant layout with multiple oversubscription levels and several tenant domains. The problem is to design a policy that, given per-device predictions or requests, outputs device power allocations that satisfy hierarchical PDN constraints and tenant SLA constraints at every control step, respect job priorities and the active/idle distinction, and distribute shortage or surplus fairly. Our Contribution To address these requirements in a single policy suitable for tight control loops, we make the following contributions: • Problem formulation and requirements. We formalize dynamic power allocation over a full tree-structured PDN topology with hierarchical capacity constraints and per-device power limits, horizontal tenant SLA constraints that couple devices across the tree, and operational requirements including job priorities and an active/idle distinction. • nvPAX: a hybrid QP/LP allocator with explicit fairness guarantees. We propose nvPAX, a constrained-optimization policy realized as a hybrid sequence of convex programs: a priority-ordered Quadratic Program (QP) for power-request satisfaction followed by max– min Linear Programs (LP) for excess power redistribution among active and then idle devices. Our optimization formulation preserves deterministic feasibility at every control step while maximizing utilization and distributing shortage/surplus in a well-defined fair 2
Tenants: Tenant A Tenant B Tenant C
Utility 10 MW Halls: 6 MW × 2 = 12 MW (1.2×)
Racks: 4 MW × 4 = 16 MW (1.6×)
Hall 1 6 MW
Rack 1 4 MW S1
Hall 2 6 MW Rack 2 4 MW
S2
S3
Rack 3 4 MW S4
S5
Rack 4 4 MW S6
S7
S8
Figure 1: Datacenter power and tenant layout illustrating hierarchical oversubscription. The tree represents the physical PDN from utility down to servers. Each node is labeled with its power capacity; notably, the sum of children’s capacities exceeds the parent’s capacity at multiple levels (e.g., 12 MW aggregate hall capacity vs. 10 MW utility feed), necessitating dynamic allocation. Tenants may have SLA power guarantees, and server colors indicate tenant domains (see legend). The dashed box highlights an example horizontal tenant SLA constraint coupling devices across the PDN, while the triangle highlights an example vertical hierarchical PDN constraint on a subtree. manner. Crucially, our hybrid optimization approach makes these large, constrained fairness problems solvable fast enough to run inside a tight control loop. • Saturation-aware surplus redistribution. To avoid leaving usable power undistributed under tight PDN and tenant constraints, nvPAX iteratively detects saturated devices and re-solves reduced LPs, improving utilization without violating fairness or feasibility. • Support for heterogeneous device types. We show how to incorporate per-device weights (i.e., supporting relative rather than absolute power allocations) so that fairness is meaningful across heterogeneous power ranges. • Baseline allocations and their limitations. We describe a greedy baseline power allocation algorithm and show that, while it can match nvPAX on balanced hierarchies, it can be substantially inferior in non-uniform PDNs with internal bottlenecks; moreover, it cannot enforce horizontal tenant SLAs. • Large-scale evaluation and scalability. Using GPU telemetry from a production datacenter with over 12,000 H100 GPUs, we evaluate nvPAX against static and greedy baseline power allocations, and measure both allocation quality and runtime. On this trace, nvPAX attains a mean satisfaction ratio of 98.92% while running within a tight control-loop budget (mean wall-clock time of 264.69 ms). We further benchmark scaling on synthetic hierarchies up to 105 devices.
2
Related Work
Datacenter power management aims to supply each power consumer with the power it needs while respecting physical limits, which include, for example, maximum power per rack or PDU so that circuit breakers are not triggered. Device or server power capping is the practice of enforcing upper limits on power consumption to prevent overloading; when capping is triggered, device performance is typically reduced (e.g., by lowering processor frequency) so that consumption stays within the allowed budget. Hierarchical power capping has been widely studied and deployed in production. SHIP [29] models the three-level power-distribution hierarchy (rack, PDU, datacenter) and uses a feedback control approach with formal stability guarantees. Similarly, Dynamo [31] coordinates power capping across Facebook’s entire datacenter fleet by monitoring power at various hierarchy levels and reactively throttling lower-priority workloads during power emergencies. Other works have explored managing distributed UPS energy for effective power capping [11] and agile, scalable datacenter power control [15]. Further research addresses sustainable and carbon-aware computing [1, 26], emphasizing the need for robust power management. However, these systems focus primarily on aggregate 3
power limits rather than modeling per-device power requests. Under power shortage, they rely on utilization-weighted proportional sharing (as in SHIP) or reactive heuristics (as in Dynamo), offering no formal guarantee of per-device fairness or strict adherence to tenant SLAs. Priority-aware capping has been deployed at scale. Sakalkar et al. [27] describe Google’s approach of pausing lower-priority workloads when power is short, so that production services keep their power. Kumbhare et al. [12] use supervised machine learning (ML) predictions to protect critical virtual machines (VMs) on Azure. Similarly, Thunderbolt [13] uses task-level QoS differentiation to throttle batch workloads while maintaining low latency for serving tasks. More recently, power oversubscription has been explored specifically for modern AI workloads. POLCA [20] characterizes LLM power consumption to enable aggressive oversubscription in inference clouds, while TAPAS [28] proposes thermal and power aware scheduling for LLM inference. Other works investigate GPU sharing and fragmentation in deep learning clusters [30]. Another solution [16] implements fuzzylogic-based prioritized capping at the rack or server level, which remains heuristic and singlelevel. None of these methods formulates allocation as a constrained optimization problem that simultaneously enforces hierarchical limits and tenant guarantees. Optimization has been applied to datacenter power, but typically for different objectives. Pelley et al. [22] use optimization for dynamic power provisioning, but their focus is on dynamically altering the PDN topology (power routing) rather than fairly allocating a fixed hierarchical budget. Linear programming has been used for energy-aware server provisioning [7] (deciding how many servers to turn on or off to meet demand), but in a flat, single-cluster model and not for per-device power allocation. Similarly, fast centralized schedulers like Firmament [5] and Apollo [4] use network flow optimization or coordinated scheduling for task placement, but do not explicitly manage hierarchical power limits. Finally, reinforcement learning and related methods [25, 10] have been explored for power-aware resource management, but they generally lack deterministic feasibility guarantees at each control step. Overall, each prior work only addresses a subset of the problem. Modern datacenters are becoming extremely large and complex, with thousands or even tens of thousands of constraints that must be satisfied simultaneously [3, 9]. Power oversubscription must therefore be managed in a way that satisfies all physical and service-level constraints at every moment while still providing high performance and efficient resource utilization. Furthermore, tenant SLAs must be respected, and mechanisms for job prioritization and fair allocation are essential to avoid job bottlenecks, which are particularly detrimental to distributed deep neural network (DNN) and large language model (LLM) workloads [6, 14, 32, 17, 23, 24, 33]. To the best of our knowledge, this paper is the first to jointly handle hierarchical PDN constraints, tenant SLA constraints, and per-device fairness in one allocation policy, realized as a sequence of convex optimization problems. We support job priorities while distinguishing active and idle jobs, and every power allocation of our algorithm is guaranteed to satisfy all physical and SLA constraints.
3
Policy Requirements and Allocation Strategy
In this section, we outline the desired requirements from a power allocation policy. Before doing so, we first define several key concepts: • Active and idle devices: An active device is one assigned to a running job; an idle device is not assigned to any running job. Device state can be classified using one of the following methods: 1. Job scheduler information: When accessible, the scheduler provides the most accurate indication of device activity, identifying whether a device is allocated to a running job or not. 2. Power-based detection: In cloud environments, where the tenants’ job-scheduler information is not available to the cloud service provider (CSP), device activity can be inferred from power consumption. • Power request: A device’s power request may be derived from predicted power consumption (e.g., via a forecasting model such as PRS [2]), or from the job specification associated with the device, such as max-P, max-Q, or other power profiles [18]. The power request of an idle device equals the minimum allowed power limit. 4
• Power allocation: The power limit applied to a device. The power allocation policy should satisfy the following requirements: 1. Feasibility (physical and service-level constraints): The policy must satisfy all physical constraints (device power limits and hierarchical PDN capacities) and all service-level constraints (e.g., tenant SLAs) at every control step, and must never output an infeasible allocation. 2. Closeness to requests: The policy should minimize the discrepancy between power requests and allocated power. This ensures that allocations closely track actual needs while still respecting all physical and service-level constraints. 3. Maximizing allocation of the available power budget: The policy should allocate as much of the available power budget as possible, subject to all constraints. The objective should not “reserve” unused power budget. 4. Idle vs. active prioritization: The policy must treat active and idle devices differently, ensuring that active devices get higher priority when distributing available power. Idle devices should only receive increased power allocations (beyond required minimum) when surplus power remains, allowing active devices to receive sufficient power and headroom, while idle devices remain near their minimum permissible power. At the same time, if surplus power remains after all active devices have been raised to their maximum allowed allocation, the allocation may also be increased for idle devices (since “idle” classification can be imperfect). 5. Priority-aware allocation: The policy must respect per-device priorities. Priority may reflect job importance, device type (e.g., networking devices such as switches may be assigned the highest priority to ensure fabric availability), or a combination of both. It should first allocate power to satisfy the requests of the highest priority devices; only after all devices at this priority level are either fully satisfied or constrained should it proceed to the next priority level, and so on. Once all per-device requests across all priority levels have been met, any remaining surplus is distributed (e.g., evenly) subject to constraints. Priority information is optional; when unavailable, all devices are assigned equal priority and the allocator reduces to a single-level fair allocation. 6. Fair handling of shortages and surplus power budget: In case of a shortage, the policy should allocate power such that lower-priority jobs are impacted first. Within each priority level, deviations from requests should be distributed evenly across devices to avoid creating bottlenecks. Once prioritized requests are satisfied, any remaining spare budget should be distributed fairly across active devices, and then to idle devices if surplus remains. 7. Heterogeneous device types: When devices have significantly different power ranges, the objective should support per-device weights or normalization (e.g., by each device’s maximum allowed power) so that “fair” deviation is meaningful across device types. 8. Computational efficiency: The algorithm computing the power allocations should be sufficiently fast to support control loops that adjust power allocations every few seconds. To satisfy the requirements outlined above, we propose a three-phase allocation procedure: 1. Phase I: Priority-ordered request satisfaction. Devices are grouped by priority and processed from the highest to the lowest priority level. At each priority level, the algorithm solves a constrained optimization problem that attempts to satisfy the power requests of devices at the current priority (that is, allocating power as close as possible to the power requested by the device), while fixing the allocations of higher-priority devices to their previously determined values and keeping lower-priority or idle devices at minimum (unless tenant lower-bound SLAs require higher allocations). 2. Phase II: Residual power redistribution among active devices. After all priority levels have been processed, any remaining power budget is redistributed among active devices, increasing their allocations beyond the values assigned in Phase I, subject to all constraints. 3. Phase III: Residual power redistribution among idle devices. Finally, if residual power remains, it is allocated to idle devices while preserving feasibility with respect to all physical and tenant constraints. 5
In the next section, we present nvPAX, a hybrid constrained-optimization instantiation of this threephase procedure. Phase I is formulated as a convex QP for priority-ordered request satisfaction, while Phases II and III are LPs for max-min surplus redistribution, all subject to the physical and service-level constraints described above. nvPAX is designed to be invoked by the datacenter power management system in a closed-loop control cycle (e.g., every 30 seconds); device failures and unexpected reductions in available power supply are therefore handled implicitly—at the next cycle, the management system passes updated device states and the current available capacity to nvPAX, which recomputes a feasible allocation from scratch. Shortening the control interval reduces the window of exposure to any such disruption.
4
Algorithmic Framework
In this section, we derive our power allocation algorithm that satisfies all requirements stated in Section 3. We start by setting the required notation in Section 4.1, and formulate the constraints in Section 4.2. Then, in Section 4.3, we present nvPAX, a hybrid QP/LP constrained-optimization approach: Phase I uses a convex quadratic program (QP) for priority-ordered request satisfaction, while Phases II and III use linear programs (LPs) for max-min surplus redistribution. This decomposition keeps all physical and tenant constraints linear while using the objective function best suited to each phase. 4.1
Notation
We start by fixing the notation used by our algorithm. We define notation for devices, their power limits and state, the power hierarchy and node capacities, and tenant domains and SLA budgets. We denote by n the number of devices in our hierarchical PDN (such as GPUs, CPUs, etc.; in general, any power consumer whose power allocation can be controlled). • Devices: Devices are indexed by i ∈ {1, . . . , n}. For each device i: – li , ui denote its minimum and maximum allowed power limits. – ai ∈ [li , ui ] denotes its allocated power (decision variable). – ri ∈ [li , ui ] denotes its requested (or predicted) power, where for idle devices ri = li . – pi ∈ {1, . . . , P } denotes its priority level, where P is the highest priority level, and higher values indicate higher priority. In practice, pi reflects the relative importance of device i (see Section 3); the priority of idle devices is irrelevant. • Device states: R ⊆ {1, . . . , n} denotes the set of active devices, and I ⊆ {1, . . . , n} denotes the set of idle devices, with R ∪ I = {1, . . . , n} and R ∩ I = ∅. If device i is idle, we set ri = li . • Power hierarchy: The physical power hierarchy forms a rooted tree. Nodes in the hierarchy are indexed by j. For each node j – Cj denotes the power capacity of node j, – Dj ⊆ {1, . . . , n} denotes the set of devices in the subtree rooted at node j. • Tenants: Tenants are indexed by k. Each tenant k is associated with a set of devices Tk ⊆ {1, . . . , n} and may impose SLA constraints on its allocations {ai : i ∈ Tk }. 4.2
Constraints
Any power allocation a1 , . . . , an must satisfy the following constraints, independent of the objective function used to determine the allocation. 1. Device power limits: Each device allocation must be within the device’s physical lower and upper limits, that is li ≤ ai ≤ ui , ∀i ∈ {1, . . . , n}. (1) 2. Hierarchical power constraints: The total power allocated to all devices under a given node in the PDN must not exceed the node’s power capacity, that is X ai ≤ Cj , ∀j, (2) i∈Dj
6
where j ranges over all nodes in the PDN. 3. Tenant SLA constraints: We assume tenant/service-level constraints can be represented as linear inequalities. Each tenant constraint is of the form X Bkmin ≤ ai ≤ Bkmax , ∀k, (3) i∈Tk min max where Bk ≥ 0 and Bk ≥ 0 are the guaranteed minimum and maximum power for tenant k, respectively. Either bound may be absent (i.e., Bkmin = 0 or Bkmax = ∞). More
general linear SLA constraints (not shown) are also supported by our algorithm and can encode, e.g., per-device minimum guarantees or combinations of budgets across subsets of devices.
4.3
nvPAX
This section provides a detailed description of our power allocation algorithm. It consists of three phases, where each phase is formulated as one or more convex optimization problems (either a QP or an LP) subject to the device, hierarchical, and tenant constraints of Section 4.2. The result of the algorithm is a feasible allocation that respects priorities, maximizes utilization of the available power budget, and distributes surplus or shortage in a well-defined fair manner. 4.3.1
Phase I: Priority-Based Allocation
Phase I allocates power to active devices as close as possible to per-device power requests while enforcing all constraints and respecting priority order. Concretely, we process priorities from highest to lowest; within each priority level, we minimize the deviation ri − ai for devices at that priority. We initialize ai ← li for all devices, then process priority levels one at a time from P down to 1, completing each level before moving to the next; once allocations for a higher-priority level have been determined, they become hard constraints for all lower-priority levels. At the start of level p, three disjoint device sets are initialized: • Unsaturated set A ← {i ∈ R : pi = p}: active devices at the current priority level whose allocations are being optimized in this iteration. • Fixed set F ← {i ∈ R : pi > p}: higher-priority devices whose allocations were finalized in earlier iterations and are held fixed. • Free set L ← {1, . . . , n}\(A∪F): lower-priority active devices and idle devices, which are not yet being optimized but must remain flexible (within [li , ui ]) to satisfy SLA constraints. For a fixed priority level, Phase I solves the QP X X min (ai − ri )2 + ε (ai − li )2 {ai }
s.t.
i∈A
i∈L
constraints (1)–(3),
(4)
ai = afixed , i ∈ F, i li ≤ ai ≤ ui , i ∈ A ∪ L. Devices in A may receive allocations above their request ri , and devices in L may P receive allocations above their minimum li : in both cases, tenant SLA constraints of the form i∈Tk ai ≥ Bkmin can force this, and fixing devices in A at ri or devices in L at li could render power allocation infeasible. The objective (4) is designed to discourage unnecessary over-allocation: it pulls allocations of devices in A toward ri and allocations of devices in L toward li , deviating only as much as the constraints require. To that end, the first term in (4) pulls current-priority devices toward their requests, while the second term keeps lower-priority and idle devices close to their minimum power unless tenant lower-bound SLAs force them upward. If no tenant lower-bound SLA is present, devices in L can be fixed at li and the regularizer is unnecessary. For heterogeneous devices (i.e., having substantially different power ranges), nvPAX also supports a normalized objective X X 2 2 min ((ai − ri )/ui ) + ε ((ai − li )/ui ) , {ai }
i∈A
i∈L
7
Algorithm 1 Phase I: Priority-based request satisfaction Require: Devices {(ri , li , ui , pi )}ni=1 ; PDN {(Cj , Dj )}; tenant SLAs {(Bk1 , Bk2 , . . . , Tk )} Ensure: Allocation {ai } satisfying constraints (1)–(3) 1: Initialize ai = li for all i 2: for each positive priority level p from highest to lowest do 3: A ← {i ∈ R : pi = p}, F ← {i ∈ R : pi > p}, L ← {1, . . . , n} \ (A ∪ F ) 4: a∗ ← solve (4) with sets A, F, L 5: ai ← a∗i for all i 6: end for 7: return {ai } which normalizes each device’s deviation by its maximum allowed power, making “fair” deviation comparable across devices with different power ranges. More generally, ui may be replaced by any positive per-device scale. The QP in (4) is strictly convex over the free variables. When there are no lower bound SLA constraints, L is pinned and the free block is A; with SLA lower bounds, ε > 0 makes the Hessian positive on A ∪ L. Variables in F are fixed by bounds, so any zero Hessian entries on fixed variables do not affect uniqueness of the solution. Thus, each priority level has a unique optimum and requires only a single QP solve. The regularization parameter ε is chosen small so that lower-priority and idle devices stay near their minimums unless feasibility requires otherwise; in practice the results are insensitive to the exact value, and we use ε = 10−5 , large enough to avoid numerical issues yet small enough not to distort the primary request-deviation objective. Algorithm 1 presents the procedure for Phase I. 4.3.2
Phase II: Max–Min Allocation for Active Devices
After Phase I, remaining power is distributed to active devices using a max-min fairness objective. For Phase II, devices are partitioned into three sets A, F, and L, initialized by A ← R, F ← ∅, L ← I. Here A contains the active devices to be optimized and the free set L contains the idle devices. In Phase II, we maximize the minimum additional power allocated across all active devices in A by solving (1) max min ai − ai , {ai } i∈A
which, via the standard epigraph transformation and a regularization term, is written as the LP X X max t + ε ai − ε ai {ai }, t
s.t.
i∈A
i∈L
(1)
∀i ∈ A,
t ≥ 0,
(5)
ai − ai
≥ t,
ai = afixed , i
∀i ∈ F ,
(1)
subject to constraints (1)–(3), where ai denotes the Phase I allocation. For heterogeneous devices, (1) (1) we can use normalized improvement by replacing ai − ai ≥ t with (ai − ai )/ui ≥ t. The +ε coefficient on A selects, among all optimal solutions, one that maximizes total active-device allocation, promoting higher utilization. The −ε coefficient on L pushes free idle devices toward li ; the LP raises them only when tenant SLA constraints require it. As in Phase I, ε is chosen small so that the max-min surplus objective dominates. After solving (5) and increasing the allocation of each active device by t, some devices may become saturated, namely, their allocation cannot be increased without violating some constraint (either its own upper bound, a tight ancestor-node capacity, the root capacity, or a tenant upper budget containing that device). We thus solve a sequence of LPs, where at each iteration we fix the allocation of saturated devices and re-solve for the remaining devices. Algorithm 2 summarizes the procedure of Phase II. 8
Algorithm 2 Phase II: Surplus allocation for active devices (1)
Require: Phase I allocation {ai }; states R, I; PDN {(Cj , Dj )}; tenant SLAs {(Bk1 , Bk2 , . . . , Tk )} Ensure: Updated allocation {ai } satisfying constraints (1)–(3) 1: A ← R, F ← ∅, L ← I 2: while A ̸= ∅ do 3: (a∗ , t∗ ) ← solve (5) with sets A, F, L 4: ai ← a∗i for all i 5: Identify saturated devices: N ← {i ∈ A : i has no positive slack to receive more surplus} 6: A←A\N 7: F ← F ∪ N {L unchanged: A ∪ F is invariant} 8: end while 9: return {ai } Algorithm 3 nvPAX: Three-phase power allocation Require: Devices {(ri , li , ui , pi )}ni=1 ; states R, I; PDN {(Cj , Dj )}; SLAs {(Bk1 , Bk2 , . . . , Tk )} Ensure: Allocation {ai } satisfying constraints (1)–(3) (1) 1: Phase I (priority-ordered request satisfaction): {ai } ← Algorithm 1 on (ri , li , ui , pi ) (2) (1) 2: Phase II (redistribute among active devices): {ai } ← Algorithm 2 given {ai }, with A ← R, F ← ∅, L ← I (3) (2) 3: Phase III (redistribute among idle devices): {ai } ← Algorithm 2 given {ai }, with A ← I, F ← R, L ← ∅ (3) 4: return {ai } 4.3.3
Phase III: Progressive Allocation for Idle Devices
Phase III distributes any remaining surplus power to idle devices, using the same max-min procedure as Phase II with A ← I and F ← R. Active devices are pinned at their Phase II allocations via (2) ai = ai for all i ∈ F. There is no free set L in this phase: all constraints, including tenant minimum SLAs, were already satisfied at the end of Phase II, and since Phase III only increases idle allocations, all tenant lower bounds are maintained or improved. The LP of Phase III maximizes the minimum power increase of idle devices, with a regularization term that promotes higher total allocation: X max t + ε ai {ai }, t
s.t.
i∈A
t ≥ 0,
(6) (2)
ai − ai
(2)
≥ t,
ai = ai ,
∀i ∈ A,
∀i ∈ F ,
(2) subject to constraints (1)–(3), where ai denotes the Phase II allocation.
The +ε coefficient selects, among all optimal solutions, one that maximizes total idle allocation. As in the preceding phases, we solve a sequence of LPs with iterative saturation detection. The procedure (2) for Phase III is the same as Algorithm 2, but starting from the Phase II allocations {ai }, while setting A ← I, F ← R, L ← ∅, and using LP (6) in place of LP (5). Algorithm 3 summarizes the complete nvPAX procedure by composing the three phases described above.
5
Experiments
We evaluate nvPAX on a large-scale simulation constructed from GPU power telemetry from a production datacenter. We compare its power allocations against static equal-share and greedy proportional baselines explained below. 9
All experiments were run on an Apple M4 Pro with 14 cores and 48 GiB RAM. The software stack was Python 3.13.9, SciPy 1.16.3, HiGHS LP solver implemented by highspy Python bindings (version 1.12.0), and Clarabel (from clarabel version 0.11.1) for the Phase I QP. This reflects a deliberate design choice: nvPAX runs entirely on CPU and requires no GPU on the management node, keeping the power-management control plane independent of the accelerator hardware it manages. 5.1
Data and Hierarchy
The dataset comprises H100 GPU power metrics sampled every 30 seconds over a three-day period (8,523 timestamps). The datacenter consists of 4 halls, with more than 12,000 GPUs. The power hierarchy is: datacenter → halls → racks → servers → devices. Each device is one GPU with li = 200 W and ui = 700 W, consistent with H100 specifications. Capacities are computed bottom-up. At each server, capacity equals number of GPUs per server multiplied by 700 W (no oversubscription at server level). At each rack, capacity equals the sum of its child server capacities multiplied by an oversubscription factor; at each hall, the sum of its rack capacities multiplied by the oversubscription factor; and at the root (datacenter), the sum of hall capacities multiplied by the oversubscription factor. Thus at every level the parent’s capacity is a fraction of the aggregate child capacity, producing hierarchical oversubscription. A single oversubscription factor (0.85 in the reported runs) is applied at each level, i.e., the parent’s capacity at each level equals the sum of its children’s capacities multiplied by this factor. Hence, the parent can supply only that fraction of the aggregate child capacity. The resulting ratio of total device maximum power to root capacity is approximately 1.63 in our hierarchy, so the network is undersupplied relative to all devices at maximum power, motivating dynamic allocation. 5.2
Request Generation
For each timestamp, each device’s power request ri is the measured GPU power from the available telemetry for that GPU. In this evaluation, we use the actual measured power as the request. Before optimization, requests are clipped to the feasible device interval [li , ui ]; thus devices whose measured power is below li enter the optimizer with request li , and similarly for ui . We assume perfect knowledge of device requests to evaluate the policy’s ability to satisfy them under constraints. A device is classified as idle if its power request is strictly below 150 W; otherwise it is active. This classification drives the Phase II/III distinction (redistribution to active devices, then to idle devices). In this experiment, all devices have priority 1. An experiment with multiple priority levels is described below. 5.3
Baseline Allocations
We compare nvPAX of Algorithm 3 against the following baseline allocations: • Static allocation: Each device receives an equal share of the root budget (Croot /n) with no redistribution of unused power. • Greedy proportional allocation: A hierarchical heuristic that mimics industry-standard proportional sharing (similar to SHIP [29]). Starting from the root, if a node is oversubscribed, it distributes its power to its children proportional to their aggregate requests, repeating recursively down to the devices. While this approach is fast and intuitive, it does not globally redistribute leftover power nor explicitly handles horizontal tenant SLA constraints. Tenant SLAs introduce horizontal constraints that couple devices across the physical hierarchy (e.g., P i∈Tk ai ≥ Bk ). Since the static and greedy baselines do not enforce such constraints, they can violate tenant SLAs even when a feasible allocation exists. Therefore, in the evaluation on real GPU telemetry we do not impose tenant SLA constraints, and compare policies only under device limits and hierarchical PDN constraints. An experiment with tenant-SLA constraints and multiple job priorities is given in Appendix B. 5.4
Evaluation Metrics
We compute the following quantities at each timestamp t. Let n denote the total number of devices, Pn and let U nvPAX = i=1 min(ri , anvPAX ) be the total power allocated by nvPAX, capped by request, i 10
at timestamp t. We define U Static and U Greedy analogously by replacing anvPAX with aStatic and i i Greedy ai , respectively. We use the following metrics: 1. Relative utilization improvement (nvPAX vs. baselines, %): The per-timestamp relative improvement is given by U nvPAX − U Static U nvPAX − U Greedy · 100, ∆U Greedy = · 100. Static U U Greedy It measures how much nvPAX improves over each baseline as a percentage of that baseline’s utilization. 2. Satisfaction ratio: U nvPAX S nvPAX = Pn , i=1 ri Pn defined when i=1 ri > 0. This is the fraction of aggregate demand that is actually met; S = 1 means every device receives at least its requested power, while S < 1 quantifies the overall shortfall. We define S Static and S Greedy analogously by replacing U nvPAX with U Static and U Greedy , respectively. ∆U Static =
5.5
Results
We ran the simulation for a duration corresponding to three consecutive days of telemetry (8,523 timestamps at 30-second spacing). Figure 2 summarizes nvPAX vs. Static: satisfaction ratio (left) and relative utilization improvement over Static (right). Greedy proportional allocation is not shown since on this trace its aggregate satisfaction is almost identical to nvPAX (mean S Greedy = 98.92% vs. S nvPAX = 98.92%), so the nvPAX–Greedy utilization gap is negligible; both policies substantially outperform Static under hierarchical oversubscription. For satisfaction ratio, nvPAX achieved mean 98.92% (std 0.48%, min 96.49%, max 100.00%); Static 81.30% (std 6.37%, min 77.07%, max 100.00%); Greedy 98.92% (std 0.48%, min 96.49%, max 100.00%). The mean improvement of nvPAX over Static was 17.62 percentage points (std 5.93%); nvPAX was at least as good as Static on every timestamp. Despite the similar performance of nvPAX and the Greedy algorithm in this simulation, they are not competitive in general. Since the Greedy algorithm makes local proportional decisions based on aggregate subtree demand, it may allocate budget to subtrees whose internal PDN bottlenecks prevent that budget from being delivered to devices. In such non-uniform hierarchies, nvPAX’s global constrained-optimization formulation reallocates power away from bottlenecked subtrees toward feasible headroom elsewhere, yielding substantially higher utilization and satisfaction. Appendix A constructs a concrete example where the Greedy algorithm is considerably inferior to nvPAX and characterizes when this behavior arises. Finally, nvPAX is strictly more expressive than Static allocation and the Greedy algorithm in that it enforces horizontal tenant SLA constraints that couple devices across the PDN. This capability is absent from these baseline algorithms. Runtime of nvPAX in this simulation (single optimize() call per timestamp): mean 264.69 ms with std 162.72 ms. 5.6
Runtime and Computational Complexity
To estimate the runtime of nvPAX and its scaling with problem size, we benchmark its performance on synthetic randomly generated hierarchies with device counts n ∈ {103 , 5·103 , 104 , 2.5·104 , 5·104 , 105 }, with five independent runs per size. We measure the wallclock time of a single optimize() call (including all phases). Figure 3 shows the results. Scalability and Runtime. We note that the presented benchmark computes per-device power allocations. In newer and emerging platforms (e.g., GB200-class servers and successors), power caps may be applied per server (typically covering 4-8 GPUs and several CPUs) rather than per device; this reduces the number of decision variables and constraints and can therefore yield substantially faster runtimes. Extrapolating from Figure 3, we expect that solving for ∼10,000 servers can be achieved in under 0.25 seconds. Additional speedups are possible via early stopping of surplus distribution in Phases II and III, warm-starting across control steps, and decomposing the hierarchy into independent 11
3000
Count
Count
6000
Static nvPAX
4000
1000
2000 0 0.6
2000
0.7
0.8
0.9
0
1.0
Satisfaction ratio
0%
10%
20%
Utilization improvement over Static
(a) Satisfaction ratio of nvPAX (S nvPAX ) and Static (S Static ). Higher is better.
(b) Relative utilization improvement of nvPAX over Static (∆U Static ).
Figure 2: nvPAX vs. Static equal-share allocation. Left: satisfaction ratio. Right: relative utilization improvement over Static. Overall, nvPAX achieves substantially higher satisfaction than Static and delivers higher useful utilization, indicating that it allocates more of the available power budget to meet demand under the constraints.
Mean solve time (s)
5.32
nvPAX fit: n 1.16
1.87 0.802 0.25 0.123 0.0245 1k
5k 10k
25k 50k100k
Number of devices, n
Figure 3: Empirical scaling of nvPAX’s optimization time on synthetic random hierarchies.
subproblems when constraints decouple. Finally, faster runtimes may be achievable with commercial QP/LP solvers (e.g., Gurobi [8]) and stronger CPUs. Computational Complexity. Phase I solves one convex QP per priority level, while Phases II and III solve a sequence of LPs with iterative saturation detection. Each QP/LP has O(n) allocation variables (plus an auxiliary scalar t in Phases II and III) and O(n + Nnodes + NSLA ) linear constraints for device bounds and physical and tenant constraints, with sparse connectivity. Thus a single solve is polynomial in the problem dimensions; practical cost is dominated by the QP/LP solver calls. Figure 3 shows that on the synthetic hierarchies, observed mean runtime scales approximately as n1.16 over 103 –105 devices.
6
Conclusion
This paper presented a hybrid QP/LP constrained-optimization algorithm for dynamic power allocation in hierarchical, multi-tenant datacenters. The algorithm extends existing power capping frameworks by supporting a full tree-structured PDN and tenant SLA constraints (e.g., minimum and maximum power guarantees per tenant), while satisfying device limits and delivering priorityaware, fair allocation. Empirical results on a large-scale simulation with thousands of GPUs and real telemetry, comparing nvPAX to Static equal-share and Greedy proportional baselines, demonstrate 12
substantial gains in utilization and request satisfaction, together with empirical scaling behavior on synthetic hierarchies up to 105 devices. For future work, we plan to integrate nvPAX into production datacenter power-management systems such as PRS [2] and DPS [19]. Beyond deployment, an important direction is to characterize and improve the closed-loop behavior of nvPAX under noisy telemetry and rapidly changing demand, including adding explicit stability/smoothness mechanisms that limit unnecessary allocation oscillations across control steps. A complementary direction is to develop an anytime, deadline-aware variant of nvPAX that returns progressively better allocations as time permits, while providing principled fallback when strict per-step latency constraints truncate later refinement phases. Finally, we plan to exploit the spatial locality of demand changes in large hierarchies, aiming to confine allocation updates to affected subtrees and reduce disruption to unrelated devices.
References [1] Bilge Acun, Benjamin Lee, Fiodar Kazhamiaka, Kiwan Maeng, Udit Gupta, Manoj Chakkaravarthy, David Brooks, and Carole-Jean Wu. Carbon explorer: A holistic framework for designing carbon aware datacenters. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ASPLOS 2023, page 118–132, New York, NY, USA, 2023. Association for Computing Machinery. [2] Nir Arad, Hadar Sivan, Gil Levy, Sridutt Bhalachandra, Larry Dennison, and Shie Mannor. Data center scale prediction-based power reservation steering, 2024. U.S. Patent Application No. 134580-1109 (NVD-109US), pending. [3] Luiz André Barroso, Jimmy Clidaras, and Urs Hölzle. The datacenter as a computer: An introduction to the design of warehouse-scale machines. Synthesis lectures on computer architecture, 8(3):1–154, 2013. [4] Eric Boutin, Jaliya Ekanayake, Wei Lin, Bing Shi, Jingren Zhou, Zhengping Qian, Ming Wu, and Lidong Zhou. Apollo: Scalable and coordinated scheduling for Cloud-Scale computing. In 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14), pages 285–300, Broomfield, CO, October 2014. USENIX Association. [5] Ionel Gog, Malte Schwarzkopf, Adam Gleave, Robert N. M. Watson, and Steven Hand. Firmament: Fast, centralized cluster scheduling at scale. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), pages 99–115, Savannah, GA, November 2016. USENIX Association. [6] Juncheng Gu, Mosharaf Chowdhury, Kang G. Shin, Yibo Zhu, Myeongjae Jeon, Junjie Qian, Hongqiang Liu, and Chuanxiong Guo. Tiresias: A GPU cluster manager for distributed deep learning. In 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19), pages 485–500, Boston, MA, February 2019. USENIX Association. [7] Brian Guenter, Navendu Jain, and Charles Williams. Managing cost, performance, and reliability tradeoffs for energy-aware server provisioning. In 2011 Proceedings IEEE INFOCOM, pages 1332–1340, 2011. [8] Gurobi Optimization, LLC. Gurobi optimizer, 2026. [9] Myeongjae Jeon, Shivaram Venkataraman, Amar Phanishayee, Junjie Qian, Wencong Xiao, and Fan Yang. Analysis of Large-Scale Multi-Tenant GPU clusters for DNN training workloads. In 2019 USENIX Annual Technical Conference (USENIX ATC 19), pages 947–960, Renton, WA, July 2019. USENIX Association. [10] Hussain Kahil, Shiva Sharma, Petri Välisuo, and Mohammed Elmusrati. Reinforcement learning for data center energy efficiency optimization: A systematic literature review and research roadmap. Applied Energy, 389:125734, 2025. [11] Vasileios Kontorinis, Liuyi Eric Zhang, Baris Aksanli, Jack Sampson, Houman Homayoun, Eddie Pettis, Dean M. Tullsen, and Tajana Simunic Rosing. Managing distributed ups energy for effective power capping in data centers. In 2012 39th Annual International Symposium on Computer Architecture (ISCA), pages 488–499, 2012. 13
[12] Alok Gautam Kumbhare, Reza Azimi, Ioannis Manousakis, Anand Bonde, Felipe Frujeri, Nithish Mahalingam, Pulkit A. Misra, Seyyed Ahmad Javadi, Bianca Schroeder, Marcus Fontoura, and Ricardo Bianchini. Prediction-Based power oversubscription in cloud platforms. In 2021 USENIX Annual Technical Conference (USENIX ATC 21), pages 473–487. USENIX Association, July 2021. [13] Shaohong Li, Xi Wang, Xiao Zhang, Vasileios Kontorinis, Sreekumar Kodakara, David Lo, and Parthasarathy Ranganathan. Thunderbolt: Throughput-Optimized, Quality-of-ServiceAware power capping at scale. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pages 1241–1255. USENIX Association, November 2020. [14] Kshiteej Mahajan, Arjun Balasubramanian, Arjun Singhvi, Shivaram Venkataraman, Aditya Akella, Amar Phanishayee, and Shuchi Chawla. Themis: Fair and efficient GPU cluster scheduling. In 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20), pages 289–304, Santa Clara, CA, February 2020. USENIX Association. [15] Daniel Miller, Neal Master, Zhengyuan Zhou, and Nicholas Bambos. Scalable data center power management via a global stress signal. In 2015 IEEE Global Communications Conference (GLOBECOM), pages 1–7, 2015. [16] Rishi Mukherjee, Shivendra Katiyar, Lori Lynn Matthews, and Elie Antoun Jreij. Kinetic power capping using fuzzy logic-based dynamic system prioritization. U.S. Patent US20240126360A1, 2024. Dell Products L.P. [17] Deepak Narayanan, Keshav Santhanam, Fiodar Kazhamiaka, Amar Phanishayee, and Matei Zaharia. Heterogeneity-Aware cluster scheduling policies for deep learning workloads. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pages 481–498. USENIX Association, November 2020. [18] Sreedhar Narayanaswamy, Pratikkumar Dilipkumar Patel, Ian Karlin, Apoorv Gupta, Sudhir Saripalli, and Janey Guo. Datacenter energy optimized power profiles, 2025. [19] NVIDIA Corporation. Nvidia domain power service (dps), 2026. [20] Pratyush Patel, Esha Choukse, Chaojie Zhang, Íñigo Goiri, Brijesh Warrier, Nithish Mahalingam, and Ricardo Bianchini. Polca: Power oversubscription in llm cloud providers. arXiv preprint arXiv:2308.12908, 2023. [21] Pratyush Patel, Esha Choukse, Chaojie Zhang, Íñigo Goiri, Brijesh Warrier, Nithish Mahalingam, and Ricardo Bianchini. Characterizing power management opportunities for llms in the cloud. Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, 2024. [22] Steven Pelley, David Meisner, Pooya Zandevakili, Thomas F. Wenisch, and Jack Underwood. Power routing: dynamic power provisioning in the data center. In Proceedings of the Fifteenth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS XV, page 231–242, New York, NY, USA, 2010. Association for Computing Machinery. [23] Yanghua Peng, Yixin Bao, Yangrui Chen, Chuan Wu, and Chuanxiong Guo. Optimus: an efficient dynamic resource scheduler for deep learning clusters. In Proceedings of the Thirteenth EuroSys Conference, EuroSys ’18, New York, NY, USA, 2018. Association for Computing Machinery. [24] Aurick Qiao, Sang Keun Choe, Suhas Jayaram Subramanya, Willie Neiswanger, Qirong Ho, Hao Zhang, Gregory R. Ganger, and Eric P. Xing. Pollux: Co-adaptive cluster scheduling for goodput-optimized deep learning. In 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21), pages 1–18. USENIX Association, July 2021. [25] Haoran Qiu, Linghao Zhang, Chen Wang2 Hubertus Franke, Zbigniew T Kalbarczyk, and Ravishankar K Iyer. Parm: Adaptive resource allocation for datacenter power capping. In Machine Learning for Systems Workshop at the Annual Conference on Neural Information Processing Systems (NeurIPS 2023), 2023. 14
[26] Ana Radovanovic, Ross Koningstein, Ian Schneider, Bokan Chen, Alexandre Nobrega Duarte, Binz Roy, Diyue Xiao, Maya Haridasan, Patrick Hung, Nick Care, Saurav Talukdar, E. Mullen, Kendal Smith, MariEllen Cottman, and Walfredo Cirne. Carbon-aware computing for datacenters. IEEE Transactions on Power Systems, 38:1270–1280, 2021. [27] Varun Sakalkar, Vasileios Kontorinis, David Landhuis, Shaohong Li, Darren De Ronde, Thomas Blooming, Anand Ramesh, James Kennedy, Christopher Malone, Jimmy Clidaras, and Parthasarathy Ranganathan. Data center power oversubscription with a medium voltage power plane and priority-aware capping. In Proceedings of the 25th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), pages 497–511. ACM, 2020. [28] Jovan Stojkovic, Chaojie Zhang, Íñigo Goiri, Esha Choukse, Haoran Qiu, Rodrigo Fonseca, Josep Torrellas, and Ricardo Bianchini. Tapas: Thermal-and power-aware scheduling for llm inference in cloud platforms. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, pages 1266–1281, 2025. [29] Xiaorui Wang, Ming Chen, Charles Lefurgy, and Tom W. Keller. Ship: Scalable hierarchical power control for large-scale data centers. In 2009 18th International Conference on Parallel Architectures and Compilation Techniques, pages 91–100, 2009. [30] Bingyang Wu, Zili Zhang, Zhihao Bai, Xuanzhe Liu, and Xin Jin. Transparent GPU sharing in container clouds for deep learning workloads. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 69–85, Boston, MA, April 2023. USENIX Association. [31] Qiang Wu, Qingyuan Deng, Lakshmi Ganesh, Chang-Hong Hsu, Yun Jin, Sanjeev Kumar, Bin Li, Justin Meza, and Yee Jiun Song. Dynamo: Facebook’s data center-wide power management system. In 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA), pages 469–480, 2016. [32] Wencong Xiao, Romil Bhardwaj, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, Zhenhua Han, Pratyush Patel, Xuan Peng, Hanyu Zhao, Quanlu Zhang, Fan Yang, and Lidong Zhou. Gandiva: Introspective cluster scheduling for deep learning. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pages 595–610, Carlsbad, CA, October 2018. USENIX Association. [33] Pengfei Zheng, Rui Pan, Tarannum Khan, Shivaram Venkataraman, and Aditya Akella. Shockwave: Fair and efficient cluster scheduling for dynamic adaptation in machine learning. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 703–723, Boston, MA, April 2023. USENIX Association.
15
A
Greedy Proportional Allocation vs. nvPAX
This appendix describes the Greedy proportional allocation baseline used in our experiments, and explains its limitations as a solution to the power allocation problems. Greedy proportional allocation is a fast, top-down heuristic: at each node in the hierarchy, the available budget is split among children in proportion to their aggregate demand, recursing until every device has an allocation. Because each split uses only local subtree information, the algorithm runs in a single pass through the tree and is easy to implement. However, it has fundamental limitations compared to a global optimizer such as nvPAX: • No cross-subtree coordination. Greedy allocation considers each parent-to-child split independently, without considering capacity bottlenecks deeper or in other branches. When demand and capacity are distributed non-uniformly across the tree, a parent may allocate too much budget to a subtree that cannot use it (due to tight internal nodes) and too little to subtrees that could. Global allocation schemes such as nvPAX avoid this by solving a single optimization problem over all devices and constraints simultaneously. • No tenant SLA support. The Greedy allocation follows the tree structure only and cannot encode horizontal constraints such as per-tenant minimum power guarantees. • No fairness objective. The Greedy allocation distributes budget proportionally to demand without an explicit fairness criterion. In balanced hierarchies where every node has ample headroom relative to its children’s demands, the Greedy allocation and nvPAX produce similar allocations (as observed in Section 5). The gap between the two methods becomes significant when the hierarchy contains non-uniform capacities or demand distributions, as shown in Section A.2 below. A.1
Greedy Proportional Algorithm
In this section, we describe in detail the Greedy allocation algorithm. Each device i has bounds [li , ui ]; requests are clipped to di ∈ [li , ui ]. Write ei = di − li ≥ 0 for extra demand above the minimum, and initialize ai ← li . For each node v in the hierarchy, let Lv be the sum of li over devices in the subtree rooted at v, and Ev the sum of ei over the same subtree. Let Cv be v’s power capacity. Define the extra capacity Xv = max{0, Cv − Lv } and the feasible extra weight Wv = min(Ev , Xv ), which is the maximum additional power above the minimal allocation that the subtree can carry without violating v’s cap. The values (Lv , Ev , Wv ) are computed bottom-up. The root receives extra budget Wroot (equivalently, one distributes total power min(Croot , Lroot + Wroot )). At each node v, any positive extra budget is split among the immediate children (child nodes and devices attached to v) in proportion to their respective weights: Wc for a child node c, and ei for a device i attached directly to v. Each child node recursively receives its share as its extra budget; each device receives its share added to ai . This top-down pass respects all node capacities by construction of Wv , but does not encode tenant SLA constraints. The Greedy allocation algorithm is summarized in Algorithm 4. A.2
Non-Uniform Hierarchy
We now construct a small example in which the Greedy algorithm’s local decisions lead to a substantially worse allocation compared to nvPAX’s global optimum. The failure arises whenever demand and internal node capacities are distributed non-uniformly across the tree, so that a subtree with high aggregate demand contains a tight internal bottleneck that prevents most of that demand from being satisfied. Consider the three-rack hierarchy in Figure 4: the datacenter cap is 10 kW and total requested power is 11.95 kW. Rack A contains a tight internal server SA1 (capacity 2.5 kW) feeding six devices that each request 0.75 kW (total demand 4.5 kW, but at most 2.5 kW can be delivered through SA1 ). A second server SA2 under rack A carries three devices at 0.15 kW each. Racks B and C are symmetric: each has one 6 kW server and ten devices at 0.35 kW. All 29 devices are marked active with priority 1 so that Greedy does not use priority tiers – any gap is purely from the hierarchical proportional rule vs. the global optimum. 16
Algorithm 4 Greedy proportional allocation. Require: Devices (ri , li , ui )ni=1 ; PDN hierarchy with node capacities Cv Ensure: Allocation {ai }ni=1 1: Initialization: 2: for each device i do 3: di ← min(max(ri , li ), ui ) {clip request to [li , ui ]} 4: ei ← di − li {extra demand above minimum} 5: ai ← li {allocate minimum} 6: end for 7: 8: Bottom-up aggregation: 9: for each node P v in post-order (leaves first) do 10: Lv ← i∈subtree(v) li {sum of device minimums} P 11: Ev ← i∈subtree(v) ei {sum of extra demands} 12: Xv ← max{0, Cv − Lv } {extra capacity above minimums} 13: Wv ← min(Ev , Xv ) {feasible extra weight} 14: end for 15: 16: Top-down distribution: 17: Call D ISTRIBUTE(root, Wroot )
Algorithm 5 D ISTRIBUTE(v, b): recursive top-down budget distribution for node v with extra budget b. Require: Node v; extra budget b ≥ 0 1: if b ≤ 0 then 2: return 3: end if P P 4: Wtot ← c∈children(v) Wc + i∈devices(v) ei 5: if Wtot = 0 then 6: return 7: end if 8: for each child node c of v do 9: bc ← min b · Wc /Wtot , Wc {proportional share, capped} 10: D ISTRIBUTE(c, bc ) {recurse into subtree} 11: b ← b − bc ; Wtot ← Wtot − Wc 12: end for 13: for each device i attached directly to v do 14: si ← min b · ei /Wtot , ei {proportional share, capped} 15: ai ← ai + si 16: b ← b − si ; Wtot ← Wtot − ei 17: end for
At the datacenter level, the Greedy algorithm splits the 10 kW budget among the three racks proportionally to their feasible extra weights Wv . Rack A reports high demand through its computed Wv , so it receives a large share. However, internally, most of that budget cannot pass through the tight server SA1 : the six high-demand devices under SA1 can receive at most 2.5 kW regardless of how much rack A is given. The surplus allocated to rack A is therefore wasted, while racks B and C, whose servers have ample capacity, receive less than they could use. P P Specifically, we evaluated satisfaction ratio S = i min(ri , ai )/ i ri with identical requests for both nvPAX and the Greedy algorithm. While nvPAX achieves S = 83.26%, the Greedy algorithm achieves S = 73.94%, a gap of +9.32 percentage points. nvPAX recognizes that rack A’s demand is largely undeliverable and redirects budget toward racks B and C, where it translates into higher satisfied demand. The condition for inferior performance of the Greedy algorithm is a mismatch between aggregate demand and deliverable capacity within a subtree. This can occur when: (i) server or rack capacities 17
Datacenter CD = 10 kW
Rack A 6 kW
Rack B 6 kW
Rack C 6 kW
SA1 : 2.5 kW
SA2 : 2.5 kW
SB1 : 6 kW
SC1 : 6 kW
6×0.75 kW
3×0.15 kW
10×0.35 kW
10×0.35 kW
Figure 4: Non-uniform hierarchy example. Rack A has a tight internal server SA1 (2.5 kW capacity vs. 4.5 kW demand), while racks B and C have ample server capacity. Greedy proportional allocation over-allocates to rack A at the datacenter level because it sees high aggregate demand, but much of that budget is wasted at the SA1 bottleneck. are heterogeneous (e.g., mixed hardware generations sharing the same PDN); (ii) workload placement concentrates high-power jobs under a subset of capacity-limited servers; or (iii) aggressive oversubscription at one level creates a tight bottleneck that Greedy algorithm’s local split cannot anticipate. Whenever such a mismatch exists, a global optimizer like nvPAX can redirect power where it is actually deliverable.
B
nvPAX with Tenant SLA Constraints
The experiment in Section 5 compares nvPAX against Static and Greedy allocations under device limits and hierarchical PDN constraints only, without tenant SLA constraints. This appendix demonstrates nvPAX’s ability to enforce horizontal tenant SLA constraints – minimum and maximum aggregate power guarantees per tenant – while maintaining high device satisfaction. B.1
Experimental Setup
Tenant and SLA Generation. We generate 100 synthetic tenants, each assigned 100 GPUs (fixed throughout the test). The SLA bounds are set to 40%–80% of the tenant’s maximum aggregate power: with 100 GPUs at 700 W each, this yields Bkmin = 28,000 W and Bkmax = 56,000 W per tenant. These bounds represent a contractual guarantee that each tenant receives at least 40% of its potential capacity while not exceeding 80%, leaving headroom for other tenants under shared PDN constraints. Device Priorities. Devices belonging to tenants are assigned random priorities pi ∈ {1, 2, 3}, where higher values indicate higher priority. Phase I of nvPAX processes devices from highest to lowest priority, ensuring that higher-priority workloads receive allocations closer to their requests before lower-priority ones. Devices not assigned to any tenant retain the default priority of 1. Data and Simulation. We use the same production datacenter GPU telemetry as in Section 5. The simulation runs for 3 days (8,523 timestamps at 30 seconds spacing) over more than 12,000 GPUs across 4 halls. Device limits are li = 200 W and ui = 700 W (H100 specifications), with oversubscription factor 0.85 and idle threshold of 150 W, identical to Section 5. B.2
SLA-Specific Metrics
In addition to the global satisfaction ratio S defined in Section 5, we report tenant-level metrics. Let Tk denote the set of devices assigned to tenant k, and let Bkmin , Bkmax be the tenant’s SLA bounds. 1. Per-tenant satisfaction ratio: P Sk =
i∈Tk min(ri , ai )
P
i∈Tk ri
is the fraction of tenant k’s aggregate demand that is satisfied. 18
2. SLA margin (lower bound headroom): P Mkmin =
min i∈Tk ai − Bk . max min Bk − Bk
A value Mkmin ≥ 0 indicates that the minimum SLA constraint is satisfied; larger values indicate greater headroom above the guaranteed minimum. B.3
Results
For global satisfaction ratio: nvPAX achieved mean 98.93% (std 0.52%, min 96.34%, max 100.00%). For per-tenant satisfaction: mean across all tenants and timestamps was 99.24% (std 0.37%, min 97.46%, max 100.00%). For SLA margin: at each timestamp, we first compute the mean Mkmin across all 100 tenants. We then average this per-timestamp mean over all 8,523 timestamps, yielding 54.44% (std 0.63%, min 44.16%, max 59.99%). To evaluate the worst-case tenant, at each timestamp we find the minimum Mkmin across all 100 tenants. Averaging this per-timestamp minimum over all 8,523 timestamps yields 33.80% (min 21.70%, max 51.44%), indicating that even the most constrained tenant at any given time maintained substantial headroom above its minimum guarantee. During the test, all timestamps had all 100 tenants satisfy their SLA constraints. There were zero minimum SLA violations and zero maximum SLA violations across all timestamps and tenants. Runtime of nvPAX with SLA constraints (single optimize() call per timestamp): mean 718.83 ms, std 97.76 ms, min 535.45 ms, max 1,863.52 ms over 8,523 timestamps. This is approximately 1.7 times slower than the non-SLA experiment (264.69 ms mean) due to the additional 200 tenant constraints (one minimum and one maximum bound per tenant). B.4
Discussion
The results demonstrate that nvPAX successfully enforces horizontal tenant SLA constraints while maintaining high device satisfaction. In this generated tenant experiment, the global satisfaction ratio with SLA constraints (98.93%) is essentially the same as the non-SLA experiment in Section 5 (98.92%). Thus, for this trace and tenant construction, the additional horizontal constraints did not reduce aggregate request satisfaction, while they did enforce contractual tenant-level bounds that the Static and Greedy baselines cannot represent. The zero minimum-SLA and maximum-SLA violations across all 8,523 timestamps validate the constrained-optimization formulation for allocation under tenant SLA constraints. Moreover, the positive lower-SLA margin indicates robustness beyond bare compliance: the mean tenant lower-SLA margin is 54.44%, and even the worst tenant at each timestamp maintains a positive average margin of 33.80% above the contractual minimum. The minimum observed worst-tenant margin remains positive (21.70%), showing that the allocation did not merely satisfy SLAs at numerical tolerance but maintained nontrivial headroom throughout the run. The cost of this additional expressiveness is runtime: the SLA-constrained run takes 718.83 ms per timestamp on average, compared with 264.69 ms for the non-SLA run. This overhead is expected because the optimizer must enforce 200 additional tenant constraints (one lower and one upper aggregate bound per tenant) while also processing nontrivial job priorities.
19