arXiv CSopen access

Energy-Aware Scheduling for Serverless LLM Serving on Shared GPUs

arXiv:2606.30391v1 [cs.DC] 29 Jun 2026

Energy-Aware Scheduling for Serverless LLM Serving on Shared GPUs Tianyu Wang

Gourav Rattihalli

Aditya Dhakal

University of Pittsburgh Pittsburgh, PA, USA

HPE Labs Milpitas, CA, USA

HPE Labs Milpitas, CA, USA

Longfei Shangguan

Dejan Milojicic

University of Pittsburgh Pittsburgh, PA, USA

HPE Labs Milpitas, CA, USA

Abstract

under fluctuating demand, there is a growing trend toward co-hosting multiple LLMs on the same GPU to multiplex requests with diverse workload characteristics [6, 7, 43, 52]. At serverless scale, the energy footprint of LLM inference is projected to rise sharply as GPU fleets expand [30]. Prior study shows that for ChatGPT-like services, inference can dominate lifecycle emissions, producing around 25× the carbon emissions of training a GPT-3 class model over a year [2]. A key driver is that modern accelerators draw high power when operated near their peak performance point. For instance, an NVIDIA H100 is configurable up to 700 W Thermal Design Power (TDP) [33]. This makes power-efficient serving increasingly important: small per-GPU inefficiencies quickly compound into a large cluster-wide energy bill. In this paper, we ask: how should a serverless LLM scheduler minimize cluster energy while preserving strict TTFT/TBT SLOs in a GPU-sharing pool? Our starting point is an empirical opportunity: existing serving stacks often leave GPU operating-point control to default hardware policies, which keep GPUs near their high-power operating region even when workload intensity is low or medium (§2). As a result, many requests complete well before their Time To First Token (TTFT) and Time Between Tokens (TBT) deadlines, leaving substantial SLO slack (§2.1). This slack creates room for energy-aware scheduling. At first glance, Dynamic Frequency and Voltage Scaling (DVFS) is the most immediate mechanism for exploiting this opportunity: when a GPU has latency headroom, the serving system can lower its frequency operating point and trade a portion of that headroom for lower power. Nevertheless, the serverless, shared-GPU setting creates three forms of coupling that make simple per-instance DVFS insufficient. • Spatial coupling from co-location. In serverless, GPUsharing setups, multiple LLMs are co-hosted on the same GPU and share one device-wide frequency operating point. However, different models and request intensities differ drastically in their individual energy-minimizing configurations (§2.2). As a result, a high-frequency-preferring workload will force a low-frequency-preferring workload to run at an unnecessarily high frequency, wasting significant energy (10%16% energy waste in §C). Conversely, lowering the frequency to save energy for one model can endanger the deadlines of

As LLM inference becomes a major cloud workload, its growing energy footprint makes cluster-wide energy optimization increasingly important. Serverless LLM serving helps platforms absorb traffic volatility by elastically sharing GPU resources across models, but this sharing also makes energy optimization difficult. Multiple co-resident models run under one device-wide operating point, while their resource demands and latency slack change across execution phases and load conditions. As a result, minimizing energy requires coordinated scheduling across request placement, runtime resource adaptation, and workload consolidation. We present Festina, a profiling-guided, power-aware control plane to minimize cluster-wide energy for serverless LLM serving. Unlike common global–local schedulers that focus on throughput or tail latency, Festina makes energyfirst decisions by jointly coordinating request placement, SM partitioning, and GPU operating points under TTFT/TBT SLOs. In our system, a lightweight global scheduler performs fast, SLO-safe, energy-aware placement using constant-time lookups from offline profiles and GPU state summaries. On each GPU, a phase-aware local scheduler continuously adapts task batching and compute resources to minimize power consumption. Festina further performs energy-aware workload consolidation to reduce GPUs’ static power consumption via SLO-aware migration. Comparison with four SOTA LLM serving systems and one DVFS-augmented system demonstrates that Festina reduces energy consumption by up to 56% while maintaining parity in SLO attainment (within a 2% margin).

1

Introduction

Large language model (LLM) inference is gradually becoming a dominant workload in today’s cloud [16]. Recent studies show that these inference workloads exhibit prominent spatial and temporal variation across regions, tenants, and time of day [7, 16, 48, 51, 54]. To cope with this volatility, major platforms such as Microsoft Azure [29] and HuggingFace [15] now offer serverless LLM serving, where the platform scales GPU resources up and down automatically to save inference cost without violating latency service level objectives (SLOs) [8]. To further improve GPU utilization 1

Table 1. Comparison to related works.

another co-resident model. Therefore, the scheduler should not focus on per-GPU or per-model frequency scheduling, but account for model placement, frequency affinity, SM availability, and queue state together. • Temporal coupling from prefill/decode dynamics. An LLM request shifts from compute-heavy prefill to memorybound decode, and these two phases have different latency sensitivity, resource demand, and energy-optimal Streaming Multiprocessor (SM) and frequency configurations (§2.2). Moreover, the live batch on a GPU changes continuously as requests arrive, finish prefill, enter decode, and complete generation. A placement decision that is energy-efficient at dispatch time can quickly become suboptimal or even unsafe as the phase mix changes (§2.3). Energy-aware serving therefore requires runtime adaptation of task batching, SM partitioning, and the shared operating point, rather than a one-shot frequency decision. • Cluster-level coupling from serverless scale-in. During load valleys, consolidating work onto fewer GPUs can save substantial static and memory-system power. But consolidation is not ordinary bin packing. Moving a model would change the co-location set on both source and destination GPUs, which can recreate frequency mismatch and alter the energy-SLO tradeoff. If active requests are moved, transferring model state or KV cache also consumes time, network bandwidth, and energy. Thus, scale-in should be triggered only when the saved idle power outweighs migration overhead and when the resulting placement remains compatible in both resource demand and preferred operating point. Therefore, the core challenge is not to choose a frequency for an isolated model instance, but to coordinate placement, SM partitioning, phase-aware runtime adaptation, and scalein under a shared GPU operating point.

Spatial sharing

Dynamic reconfiguration

Energy aware

Phase aware

ServerlessLLM [8] MuxServe [7] Prism [58] GreenLLM [25] DynamoLLM [41] Dilu [27] Aegaeon [52] VoltanaLLM [57]

No Yes Yes No No Yes Yes No

Partial No Partial No Partial Yes Yes No

No No No Yes Yes No No Yes

No No No Yes No No Partial Yes

Festina (ours)

Yes

Yes

Yes

Yes

to use (§3.4). This runtime adaptation lets Festina track prefill/decode shifts and workload drift without relying on static per-model configurations. To keep control-plane overhead low at serverless scale, both global and local schedulers use an offline-profiled look-up table (LUT) that maps request features and resource configurations to latency and power (§3.2). At runtime, energy and latency reasoning reduces to constant-time table queries plus lightweight state checks. To further minimize the substantial power consumption during periods of low GPU utilization, we propose an active scale-in mechanism where the global scheduler periodically evaluates cluster-wide load to identify opportunities for workload consolidation and adaptively triggers an LLM replacement to deactivate underutilized GPUs by co-locating models with similar optimal frequencies (§3.5). To avoid disrupting ongoing inference, we further design a latency-aware migration algorithm that moves active requests only when their remaining execution time exceeds migration overhead. We implement Festina based on vLLM [46]. Experiments based on industrial traces show that Festina can save up to 56% energy while maintaining parity in SLO attainment (within a 2% margin) in comparison to the state-of-the-art (SOTA) LLM serving systems. To put this in perspective, in an industry AI cloud of 10,000 NVIDIA H100 GPUs, Festina could potentially save over 2.8 million kWh of electricity per month: an amount equivalent to the monthly energy consumption of approximately 2,600 average U.S. households. Sensitivity analysis confirms the robustness of Festina across diverse workload compositions (e.g., prefill-heavy, decodeheavy) and validates its feasibility of energy-efficient execution under a prefill-decode disaggregated architecture. To the best of our knowledge, Festina is the first serverless LLM serving system that explicitly targets cluster-wide energy minimization under a GPU-sharing regime by coordinating request placement, phase-aware task scheduling, SM partitioning, shared operating-point control, and scale-in migration under strict TTFT/TBT SLOs. This paper makes the following contributions:

We present Festina, a two-tier control plane for energyefficient serverless LLM serving on shared GPUs. At the cluster level (§3.3), a lightweight global scheduler performs queue-aware, SLO-safe request placement. Rather than simply selecting the least-loaded GPU, it chooses a feasible GPU whose current operating point, available SMs, memory capacity, and queued work best match the incoming request. This avoids placing requests on GPUs where they would either violate SLOs or create unnecessary operating-point mismatch. However, as an LLM request quickly transitions from compute-heavy prefill to memory-bound decode, and the batch on a GPU changes as other requests arrive and finish, a request placement that is energy optimal at dispatch time can soon become suboptimal mid-execution, wasting energy and increasing SLO violation risk when local contention deviates from what the global scheduler assumed. Festina bridges this gap with per-GPU local scheduler that continuously observes the live batch composition and jointly decides which prefill and decode tasks to run, how to partition SMs among them, and which shared operating point 2

Norm. latency

1485

0.4

1335

0.2

1185

GPU Index

5

6

7

30

45

60 sec

1035 1785 1635

0.8 > 52% slack

0.6

1485

0.4

1335

0.2

1185

Aegaeon 0

15

30

45

(a) Normalized latency of three co-located LLMs and real-time GPU frequency.

60 sec

1035

MuxServe

1 0.8 0.6 0.4 0.2

0

0.2

0.4 0.6 0.8 SLO slack

Aegaeon

0 1 1 0.8 0.6 0.4 0.2

0

0.2

0.4 0.6 0.8 SLO slack

0 1

(b) CDF of SLO slack.

Figure 2. (a) Normalized latency of three co-located LLMs and real-time GPU frequency. (b) CDF of SLO slack for three LLMs during serving requests.

• We formulate energy-efficient serverless LLM serving on shared GPUs as a coupled scheduling problem, where request placement, SM partitioning, phase-aware batching, shared operating-point control, and consolidation should be jointly decided under TTFT/TBT SLOs. • We design Festina, a multi-timescale control plane that combines queue-aware global placement, phase-aware local batching, joint SM/operating-point adaptation, and frequency-affinity-aware consolidation with migrationcost checks and efficient transitions. • We implement Festina on vLLM and evaluate it on an H100 testbed with industrial LLM serving traces. Festina cuts cluster energy consumption by up to 56% over SOTA serving systems while preserving SLO attainment within 2%.

used by these systems, GPUs frequently remain close to the TDP-limited frequency of 1635 MHz, despite substantial variation in the workload intensity. To understand whether this high operating point is necessary for meeting latency constraints, we measure request latency under the same setup. Figure 2(a) shows the normalized serving latency of three co-located LLMs, where a value below one means that the request completes within its SLO. Across the trace, all three models remain well below the SLO boundary. Under MuxServe, even the heaviest model, Mistral-24B, remains at least 24% below the SLO limit at its worst point; Qwen-7B and Llama2-13B exhibit even larger margins for most of the trace. Aegaeon shows greater slack, with more than 52% headroom, due to its prefill-decode disaggregated architecture. Figure 2(b) further shows that this slack persists throughout the trace rather than appearing only during short idle intervals. This behavior suggests an important opportunity. Existing serving stacks are optimized primarily for throughput, utilization, or SLO attainment, and typically leave GPU operating-point control to default hardware policies. As a result, the GPU can continue operating near a high-power point even when requests already finish well before their TTFT/TBT deadlines. Such unused latency headroom creates room for energy-aware scheduling: the serving system can trade part of the slack for lower power, as long as doing so does not increase queuing delay enough to violate SLOs.

Motivation: Energy Coupling in Shared-GPU LLM Serving

In this section, we conduct measurement studies to explain why energy-efficient serverless LLM serving cannot be reduced to standalone DVFS. We first show that existing serverless GPU-sharing systems leave substantial SLO slack while GPUs remain near their high-power operating region, which creates an opportunity for energy-aware control. We then demonstrate that GPU frequency affects energy in the expected U-shaped manner, consistent with prior DVFS studies [25, 41, 57]. However, our key finding is that the energyoptimal configuration shifts with the model, workload density, execution phase (i.e., prefill/decode), and SM allocation. These shifts make energy optimization a coupled scheduling problem over model placement, SM partitioning, phaseaware runtime adaptation, and just-in-time scale-in. 2.1

15

(SLO) 1

0

Figure 1. The frequency distribution of eight GPUs in MuxServe [7] and Aegaeon [52] setups. They all stay at a high 1635 MHz throughout the model serving session.

2

MuxServe 0

Norm. latency

SM Clock (MHz) SM Clock (MHz)

4

1635

CDF

3

1785

0.6

0

2

Freq.

CDF

1

Mistral3.1-24b

Frequency (MHz)

0

Llama2-13b

> 24% slack

0.8

(b) Aegaeon 1600 1500 1400 1300

Qwen2.5-7b

(SLO) 1

Frequency (MHz)

(a) MuxServe 1600 1500 1400 1300

Observation One: Serverless GPU-sharing workloads often have substantial SLO slack while GPUs remain near high-power operating points, creating room for energyaware scheduling.

SLO Slack Creates Room for Energy-Aware Scheduling

We begin by examining how existing serverless GPU-sharing systems use GPU operating points under realistic serving workloads. We run MuxServe [7] and Aegaeon [52] using MuxServe’s traces and record GPU frequencies across eight NVIDIA H100 GPUs in a serverless serving setup. Figure 1 shows a clear pattern: under the default hardware policy

2.2

Energy-Optimal Operating Points Shift Across Workloads and Phases

The previous result shows that serverless LLM requests often have latency slack. A natural next question is whether changing the GPU operating point can safely convert this 3

sweet spot bs=128, seq=128 bs=128, seq=512 Qwen2.5:7b -- Prefill

1.8

1.3

1.6

1.2

1.4

1.1

1.2

sweet spot in=1024, out=128 in=4096, out=128 Qwen2.5:7b -- Decode

Norm. energy

Norm. energy Norm. energy Norm. energy

1.4

1 1 345 525 705 885 1065 1245 1425 1605 1785 345 525 705 885 1065 1245 1425 1605 1785 MHz MHz 1.4 1.8 Llama2:13b -- Prefill Llama2:13b -- Decode 1.3

1.6

1.2

1.4

1.1

1.2

1.4 5 1.3

1.2 1.1

1.1 5 1 1 345 525 705 885 1065 1245 1425 1605 1785 345 525 705 885 1065 1245 1425 1605 1785

Frequency

MHz

Frequency

MHz

Figure 3. Impact of GPU frequency on energy consumption. Across all three LLMs, energy consumption follows a Ushaped curve during both prefill and decode. Red circles mark the energy-optimal frequency, while annotations indicate the remaining SLO slack at those operating points.

0.8 0.7

S3

17% 8% (512, 128) (1024, 128) (4096, 512) (input length, output length)

Observation Two: Consistent with prior DVFS studies, LLM serving exhibits a U-shaped energy-frequency tradeoff. The key challenge in serverless GPU sharing is that the energy sweet spot shifts across models, phases, and workload intensity, creating conflicts among co-resident workloads that share one device-wide operating point.

slack into energy savings. Following common DVFS characterization methodology [41, 54, 57], we profile phase-wise energy consumption of representative LLMs under different GPU frequencies and workload intensities. We use the same models as before and measure both prefill and decode energy under different input/output lengths. Figure 3 shows that energy follows the expected U-shaped trend with respect to GPU frequency. At low frequencies, execution time increases and dominates total energy. While at high frequencies, power increases and dominates total energy. The minimum (i.e., sweet spot) appears at an intermediate frequency, reflecting the classic power–time tradeoff: 𝐸𝑛𝑒𝑟𝑔𝑦 = 𝑃𝑜𝑤𝑒𝑟 × 𝑇 𝑖𝑚𝑒

0.9

S2

Figure 4. The normalized energy for three distinct settings under various workloads (i.e., input length). Accordingly, when multiple LLMs are co-hosted on the same GPU, each individual model’s energy sweet spot can conflict because the GPU exposes one device-wide operating point. As a result, an operating point that saves energy for one model or phase may waste energy for another, while an overly aggressive downshift may slow co-resident work enough to threaten SLOs.

1 1 345 525 705 885 1065 1245 1425 1605 1785 345 525 705 885 1065 1245 1425 1605 1785 MHz MHz 1.4 1.6 Mistral3.1:24b -- Prefill Mistral3.1:24b -- Decode 1.3

S1

1

2.3

Frequency Scaling Alone Is Insufficient Under GPU Sharing

The previous subsection shows that frequency is an important actuator. We now ask whether frequency control alone is sufficient once multiple LLMs share a GPU. In a GPUsharing setting, SM partitioning is another primary determinant of execution time because it controls how much compute budget each co-resident workload receives. Frequency and SM allocation are therefore coupled: lowering frequency reduces dynamic power but can increase execution time, while changing SM allocation changes the latency impact of any frequency choice. To quantify this coupling, we compare three configurations that progressively expand the optimization scope: • S1: Performance-first baseline. We select the SM partitioning that minimizes execution time and leave GPU frequency under default hardware control. • S2: Frequency-optimized. We keep the latency-optimal SM partitioning from S1, but tune GPU frequency to minimize energy while preserving SLOs. • S3: Jointly optimized. We jointly select SM partitioning and GPU frequency to minimize energy under the same SLO constraints. Figure 4 plots normalized energy under these configurations. Comparing S1 and S2 confirms that frequency scaling remains useful in GPU-sharing scenarios, reducing energy by 17% in our measured setting. However, comparing S2 and S3 shows that frequency scaling alone leaves additional savings on the table: jointly reconfiguring SM partitioning and frequency reduces energy by another 8%. This additional gain

(1)

This U-shaped behavior is consistent with prior DVFS studies [41, 54, 57] and selecting the energy-minimizing operating point yields significant savings in Figure 3: at least 16% for prefill and 27% for decode compared to the TDP limit, while still satisfying latency SLOs. However, the real challenge is that for shared-GPU serving, the sweet spot is not fixed. We conclude three sources of variation from Figure 3: • Workload sensitivity. For the same LLM, as workload intensity increases, e.g., with longer input sequences, the energy-minimizing operating point shifts toward higher frequencies because the workload becomes more computeintensive and benefits more from higher throughput. • Phase sensitivity. The prefill phase generally prefers a higher frequency than the decode phase for the same LLM, which aligns with the compute-heavy nature of prefill and the memory-bound nature of decode [25]. • Model sensitivity. Larger models tend to favor higher frequencies than smaller models under comparable serving conditions because they impose heavier compute demand. 4

• Local Tier: Joint SM-Frequency, Phase-Level Adaptation (§3.4). Because an LLM request shifts from computeheavy prefill to memory-bound decode, the energy-optimal SM/frequency setting can change during execution. Each GPU runs a local scheduler that monitors the live batch mix and dynamically adjusts SM partitioning and GPU frequency to track these phase changes without violating SLOs. • Cluster-Level Consolidation: Active Scale-In (§3.5). To reduce cluster-wide static power (e.g., HBM) under low load, Festina periodically re-evaluates placement and consolidates workloads onto fewer GPUs, deactivating underutilized devices when possible while preserving frequency affinity. When consolidation requires moving in-flight work, we use an SLO-aware, pipelined migration mechanism to minimize service disruption (details in §F). 3.2

Figure 5. System overview of Festina. Festina adopts a twotier hierarchy for energy-aware serverless LLM serving.

3.2.1 Look-Up Table Construction. Prior to runtime deployment, Festina constructs a high-fidelity look-up table that records the inference latency and power metrics of each LLM under different request intensity and GPU hardware resource configurations, as shown in Figure 6. We perform a comprehensive grid search across three primary dimensions: GPU clock frequency, Streaming Multiprocessor (SM) allocation, and input sequence length. Profiling Grid and Bounds. We sweep GPU clock frequencies from 795 MHz to 1635 MHz with a stride of 90 MHz. The lower bound of 795 MHz is chosen based on findings from EVeREST [59], which demonstrate that reducing frequency below this threshold yields negligible voltage reduction, thereby halting energy savings. The upper bound corresponds to the hardware’s TDP for the specific GPU type (e.g., 1635 MHz for NVIDIA H100 GPU), consistent with prior energy-aware frameworks [41, 59]. We also vary SM allocation from 10% to 100% with a stride of 10%. To ensure our profiling captures representative production demands, we define the input sequence range based on real-world LLM workload traces from Azure [41], sweeping from 256 to 8192 tokens (same as DynamoLLM [41]) to encompass the vast majority of standard inference requests. Metric Collection and Interpolation. For each configuration tuple (Frequency, SM%, Input Length), we record three critical metrics: Time To First Token (TTFT), Time Between Tokens (TBT), and average power consumption. We also quantify the one-time latency of loading model weights into GPU memory. To estimate performance for input lengths not explicitly covered by our grid, we employ polynomial regression, a standard technique for continuous resource modeling in LLM serving [13, 41, 49]. Profiling Overhead and Reusability. Note that offline profiling is a one-time, low-overhead setup step. In our implementation, the full profiling procedure completes in under 40

appears because changing SM allocation shifts the latencyenergy tradeoff and can move the energy-minimizing frequency itself. More importantly, our experiments also show that the energy-optimal configuration is not static. For instance, under the workload (1024 input tokens, 128 output tokens), the ideal (SM partitioning, frequency) setting shifts from (30-3040 SMs, 1155 MHz) in S2 to (20-40-40 SMs, 1065 MHz) in S3. In real-world LLM serving, workloads are inherently dynamic: request rates fluctuate and token lengths vary continuously. Observation Three: Significant energy-saving potential exists in GPU-shared environments, but maximizing efficiency requires the co-reconfiguration of SM partitioning and frequency and the adaptation to workloads.

3

Festina Design

As shown in Figure 5, Festina uses a two-tier hierarchy for energy-aware serverless LLM serving: a lightweight global scheduler makes fast, SLO-safe energy-aware placement decisions across the GPU pool, while a per-GPU local scheduler performs phase-aware runtime adaptation by jointly tuning SM partitioning and GPU operating point to save energy. Both tiers are unified by the same offline-profiled look-up table (§3.2), turning energy and latency reasoning into constanttime queries that scale with cluster size. 3.1

One-Time Offline Profiling

Our offline profiling consists of Look-Up Table construction and output token length predictions.

System Overview

• Global Tier: Light-Weight, Frequency-Aware Dispatching (§3.3). On each request arrival, the global scheduler uses the LUT and compact per-GPU state summaries to select an SLO-feasible GPU whose current operating point best matches the request’s predicted intensity. This keeps dispatch overhead low while avoiding energy waste from frequency-mismatched execution. 5

Look-up table (LUT) KV per Frequency Prefill SM% Length TTFT (ms) token (KB) (MHz) power (W) 1015 256 … … 20 1635 256 … … Mistral80KB nemo:12b 1015 1024 … … 30 1635 1024 … … LLM

GS

Reqmem

LS

LS

220 < 200

SLO checking 152 < 200

… … … …

this, we propose a frequency-aware dispatcher that routes each request to a GPU whose current clock frequency best matches the request’s workload, while still meeting the SLO. As illustrated in Figure 6, the dispatcher works in three phases: (i) feasibility checking, (ii) GPU selection, and (iii) resource scaling. We elaborate on each phase below. • Phase One: Pooling-based Feasibility Checking. When a request arrives, the global dispatcher first predicts its output token length and adds a 5% safety margin to account for estimation error (§3.2.2). With request input length 𝑅𝑒𝑞𝑖𝑛 and this padded length 𝑅𝑒𝑞𝑜𝑢𝑡 , it derives the request’s memory demand, denoted as 𝑅𝑒𝑞𝑚𝑒𝑚 . Next, the dispatcher polls all local schedulers to determine which GPUs can accommodate the request. Each local scheduler maintains the state of its running and queued requests and uses the offline-profiling look-up table (§3.2) to translate these queued workloads into a time-varying estimate of SM and memory utilization. From this local view, the scheduler computes (1) the Earliest Available Timestamp 𝑇𝑎𝑣𝑎𝑖𝑙 at which at least 𝑅𝑒𝑞𝑚𝑒𝑚 memory can be allocated, and (2) the number of free SMs available at that time, 𝑆𝑀 𝑓 𝑟𝑒𝑒 . It then returns (𝑇𝑎𝑣𝑎𝑖𝑙 , 𝑆𝑀 𝑓 𝑟𝑒𝑒 ) to the global dispatcher. Finally, the global dispatcher performs an SLO feasibility check for each GPU. A GPU is considered feasible only if a request starting at𝑇𝑎𝑣𝑎𝑖𝑙 , with 𝑆𝑀 𝑓 𝑟𝑒𝑒 SMs, can complete before its SLO deadline under the GPU’s current clock frequency 𝐹𝑐𝑢𝑟𝑟 . This phase ensures that subsequent energy-oriented decisions are made only among candidates that do not compromise SLO compliance. As shown in Phase 1 of Figure 6, only GPUs with estimated timestamp < 200 (i.e., SLO target) are feasible for the next phase. • Phase Two: Energy-Minimal GPU Selection. Given the feasible set of candidates C, the global dispatcher chooses the target GPU that minimizes the request’s expected energy consumption at the GPU’s current operating frequency. For each candidate 𝐺𝑃𝑈𝑘 ∈ C with current frequency 𝑓𝑘 and SM availability 𝑆𝑘 , the dispatcher estimates the energy to serve the request as:

Decode power (W) … … … …

energy comparison

LS

Tavail , SMfree

TBT (ms)

Skip

42

30

select

Phase2: Minimum-Energy Selection pass all fail

Sort by Tavail & SLO checking (TDP) select fail

SLO = 200 185 < 200

Phase1: Feasibility Checking GS Global Scheduler

If SLO checking fails for all activate a new GPU new select

Phase3: Fallback and Scaling LS Local Scheduler

Figure 6. Dispatching flow of the global scheduler. minutes on an NVIDIA H100. Although profiling is hardwareand model-specific, it is required only once per GPU type and model. Moreover, serverless providers typically offer a small, fixed set of GPU SKUs and model types; therefore, the total number of distinct profiles needed in practice is limited.

Portion (%)

3.2.2 Output Token Length Prediction. To accurately estimate total inference latency and determine if a specific GPU frequency can satisfy a request’s SLO, Festina should account for the duration of each task’s prefill and decode phase. Prior works [18, 40, 41, 65] have shown that the output token length is highly predictable. We thus borrow the same approach to predict the output token length using a highaccuracy output length predictor [41, 65]. Figure 7 illustrates the distribution of prediction errors of two LLMs under real-world LLM workload traces [41, 64]. As observed, the pre50 dictor exhibits a tight 40 error bound, with the 30 vast majority of pre20 10 dictions deviating by 0-10 -5 -2.5 +2.5 +5 +10 less than 5% from the Error deviation (%) ground truth (only 2% Qwen3-8B Chatglm-6b of predictions have an Figure 7. Distribution of relaerror > +5%). Suggested tive error percentages, categoby these results, we rized by deviation ranges. provide a modest extra 5% memory buffer, which sufficiently accommodates this variance without incurring significant over-provisioning waste. With this, the global scheduler can pre-calculate the expected latency for any given frequency-SM configuration, ensuring that dispatching decisions are both energy-efficient and SLO-compliant. 3.3 The Global Scheduler: Light-Weight, Frequency-Aware Request Dispatching

𝐸𝑘 = Power(𝑓𝑘 ) × Latency(𝑓𝑘 , 𝑆𝑘 , 𝑅𝑒𝑞𝑖𝑛 , 𝑅𝑒𝑞𝑜𝑢𝑡 )

(2)

where Power(·) and Latency(·) are obtained from the lookup table. The scheduler dispatches the request to the GPU 𝑘 ∗ that minimizes the energy cost:   𝑘 ∗ = argmin 𝐸𝑘 (3) 𝑘∈C

In Phase 2 of Figure 6, the GPU with lower energy consumption (i.e., 30 Joules) is selected to process the request. This objective implicitly favors frequency alignment. Because energy exhibits a convex dependence on frequency (§2.2), the estimated energy 𝐸𝑘 increases when a candidate’s current frequency 𝑓𝑘 is far above or below the request’s energy-optimal operating point (i.e., frequency sweet spot), making such mismatches less likely to be selected.

The energy-optimal GPU frequency depends strongly on the request’s workload intensity: running at a mismatched frequency can substantially increase energy consumption, while running too slowly risks SLO violations. To address 6

Sort by (priority) Score Index Score Mem. (MB) SM (%) D1 3.8 400 20 D3 3 500 20 D2 2.8 650 20

In above equation, Latency(𝑡𝑎𝑠𝑘) is the profiled execution time of the scheduling unit (one decode step for decode tasks, and a prefill unit for prefill tasks), and Slack𝑡𝑎𝑠𝑘 is the remaining time budget to the relevant SLO deadline (i.e., TBT for decode; TTFT for prefill). Age𝑡𝑎𝑠𝑘 is the time a task has waited since becoming runnable. We set 𝛽 = 0 for decode tasks (they are continuously active and repeatedly re-evaluated), and use 𝛽 = 1 for prefills so that waiting gradually reduces effective slack, preventing starvation. Task Batching. Using this unified score, the scheduler maintains a single priority queue containing both runnable decodes and pending prefills. At each epoch, it constructs the next batch by selecting tasks in descending Score𝑡𝑎𝑠𝑘 order subject to the GPU’s instantaneous memory capacity and the chosen SM partition. This resource-aware selection is illustrated in Figure 8. The scheduler iteratively fills the batch. However, if the remaining resources (e.g., 40% SM capacity) are insufficient for the highest-priority task (P1), the scheduler bypasses it to dispatch a fitting lower-priority task (P3). This strategy mitigates head-of-line blocking and maximizes resource occupancy. Consequently, the policy dynamically balances trade-offs: it prioritizes decodes when TBT slack is tight, yet elevates prefills as their TTFT deadlines approach, ensuring SLO compliance across both phases. Runtime Resource Allocation. Given a candidate batch, the scheduler jointly chooses GPU clock frequency and SM partitioning to minimize energy while meeting the batch’s SLO constraints. Similar in spirit to DVFS-based energy optimization [56], we exploit the fact that valid GPU clock settings are discrete and few (typically < 10 levels [41, 59]). Rather than solving a complex continuous optimization, we perform a lightweight frequency sweep, exhibited in Step 3 of Figure 8: for each candidate frequency 𝑓 , we use the offline look-up table to (1) compute, for each task in the batch, the minimum SM allocation required to satisfy its deadline at frequency 𝑓 , and (2) check whether the aggregate SM demand fits within the device budget (i.e., < 100% SMs). For feasible configurations, we estimate total energy using the profiled power and latency models, and select the frequency/partition pair with the lowest predicted energy. Because the sweep iterates over a small constant-sized frequency set and uses only a look-up table, its overhead is negligible (i.e., 0.1 ms), scaling as 𝑂 (|F | · 𝑁 ) for 𝑁 tasks and |F | frequency levels. Handle mis-predictions. In rare instances where the global scheduler underestimates output length, a request may exceed its allocated memory budget, triggering an Out-ofMemory (OOM) error. To resolve this, the local scheduler immediately assigns the highest priority to the mispredicted request. If memory remains insufficient to complete this request, the local scheduler evicts the running request that incurs the minimum recomputation overhead to free up space.

Index Score Mem. (MB) SM (%) P1 2.4 800 50 P3 2.1 500 40 P2 2 1400 70

Step 1 & 2: Priority computation & Priority-based task batching Freq. (MHz) SM(P1) SM(D1) SM(D2) SM(D3) < 100% SMs Energy (J) 900 50 20 30 20 No 36 1100 40 20 20 20 Yes 40 1300 30 20 20 20 Yes 48

Step 3: Runtime energy-minimizing frequency and SM adjustment

Figure 8. Working flow of the local scheduler. It should be noticed that when multiple candidates have near-identical energy (e.g., within 2%), Festina uses a stabilityoriented tie-breaker: it first selects the GPU with the most free memory to reduce out-of-memory risk, and then the GPU with the largest 𝑆𝑀 𝑓 𝑟𝑒𝑒 to improve throughput. • Phase Three: Fallback and Scaling. If the set of candidates C is empty, which means no GPU can meet the SLO at its current frequency, the global dispatcher relaxes the frequency constraint. It orders GPUs by the earliest available timestamp and evaluates feasibility assuming each GPU boosts to its TDP-limited maximum frequency (𝐹𝑚𝑎𝑥 = 1635 MHz). The request is dispatched to the first GPU capable of meeting the SLO under 𝐹𝑚𝑎𝑥 . If the deadline remains infeasible even at 𝐹𝑚𝑎𝑥 , Festina triggers a scale-out action to activate a new GPU, as exhibited in Phase 3 of Figure 6. For this newly activated GPU, the global scheduler queries the LUT to identify the optimal frequency, minimizing energy usage without violating SLOs. This scheduling process has negligible overheads (< 1 ms for 10,000 GPUs) in §D. 3.4

The Local Scheduler: Joint Frequency- SM Adaptation for Energy Efficiency

The local scheduler deployed on each GPU decides which prefill and/or decode tasks to run next and adjusts SM partitions and GPU clock for those tasks, performing fine-grained resource management to optimize energy efficiency. Figure 8 illustrates the working flow of the local scheduler. Motivated by prior LLM serving work on stall-free scheduling [3, 7, 9, 17, 36], the local scheduler allows out-of-order interleaving between prefill and decode so that a request’s short decode steps can still make progress even when another request’s long prompts are present. The key challenge is that these two task types operate on very different time scales: decode deadlines are at the millisecond level, whereas prefill deadlines are typically at the second level. Unified Priority Score. To compare these heterogeneous tasks directly, we normalize them using a single urgency metric: the Slack Consumption Ratio (Score𝑡𝑎𝑠𝑘 ). It measures the fraction of the remaining time budget a task would consume if scheduled next; higher values indicate higher urgency : Score𝑡𝑎𝑠𝑘 =

Latency(𝑡𝑎𝑠𝑘) Slack𝑡𝑎𝑠𝑘 − 𝛽 · Age𝑡𝑎𝑠𝑘

(4) 7

Estimate SMl Estimate Ml

Phase 1: Resource Profiling 25% 25% 20% 55%

50%

Table 2. TTFT/TBT SLOs from DynamoLLM [41]. 20%

35%

35%

Short Medium Long

Phase 2: Capacity Planning safety margin

BestFit SMeff Min #GPUs

Meff Nmin = Max(NSM = 1, Nmem = 2)

Meff

1325 (MHz)  1250  1200  1125 EWR Place

freq. = 1325

freq. = 1250

freq. = 1325 freq. = 1325

freq. = 1250 mem. EWR swap

freq. = 1250 freq. = 1250

Place

Fallback freq. = 1325

swap freq. = 1250

Figure 9. EWR-aware LLM placement reconfiguration. 3.5

TBT SLO

< 256 < 1024 ≤ 8192

< 100 < 350 ≥ 350

250 ms 400 ms 2000 ms

100 ms 100 ms 100 ms

3.5.2 EWR-Aware Workload Replacement. Our algorithm (Algo.1 in §E) proceeds in three phases to minimize active GPUs while maintaining a low EWR, as shown in Figure 9 (explanation for the running example in §E). Phase One: Profiling & Resource Estimation. For each active LLM 𝑙, the local scheduler calculates the preferred configuration (𝑆𝑀𝑙 , 𝐹𝑙 ) that minimizes energy while meeting SLOs, based on the average input/output lengths in the past time window (i.e., Δ = 5 minutes). The detailed computation process is listed in §E. Simultaneously, it estimates the memory requirement, 𝑀𝑙 , using Little’s Law [32] to avoid static over-provisioning. Specifically, the local scheduler calculates the expected request concurrency using the formula 𝑁𝑐𝑜𝑛𝑐 = 𝜆 · W, where 𝜆 is the measured request arrival rate and W is the average processing latency given the GPU configuration (𝑆𝑀𝑙 , 𝐹𝑙 ). The total memory pressure is then derived by summing the static model weights and the KV cache required for 𝑁𝑐𝑜𝑛𝑐 active sequences, expressed as: 𝑀𝑙 = Weights + 𝑁𝑐𝑜𝑛𝑐 · Size𝐾𝑉 . Phase Two: Global Capacity Planning. Before choosing specific GPUs, the global scheduler computes the minimum number of GPUs (𝑁𝑔𝑝𝑢 ) required to serve the aggregate workload. However, naïve packing strategies are susceptible to resource thrashing, where minor fluctuations in LLM workloads (e.g., request rate) trigger a scale-out shortly after consolidation, negating energy savings and incurring coldstart penalties. To mitigate resource thrashing, we reserve a safety margin (e.g., 𝛿 = 5% in this work) for both SMs and memory on active GPUs. We define the effective capacity as 𝑆𝑀𝑒 𝑓 𝑓 = (1 − 𝛿) · 𝑆𝑀𝑐𝑎𝑝 and 𝑀𝑒 𝑓 𝑓 = (1 − 𝛿) · 𝑀𝑐𝑎𝑝 . Then, a bin-packing lower bound is solved for 𝑆𝑀𝑙 and 𝑀𝑙 with respect to 𝑆𝑀𝑒 𝑓 𝑓 and 𝑀𝑒 𝑓 𝑓 , and the maximum is used as 𝑁𝑔𝑝𝑢 . This step identifies the smallest feasible active GPU pool for the current workload while retaining headroom to absorb minor workload fluctuations without triggering scale-out. Phase Three: Workload Replacement. Given 𝑁 gpu active GPUs, the scheduler places LLMs to reduce frequency skewness by grouping models with similar preferred frequencies. It sorts all LLMs by 𝐹𝑙 and assigns them in order. For each

Place freq. = 1325

TTFT SLO

EWR = 0 indicates perfect frequency alignment (no waste), while larger values imply increasing energy loss due to consolidation-induced frequency skew.

mem.

EWR-driven swapping

Output

We then derive the Energy Waste Ratio as a normalized inefficiency metric: 𝐸 waste EWR = (7) 𝐸 optimal

Phase 3: EWR-Minimization Placement

Sort by freq. Direct placement

S M L

Input

Energy-Aware Workload Replacement

To further reduce the cluster-wide energy overhead, Festina periodically consolidates workload by migrating LLM workloads off lightly loaded GPUs onto more utilized ones and deactivates the freed GPUs. To make consolidation energyaware, we introduce a metric called Energy Waste Ratio (EWR) (§3.5.1) and then design an EWR-aware reconfiguration algorithm that consolidates aggressively while minimizing the energy penalty from frequency misalignment (§3.5.2). 3.5.1 Energy Waste Ratio (EWR). We quantify consolidation-induced energy waste as the gap between the power implied by a GPU’s actual operating frequency after colocation and the power each LLM would ideally consume at its own energy-optimal frequency. Since GPU dynamic power scales approximately cubically with frequency (𝑃 ∝ 𝑓 3 ) [10, 11, 25, 50], the cluster-wide energy waste can be expressed (up to a constant factor) as: ∑︁ ∑︁ 𝐸 waste = 𝑡𝑙 · (𝐹𝑔3 − 𝐹𝑙3 ) (5) 𝑔∈GPUs 𝑙 ∈𝑔

where 𝐹𝑔 represents the operating frequency of GPU 𝑔 (determined by the highest frequency of co-located LLMs); 𝐹𝑙 is the desired energy-optimal frequency of LLM 𝑙; 𝑡𝑙 is the model execution time. As a reference, we define the idealized energy if every LLM ran at its preferred frequency: ∑︁ ∑︁ 𝐸 optimal = 𝑡𝑙 · 𝐹𝑙3 (6) 𝑔∈GPUs 𝑙 ∈𝑔 8

LLM, it identifies a Target GPU (the active GPU with the closest operating frequency) and a Candidate GPU (the second closest) based on the following rules: • Direct Placement: If the Target GPU satisfies resource constraints (≥ 𝑆𝑀𝑙 , 𝑀𝑙 ), the LLM is assigned here. This minimizes 𝐹𝑔3 − 𝐹𝑙3 , keeping EWR low. • EWR-Driven Swapping: If the Target GPU is saturated, the scheduler attempts to migrate an "outlier" incumbent (the resident LLM with the lowest 𝐹𝑙 ) to the Candidate GPU. If the swap is resource-valid and reduces the total 𝐸 waste , the outlier is moved to accommodate the new LLM. • Fallback: If prior actions fail, the scheduler iteratively checks the next closest GPUs in the sorted list and assigns the LLM to the first GPU found with sufficient resources. We give a running example of phase three in §E. The implementation of minimizing communication cost during workload placement is detailed in §F. When to trigger workload replacement? We trigger a replacement check when an instance’s keep-alive timer expires, indicating an LLM has become idle and can be evicted without disruption. A replacement is performed only if capacity planning (with a safety margin for small fluctuations) shows the workload can be served with fewer GPUs (𝑁 gpu < 𝑁 active ), enabling scale-in with low control-plane overhead.

4

LLMs from various families, including Qwen, Llama, Yi, and Mistral. Models range from 3B to 14B parameters, consistent with prior studies [7, 52]. We evaluate our system using the ShareGPT dataset. Following prior works [7, 8, 27, 52, 53], we generate workloads with a scaled Poisson process and random sampling from the dataset. To stress-test Festina under prefill-heavy and decode-heavy scenarios, we also use two variants from Aegaeon [52]: ShareGPT-ix2 (2× input length) and ShareGPT-ox2 (2× output length). Metrics. We measure energy consumption and SLO attainment, adopting SLO constraints from DynamoLLM [41] (Table 2). Requests exceeding 8192 tokens are excluded. Energy is profiled from the first request arrival to the final completion. We report Normalized Energy Consumption, calculated as the total energy consumed by a baseline system divided by that of Festina for the same workload. A value greater than 1.0 indicates higher energy usage than Festina. Baselines. We compare Festina against four open-sourced SOTA serving systems representing the latest works in serverless auto-scaling, GPU multiplexing, and PD-disaggregation. • ServerlessLLM [8]: A specialized serverless framework designed for fast auto-scaling. It minimizes cold-start latency through rapid checkpoint loading and implements efficient live migration mechanisms. • MuxServe [7]: A static multiplexing system that optimizes SM partitioning and model placement offline. To ensure a fair comparison with our dynamic approach, we extended MuxServe to support co-locating more than two models per GPU and added a keep-alive mechanism to offload idle models to host memory. • Dilu [27]: A dynamic serving system that optimizes cluster utilization by scheduling models according to real-time demand. It employs on-the-fly SM re-partitioning to minimize latency for co-located workloads. • Aegaeon [52]: The state-of-the-art system for prefill-decode disaggregated serverless serving. Aegaeon decouples prefill and decode phases onto separate nodes, using fine-grained KV cache management to reduce data transmission overhead and MPS-based sharing to maximize GPU occupancy. • AegVoltana: We augment Aegaeon with VoltanaLLM [57] to construct a DVFS-aware baseline. Because VoltanaLLM lacks GPU multiplexing support, Aegaeon handles request dispatching and model placement, while VoltanaLLM governs per-GPU runtime frequency scaling. • Festina+: Festina+ simulates the effects of dynamic HBM frequency scaling (a feature not yet supported on NVIDIA H100) during idle periods. We include this to establish an upper bound on the potential energy savings our system can achieve through memory frequency adjustments after scale-in LLM placement reconfiguration in Section 3.5. Although DynamoLLM [41], GreenLLM [25], and VoltanaLLM [57] are closely related in spirit, they are not designed

Evaluation

Festina is implemented in approximately 2,800 lines of Python and CUDA/C++ code. The scheduler is built using Python’s asyncio [39] library to enable high-concurrency instance orchestration. The backend is built upon vLLM [46] to benefit from state-of-the-art optimizations such as FlashAttention [5] and PagedAttention [20]. We utilize pyNVML [38] to adjust GPU frequency with minimal runtime overhead. We run experiments to answer the following questions: (1): How does Festina balance SLO attainment with energy efficiency compared to SOTA serverless LLM systems? (2): Can Festina maintain performance across different workload intensities and types? (3): What is the contribution of each design module? (4): How sensitive is the system to configuration parameters like safety margins and disaggregated setups? 4.1

Experimental Setup

Testbed. We conduct experiments on a serverless testbed consisting of eight NVIDIA H100 GPUs, interconnected via NVLink. All GPUs support Multi-Process Service (MPS) for SM partitioning. Models, Datasets, and Workloads. Our evaluation focuses on LLMs that fit on a single GPU, since Festina targets serverless GPU sharing where each GPU can co-host multiple models. We also confirm that substantial SLO slack also appears for larger, multi-GPU deployments (e.g., a 70B Llama2 model) in §B. With that, we select representative 9

Norm. energy

RPS=2

2.5

3

3.5

4

32 LLMs

RPS=0.5

1.75 1.5 1.25 1 0.75 0.5

24 LLMs

RPS=1

6000

#Requests

Norm. energy Norm. energy

1.75 1.5 1.25 1 0.75 0.5

16 LLMs

1

ServerlessLLM

1.5

MuxServe

2

Dilu

2.5

Aegaeon

1.5

2

2.5

3

32 LLMs at RPS=2.5

5500

(MHz) 1800 1600

5000

1400

4500

1200

4000

AegVoltana

0

10

Festina

20

Festina+

30

40

Request-per-minutes

50

1000 60 (mins)

Frequency

1.75 1.5 1.25 1 0.75 0.5

Figure 10. Normalized energy consumption relative to Festina under varying RPS loads and numbers of LLMs, alongside a runtime trace of GPU frequency and request arrival rate for the 32-LLM scenario at 2.5 RPS. can save at least 22% energy under a heavy-loaded scenario. If the vendors provide the function to adjust HBM frequency, our system (Festina+) can further save up to 48% energy.

for the serverless GPU-sharing setting we study, where multiple LLMs are co-hosted on a single GPU, and the scheduler must jointly consider co-location, SM partitioning, and SLOs under bursty arrivals. A detailed discussion is in §I. 4.2 Head-to-head Comparison with SOTAs

4.2.2 SLO Attainment. Next, we measure the latency SLO attainment of all systems under the same setup as the previous experiment. As shown in Figure 11, Festina outperforms ServerlessLLM, MuxServe, and Dilu by up to 27%, 16%, and 4%, respectively. Also, Festina exhibits comparable performance to Aegaeon and AegVoltana, with gaps of less than 2% and 1%, respectively. Specifically, Festina leverages MPS to allow multiple LLMs to share the same GPU and to process requests concurrently, which improves device utilization and shortens the overall execution time. However, ServerlessLLM utilizes a temporal sharing manner that each LLM occupies the entire GPU in turn, which under-utilizes GPUs and prolongs the waiting time of co-located LLMs. This leads to significant SLO violation, especially when request-per-second becomes high. MuxServe relies on static offline partitioning, preventing models from utilizing idle SMs beyond their pre-assigned quota. Dilu, despite supporting dynamic resizing, employs a First-Come-First-Served (FCFS) policy that exacerbates Head-of-Line (HOL) blocking, particularly for short decode tasks stuck behind long prefills. Festina overcomes these limitations by combining dynamic, on-the-fly SM reallocation with out-of-order task batching, effectively mitigating HOL blocking and maximizing SLO attainment. For Aegaeon and AegVoltana, they benefit from the PDdisaggregated architecture. This design inherently eliminates intra-batch interference, ensuring that latency-sensitive decode tasks are never stalled by computation-heavy prefill tasks. In contrast, Festina utilizes a PD-aggregated (colocated) setup. Consequently, it remains susceptible to minor interference when prefill and decode tasks share the same execution batch, leading to the observed marginal difference in SLO attainment.

4.2.1 Energy Consumption. Figure 10 shows the normalized energy consumption under various average request-persecond (RPS) and numbers of LLMs settings, where a value greater than 1.0 indicates higher energy usage than Festina. Festina consistently uses less energy compared to all baselines, where ServerlessLLM, MuxServe, Dilu, Aegaeon, and AegVoltana consume up to 1.56×, 1.37×, 1.36×, 1.31×, and 1.3× energy than Festina, respectively. The reason behind this energy gap is two-fold. (i): the global scheduler of Festina will dispatch requests to GPUs based not only on SLO constraints but also on frequency preference. This means co-located LLMs will process their requests at a frequency that won’t hugely deviate from their own optimal frequency, while other baselines don’t employ such a frequency-aware dispatching mechanism. (ii): these baselines target higher SLO attainment, which means they adjust resource allocation to speed up inference latency without optimizing energy efficiency. In contrast, the local scheduler of Festina dynamically reconfigures not only GPU frequency but also SM allocation to meet SLO constraints and minimize energy consumption. Although AegVoltana adjusts per-GPU frequency based on SLO slack, it lacks frequency-aware request dispatching and model placement. As a result, models with mismatched frequency preferences are colocated on the same GPU and forced to operate at a frequency dictated by their neighbors rather than their own energy-optimal point. This limited energy reduction underscores the importance of jointly SLO- and frequency-aware scheduling, which is the core capability of Festina. We further plot the GPU frequency of all systems with respect to the workload variation in Figure 10. As shown, the GPU frequency in Festina closely follows the workload pattern up and down, whereas all other baseline systems keep the GPU frequency at a consistently high level across the entire one-hour observation window. In summary, Festina

4.3

Ablation Study

Next, we quantify the contribution of each design part toward energy efficiency, isolating six distinct design features: 10

SLO attainment

MuxServe

Dilu

Aegaeon

AegVoltana

Festina 100%

90% 80% 70% 60%

90%

RPS=2

2.5

16 LLMs 3 3.5

4

RPS= 1 1.5

24 LLMs 2 2.5

3

RPS= 0.5 1

32 LLMs 1.5

2

2.5

80% 70% 60%

SLO attainment

ServerlessLLM 100%

ShartGPT-ix2

100%

1.4

100%

1.3

90% 85%

95%

80%

RPS=2

2.5

3

3.5

4

ShartGPT-ix2

95%

1.3

95%

1.3

1.2

90%

1.2

90%

1.2

1.1

85% 80%

1.1 1

85% 80% 100% 95%

32 LLMs 1.4 1.3 1.2 1.1

1

RPS=1

1.5

2

2.5

ShartGPT-ox2

3

100%

95%

95%

24 LLMs 1.4 1.3

90% 85%

1.2 1.1

90% 85%

1.2 1.1

90% 85%

1

80%

1

80%

RPS=2

2.5

3

3.5

SLO curve:

4

1.4

100%

16 LLMs 1.4 1.3

80%

ShartGPT-ix2

1.4

24 LLMs

Aegaeon

RPS=1

1.5

Festina

2

2.5

3

Energy bar:

RPS=0.5

1

1.5

2

ShartGPT-ox2

RPS=0.5

1

1.5

32 LLMs

2.5

2

2.5

1.1 1

1

Norm. energy

ShartGPT-ox2

100%

16 LLMs

Norm. energy

SLO attainment SLO attainment

Figure 11. SLO attainment under various average request-per-second (RPS).

Aegaeon (normalized to Festina)

Average norm. energy

Figure 12. SLO attainment and normalized energy consumption under various RPS and # of LLMs. 1. SLO-aware Dispatching: Route requests based on resource availability and deadline constraints. 2. Frequency-aware Dispatching: Route requests to match frequency affinities while maintaining SLO compliance. 3. Fine-grained Runtime Management: Dynamically adjust SM partitioning and GPU frequency. 4. SLO-aware Placement: Reallocate LLMs across GPUs based on resource demands. 5. Frequency-Affinity Placement (EWR-aware): Reallocate LLMs based on resource and frequency demands. 6. Idle Power Management: Simulate theoretical energy savings via GPU shutdown or HBM frequency scaling. These features are enabled incrementally from Stage-1 to Stage-6. We designate Stage-5 as the standard Festina configuration, representing the fully implemented system on our testbed. Stage-6 represents Festina+. We measure energy consumption under the same workloads in Section 4.2. Figure 13 details the energy consumption for each stage, normalized to Festina (Stage-5). Stage-1 (baseline) exhibits the highest consumption due to the absence of energy-aware optimizations. Stage-2 introduces frequency-aware dispatching, yielding around 12% energy reduction. By clustering requests with similar optimal frequencies, this stage prevents the hardware from defaulting to the TDP limit for mismatched workloads. Stage-3 adds fine-grained runtime management, delivering a substantial 25% energy saving. Stage-4 incorporates standard SLO-aware placement (similar to [27, 52]), which offers a modest 2% energy saving. While this consolidates resources based on demand, it fails

1.4 1.2 1 0.8

16 LLMs Stage-1

24 LLMs Stage-2

Stage-3

Stage-4

32 LLMs Stage-5 (Festina)

Stage-6 (Festina+)

Figure 13. Breakdown of average energy consumption across incremental stages. Results are normalized to Stage-5 (the standard Festina configuration). to account for phase dominance (i.e., whether energy consumption is dominated by compute-bound prefill or memorybound decode). Stage-5 (Festina) addresses this by enabling EWR-aware placement. By consolidating workloads based on frequency preference, we achieve an additional 11% energy saving. Finally, Stage-6 (Festina+) demonstrates that if idle GPUs can be fully shut down or HBM frequency can be lowered, a further 16% energy reduction is achievable. This points to a promising direction, once such hardware-level controls become feasible. 4.4

Micro-benchmarks

Workload Sensitivity. We further evaluate Festina under extreme workload distributions using ShareGPT-ix2 (prefillheavy) and ShareGPT-ox2 (decode-heavy) from Aegaeon. Figure 12 compares the SLO attainment and energy consumption against the state-of-the-art baseline, Aegaeon. For the prefill-heavy workload (ShareGPT-ix2), Festina maintains SLO attainment comparable to Aegaeon while achieving energy savings of 18% to 30%. We observe that energy gains diminish as workload intensity (RPS and number of LLMs) increases. This is because high-intensity prefill workloads generate a massive influx of tokens, requiring 11

SLO attainment

16 LLMs SLO PD-A

24 LLMs SLO PD-D

32 LLMs Energy PD-D

1 0.98 0.96 0.94 0.92 0.9

utilization through advanced resource management. ServerlessLLM [8] and Medusa [60] prioritize auto-scaling efficiency by reducing cold-start latency and accelerating checkpoint migration. To improve GPU utilization, systems like MuxServe [7], Prism [58], Dilu [27], and Aegaeon [52] enable spatial sharing, allowing multiple models to co-locate on a single GPU. These works employ dynamic reconfiguration, ranging from coarse-grained GPU allocation to fine-grained SM partitioning, to maximize throughput. However, their primary objective is performance (SLO attainment) or utilization, where they do not explicitly target energy efficiency or leverage frequency scaling to reduce power consumption. Energy-aware serving systems. Recent systems such as DynamoLLM [41], GreenLLM [25], and VoltanaLLM [57] use DVFS to reduce LLM serving energy. However, they are designed around assumptions that do not hold in serverless multi-tenant serving. First, they largely assume that a GPU is dedicated to one model or one serving pipeline. Energy optimization can therefore be reduced to choosing a frequency based on workload intensity, such as input/output length or serving phase. In contrast, Festina targets a serverless regime where multiple tenants may share one GPU. Co-location couples the energy and latency behavior of different requests and makes frequency selection a shared scheduling decision rather than a per-model control knob. Second, prior DVFS systems do not deal with sharedfrequency contention. Co-resident models share one devicewide clock while competing for SMs, so the frequency cannot be tuned independently for each workload. The GPU clock may be pinned by the most demanding co-tenant, forcing other requests to run at a higher-than-needed frequency and wasting energy. Festina addresses this issue through frequency-aware dispatching, which routes requests to GPUs whose current clocks better match their energy-optimal frequencies, reducing energy waste from frequency mismatch. Third, co-location expands the control space beyond what single-model DVFS systems can efficiently enumerate. Once multiple tenants share a GPU, frequency selection must be coordinated with SM partitioning across requests. The joint configuration space grows rapidly with the number of colocated tasks, making exhaustive per-batch search impractical. Festina instead uses a lightweight frequency sweep and phase-aware (SM, frequency) adaptation, keeping scheduling overhead low while accounting for the distinct energyoptimal operating points of prefill and decode. These differences are structural rather than missing features that can be added incrementally. DynamoLLM relies on fixed per-pool frequencies from offline profiling and rigid per-LLM pools, which conflict with dynamic serverless colocation. GreenLLM and VoltanaLLM introduce phase-specific frequency control, but they rely on PD-disaggregated serving with static prefill/decode partitions and do not provide autoscaling or multi-tenant spatial multiplexing. Adapting these

Norm. energy

100% 98% 96% 94% 92% 90%

Figure 14. Average SLO attainment and energy consumption (normalized to the PD-aggregated setup) comparing PD-A and PD-D configurations.

the GPU to operate near its TDP limit to meet deadlines, thereby reducing the opportunity for frequency downscaling. Conversely, for the decode-heavy workload (ShareGPT-ox2), Festina achieves both higher energy efficiency (saving 28% to 35% compared to Aegaeon) and better SLO attainment in specific high-load scenarios (e.g., 32 LLMs at 2.5 RPS). The performance advantage stems from resource flexibility: Aegaeon relies on static partitioning between prefill and decode nodes, leading to bottlenecks when decode demand outstrips the fixed decode resources. In contrast, Festina allows decode tasks to opportunistically utilize free SMs from prefill tasks. Regarding energy, Aegaeon relies on the default hardware policy that aggressively boosts frequency to the TDP regardless of phase characteristics. Festina proactively lowers the GPU frequency for the decode phase, yielding substantial energy savings without violating SLOs. Extension to PD-disaggregation. We further evaluate Festina in the prefill-decode disaggregated (PD-D) architecture and compare it against the PD-aggregated (PD-A) setup using the same workloads as in §4.2. For PD-D, we adopt Aegaeon’s prefill–decode separation strategy. Results are presented in Figure 14. Regarding SLO attainment, the PD-D setup exhibits a slight degradation of approximately 1.5% compared to PD-A. This drop is attributed to the overhead of KV cache transmission, which currently lacks low-level optimization in our testbed, delays the initiation of decode tasks, and extends total execution time. However, regarding energy efficiency, the PD-D setup yields an additional 7% saving over PD-A. This improvement stems from the isolation of workload phases: prefill and decode tasks possess significantly divergent optimal frequencies. By separating them onto dedicated GPUs, the local schedulers can enforce precise frequency tuning for each phase without the interference caused by mixing frequency-mismatched prefill and decode tasks. The impact of safety margin can be found in §G. The energy consumption of migration can be found in §H.

5

Related Works

High-Performance & Spatial Sharing Systems. Numerous systems focus on optimizing inference latency and device 12

systems to serverless serving would require co-locationaware dispatch, shared-frequency conflict resolution, joint SM/frequency adaptation, and frequency-aware scale-in or migration. These mechanisms are central to Festina rather than extensions to prior designs. Table 1 summarizes the comparison between Festina and prior systems.

6

[10] Yunchu Han, Zhaojun Nan, Sheng Zhou, and Zhisheng Niu. 2025. Joint Memory Frequency and Computing Frequency Scaling for Energyefficient DNN Inference. arXiv preprint arXiv:2509.17970 (2025). [11] Sarah L Harris and David Harris. 2021. Digital design and RISCV computer architecture textbook. In 2021 ACM/IEEE Workshop on Computer Architecture Education (WCAE). IEEE, 1–5. [12] Zicong Hong, Jian Lin, Song Guo, Sifu Luo, Wuhui Chen, Roger Wattenhofer, and Yue Yu. 2024. Optimus: Warming serverless ml inference via inter-function model transformation. In Proceedings of the Nineteenth European Conference on Computer Systems. 1039–1053. [13] Zhengding Hu, Vibha Murthy, Zaifeng Pan, Wanlu Li, Xiaoyi Fang, Yufei Ding, and Yuke Wang. 2025. HedraRAG: Co-Optimizing Generation and Retrieval for Heterogeneous RAG Workflows. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles (Lotte Hotel World, Seoul, Republic of Korea) (SOSP ’25). Association for Computing Machinery, New York, NY, USA, 623–638. doi:10.1145/3731569.3764806 [14] Tao Huang, Pengfei Chen, Kyoka Gong, Jocky Hawk, Zachary Bright, Wenxin Xie, Kecheng Huang, and Zhi Ji. 2024. ENOVA: Autoscaling towards Cost-effective and Stable Serverless LLM Serving. arXiv:2407.09486 [cs.DC] https://arxiv.org/abs/2407.09486 [15] Hugging Face. [n. d.]. Hugging Face: The AI Community Building the Future. Hugging Face. https://huggingface.co/ Accessed: 2026-02-04. [16] Shashwat Jaiswal, Kunal Jain, Yogesh Simmhan, Anjaly Parayil, Ankur Mallick, Rujia Wang, Renee St Amant, Chetan Bansal, Victor Rühle, Anoop Kulkarni, et al. 2025. Serving models, fast and slow: optimizing heterogeneous llm inferencing workloads at scale. arXiv preprint arXiv:2502.14617 (2025). [17] Jinwoo Jeong and Jeongseob Ahn. 2025. Accelerating LLM Serving for Multi-turn Dialogues with Efficient Resource Management. Association for Computing Machinery, New York, NY, USA, 1–15. https://doi.org/ 10.1145/3676641.3716245 [18] Yunho Jin, Chun-Feng Wu, David Brooks, and Gu-Yeon Wei. 2023. S3: increasing GPU utilization during generative inference for higher throughput. In Proceedings of the 37th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS ’23). Curran Associates Inc., Red Hook, NY, USA, Article 791, 13 pages. [19] Harold W Kuhn. 1955. The Hungarian method for the assignment problem. Naval research logistics quarterly 2, 1-2 (1955), 83–97. [20] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles. 611–626. [21] Baolin Li, Tirthak Patel, Siddharth Samsi, Vijay Gadepally, and Devesh Tiwari. 2022. MISO: Exploiting Multi-Instance GPU Capability on Multi-Tenant GPU Clusters. In Proceedings of the 13th Symposium on Cloud Computing (San Francisco, California) (SoCC ’22). Association for Computing Machinery, New York, NY, USA, 173–189. doi:10.1145/ 3542929.3563510 [22] Bingyao Li, Yueqi Wang, Tianyu Wang, Lieven Eeckhout, Jun Yang, Aamer Jaleel, and Xulong Tang. 2024. STAR: Sub-Entry Sharing-Aware TLB for Multi-Instance GPU. In 2024 57th IEEE/ACM International Symposium on Microarchitecture (MICRO). IEEE, 309–323. [23] Yanying Lin, Shijie Peng, Chengzhi Lu, Chengzhong Xu, and Kejiang Ye. 2025. FlexPipe: Adapting Dynamic LLM Serving Through Inflight Pipeline Refactoring in Fragmented Serverless Clusters. arXiv preprint arXiv:2510.11938 (2025). [24] Qianli Liu, Zicong Hong, Peng Li, Fahao Chen, and Song Guo. 2025. Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Management. In IEEE INFOCOM 2025 - IEEE Conference on Computer Communications. 1–10. doi:10.1109/INFOCOM55648.2025. 11044533

Conclusion

We have presented the design, implementation, and evaluation of Festina, a power-efficient serverless LLM serving system. Festina leverages a global scheduler and per-GPU local scheduler to jointly optimize the request placement and GPU resources to minimize the energy consumption of LLM serving without violating each request’s latency SLOs. Moreover, Festina leverages an EWR-aware mechanism that consolidates workloads via SLO-aware migration to further reduce static memory power consumption. Evaluations show that Festina can save up to 56% energy compared to existing SOTA LLM serving systems while maintaining SLO attainment within a 2% margin.

References [1] Jing Chen, Madhavan Manivannan, Bhavishya Goel, and Miquel Pericàs. 2023. JOSS: Joint Exploration of CPU-Memory DVFS and Task Scheduling for Energy Efficiency. In Proceedings of the 52nd International Conference on Parallel Processing (Salt Lake City, UT, USA) (ICPP ’23). Association for Computing Machinery, New York, NY, USA, 828–838. doi:10.1145/3605573.3605586 [2] Andrew A Chien, Liuzixuan Lin, Hai Nguyen, Varsha Rao, Tristan Sharma, and Rajini Wijayawardana. 2023. Reducing the Carbon Impact of Generative AI Inference (today and in 2035). In Proceedings of the 2nd workshop on sustainable computer systems. 1–7. [3] Seungbeom Choi, Jeonghoe Goo, Eunjoo Jeon, Mingyu Yang, and Minsung Jang. 2025. ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictor. arXiv:2505.09142 [cs.DC] https: //arxiv.org/abs/2505.09142 [4] Seungbeom Choi, Sunho Lee, Yeonjae Kim, Jongse Park, Youngjin Kwon, and Jaehyuk Huh. 2022. Serving Heterogeneous Machine Learning Models on Multi-GPU Servers with Spatio-Temporal Sharing. In 2022 USENIX Annual Technical Conference (USENIX ATC 22). USENIX Association, Carlsbad, CA, 199–216. https://www.usenix.org/ conference/atc22/presentation/choi-seungbeom [5] Tri Dao. 2023. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691 (2023). [6] Aditya Dhakal, Sameer G Kulkarni, and KK Ramakrishnan. 2020. Gslice: controlled spatial sharing of gpus for a scalable inference platform. In Proceedings of the 11th ACM Symposium on Cloud Computing. 492–506. [7] Jiangfei Duan, Runyu Lu, Haojie Duanmu, Xiuhong Li, Xingcheng Zhang, Dahua Lin, Ion Stoica, and Hao Zhang. 2024. MuxServe: flexible spatial-temporal multiplexing for multiple LLM serving. In Proceedings of the 41st International Conference on Machine Learning (Vienna, Austria) (ICML’24). JMLR.org, Article 473, 13 pages. [8] Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yuvraj Patel, and Luo Mai. 2024. {ServerlessLLM}:{LowLatency} serverless inference for large language models. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). 135–153. [9] Shiwei Gao, Qing Wang, Shaoxun Zeng, Youyou Lu, and Jiwu Shu. 2025. Weaver: Efficient {Multi-LLM} Serving with Attention Offloading. In 2025 USENIX Annual Technical Conference (USENIX ATC 25). 587–595. 13

[25] Qunyou Liu, Darong Huang, Marina Zapater, and David Atienza. 2025. GreenLLM: SLO-Aware Dynamic Frequency Scaling for EnergyEfficient LLM Serving. arXiv preprint arXiv:2508.16449 (2025). [26] Chiheng Lou, Sheng Qi, Chao Jin, Dapeng Nie, Haoran Yang, Xuanzhe Liu, and Xin Jin. 2025. Towards Swift Serverless LLM Cold Starts with ParaServe. arXiv preprint arXiv:2502.15524 (2025). [27] Cunchi Lv, Xiao Shi, Zhengyu Lei, Jinyue Huang, Wenting Tan, Xiaohui Zheng, and Xiaofang Zhao. 2025. Dilu: Enabling GPU Resourcingon-Demand for Serverless DL Serving via Introspective Elasticity. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 (Rotterdam, Netherlands) (ASPLOS ’25). Association for Computing Machinery, New York, NY, USA, 311–325. doi:10.1145/3669940.3707251 [28] Xupeng Miao, Chunan Shi, Jiangfei Duan, Xiaoli Xi, Dahua Lin, Bin Cui, and Zhihao Jia. 2024. Spotserve: Serving generative large language models on preemptible instances. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. 1112–1127. [29] Microsoft. [n. d.]. Microsoft Azure: Cloud Computing Services. Microsoft. https://azure.microsoft.com/en-us Accessed: 2026-02-04. [30] MIT Technology Review. 2026. We did the math on AI’s energy footprint. Here’s the story you haven’t heard. https://www.technologyreview.com/ 2025/05/20/1116327/ai-energy-usage-climate-footprint-big-tech/ Accessed: 2026-01-26. [31] Salman Mohamadi, Ghulam Mujtaba, Ngan Le, Gianfranco Doretto, and Donald A Adjeroh. 2023. ChatGPT in the age of generative AI and large language models: a concise survey. arXiv preprint arXiv:2307.04251 (2023). [32] C Newell. 2013. Applications of queueing theory. Vol. 4. Springer Science & Business Media. NVIDIA H100 Tensor Core GPU Architec[33] NVIDIA. 2022. ture. https://resources.nvidia.com/en-us-hopper-architecture/nvidiah100-tensor-c Whitepaper. [34] NVIDIA. 2026. NVIDIA GPU Management and Deployment - MultiProcess Service. https://docs.nvidia.com/deploy/mps/index.html [35] NVIDIA. 2026. NVIDIA Multi-Instance GPU User Guide. https: //docs.nvidia.com/datacenter/tesla/mig-user-guide/ [36] Archit Patke, Dhemath Reddy, Saurabh Jha, Haoran Qiu, Christian Pinto, Chandra Narayanaswami, Zbigniew Kalbarczyk, and Ravishankar Iyer. 2025. Queue management for slo-oriented large language model serving. arXiv:2407.00047 [cs.DC] doi:10.1145/3698038.369852 [37] Qiangyu Pei, Yongjie Yuan, Haichuan Hu, Qiong Chen, and Fangming Liu. 2023. Asyfunc: A high-performance and resource-efficient serverless inference system via asymmetric functions. In Proceedings of the 2023 ACM Symposium on Cloud Computing. 324–340. [38] PyPI Contributors. [n. d.]. nvidia-ml-py: Python Bindings to the NVIDIA Management Library (NVML). https://pypi.org/project/ nvidia-ml-py/. Accessed: 2026-02-04. [39] Python Software Foundation. [n. d.]. asyncio — Asynchronous I/O. https://docs.python.org/3/library/asyncio.html. Accessed: 2026-02-04. [40] Haoran Qiu, Weichao Mao, Archit Patke, Shengkun Cui, Saurabh Jha, Chen Wang, Hubertus Franke, Zbigniew T Kalbarczyk, Tamer Başar, and Ravishankar K Iyer. 2024. Efficient interactive llm serving with proxy model-based sequence length prediction. arXiv preprint arXiv:2404.08509 (2024). [41] Jovan Stojkovic, Chaojie Zhang, Íñigo Goiri, Josep Torrellas, and Esha Choukse. 2025. DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency. In 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA). 1348–1362. doi:10.1109/HPCA61900.2025.00102 [42] Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. 2024. Llumnix: Dynamic Scheduling for Large Language Model Serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association,

Santa Clara, CA, 173–191. https://www.usenix.org/conference/osdi24/ presentation/sun-biao [43] Cheng Tan, Zhichao Li, Jian Zhang, Yu Cao, Sikai Qi, Zherui Liu, Yibo Zhu, and Chuanxiong Guo. 2021. Serving DNN models with multi-instance gpus: A case of the reconfigurable machine scheduling problem. arXiv preprint arXiv:2109.11067 (2021). [44] Gemini Team, R Anil, S Borgeaud, Y Wu, JB Alayrac, J Yu, R Soricut, J Schalkwyk, AM Dai, A Hauth, et al. 2024. Gemini: A family of highly capable multimodal models, 2024. arXiv preprint arXiv:2312.11805 10 (2024). [45] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, MarieAnne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023). [46] vLLM Project. 2026. vLLM Documentation. https://docs.vllm.ai/en/ latest/. Accessed: 2026-02-04. [47] Tianyu Wang, Sheng Li, Bingyao Li, Yue Dai, Ao Li, Geng Yuan, Yufei Ding, Youtao Zhang, and Xulong Tang. 2024. Improving GPU Multi-Tenancy Through Dynamic Multi-Instance GPU Reconfiguration. arXiv preprint arXiv:2407.13126 (2024). [48] Yuxin Wang, Yuhan Chen, Zeyu Li, Xueze Kang, Yuchu Fang, Yeju Zhou, Yang Zheng, Zhenheng Tang, Xin He, Rui Guo, et al. 2025. Burstgpt: A real-world workload dataset to optimize llm serving systems. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2. 5831–5841. [49] Zibo Wang, Yijia Zhang, Fuchun Wei, Bingqiang Wang, Yanlin Liu, Zhiheng Hu, Jingyi Zhang, Xiaoxin Xu, Jian He, Xiaoliang Wang, Wanchun Dou, Guihai Chen, and Chen Tian. 2025. Using Analytical Performance/Power Model and Fine-Grained DVFS to Enhance AI Accelerator Energy Efficiency. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 (Rotterdam, Netherlands) (ASPLOS ’25). Association for Computing Machinery, New York, NY, USA, 1118–1132. doi:10.1145/3669940.3707231 [50] Adam Wierman, Lachlan LH Andrew, and Antony Tang. 2009. Poweraware speed scaling in processor sharing systems. In IEEE INFOCOM 2009. IEEE, 2007–2015. [51] Tian Xia, Ziming Mao, Jamison Kerney, Ethan J Jackson, Zhifei Li, Jiarong Xing, Scott Shenker, and Ion Stoica. 2025. SkyLB: A LocalityAware Cross-Region Load Balancer for LLM Inference. arXiv preprint arXiv:2505.24095 (2025). [52] Yuxing Xiang, Xue Li, Kun Qian, Yufan Yang, Diwen Zhu, Wenyuan Yu, Ennan Zhai, Xuanzhe Liu, Xin Jin, and Jingren Zhou. 2025. Aegaeon: Effective GPU Pooling for Concurrent LLM Serving on the Market. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles (Lotte Hotel World, Seoul, Republic of Korea) (SOSP ’25). Association for Computing Machinery, New York, NY, USA, 1030–1045. doi:10.1145/3731569.3764815 [53] Yuxing Xiang, Xue Li, Kun Qian, Wenyuan Yu, Ennan Zhai, and Xin Jin. 2025. ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production. arXiv:2505.09999 [cs.DC] https://arxiv.org/abs/2505.09999 [54] Kaiqiang Xu, Decang Sun, Han Tian, Junxue Zhang, and Kai Chen. 2025. {GREEN}: Carbon-efficient Resource Scheduling for Machine Learning Clusters. In 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25). 999–1014. [55] Yanan Yang, Laiping Zhao, Yiming Li, Huanyu Zhang, Jie Li, Mingyang Zhao, Xingzhen Chen, and Keqiu Li. 2022. Infless: a native serverless system for low-latency, high-throughput inference. In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems. 768–781. [56] Jie You, Jae-Won Chung, and Mosharaf Chowdhury. 2023. Zeus: Understanding and optimizing {GPU} energy consumption of {DNN} training. In 20th USENIX Symposium on Networked Systems Design and 14

Implementation (NSDI 23). 119–139. [57] Jiahuan Yu, Aryan Taneja, Junfeng Lin, and Minjia Zhang. 2025. VoltanaLLM: Feedback-Driven Frequency Control and State-Space Routing for Energy-Efficient LLM Serving. arXiv preprint arXiv:2509.04827 (2025). [58] Shan Yu, Jiarong Xing, Yifan Qiao, Mingyuan Ma, Yangmin Li, Yang Wang, Shuo Yang, Zhiqiang Xie, Shiyi Cao, Ke Bao, et al. 2025. Prism: Unleashing GPU Sharing for Cost-Efficient Multi-LLM Serving. arXiv preprint arXiv:2505.04021 (2025). [59] Anna Yue, Pen-Chung Yew, and Sanyam Mehta. 2025. EVeREST: An Effective and Versatile Runtime Energy Saving Tool for GPUs. In Proceedings of the 30th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming (Las Vegas, NV, USA) (PPoPP ’25). Association for Computing Machinery, New York, NY, USA, 57–69. doi:10.1145/3710848.3710875 [60] Shaoxun Zeng, Minhui Xie, Shiwei Gao, Youmin Chen, and Youyou Lu. 2025. Medusa: Accelerating serverless LLM inference with materialization. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1. 653–668. [61] Huaizheng Zhang, Yuanming Li, Wencong Xiao, Yizheng Huang, Xing Di, Jianxiong Yin, Simon See, Yong Luo, Chiew Tong Lau, and Yang You. 2023. MIGPerf: A Comprehensive Benchmark for Deep Learning Training and Inference Workloads on Multi-Instance GPUs. arXiv preprint arXiv:2301.00407 (2023). [62] Wei Zhang, Weihao Cui, Kaihua Fu, Quan Chen, Daniel Edward Mawhirter, Bo Wu, Chao Li, and Minyi Guo. 2019. Laius: Towards Latency Awareness and Improved Utilization of Spatial multitasking accelerators in datacenters. In Proceedings of the ACM International Conference on Supercomputing (Phoenix, Arizona) (ICS ’19). Association for Computing Machinery, New York, NY, USA, 58–68. doi:10.1145/3330345.3330351 [63] Yijia Zhang, Qiang Wang, Zhe Lin, Pengxiang Xu, and Bingqiang Wang. 2024. Improving GPU Energy Efficiency through an Applicationtransparent Frequency Scaling Policy with Performance Assurance. In Proceedings of the Nineteenth European Conference on Computer Systems (Athens, Greece) (EuroSys ’24). Association for Computing Machinery, New York, NY, USA, 769–785. doi:10.1145/3627703.3629584 [64] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. 2024. LMSYS-Chat1M: A Large-Scale Real-World LLM Conversation Dataset. In The Twelfth International Conference on Learning Representations. https: //openreview.net/forum?id=BOfDKxfwt0 [65] Zangwei Zheng, Xiaozhe Ren, Fuzhao Xue, Yang Luo, Xin Jiang, and Yang You. 2023. Response length perception and sequence scheduling: An llm-empowered llm inference pipeline. Advances in Neural Information Processing Systems 36 (2023), 65517–65530.

A

Background

A.1

LLM Inference

Contemporary LLMs [31, 44, 45] are predominantly decoderonly Transformers composed of repeated blocks that consist of multi-head attention and feed-forward layers. Given input prompts, LLMs generate tokens in an autoregressive manner. The LLM inference splits into two distinct phases: prefill and decode. In the prefill phase, the input prompt is processed once to build the key-value (KV) cache [20], an optimization to eliminate redundant computation of key-value matrix contents across decoding iterations, and emit the first output token. The prefill phase, characterized by long input prompts, heavily utilizes computation resources (i.e., SMs). At the end of the prefill phase, a new token is generated, then used as the input in the decode phase. In the decode phase, the LLM generates one new token per iteration until it reaches a predefined sequence length or the end-of-sentence marker. Each iteration reuses the stored KV cache. This KV cache optimization typically shifts the bottleneck from computation to memory bandwidth, making the decode phase memorybound. Moreover, Time-to-First-Token (TTFT) captures latency from request arrival to the first output token, used as the performance metric for the prefill phase. Time-BetweenTokens (TBT) captures latency between successive output tokens, used as the performance metric for the decode phase. A.2

Serverless LLM Serving

Conventional server-centric LLM deployments have large resource footprints and require extensive server management. In contrast, cloud vendors and researchers have advanced serverless serving systems, which elastically scale inference workers to match demand. With pay-as-you-go pricing—charging only for the time each worker runs—serverless is cost-effective for bursty, long-tail workloads [12, 37, 55]. Recent serverless LLM solutions [8, 26, 60] focus on reducing cold-start latency, incurred when no worker hosts the target LLM and a new one must be initialized. However, they still rely on static, exclusive GPU allocations optimized for the most demanding workloads, which leads to underutilization, GPU waste, and energy inefficiency for lighter workloads with shorter inputs. A.3

GPU Sharing

Modern GPUs support two advanced sharing mechanisms: (i) Multi-Process Service (MPS) [34] and (ii) Multi-Instance GPU (MIG) [35]. MPS partitions compute resources (SMs) to reduce compute contention, while keeping memory shared, making it suitable for memory-hungry workloads like LLMs. It also supports runtime SM repartition without disrupting running models. MIG partitions both compute and memory at the GPC level, providing full physical isolation. However, it offers only 5 fixed partition sizes (1, 2, 3, 4, or 7 GPCs), 15

Qwen2.5-7b

Llama2-70b

Freq.

1785

0.8

1635

0.6

1485

0.4

1335

0.2 0

1185

MuxServe 0

15

30

45

60 (a) Normalized latency of three co-located LLMs sec

1035

MuxServe

and real-time GPU frequency.

D

1

0.6 0.4

We evaluate the scalability of Festina’s control plane by measuring the scheduling latency of both the global and local schedulers across varying cluster sizes and queue depths (i.e., number of pending tasks). Our simulations, which isolate algorithmic overhead by excluding network variance, demonstrate that Festina remains highly responsive at scale. As detailed in Table 3, the global scheduler maintains a dispatch latency of under 1 ms even when managing a cluster of 10,000 GPUs. Similarly, the local scheduler determines the next task batch in less than 0.1 ms, even with 100 tasks in the queue. These results confirm Festina ’s robustness and ability to handle massive-scale deployments with negligible overhead.

0.2 0

0.2

0.4 0.6 0.8 SLO slack

0 1

(b) CDF of SLO slack.

Figure 15. (a) Normalized latency of three co-located LLMs and real-time GPU frequency. (b) CDF of SLO slack for three LLMs during serving requests. often with too little memory for LLMs, and reconfiguration is possible only when involved GPCs are idle, incurring checkpointing and initialization overheads [21, 22, 47, 61]. Due to MIG’s coarse granularity, limited memory, and costly reconfiguration, MPS has emerged as the preferred mechanism for GPU-shared LLM serving [7, 27, 52]. We align with this established paradigm and employ MPS in this work. A.4

Table 3. Scheduling latency of Festina’s control plane (global scheduler and local scheduler) under various cluster sizes and queue sizes.

Dynamic Voltage and Frequency Scaling

Dynamic voltage and frequency scaling (DVFS) is a prevalent energy management technique for CPUs and GPUs to balance performance and power consumption. By adjusting operating voltage and frequency, DVFS exploits the physical relationship where power consumption (𝑃) is proportional to the square of the voltage (𝑉 ) and the frequency (𝑓 ), expressed as 𝑃 ∝ 𝑉 2 𝑓 [11, 49]. While CPUs often benefit from mature DVFS governors, GPU-based strategies remain comparatively under-optimized [1, 49, 63]. Furthermore, applying DVFS in the context of LLM serving poses a unique challenge: energy efficiency strategies must be implemented without compromising SLOs.

B

Global Scheduler (Linear Scan)

SLO Slack for Large LLMs

E

We utilize the same trace described in Section 2.1 and employ MuxServe as the serving platform with two H100s due to memory space shortage. As illustrated in Figure 15, even when co-locating large LLMs such as Llama2-70b, we observe a minimum of 18% SLO slack. This confirms that the opportunity to enhance energy efficiency by lowering frequency remains viable, even when serving substantial models. We observe consistent results when utilizing Aegaeon as the serving platform.

C

Scheduling Latency Breakdown

0.8

CDF

Norm. latency

Llama2-13b > 18% slack

Frequency (MHz)

(SLO) 1

Local Scheduler (Priority Batching)

Cluster Size (GPUs)

Latency (ms)

Queue Size (Tasks)

Latency (ms)

100 1,000 5,000 10,000

0.011 0.103 0.509 0.988

10 25 50 100

0.0544 0.0612 0.0768 0.0942

EWR-Aware Placement Reconfiguration

Determining Preferred Configurations. Before reconfiguration, the local scheduler computes the preferred configuration (𝑆𝑀𝑙 , 𝐹𝑙 ) for each model. This calculation utilizes the average input/output lengths (𝑙𝑒𝑛𝑖𝑛 , 𝑙𝑒𝑛𝑜𝑢𝑡 ) observed over a sliding window (Δ = 5 minutes). First, the scheduler queries the Look-Up Table (LUT) to identify the optimal independent configurations for the prefill and decode phases—denoted as (𝑆𝑀𝑝 , 𝐹𝑝 ) and (𝑆𝑀𝑑 , 𝐹𝑑 )—that minimize energy consumption while satisfying SLOs. The LUT also provides the expected power consumption (𝑃𝑝 , 𝑃𝑑 ) and execution duration (𝑇𝑝 ,𝑇𝑑 ) for each phase under these settings. To derive a unified configuration (𝑆𝑀𝑙 ), we compute the time-weighted average of the required SMs, ensuring the allocation reflects the phase occupying the majority of the execution time:

Energy Waste from Frequency Mismatch

Co-locating multiple LLMs improves utilization and enables scale-in, but different models exhibit different energy-optimal frequencies. Because GPU frequency is effectively devicewide and often pinned by the most demanding workload, frequency-agnostic packing can force low-frequency workloads to run at an unnecessarily high clock, wasting energy (e.g., we observe up to 10–16% overhead under mismatched co-location). Therefore, cluster-level placement and consolidation must explicitly account for frequency affinity.

𝑆𝑀𝑙 =

𝑆𝑀𝑝 · 𝑇𝑝 + 𝑆𝑀𝑑 · 𝑇𝑑 𝑇𝑝 + 𝑇𝑑

(8)

Similarly, we compute the preferred frequency (𝐹𝑙 ) as the energy-weighted average of the phase-specific frequencies. 16

SLO attainment

1.1

100%

97.5%

1.05

95%

24 LLMs

32 LLMs

1.1

100%

97.5%

1.05

97.5%

1.05

1

95%

1

95%

1

92.5%

0.95

92.5%

0.95

92.5%

0.95

90%

0.9

90%

0.9

90%

15%

10%

5%

Safety Margin

0%

15%

10%

5%

Safety Margin

SLO attainment

0%

15%

10%

5%

Safety Margin

1.1

0%

0.9

Norm. energy

16 LLMs

100%

Energy consumption

Figure 16. Impact of varying safety margins on average SLO attainment and energy consumption. Energy values are normalized to the 5% margin baseline. Same Workloads in Section 4.2 are utilized. This weighting scheme prioritizes the frequency requirements of the more energy-intensive phase:

on GPU 𝑢 to GPU 𝑣, with cost 𝑒𝑢𝑣 =

𝐹𝑙 =

𝑃𝑝 · 𝑇𝑝 · 𝐹𝑝 + 𝑃𝑑 · 𝑇𝑑 · 𝐹𝑑 𝑃𝑝 · 𝑇𝑝 + 𝑃𝑑 · 𝑇𝑑

(9) where Bytes(·) accounts for the model weights and any request state selected for migration, and BW(𝑢, 𝑣) is the measured interconnect bandwidth. We then solve the minimumcost assignment using the Kuhn–Munkres (Hungarian) algorithm [19] to obtain a migration plan with low aggregate transfer overhead. Selective request migration (finish locally vs. move). To avoid unnecessary transfers, we adopt a simple cost-based rule, similar in spirit to prior live-migration systems for LLM inference: for each in-flight request, we compare the estimated remaining computation time if it finishes on the source GPU against the estimated migration time under the chosen mapping [8, 24]. Requests that are close to completion are allowed to finish in place, while requests with substantial remaining work are migrated, reducing both network traffic and reconfiguration disruption. What to transfer (tokens/prompt vs. KV cache). For requests selected for migration, we further choose between two state-transfer modes. Following the same principle as ServerlessLLM and related work, we may transfer only the minimal token state (input prompt and generated tokens) and recompute the KV cache on the destination, or transfer both tokens and KV cache when the link is fast and recomputation would be slower [28]. This decision is made per request using lightweight time estimates for (i) KV recomputation and (ii) KV transmission, favoring the cheaper option. Staged, memory-safe transfer. Finally, to prevent out-ofmemory (OOM) on the destination GPU during reconfiguration, we perform transfers in stages using a simple reservation/credit protocol inspired by the coordinated, multi-stage migration style in Llumnix [42]. The destination scheduler first reserves memory for locally retained requests, then admits incoming transfers only when sufficient free space is available. As local execution frees memory, the destination releases additional credits, allowing the source to continue sending state incrementally. This staged approach bounds transient memory pressure and avoids service stalls caused by OOM during migration.

Placement reconfiguration algorithm. The pseudo code of the reconfiguration algorithm is shown in Algo. 1. A Running Example of Phase Three. As shown in Figure 9, three LLMs in purple, blue, and green are successfully placed onto their preferred nodes without contention (Direct Placement). Next (EWR-driven swapping), when trying to place the red LLM on the second GPU, which gives the closest frequency, the memory is insufficient to hold the red LLM. The green LLM is then moved from the second GPU to the first, due to sufficient resources on the first GPU for the green LLM and the reduced overall EWR after migration. After migrating the green LLM to the first GPU, the red LLM is placed on the second GPU. If the prior two actions fail to find a GPU to deploy the red LLM, the scheduler iteratively checks the next closest GPUs in the sorted list. The LLM is placed on the first active GPU found with sufficient resources. In this case, the red LLM is placed on the first GPU, despite encountering a huge frequency deviation.

F

Bytes(𝑢 → 𝑣) , BW(𝑢, 𝑣)

Efficient State Transfer Across Different GPUs

After the global controller computes a new placement, the system needs to (re)initialize the inference engine and move the necessary execution state (model weights and, for inflight requests, prompt/KV state) before normal serving can resume. Because these states can be large, naïvely transferring everything can dominate reconfiguration time and visibly disrupt inference. Transmission plan (device mapping). Motivated by prior work on minimizing reconfiguration communication cost (e.g., SpotServe [28]), we compute an explicit mapping from source GPUs (current placement) to destination GPUs (target placement) that minimizes total transfer time. We model the mapping as a weighted bipartite assignment problem: each edge (𝑢, 𝑣) represents transferring the state currently 17

Regarding energy, higher margins (10% and 15%) result in higher consumption, which strict resource reservation restricts opportunities for workload consolidation. While the 0% margin achieves the lowest energy consumption (approximately 2% lower than the 5% setting), at the cost of SLO violations. Consequently, we empirically select a 5% safety margin as the optimal equilibrium between SLO adherence and energy efficiency.

Algorithm 1: EWR-Aware Placement Reconfiguration Input : List of active LLMs L; GPU Constraints (𝑆𝑀𝑒 𝑓 𝑓 , 𝑀𝑒 𝑓 𝑓 ) Output : Map of LLMs to GPUs // Phase 1: Resource Profiling 1 foreach LLM 𝑙 ∈ L do 2 𝑙 .𝑆 ← EstimateSM(model, avg_in, avg_out) 3 𝑙 .𝑀 ← EstimateMem(model_size, max_kv) 4 𝑙 .𝐹 ← CalcFreq(model, 𝑙 .𝑆, avg_in, avg_out) 5 end // Phase 2: Capacity Planning 6 𝑁𝑠𝑚 ← 𝐵𝑒𝑠𝑡𝐹𝑖𝑡𝐷𝑒𝑐𝑟𝑒𝑎𝑠𝑖𝑛𝑔(L.𝑆, 𝑆𝑀𝑒 𝑓 𝑓 ) 7 𝑁𝑚𝑒𝑚 ← 𝐵𝑒𝑠𝑡𝐹𝑖𝑡𝐷𝑒𝑐𝑟𝑒𝑎𝑠𝑖𝑛𝑔(L.𝑀, 𝑀𝑒 𝑓 𝑓 ) 8 𝑁𝑚𝑖𝑛 ← max(𝑁𝑠𝑚 , 𝑁𝑚𝑒𝑚 ) 9 Initialize active GPUs set G with count 𝑁𝑚𝑖𝑛 // Phase 3: EWR Minimization Loop 10 Sort(L by 𝑙 .𝐹 ) // Group by Frequency 11 foreach LLM 𝑙 ∈ L do 12 𝑔𝑡𝑎𝑟𝑔𝑒𝑡 ← arg min𝑔∈ G |𝑔.𝑓 𝑟𝑒𝑞 − 𝑙 .𝐹 | 13 𝑔𝑐𝑎𝑛𝑑 ← arg min𝑔∈ G\𝑔𝑡𝑎𝑟𝑔𝑒𝑡 |𝑔.𝑓 𝑟𝑒𝑞 − 𝑙 .𝐹 | 14 if Fits (𝑔𝑡𝑎𝑟𝑔𝑒𝑡 , 𝑙) then 15 Assign 𝑙 to 𝑔𝑡𝑎𝑟𝑔𝑒𝑡 16 else 17 𝑙𝑜𝑢𝑡 ← Incumbent on 𝑔𝑡𝑎𝑟𝑔𝑒𝑡 with lowest 𝐹 // Simulate Swap to check EWR reduction 18 𝐸𝑐𝑢𝑟𝑟 ← CalcEWR (𝑔𝑡𝑎𝑟𝑔𝑒𝑡 ∪ {𝑙 }, 𝑔𝑐𝑎𝑛𝑑 ) 19 𝐸𝑠𝑤𝑎𝑝 ← CalcEWR (𝑔𝑡𝑎𝑟𝑔𝑒𝑡 \ {𝑙𝑜𝑢𝑡 } ∪ {𝑙 }, 𝑔𝑐𝑎𝑛𝑑 ∪ {𝑙𝑜𝑢𝑡 }) 20 if Fits (𝑔𝑐𝑎𝑛𝑑 , 𝑙𝑜𝑢𝑡 ) and Fits (𝑔𝑡𝑎𝑟𝑔𝑒𝑡 \ 𝑙𝑜𝑢𝑡 , 𝑙) and 𝐸𝑠𝑤𝑎𝑝 < 𝐸𝑐𝑢𝑟𝑟 then 21 Move 𝑙𝑜𝑢𝑡 to 𝑔𝑐𝑎𝑛𝑑 ; Assign 𝑙 to 𝑔𝑡𝑎𝑟𝑔𝑒𝑡 22 else 23 if Fits (𝑔𝑐𝑎𝑛𝑑 , 𝑙) then 24 Assign 𝑙 to 𝑔𝑐𝑎𝑛𝑑 25 else 26 Activate new GPU 𝑔𝑛𝑒𝑤 and assign 𝑙 27 G ← G ∪ {𝑔𝑛𝑒𝑤 } 28 end 29 Update 𝑔.𝑆𝑀, 𝑔.𝑀, 𝑔.𝑓 𝑟𝑒𝑞 for all affected GPUs 30 end

G

H

Energy Consumption during Migration

We measured migration energy on our H100/NVLink testbed. Migrating Llama2-13B with a 4K-token KV cache costs 10.67 J, recovered after 108 ms of avoided idle power; migrating Qwen3-32B costs 40.76 J, recovered in 413 ms. Over a 5minute consolidation window, disabling one GPU saves 29.6 kJ: 2788×/727× the two migration costs.

I

Prior Energy-Efficient LLM Serving Systems

Recent research has increasingly focused on improving the energy efficiency of LLM serving. We explain these representative systems and highlight their difference with Festina. DynamoLLM [41]. This system targets server-centric, singleLLM scenarios. It classifies requests into groups based on input and estimated output lengths. It then assigns a static, energy-optimal frequency to each group to minimize energy consumption. GreenLLM [25]. GreenLLM is particularly designed for a PD-disaggregated setup. It leverages offline profiling to dynamically adjust frequencies for prefill batches and employs a feedback-driven mechanism to adjust decode frequencies to save energy. VoltanaLLM [57]. Similar to GreenLLM, VoltanaLLM targets single-LLM serving under PD-disaggregated setups. It independently adjusts the frequency for prefill and decode nodes based on current workloads, aiming to minimize energy usage while satisfying SLO constraints. Limitations in Serverless Contexts. While these works effectively address energy efficiency in dedicated environments, they are not well-suited for shared, serverless scenarios due to five critical limitations: 1. Lack of Auto-scaling: GreenLLM and VoltanaLLM rely on static node partitioning for prefill and decode phases in disaggregated setups. They lack the dynamic auto-scaling capabilities required for serverless serving. Without scaleout support, meeting SLOs under high load is difficult; conversely, without scale-in support, GPUs remain under-utilized during low load, wasting resources and preventing other tenants from occupying GPUs. 2. Absence of Spatial Multiplexing: These systems do not support spatial multiplexing for co-located multi-tenancy. Dedicating a single high-performance GPU (e.g., NVIDIA H100) to one LLM often results in severe under-utilization [7,

Impact of Safety Margin

The safety margin serves as a resource buffer during placement reconfiguration, designed to absorb minor workload fluctuations and prevent rapid oscillation between scale-in and scale-out operations. Figure 16 illustrates the trade-off between SLO attainment and energy consumption across margins ranging from 0% to 15%. Regarding SLO attainment, a 0% margin is insufficient, resulting in a 2% performance drop compared to a 5% margin. In contrast, margins of 5%, 10%, and 15% yield comparable reliability, with performance variances below 0.5%. 18

27, 52]. However, naïve sharing without dedicated resource management (e.g., SM partitioning, frequency control) leads to significant interference and SLO violations [4, 6, 62]. The lack of fine-grained management limits the effectiveness of these systems in shared environments. 3. No Shared-Frequency Conflict Resolution: Prior systems assume each frequency setting governs a single workload. Under co-location, all tenants on a GPU share one device-wide clock, so the most demanding co-tenant pins the frequency and forces models that prefer lower frequencies to over-clock, eroding DVFS savings. None of these systems provide a dispatching mechanism that groups frequencycompatible tenants to avoid this conflict. 4. Inflexible Phase Management: DynamoLLM applies a single static frequency for both prefill and decode phases,

neglecting their distinct computational intensities. While GreenLLM and VoltanaLLM provide phase-specific adjustments, they are tailored for disaggregated setups and do not support prefill-decode mixed scenarios, where PD-aggregated serving is common in current serverless scenarios [8, 14, 23, 27, 60]. 5. Decoupled, Not Joint, Control: These systems treat frequency as the sole energy knob. In a shared GPU, SM partitioning and frequency are coupled—the SM allocation shifts each tenant’s latency/energy-optimal frequency—so optimizing either in isolation is sub-optimal. Prior controllers offer no mechanism for the joint SM/frequency adaptation that co-located serving demands.

19