InferScale: Kubernetes-Native Predictive Autoscaling for SLO-Aware AI Inference

InferScale: Kubernetes-Native Predictive Autoscaling for SLO-Aware AI Inference Wen Huang

arXiv:2610.34006v1 [cs.DC] 25 Oct 2026

Independent Researcher alexwhuang@outlook.com

Abstract—Kubernetes Horizontal Pod Autoscaler (HPA) is widely used for inference services, but reactive scaling is often too slow when pods take tens of seconds to become ready. The problem gets worse in shared clusters, where independent HPAs compete for the same resource budget and can hurt different tenants from run to run. We present InferScale, a Kubernetesnative autoscaler that combines inference-aware signals, shorthorizon demand prediction, SLO-state control, fast-path spike handling, and tenant-priority headroom without replacing the default scheduler. We evaluate InferScale on Azure Kubernetes Service in two settings. On a real ResNet-50 ONNX workload with 21-second cold start, repeated ramp-burst runs improve SLO compliance from 47.8% ± 1.5% under a stronger external-metric HPA to 65.3% ± 9.0% under InferScale. In repeated controlled multitenant contention experiments, InferScale cuts Gold-tier SLO standard deviation from 16.1 to 1.2 percentage points and makes contention outcomes follow the priority policy more closely. Repeated single-service ablations show 6–8 percentage point degradation, with an exploratory priority ablation showing a 27-point drop under contention. The main takeaway is that, in startup-sensitive inference serving, the practical value of prediction lies less in precise forecasting than in moving the controller early enough to offset cold-start delay. Index Terms—Kubernetes, autoscaling, AI inference, SLOaware systems, predictive scaling, multi-tenant cloud systems

I. I NTRODUCTION AI inference is now a normal Kubernetes workload. Teams deploy model servers as standard Deployments, expose them over HTTP, and expect the platform to scale them the same way it scales other services. That is convenient operationally, but inference serving has one awkward property: new pods are often expensive to start. A serving pod may need to download model artifacts, initialize a runtime, allocate memory, and run warmup requests before it is actually useful. In our ONNX Runtime setup, a ResNet-50 pod takes about 21 seconds to become ready even on a modest 500m-CPU allocation. HPA reacts only after a metric crosses a threshold [1], and the rest of the path— scrape, reconcile, create, ready—adds more delay. In our AKS deployment, that whole loop takes roughly 60–90 seconds for real model serving. For a short burst, that is simply too late. We first expected a stronger reactive baseline to narrow the gap. It helped, but not enough. In a motivation experiment with an external-metric HPA, the autoscaler still violated the 200 ms P95 SLO for 60 of 61 burst seconds. Static provisioning held the SLO, but only by keeping extra replicas around when they

were not needed. That gives the basic tradeoff for this paper: react and miss, or provision and waste. Shared clusters make the problem less predictable. When several inference services share the same node pool, independent HPAs can hurt different tenants in different runs even under the same workload and budget. Large production systems have long faced the same resource-sharing tension at cluster scale [2]. For a cloud platform team, that is a bad property. The problem is not just average latency; it is that the policy outcome is hard to reason about. InferScale is our attempt to fix that without changing the deployment model. It stays in the standard Kubernetes control plane and does not replace the scheduler. The controller uses inference-aware signals, short-horizon demand prediction, SLO-state control, fast-path spike handling, and tenantpriority headroom to move earlier when startup delay is the real bottleneck. The contribution is not a new forecasting algorithm in isolation; it is the integration of early demand signal, SLO-state response, and priority-aware headroom into a Kubernetes-native control loop and the evaluation of where that integration helps. The main contributions are: • We show, with both real-model and controlled evidence, that reactive HPA is too slow for bursty inference workloads when pod startup is non-trivial. • We present InferScale, a Kubernetes-native predictive autoscaler that combines short-horizon demand estimation, SLO-state control, fast-path spike handling, and tenantaware guardrails within the standard control plane. • We evaluate InferScale on AKS and show that repeated real-model ramp-burst runs still improve SLO compliance over both CPU HPA and a stronger external-metric HPA, while also providing more consistent protection for highpriority tenants under repeated shared-cluster contention. • Ablation shows that under startup-sensitive bursts, early trend exposure matters at least as much as the forecasting component itself. Removing individual components degrades repeated single-service SLO compliance by 6–8 percentage points, while an exploratory priority ablation shows a 27-point drop under contention. II. BACKGROUND AND M OTIVATION A. Why reactive autoscaling struggles with inference HPA works well when pods start quickly and the scaling metric tracks load tightly. Inference weakens both assump-

Author accepted manuscript. Published in IEEE CLOUD 2026, pp. 414–421. DOI: https://doi.org/10.1109/CLOUD72782.2026.00054 © 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

A. Architecture

Fig. 1. Motivation experiment. External-metric HPA violates the SLO for nearly the entire burst because reactive decisions arrive after the useful window.

tions. Model-serving pods download artifacts, initialize runtimes, and run warmup inferences before becoming ready— 21 seconds in our ResNet-50 setup. Meanwhile the uservisible objective is tail latency, not the indirect signal that the autoscaler watches. The resulting reaction path is long: scrape metrics, observe a threshold crossing, create replicas, and wait for readiness. In our AKS deployment, total response time reaches 60–90 seconds. A short burst can therefore cause sustained SLO violations before new capacity arrives. B. Motivation experiment We ran a controlled motivation experiment on AKS: a burst from 20 to 120 QPS against an inference service with a 45second startup delay. This setup is separate from the later real-model ResNet-50 evaluation and is used only to illustrate why stronger reactive baselines can still miss short bursts. As Figure 1 shows, CPU HPA barely reacted. External-metric HPA reacted, but too late—it violated the 200 ms P95 target for 60 of 61 burst seconds. Static provisioning with six replicas held the SLO at the cost of persistent over-provisioning. C. Why shared-cluster pressure matters In production clusters, multiple inference services share a node pool. When resources become scarce, independent reactive autoscalers can interact badly: one tenant scales first and consumes most of the budget, starving others. We observed exactly this—HPA produced unstable per-tenant outcomes across repeated trials with identical workloads and budgets. Beyond simply scaling faster, a useful autoscaler should also behave predictably under shared-cluster pressure. III. I NFER S CALE D ESIGN InferScale is a Kubernetes operator that manages a custom resource (InferenceAutoscaler) specifying the target Deployment, SLO, replica bounds, forecast window, and tenant class. The controller queries Prometheus every 5 seconds, computes a desired replica count, and patches the Deployment through the Kubernetes API.

Figure 2 shows the architecture. InferScale follows the standard Kubernetes operator pattern. An InferenceAutoscaler custom resource specifies the target Deployment together with policy inputs such as SLO, tenant class, replica bounds, forecast window, and cooldown. The controller then executes a reconcile loop every 5 seconds. Each reconciliation cycle has four stages: 1) Collect request rate, queue depth, P95 latency, and readyreplica count from Prometheus. 2) Predict near-future demand using a short-horizon forecasting model. 3) Decide a target replica count from predicted demand, estimated capacity, and current SLO state. 4) Guard the decision with bounded scaling and cooldownbased scale-down before patching the Deployment replica count through the Kubernetes API. InferScale does not modify the serving runtime or replace the default scheduler. Existing Kubernetes services can adopt it by exporting the required metrics and attaching an autoscaler custom resource. In the reported experiments, the controller uses a 60-second forecast window. This design still inherits the latency of scraped metrics. Prometheus scraping, adapter propagation, and controller reconciliation mean InferScale cannot observe a burst at the exact instant it begins. The design therefore should not be read as eliminating monitoring delay. Instead, it tries to use the first available rising signal more effectively than a threshold-only reactive controller. Once the signal appears, the predictor and fast path move the replica target before the service has already spent a full startup interval in violation. B. Inference-aware signals and demand estimation InferScale uses a small, portable signal set—QPS, queue depth, P95 latency, and ready replicas—that is easy to expose from common model-serving stacks. It then estimates nearfuture request rate using a lightweight Holt-style forecasting model [3]: ℓt = αxt + (1 − α)(ℓt−1 + bt−1 )

(1)

bt = β(ℓt − ℓt−1 ) + (1 − β)bt−1 ,

(2)

x̂t+k = ℓt + kbt .

(3)

We use the predictor for a short horizon rather than attempting long-range workload forecasting. In this setting, the purpose of prediction is not perfect accuracy; it is to expose enough trend signal for the controller to move before the burst fully lands. We kept the predictor simple on purpose. A heavier forecasting model might improve fit on some traces, but it would not change the basic control problem this paper focuses on: once startup takes tens of seconds, the controller mainly needs a reliable early signal, not a sophisticated long-range forecast. That is why we spend more of the design budget on SLO-state

D. Tenant-aware control and guardrails

InferScale control architecture Autoscaler CR SLO target Tenant class Replica bounds Forecast window Cooldown ⇓ config

Prometheus QPS P95 latency Queue depth Ready replicas

⇓ metrics

Kubernetes API Read/patch Deployment Replica updates

⇓ actuation

InferScale Controller (5s reconcile loop) Collect metrics → predict demand → estimate capacity → evaluate SLO state → guarded replica decision ⇓ desired / safe replicas Target Deployment / Pods ONNX Runtime server Ready after download + init + warmup Exports serving metrics Fig. 2. InferScale architecture. The custom resource provides policy, Prometheus provides serving metrics, and the controller updates Deployment replicas through the Kubernetes API.

control, fast-path handling, and priority-aware decisions than on predictor complexity. Baseline replica demand is computed from the larger of current and predicted QPS divided by estimated per-replica capacity. Capacity is tracked with an EWMA over healthy observations only. Under overload, observed throughput drops due to queueing and readiness lag, not because true capacity changed; updating from those samples would underestimate capacity and amplify oscillation. C. SLO-state control and fast path The controller evaluates the ratio between observed P95 latency and the configured SLO target and classifies the service as Healthy, Pressure, or Violation. In the reported experiments, Pressure begins at a P95/SLO ratio of 0.7 and Violation begins at 1.0. Headroom increases with severity: Pressure triggers more aggressive scaling than Healthy, while Violation adds an emergency step. A separate fast path handles rapid load increases. In the shared-cluster experiments, fast-path spike detection is triggered when the relative QPS growth over the recent control window exceeds 0.5, allowing the controller to add headroom before the smoothed trend fully converges. This matters near the rising edge of a burst, where a predictor alone can still lag.

InferScale supports tenant classes such as Gold, Silver, and Bronze. In the reported multi-tenant experiments, each custom resource specifies a tenant class, a per-tenant P95 SLO, a 60-second forecast window, and a 120-second scale-down cooldown. Under scarcity, the controller uses tenant class to differentiate headroom so that independent scaling decisions do not collapse into winner-takes-all behavior. This mechanism is related to keeping spare capacity for important tenants, but it is applied through the autoscaling policy rather than through a separate warm-pool manager. Static extra capacity is a strong operational baseline for burst protection, especially for high-priority tenants, but it also pays idle cost continuously. InferScale instead adds headroom when the observed and predicted state justify it. Warm pools and preprovisioned standby pods are therefore complementary: they would reduce the physical cold-start bound that any autoscaler faces, while InferScale decides when that capacity should be requested. To remain practical under noisy scraped metrics, InferScale combines bounded per-cycle scaling with cooldown-based scale-down. The exact thresholds are configuration parameters; in the experiments reported here, the scale-down cooldown is set to 120 seconds. The goal of these guardrails is not sophistication, but stability. IV. E VALUATION We evaluate three questions: • Q1. Does InferScale outperform reactive baselines on real model serving? • Q2. Does it produce more predictable outcomes under multi-tenant contention? • Q3. Which components matter most? A. Setup Cluster. All experiments run on AKS in West US 2 with two D2s_v3 system nodes and three D4s_v3 user nodes. Real-model workload. Each pod requests 500m CPU and 512 MiB (Guaranteed QoS). The container downloads ResNet50 from Azure Blob, loads it into ONNX Runtime, runs 60 warmup inferences, and then reports ready. Cold start is approximately 21 s. Effective capacity is approximately 2– 3 QPS per pod at the configured 1500 ms P95 target. Multi-tenant workload. Three tenants (Gold/Silver/Bronze) share the cluster using a controlled service with configurable latency and startup delay. This isolates allocation behavior and enables repeatable trials. Comparison fairness. All single-service methods use the same pod resource requests, the same cluster budget, and the same min/max replica bounds. The multi-tenant experiments likewise use the same cluster, per-tenant resource requests, and replica bounds across methods. Differences therefore come from control decisions rather than a looser resource budget. Unless otherwise noted, we use a 60-second forecast window, a 120-second scale-down cooldown, and fixed forecasting and guardrail settings across InferScale runs.

Baselines. Real-model experiments compare InferScale against both CPU HPA (target 50% utilization), the operational baseline many deployments start with, and a stronger externalmetric HPA based on queue depth. The external-metric HPA uses the same exported queue-depth metric as InferScale, a target value of 8 queued requests, and the same single-service replica bounds (1–10). In the multi-tenant experiments, each tenant has its own external-metric HPA with the same queue target; Gold is allowed up to 10 replicas, while Silver and Bronze are allowed up to 8, matching the InferScale custom resources. We did not add extra smoothing or forecasting to this baseline, because the point of comparison is a well-tuned reactive controller over inference-specific metrics. The motivation experiment additionally includes static provisioning, showing that reactive control remains limited by startup delay while static capacity pays an idle-cost penalty. We separate the single-service and multi-tenant studies because they answer different questions. The real-model study tests whether the controller helps under realistic startup and serving costs. The controlled multi-tenant study asks whether the allocation outcome becomes easier to predict under sharedcluster pressure. This is still not a full real-model multi-tenant validation. The baseline choice follows the same logic. The motivation experiment includes external-metric HPA and static provisioning to show that even a stronger reactive controller still misses short bursts. Q1 includes both CPU HPA and external-metric HPA so the paper is not reduced to a CPU-HPA comparison. Q2 keeps the stronger reactive baseline as well, so the contention result is not attributed to a weak comparator. The paper also uses three different evidence levels. CPU HPA is kept as an operational reference even though severe overload reduces usable coverage. The stronger reactive baseline is repeated where the main concern is comparator stability. Ramp-burst single-service results therefore rely most on repeated comparisons, whereas diurnal and priority-ablation results are interpreted more conservatively. Reproducibility. The implementation and experiment setup are configuration-driven. The controller behavior is defined through Kubernetes manifests and autoscaler custom resources, while the reported runs are generated by the same load scripts, Prometheus queries, and deployment templates across baselines. The artifact set includes controller and HPA manifests, repeated-run directories with per-run logs and replica traces, and the figure-generation scripts used in the paper. We do not claim push-button reproduction on arbitrary clusters, but the pipeline from configuration to metrics to reported tables is explicit enough to audit. Different parts of the evaluation have different noise profiles. Burst-heavy single-service runs are sensitive to startup timing and control cadence, so repeated comparison matters there. By contrast, the Gold-tier multi-tenant result remains stable across repeated contention runs, while the lower-tier story is better understood as a policy tradeoff than as a lowvariance performance claim.

TABLE I R EAL - MODEL VALIDATION (R ES N ET-50, ONNX RUNTIME , P95 < 1500 MS ). L ATENCY IN MS . CPU-HPA IS SHOWN AS THE OPERATIONAL SINGLE - RUN REFERENCE BECAUSE LOADGEN TIMED OUT UNDER OVERLOAD . R AMP E XTERNAL -HPA AND I NFER S CALE RESULTS ARE REPEATED MEANS OVER THREE RUNS . D IURNAL E XTERNAL -HPA AND I NFER S CALE RESULTS ARE REPRESENTATIVE SINGLE - RUN VALUES FROM THE SAME SETUP. Experiment

SLO%

P95 med

Avg Repl.

Max Repl.

ramp-cpu-hpa ramp-ext-hpa ramp-inferscale

30.6 47.8 ± 1.5 65.3 ± 9.0

3494 1650 1097

1.5 2.1 5.5

3 7 7

diurnal-cpu-hpa diurnal-ext-hpa diurnal-inferscale

58.6 74.9 80.3

1268 618 572

2.0 5.7 5.1

3 10 7

B. Q1: Real-model validation Table I shows the updated real-model comparison. Externalmetric HPA is a substantial upgrade over CPU HPA, and repeated ramp-burst runs still favor InferScale: 65.3% ± 9.0% versus 47.8% ± 1.5%. We repeated the stronger reactive comparator and the main method in this ramp setting because the main concern was whether the apparent gain came from baseline or controller instability rather than from the design itself. The repeated ramp results also sharpen the paper’s scope. Under the ramp burst, the gap remains large. Under the diurnal pattern, the two methods are much closer (74.9% vs 80.3%), so we do not treat that regime as a headline win. The contrast between the two workloads is itself informative. In the ramp case, the system spends most of the critical interval deciding and waiting: by the time a reactive controller has enough evidence to scale, much of the useful window is already gone. In the diurnal case, load changes more gradually, so a stronger reactive baseline has time to catch up and the predictive advantage shrinks. The repeated ramp runs are informative in another way as well. InferScale has noticeably higher variance than the stronger reactive baseline. That is not necessarily a bug. External HPA is more deterministic because it waits for a clearer signal and reacts later. InferScale moves earlier, so it is more exposed to timing details in scrape cadence and control alignment. What matters is that the separation between the two systems still holds at the regime level: the worst InferScale ramp run remains above the best repeated external-HPA run. If an operator only cared about minimizing run-to-run variance, the stronger reactive baseline would look attractive. But that stability comes from reacting late and giving up the early part of the burst almost entirely. InferScale accepts more timing sensitivity in exchange for consistently operating in a better part of the latency curve during the interval that actually matters. A closer look at the time series explains the difference. In the ramp run, the reactive baselines remain underprovisioned through the 6 QPS and 12 QPS phases, then begin scaling only after the service is already in sustained distress. InferScale, by contrast, reaches six replicas around the start of the heavy-

TABLE II R AMP - BURST PHASE ANALYSIS ( REAL MODEL , CPU HPA VS I NFER S CALE ). L ATENCY IN MS .

System

0–60s 2 QPS

60–120s 6 QPS

120–180s 12 QPS

180–240s 18 QPS

CPU HPA SLO% IS SLO%

100% 100%

0% 91%

0% 66%

0% 8%

CPU HPA P95 IS P95

628 306

3600 778

3606 1395

5380 2165

TABLE III S CALING TIMELINE UNDER RAMP BURST ( REAL MODEL , CPU HPA VS I NFER S CALE ).

Fig. 3. Ramp-burst latency traces for CPU HPA and InferScale. The burst window is short relative to pod startup time, so waiting for clear reactive evidence leaves the controller behind the workload. InferScale moves earlier and keeps the service closer to the target operating region.

load phase and seven replicas near the peak. This does not mean prediction is perfect. It means the controller moves early enough to keep the service closer to its operating region. Figure 3 makes the same point visually. CPU HPA spends most of the burst above the SLO line and only begins to recover after load is already falling. InferScale still violates the SLO near the peak, but it spends much more of the burst in the recoverable region rather than in prolonged distress. What matters here is not only the peak value, but how long the system stays in trouble once it falls behind. In the CPU-HPA run, the service enters overload and more or less stays there until demand is already on the way down. In the InferScale run, the service still gets stretched near the peak, but it does not spend the whole burst in that state. From an operator’s perspective, those two traces feel very different even before looking at the aggregate SLO numbers: one is a long period of visible service degradation, the other is a shorter period of strain followed by recovery. There is also an operational asymmetry. Under CPU HPA, the load generator timed out during sustained overload. External HPA and InferScale both complete with full coverage. The missing CPU-HPA data are therefore not just a measurement annoyance—they are part of the operational outcome of allowing the service to enter prolonged distress. The aggregate CPU-HPA ramp value in Table I is computed over available samples and includes recovery-period observations, whereas Table II focuses on the four ramp phases where the controller falls behind. Table II breaks the ramp result down by phase for the operational-default baseline, CPU HPA, versus InferScale. At 6 QPS, InferScale is already at six replicas and largely within SLO while CPU HPA is in complete violation with one replica. At 12 QPS, InferScale maintains 66% compliance near the operating boundary; CPU HPA still shows zero. Even at 18 QPS—which we observe to overload seven pods at the target SLO in this setup—InferScale degrades more

Time

CPU HPA replicas

InferScale replicas

0s 30 s 60 s 120 s 180 s 330 s

1 1 1 1 2 (first scale) 3 (too late)

1 3 (early scale-up) 6 6 7 6

gracefully (P95 2165 ms vs 5380 ms). The stronger external HPA baseline in Table I narrows the gap, but this detailed phase view shows why early scaling still matters. The absolute ramp SLO remains below 100% not because the controller stops helping, but because the combination of 21-second cold start and bounded cluster capacity leaves part of the burst unsupportable at the target SLO. The diurnal pattern tells a similar story: during the recovery phase (180–270s, 6 QPS), InferScale achieves 93% SLO compliance versus CPU HPA’s 78%, because it already has enough replicas from its earlier scaling response. Scaling timeline. InferScale’s first scale-up occurs within roughly the first 15 seconds (by the fourth 5-second sample), reaching six replicas before the heavy-load phase begins. CPU HPA first detects sustained pressure and issues a scale-up decision at roughly 168 seconds; by 180 seconds, the effect is visible in the observed replica count. This more-than-150second head start is the core mechanism behind the aggregate numbers. Table III highlights the timing gap between CPU HPA and InferScale. By the time CPU HPA creates its second replica at 180 seconds, InferScale is already at maximum capacity with seven replicas. CPU HPA reaches three replicas only at 330 seconds, when load has already dropped to 2 QPS. C. Q2: Multi-tenant contention Three tenants (Gold/Silver/Bronze) share a cluster with approximately 10.5 allocatable CPUs while total demand reaches 240 QPS during the stress phase. The key finding is not that all tenants become uniformly stable. Rather, InferScale makes contention outcomes more structured. Under the external-metric HPA baseline, the victim tenant changes from trial to trial: in one run Silver collapses

TABLE IV M ULTI - TENANT CONTENTION ( CONTROLLED SERVICE , MEAN ± STD , n=5).

Metric

External HPA

InferScale

Gold SLO attainment Silver SLO attainment Bronze SLO attainment

89.9 ± 16.1% 84.3 ± 22.1% 94.7 ± 11.8%

97.9 ± 1.2% 89.3 ± 23.8% 94.7 ± 11.9%

142 153 134

130 191 179

Gold median P95 Silver median P95 Bronze median P95

TABLE V P ER - TRIAL MULTI - TENANT SLO ATTAINMENT (%, n=5).

Tenant

t1

t2

Ext. HPA t3

Gold Silver Bronze

98.6 68.3 73.5

100 100 100

61.8 100 100

t4

t5

98.3 100 100

90.5 53.3 100

t1

InferScale t2 t3

t4

t5

97.3 100 100

97.4 100 100

96.4 99.9 73.3

99.1 46.7 100

99.0 100 100

while Gold remains healthy; in another, Gold is the one that fails. Under InferScale, the Gold-tier tenant is consistently protected, with 97.9% ± 1.2% SLO attainment and a worst case of 96.4%, which still exceeds the external-HPA Gold mean of 89.9%. Table V makes the distinction concrete. The external-metric HPA baseline produces trial-dependent victim selection: trial 1 hurts Silver, trial 3 hurts Gold, and trial 5 again hurts Silver. InferScale does not remove contention effects for all tenants. Instead, it makes the degradation pattern more policy-aligned: Gold remains consistently protected, while lower-priority tenants absorb more of the contention when resources are tight. This tradeoff is visible in the replica distributions as well. Under the external-metric HPA baseline, whichever tenant happens to scale first can capture most of the available headroom. Under InferScale, allocations are still differentiated, but the controller uses that differentiation to preserve Goldtier SLOs rather than allowing the losing tenant to vary unpredictably from trial to trial. The benefit is therefore not “universal stability,” but predictable priority enforcement under scarcity. This is not a free win: under scarcity, preserving Gold-tier SLOs can lead to worse latency for lower-priority tenants, which is visible in the Silver and Bronze median P95 values in Table IV. The Silver t5 and Bronze t4 outliers illustrate the limit of autoscaling without scheduler coordination: once contention becomes acute, a scheduler-blind controller can enforce policy priorities more reliably than it can eliminate lower-tier degradation. Figure 4 shows the same pattern visually. Averages hide part of the change: the cluster does not suddenly have enough capacity for everyone. What changes is who pays when capacity runs out. With the reactive baseline, that cost lands in different places in different runs. With InferScale, the cost lands more often where the policy says it should land. For a platform operator, that is the practical difference between an unstable service and a stable policy.

Fig. 4. Per-tenant SLO attainment across repeated trials (n=5). Bars show means and whiskers show the observed min–max range, bounded by the feasible 0–100% SLO interval. The external-metric HPA baseline yields trialdependent victim selection, whereas InferScale consistently protects the Goldtier tenant and makes degradation more priority-aligned under scarcity. TABLE VI A BLATION : SINGLE - SERVICE REAL MODEL (P95 < 1500 MS ). R AMP RESULTS ARE REPEATED MEANS (n=3); DIURNAL RESULTS ARE REPRESENTATIVE SINGLE - RUN VALUES .

Variant

Scenario

SLO%

∆

Full InferScale No-Predict No-FastPath

Ramp Ramp Ramp

65.3 ± 9.0% 59.5 ± 1.3% 57.3 ± 1.8%

— −5.8 pp −8.0 pp

Full InferScale No-Predict

Diurnal Diurnal

80.3% 74.0%

— −6.3 pp

D. Q3: Ablation Table VI shows that prediction and fast-path handling are complementary: in repeated ramp runs, removing prediction reduces SLO by 5.8 pp and removing fast-path handling reduces it by 8.0 pp. The predictor captures the upward trend, whereas fast-path reacts at the rising edge. Taken together, these results suggest that the practical gain does not come from prediction alone. Under startup-sensitive ramp bursts, exposing and acting on rising-demand signal early is at least as important as the forecasting component itself for preserving SLOs. The single-run diurnal result suggests a smaller effect under smooth load (−6.3 pp), which matches the broader observation that stronger reactive control catches up when demand changes more gradually. Table VII reports the priorityremoval ablation. Priority control matters under contention, but we treat this result as exploratory rather than definitive: without priority differentiation, Silver drops from 100% to 73%, which is consistent with the role of structured priority enforcement but not yet a stable estimate. E. Overhead We do not claim a comprehensive overhead study. In our setting, the dominant costs are pod startup (about 21 s in the real-model study), readiness delay, and request latency under overload rather than the autoscaler logic itself. InferScale

TABLE VII A BLATION : PRIORITY REMOVAL UNDER MULTI - TENANT CONTENTION ( SINGLE TRIAL , EXPLORATORY DIRECTIONAL EVIDENCE ).

Variant Full InferScale No-Priority

Silver SLO%

∆

100% 73%

— −27 pp

follows the standard operator pattern, queries a small set of Prometheus metrics, and patches Deployment replicas through the Kubernetes API. It does not modify the serving runtime or replace the scheduler. In that sense, the control-path cost is small relative to the startup and serving costs that dominate the evaluated scenarios. V. D ISCUSSION AND L IMITATIONS InferScale helps when there is enough trend signal to move before the burst fully lands. A pure step increase with no precursor is still hard because cold start is a physical limit. The paper also has a clear boundary. The real-model results are single-service, while the multi-tenant results use a controlled service. We did explore a real-model multi-tenant ONNX setting, but it did not strengthen the paper. Cold start remained dominant, and the per-pod CPU setting was not tight enough for scarcity-based prioritization to show up reliably. We therefore keep the two evidence lines separate. The updated multi-tenant results are useful partly because they are not a perfect win. InferScale does not make every tenant stable. Under the external-metric HPA baseline, the losing tenant changes from run to run. Under InferScale, Gold stays protected much more reliably, and the cost of that decision shows up on lower-priority tenants when the cluster is tight. This matters operationally: for a platform team, a predictable policy tradeoff is easier to reason about than random victim selection. There is also a measurement caveat in the real-model study. The CPU-HPA runs contain fewer data points because the load generator timed out under sustained overload. We therefore report SLO attainment over the available windows rather than over the nominal full schedule. This is not ideal, but the failures are part of the operational outcome rather than just a logging artifact. The current results come from one model family, two singleservice traffic patterns, and one repeatable shared-cluster setup. That is enough to support the paper’s main takeaways, but not universal gains across all inference workloads. Finally, this is an autoscaling paper, not a scheduling paper. The controller decides when and how much to scale; it does not decide where pods land. That choice keeps the design simple, but it also explains some of the lower-tier outliers under acute contention. If tighter guarantees are needed for lower-priority tenants as well, scheduler coordination is the next place to look. Generalizability. The current evaluation uses CPU-based inference with ONNX Runtime. GPU-backed inference would

change startup and capacity dynamics, and evaluating that regime is outside the scope of this paper. For LLM serving, capacity should likely be expressed in prefill/decode token throughput, KV-cache pressure, batching state, and GPU memory headroom rather than only QPS per replica. Startup also includes image pull, weight loading, GPU memory allocation, and sometimes cache warmup. The same underlying problem still matters there, but the capacity model, metrics, and guardrail settings would need workload-specific recalibration. InferScale also targets online inference rather than training jobs. Elastic training has different objectives, such as throughput, job completion time, checkpoint cost, and gang scheduling constraints; we do not claim that the current controller directly generalizes to that setting. VI. R ELATED W ORK Inference serving systems. Clipper [4] established the case for low-latency prediction serving. INFaaS [5], Cocktail [6], and InferLine [7] study adaptation, provisioning, and scaling for inference pipelines under changing demand. These systems reason about model selection, serving pipelines, and modelserving resources inside or close to the serving stack. InferScale is complementary: it operates at the Kubernetes replicacontrol layer and keeps the serving runtime unchanged. This makes the design less specialized than a serving-system scheduler, but easier to attach to ordinary Kubernetes deployments that already export request, latency, and queue metrics. Kubernetes and serverless autoscaling. Kubernetes HPA [1] provides the default replica-control mechanism used by many deployments. KEDA [8] and Knative [9] broaden the trigger set and support event-driven or scale-to-zero operation. These systems are valuable operationally, and our externalmetric HPA baseline uses inference-specific queue depth to avoid a weak CPU-only comparison. However, they remain largely reactive: once the metric crosses the target, pod startup and readiness still determine when capacity arrives. InferScale keeps the same Kubernetes control surface but changes the control policy around startup-sensitive inference. Predictive autoscaling. Predictive autoscaling has a long history in cloud systems, and MArk [10] is especially relevant because it uses forecasting to reduce reaction lag for machinelearning inference. InferScale shares that motivation but differs in three ways. First, it operates directly at the Kubernetes replica-control layer rather than assuming a specialized serving environment. Second, prediction is only one input to an SLOstate controller with fast-path spike handling. Third, the paper studies shared-cluster priority handling, where the question is not only whether the average service scales earlier but whether contention produces predictable policy outcomes. Resource management and cloud control. Borg [2], Autopilot [11], and FIRM [12] represent the broader clustermanagement and production autoscaling context in which our problem sits. These systems show that safe automation, resource isolation, and SLO-oriented control matter at production scale. InferScale addresses a narrower problem at the Kubernetes replica-control layer, but the same lesson applies:

useful autoscaling should be judged by operational outcomes, not only by prediction accuracy. Modern AI serving. vLLM [13] and Llumnix [14] focus on runtime-level optimizations such as PagedAttention, memory management, and request migration for LLM serving. Those mechanisms change the capacity surface exposed to an autoscaler: token throughput, batching efficiency, and KV-cache pressure may matter more than CPU utilization or raw QPS. InferScale does not replace such runtime mechanisms. Instead, it could consume their exported metrics in the same way it consumes queue depth and P95 latency in this paper. VII. C ONCLUSION We presented InferScale, a Kubernetes-native predictive autoscaler for AI inference that combines lightweight trend prediction, SLO-state control, and tenant-priority headroom without replacing the default scheduler. In repeated rampburst runs, InferScale lifts SLO compliance from 47.8% ± 1.5% under a stronger external-metric HPA to 65.3% ± 9.0%. Under repeated multi-tenant contention, the main benefit is not uniform stability but more reliable Gold-tier protection and a more policy-aligned degradation pattern. The main limitation is still evidence composition: the realmodel results are single-service while the multi-tenant results use a controlled service. The clearest next steps are real multi-tenant model serving under active scarcity and tighter coordination between autoscaling and scheduling. R EFERENCES [1] Kubernetes Community, “Horizontal pod autoscaler,” https://kubernetes. io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/, 2024. [2] A. Verma, L. Pedrosa, M. R. Korupolu, D. Oppenheimer, E. Tune, and J. Wilkes, “Large-scale cluster management at google with borg,” in Proceedings of the Tenth European Conference on Computer Systems (EuroSys), 2015. [3] C. C. Holt, Forecasting Seasonals and Trends by Exponentially Weighted Moving Averages. Carnegie Institute of Technology, 1957. [4] D. Crankshaw, X. Wang, G. Zhou, M. J. Franklin, J. E. Gonzalez, and I. Stoica, “Clipper: A low-latency online prediction serving system,” in 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 2017). USENIX Association, 2017, pp. 613–627. [5] F. Romero, Q. Li, N. J. Yadwadkar, and C. Kozyrakis, “INFaaS: Automated model-less inference serving,” in Proceedings of the 2021 USENIX Annual Technical Conference (ATC 2021). USENIX Association, 2021, pp. 397–411. [6] J. R. Gunasekaran, C. S. Mishra, P. Thinakaran, B. Sharma, M. T. Kandemir, and C. R. Das, “Cocktail: A multidimensional optimization for model serving in cloud,” in 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 2022). USENIX Association, 2022, pp. 1041–1057. [7] D. Crankshaw, G.-E. Sela, X. Mo, C. Zumar, I. Stoica, J. E. Gonzalez, and A. Tumanov, “Inferline: Latency-aware provisioning and scaling for prediction serving pipelines,” in Proceedings of the 11th ACM Symposium on Cloud Computing (SoCC), 2020.

[8] CNCF, “KEDA: Kubernetes event-driven autoscaling,” https://keda.sh/, 2024. [9] Knative Community, “Knative serving,” https://knative.dev/docs/ serving/, 2024. [10] C. Zhang, M. Yu, W. Wang, and F. Yan, “MArk: Exploiting cloud services for cost-effective, SLO-aware machine learning inference serving,” in Proceedings of the 2019 USENIX Annual Technical Conference (ATC 2019). USENIX Association, 2019, pp. 1049–1062. [11] K. Rzadca, P. Findeisen, J. Swiderski, P. Zych, P. Broniek, J. Kusmierek, P. Nowak, B. Strack, P. Witusowski, S. Hand, and J. Wilkes, “Autopilot: Workload autoscaling at google scale,” in Proceedings of the Fifteenth European Conference on Computer Systems (EuroSys), 2020. [12] H. Qiu, S. S. Banerjee, S. Jha, Z. T. Kalbarczyk, and R. K. Iyer, “FIRM: An intelligent fine-grained resource management framework for SLOoriented microservices,” in 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2020). USENIX Association, 2020, pp. 805–825. [13] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with PagedAttention,” in Proceedings of the 29th Symposium on Operating Systems Principles (SOSP 2023). Association for Computing Machinery, 2023, pp. 611–626. [14] B. Sun, Z. Huang, H. Zhao, W. Xiao, X. Zhang, and Y. Li, “Llumnix: Dynamic scheduling for large language model serving,” in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2024). USENIX Association, 2024, pp. 173–191.

Related documents

• InferScale: Kubernetes-Native Predictive Autoscaling for SLO-Aware AI Inference#1456829
• TriCalRAG: A Three-Strategy, Retrieval-Augmented Benchmark for On-Premise LLM-Based Root Cause Analysis in AIOps#1084686
• TriCalRAG: A Three-Strategy, Retrieval-Augmented Benchmark for On-Premise LLM-Based Root Cause Analysis in AIOps#919355
• Open Science, Open Data, and Open Scholarship: European Policies to Make Science Fit for the Twenty-First Century#703879
• ROAR: Unifying Runs across Heterogeneous AI-Driven Research Systems#1199240