arXiv:2609.20874v1 [cs.PF] 16 Sep 2026
Decomposing Predictive Kubernetes Autoscaling for Large Language Model Serving Under Long Startup Delays Tianrui Liu
Xiaohai Hu
University of California, San Diego La Jolla, CA, USA [email protected]
University of Washington Seattle, WA, USA [email protected]
Abstract—Large language model (LLM) inference deployed on Kubernetes faces an autoscaling challenge that conventional web services do not: new serving replicas take two to ten minutes to start because multi-gigabyte model weights must be loaded, which makes purely reactive scaling structurally late. We ask a sharp question: among the components of a predictive autoscaler, which ones actually matter under such long actuation delays? We answer it by decomposing predictive autoscaling into four factors—token-aware demand tracking, startup-delay lookahead, a bounded uncertainty margin, and plant-state observation—and measuring each factor’s contribution in isolation on productionderived heavy-tailed workloads generated by ServeGen. Our main finding is that a simple exponentially weighted moving average (EWMA) predictor with delay-aware lookahead and an upper confidence bound (UCB) margin captures most of the benefit, reducing time-to-first-token (TTFT) service-level-objective (SLO) violations from 53% (reactive, queries-per-second based) to 0.5% across five random seeds; lookahead alone is the single largest factor, a 14× reduction. Kalman filter variants do not consistently improve the cost–SLO tradeoff. Controlled experiments isolate the reason token granularity is necessary: context length, through key–value cache pressure, degrades TTFT far more than request rate at matched throughput. Finally, a validation on a real Kubernetes cluster (Qwen2.5-7B, A100, vLLM) confirms the central mechanism: a delay-aware lookahead controller cuts TTFT violations from 63.5% to 3.7% relative to reactive KEDA scaling. We distinguish throughout which findings are specific to Kubernetes actuation and which are general to LLM serving. Index Terms—LLM serving, autoscaling, Kubernetes, predictive control, cold start, KV cache, service-level objectives
I. I NTRODUCTION Large language models (LLMs) such as GPT, Llama, and Qwen are increasingly deployed as online inference services on Kubernetes clusters. Autoscaling these services—adjusting the number of serving replicas to track offered load—is harder than autoscaling traditional web services for two distinct reasons. The first is general to LLM serving, independent of the orchestrator; the second is specific to the Kubernetes actuation path. We are careful to separate them throughout this paper, because conflating them has led prior work to attack the wrong part of the problem. General to LLM serving: load is not requests. Each request processes a variable number of input tokens (the
prompt) and generates a variable number of output tokens (the completion). Inference proceeds in two phases with different resource profiles. The prefill phase processes all input tokens in one parallel forward pass; it is compute-bound and sets the time-to-first-token (TTFT). The decode phase emits output tokens one at a time auto-regressively; it is memory-bandwidthbound and sets the time-per-output-token (TPOT). During decode, the model holds the attention key and value tensors for every active sequence in GPU memory—the “KV cache.” A request with 4096 input tokens consumes roughly 16× the prefill compute and KV-cache memory of one with 256 tokens, yet a queries-per-second (QPS) signal treats the two identically. The KV cache, not GPU compute, is frequently the binding capacity constraint: compute can sit idle while a full cache blocks new admissions. Specific to Kubernetes: actuation is slow. Launching a new LLM serving pod requires pulling and loading model weights (14–140 GB for 7B–70B parameter models), initializing CUDA contexts, and capturing execution graphs. This takes two to ten minutes—orders of magnitude longer than the seconds it takes to start a stateless web pod. By the time a reactive controller observes overload and provisions capacity, the transient that triggered scaling has already caused sustained SLO violations. This is a delayed-actuation control problem: any reactive policy is structurally late by the startup delay ∆. Together these properties mean the autoscaler must predict future demand and act ∆ ahead of time. But a predictive autoscaler has many moving parts—demand estimators, lookahead horizons, uncertainty margins, plant-state observers, and increasingly elaborate state estimators such as Kalman filters. Which of these actually matter? Should practitioners invest in sophisticated predictors, or does a simple recipe capture the benefit? This is the central question of the paper, and to our knowledge it has not been answered with a controlled factor study. Approach. We implement a single predictive-autoscaling framework and instantiate it as a family of variants that differ in exactly one design factor at a time, so that each factor’s contribution can be read off directly. We evaluate on production-derived heavy-tailed workloads in a calibrated
event-driven simulator, complemented by a real-cluster validation of the central mechanism. We decompose the design into four factors: 1) Token-aware demand tracking: monitoring prefill/decode token rates rather than QPS. Reduces violations from 53% to 21%. 2) Startup-delay lookahead: predicting demand at t + ∆ instead of reacting to the present. Reduces violations from 21% to 1.4%—the largest single-factor improvement. 3) Bounded uncertainty margin: scaling for the upper confidence bound of predicted demand rather than the point estimate. Reduces violations from 1.4% to 0.5%. 4) Plant-state observation: tracking queue depth and KVcache pressure in addition to demand. Marginal SLO improvement at higher replica cost. Contributions. (1) A controlled factor decomposition showing that delay-aware lookahead is the dominant factor in predictive LLM autoscaling, with the UCB margin adding incremental robustness and more complex predictors (Kalman filters) yielding diminishing or negative returns (Section IV). (2) System identification and controlled experiments establishing that token granularity is necessary because context length, via KV-cache pressure, dominates TTFT degradation, while QPS alone is insufficient (Section IV-E). (3) A real-Kubernetes validation of the central timing mechanism: predictive lookahead reduces TTFT violations from 63.5% to 3.7% versus reactive KEDA (Section V). (4) An explicit accounting of which conclusions are Kubernetes-specific, which are general to LLM serving, and which depend on relative (not absolute) simulator fidelity. II. BACKGROUND AND S YSTEM M ODEL A. LLM Serving Systems State-of-the-art serving systems such as vLLM [1] and SGLang [2] employ continuous batching [3], dynamically admitting and retiring requests at each decode iteration rather than waiting for a batch to finish. The KV cache is managed via PagedAttention [1] in fixed-size blocks. Crucially, vLLM gates new request admission on KV-cache block availability (free_blocks ≥ watermark), not on compute utilization: when free blocks fall below the watermark, arriving requests wait regardless of how idle the GPU is. This cachegated admission rule is the architectural fact behind our system-identification result (Section IV-E) and is general to LLM serving, not specific to Kubernetes. B. Kubernetes Autoscaling The Kubernetes Horizontal Pod Autoscaler (HPA) [4] is a proportional controller, desired = ⌈ready × metric/target⌉, with a tolerance band and configurable stabilization windows. KEDA [5] extends HPA with external metric sources such as a Prometheus query on queue depth. Both are reactive and orchestrator-generic: neither distinguishes token-heavy from token-light requests at equal QPS (an LLM-serving gap), and neither compensates for the multi-minute pod startup
delay (the Kubernetes-specific gap). Our predictive framework addresses both, but our factor study shows the second gap is the one that dominates outcomes. C. System Model We make the serving topology and its assumptions explicit, since the autoscaler’s capacity arithmetic depends on them. Replicas. The deployment consists of R homogeneous replicas, each a full single-GPU copy of the model (tensor-parallel-size=1). Prefill and decode are colocated within a replica (continuous batching over a shared KV cache), not disaggregated across separate pools as in DistServe [6] or Splitwise [7]; the disaggregated case is out of scope and noted as future work. Load balancing. A request is dispatched to the ready replica with the shortest queue (join-shortest-queue). This keeps per-replica load approximately balanced, which is what licenses the aggregate-demand-divided-by-per-replica-capacity arithmetic below. Capacity constants. Each replica sustains a prefill throughput Cp and a decode throughput Cd (tokens/s). We estimate Cp =8000 and Cd =500 tokens/s per replica by profiling the simulator at its saturation point and matching it to Vidur’s [8] hardware-calibrated capacity wall (∼20 QPS per Llama-2-7B replica on an A100). These constants are deployment-specific inputs, not tuned per experiment; Section IV reports sensitivity to the resulting operating point via a parameter sweep. Control variables. The autoscaler observes, at a polling interval T , the aggregate prefill token rate pt , decode token rate dt , queue depth qt , and KV-cache utilization kt , all exported by vLLM as Prometheus metrics. It actuates a single integer, the replica count R, subject to a startup delay ∆ on scale-up. III. P REDICTIVE AUTOSCALING F RAMEWORK All variants share a four-step structure; they differ only in how demand is estimated and what state is observed, which is what lets us attribute outcomes to individual factors. Step 1: Demand estimation. At each interval (T =15 s) estimate pt and dt from vLLM’s prompt tokens total and generation tokens total counters. Step 2: Delay-aware lookahead. Predict demand at t + ∆ (the startup delay, typically 180 s for a 7B model on an A100). Using a backward-difference slope, p̂t − p̂t−T , p̂t+∆ = p̂t + ṗˆt (∆ − T2 ). (1) ṗˆt = T The ∆ − T /2 correction accounts for the slope being centered at t − T /2. For the first two intervals (t < 2T ) the slope is undefined; we set ṗˆt =0 and rely on the reactive floor p̂t plus the margin below. Step 3: Bounded uncertainty margin. Scale for an upper confidence bound rather than the point estimate. For each demand dimension i ∈ {p, d}, ducb = x̂t+∆,i + β σeff,i , i
σeff,i = min(σi , γ |x̂t+∆,i |), (2)
with β=1.5 and γ=0.8 unless noted. The cap is essential: without it, propagated uncertainty grows unboundedly with
the horizon, causing pathological over-provisioning at long ∆. EWMA-UCB derives σi from the EWMA of squared prediction errors; Kalman variants derive it from the covariance propagated over the horizon. Step 4: Replica computation. Convert demand to replicas via the capacity constants of Section II-C: ' ! & ducb ducb p ⋆ d , , Rmin . (3) R = max Cp Cd Scale-up is applied immediately; scale-down uses a 300 s stabilization window following HPA convention.
TABLE I FACTOR DECOMPOSITION ON THE S ERVE G EN WORKLOAD (5 SEEDS , MEAN± STD ). E ACH ROW ADDS ONE FACTOR RELATIVE TO THE ROW ABOVE ; K ALMAN VARIANTS SWAP IN A MORE COMPLEX ESTIMATOR .
Policy
Viol%
Repl
Design factor
HPA-QPS Token-Reactive EWMA EWMA-UCB KF-4state KF-6state Reactive-Plant
52.5±3.1 20.9±1.2 1.44±.68 0.54±.32 5.91±6.9 0.47±.30 19.0±1.2
2.1 4.8 9.9 9.5 9.2 11.1 6.0
(reactive, QPS) + Token demand + Lookahead + UCB margin + KF estimation + Plant state (react to q/KV)
Static(8)
0.18±.06
8.0
(hindsight oracle)
A. Variants The variants below each add exactly one factor relative to the previous, isolating its effect. Token-Reactive: tracks pt , dt via EWMA but scales on current smoothed demand, with no lookahead and no margin. Isolates token awareness from prediction. EWMA: adds delay-aware lookahead (predicts at t + ∆); demand only, no margin. The simplest token-aware predictive autoscaler. EWMA-UCB: adds the bounded margin. Maintains p̂t = αpt + (1−α)p̂t−1 (α=0.3), slope by consecutive differences, lookahead via Step 2, and σ from the EWMA of squared errors. Despite its simplicity it captures most of the benefit. KF-4state: replaces the EWMA estimator with a Kalman filter [9] over x = [pt , ṗt , dt , d˙t ]⊤ using a constant-velocity model (per stream Fs = [ 10 T1 ], Hs = [ 1 0 ], block-diagonal over prefill/decode). It adapts trust between observation and trend via the Kalman gain and yields a calibrated σ for the margin. KF-6state: extends the state with queue qt and KV pressure kt , with transition rows coupling queue growth to prefill demand (gain aq , drain bq ) and KV growth to decode demand (ak , bk ), so demand uncertainty propagates into plant-state predictions. Reactive-Plant: observes qt , kt but does not predict ahead; scales when thresholds are crossed. A Chiron-style [10] backpressure approach without delay compensation. IV. E VALUATION A. Experimental Setup Simulator. We build an event-driven simulator (SimPy) modeling continuous batching, chunked prefill, and KVcache block √ tracking, with sub-linear batch scaling titer (b) = 10 + 1.89 b − 1 ms calibrated against Vidur [8] (validated to within 9% of real A100 systems). Our simulator matches Vidur’s capacity wall (∼20 QPS per Llama-2-7B replica) and TPOT scaling (within 1.3×) but over-estimates absolute TTFT by roughly 2× due to simplified scheduling. We therefore restrict all simulator-based claims to relative comparisons between policies; absolute latency values are not used as evidence. The real-cluster validation (Section V) covers the absolute regime.
Workloads. We generate production-representative workloads with ServeGen [11], fit to Alibaba’s Bailian platform (3.5B requests over four months). Input lengths follow Pareto/log-normal distributions (mean 622, p95 1649, capped at 4096); outputs follow an exponential (mean 244, p50 66)— substantially heavier-tailed than Gaussian synthetic loads. Arrivals follow a burst pattern (5→15 req/s) with Gamma interarrivals. We also use a synthetic burst trace (Gaussian 256/128token lengths) for the cold-start sweep. Baselines and their tuning. HPA-QPS (8 req/s target per replica), KEDA-Queue (threshold 5 waiting requests per replica), and Static(N ) (a fixed allocation chosen after seeing the workload—a hindsight lower bound, not an online policy). All dynamic methods start at 2 replicas; default ∆=180 s. We do not claim the reactive thresholds are optimal. We swept neighboring thresholds for HPA and KEDA and the qualitative conclusion was unchanged: no reactive configuration reached the low-violation region without approaching static over-provisioning, because the limitation is structural (actuation latency), not threshold choice. The Pareto analysis below sweeps the predictive parameters so that no single configuration carries the argument, and Static(N ) provides the matched-cost over-provisioning reference the reactive baselines cannot reach online. Metrics. TTFT SLO violation rate (TTFT>2000 ms among completed requests), average replica count (proportional to GPU cost), and a conservative failure rate (violations + dropped + unfinished, over total). We report mean±std over five seeds. B. Factor Decomposition Table I is our main result; three patterns emerge. Lookahead is the largest single factor. Going from reactive token-aware scaling (20.9%) to EWMA with delay-aware lookahead (1.44%) is a 14× reduction. Without lookahead, even correct demand magnitude leaves the system structurally late by ∆. This is the factor practitioners should prioritize. Token granularity is the prerequisite. QPS-based HPA (53%) degrades to majority-failure, whereas token-aware reactive scaling (20.9%) already captures the right demand
EWMA-UCB KF-4state
KEDA-Queue
35
TTFT SLO Violation Rate (%)
TTFT SLO violation rate (%)
60
HPA-QPS
40
EWMA KF-4state
30
EWMA-UCB
25 20 15 10 5 0 0
100
200
300
400
500
600
Pod startup delay Δ (s)
KF-6state Static
50
40
30
20
10
0 4
5
6
7
8
9
10
11
12
Avg Replicas (Cost)
Fig. 1. TTFT SLO violation rate vs. pod startup delay ∆ on the synthetic burst workload (5 seeds). Predictive-UCB methods stay near zero up to 300 s; reactive baselines degrade monotonically with ∆.
magnitude. Token awareness is necessary but not sufficient; it must be paired with lookahead. The UCB margin adds robustness at no cost; complex predictors do not pay off. EWMA-UCB (0.54%, 9.5 replicas) vs. EWMA (1.44%, 9.9 replicas) shows the bounded margin catches tail under-predictions while the cap (γ=0.8) prevents over-provisioning—a 3× violation reduction at no replica cost (the 0.4-replica difference is within one standard deviation). Sophistication does not help: KF-4state is worse (5.9%, high variance ±6.9) because its constant-velocity model is mismatched to heavy-tailed bursts, and KF-6state reaches a marginally lower 0.47% only at 17% higher replica cost. Reactive-Plant (19%) confirms that observing plant state without predicting ahead is insufficient. C. Cold-Start Sensitivity Figure 1 varies ∆ on a 5→18 req/s ramp. Reactive baselines degrade monotonically: their violation rate is a direct function of how long the system stays under-provisioned waiting for capacity. Predictive-UCB methods stay near zero across 30– 600 s, because lookahead shifts the decision point earlier by exactly ∆ and the margin absorbs the larger prediction error at longer horizons. A practical consequence: one need not estimate ∆ precisely—any conservative upper bound suffices. We quantify the cost of that conservatism next. D. Cost–SLO Pareto Frontier To show the findings are not artifacts of a single parameter choice, we sweep β ∈ {0.5, 1.0, 1.5, 2.0, 3.0} for EWMAUCB and KF-4state and queue/KV thresholds for KF-6state (55 configurations). Figure 2 shows the frontier. EWMA-UCB at low β (aggressive) sits at 7–10 replicas with 3–8% violations and moves toward the Table I point as β grows; KF-6state reaches sub-1% only at 9+ replicas. No dynamic method dominates the hindsight-optimal Static(8). The adaptation premium—the extra cost of scaling without prior capacity knowledge—is ∼1.5 replicas for EWMA-UCB (9.5 vs. 8.0), the fundamental price of adapting to unknown variation rather
Fig. 2. Cost–SLO Pareto frontier on ServeGen (55 configurations across policies and parameter sweeps). Static(N ) is a hindsight-tuned lower bound, not deployable online. No dynamic method dominates Static(8); the adaptation premium is ∼1.5 replicas.
than manually right-sizing. For stable, predictable workloads, a right-sized static allocation remains more cost-efficient; the premium is justified precisely when load is uncertain. E. System Identification: What Drives Queue Buildup? To understand why plant-state observation helps only marginally, we apply system identification, fitting a model of the one-step queue change from current state over 1907 transitions (3 ServeGen seeds × 4 replica counts: 4, 6, 8, 10). A data-driven Ridge model attains one-step queue mean absolute error (MAE) of 0.04, versus 7.40 for the hand-written linear coupling assumed inside KF-6state—so the coupling KF-6state relies on is a poor model of the real dynamics, which is consistent with its failure to improve the tradeoff. Single-feature ablations show queue dynamics are strongly autoregressive (R2 =0.64 from queue history alone), while kv cache usage and request rate individually carry almost no marginal signal (R2 =0.001 and −0.004). This does not mean cache state is unimportant; rather, in observational traces demand, queue history, and KV usage are tightly correlated (r>0.89), so linear attribution among them is unreliable. To break the correlation we turn to controlled experiments. Table II provides the causal evidence the regression cannot. Holding QPS fixed at 8 and raising input length 32× (128→4096) raises KV usage from 2% to 34% and TTFT 17×—at identical request rate. Conversely, raising QPS 2.5× at short context degrades TTFT only modestly (55→95 ms). In systems with cache-gated admission, token weight matters more than request count—a finding general to LLM serving. The architectural basis is vLLM’s admission rule (Section III): longer contexts consume more cache per request and trigger admission blocking sooner, regardless of GPU idle time. This is why token-aware metrics, not QPS, are the right autoscaling signal, and it explains the first jump in Table I.
TABLE II C ONTROLLED ISOLATION (4 REPLICAS , SINGLE SEED ). AT MATCHED QPS, CONTEXT LENGTH DRIVES KV- CACHE PRESSURE AND TTFT FAR MORE THAN REQUEST RATE DOES AT SHORT CONTEXT.
Input Len
QPS
KV%
Queue/rep
TTFT P95
128 512 2048 4096
8 8 8 8
1.9 5.2 17.8 34.0
0.01 0.01 0.01 0.13
40 ms 89 ms 89 ms 693 ms
256 256 256 256
8 12 16 20
2.9 4.8 7.4 10.0
0.01 0.01 0.03 0.03
55 ms 69 ms 83 ms 95 ms
Results. Table III confirms the mechanism. The predictive controller detected the rising demand slope at t≈45 s—when active requests climbed from 0 to 19, before any queue formed—and scaled from 1 to 2 replicas; the new replica became ready at t≈105 s, just as the burst entered its highQPS phase. KEDA detected queue buildup only at t≈90 s, and its capacity arrived at t≈150 s, 45 s into the overload. The 45-second-earlier scale-up cut TTFT violations from 63.5% to 3.7% (P95 24.0 s→1.6 s). The direction and magnitude match the simulator’s qualitative prediction, supporting the relativecomparison claims of Section IV while remaining honest that absolute simulator latency is off by ∼2×. VI. R ELATED W ORK
TABLE III H ARDWARE RESULTS (Q WEN 2.5-7B, A100, V LLM V 0.11, ∆=60 S ).
Controller KEDA-Queue (reactive) Predictive (lookahead)
Viol%
P95
Scale-up time
Ready time
63.5
24.0 s
t≈90 s
t≈150 s
3.7
1.6 s
t≈45 s
t≈105 s
V. H ARDWARE VALIDATION The simulator establishes relative policy behavior; a real cluster is needed to confirm the central timing mechanism holds in the absolute regime. We therefore validate on hardware. We are explicit about scope: this experiment validates delay-aware lookahead as the decisive control-plane mechanism, not the full token-decomposed EWMA-UCB policy or its multi-seed statistics. Deployment. Qwen2.5-7B-Instruct on a Kubernetes cluster of NVIDIA A100 40 GB GPUs (p4d.24xlarge nodes), one GPU per vLLM pod, scaling the Deployment from 1 to 4 single-GPU replicas. Key configuration: tensor-parallel-size=1 (one GPU per replica); max-num-seqs=32 (matching the simulator’s batch cap); • gpu-memory-utilization=0.9.
• •
We inject a controlled 60 s startup delay via an init-container sleep, which isolates actuation delay from unrelated imagepull and model-load variability. Controllers (same Deployment, 1–4 replicas). KEDAQueue (reactive): a KEDA ScaledObject scales up when sum(vllm:num_requests_waiting)>5; 15 s polling. Predictive (lookahead): a controller reads vLLM metrics every 15 s, applies EWMA (α=0.3) to active demand (running + waiting), estimates the slope, predicts demand at t+∆ (∆=60 s, matching the injected delay), and adds the UCB margin (β=1.5, γ=0.8) before computing R⋆ . Workload (identical for both). 5→15→5 QPS over 5 minutes, mixed Pareto context lengths (256–4096), max_tokens=256, driven from a co-located client.
LLM serving optimization. vLLM [1] introduced PagedAttention; Orca [3] pioneered continuous batching; SarathiServe [12] chunks prefill; DistServe [6] and Splitwise [7] disaggregate prefill and decode; ServerlessLLM [13] reduces cold-start latency via optimized loading. These optimize intranode serving; we study inter-node autoscaling policy and treat the serving engine as given. LLM autoscaling. llm-d [14] provides Kubernetes-native distributed inference with KV-cache-aware routing and a workload-variant autoscaler that scales prefill and decode pools independently. Chiron [10] combines local batch-size adaptation with global queue-based instance scaling using queue-wait estimation as an SLO proxy. Both are important advances but remain fundamentally reactive—they scale after detecting overload. Our contribution is orthogonal: a controlled analysis of which predictive factors matter, with delay-aware lookahead identified as dominant. Recent predictive serving systems target demand forecasting directly; a head-to-head quantitative comparison under matched startup delay and workload is an important next step we leave to future work, as it requires reproducing those systems’ control planes on the same cluster. Predictive autoscaling (general). SHEPHERD [15] forecasts DNN-serving demand; Autopilot [16] predicts resource needs at Google scale; classical feedback-control theory [17] underpins such controllers. None addresses LLM-specific token costs, KV-cache dynamics, two-phase latency, or multiminute startup delays. Our empirical finding—that for LLM autoscaling, predictor sophistication matters far less than the delay-aware UCB framework—is, to our knowledge, new. VII. D ISCUSSION AND C ONCLUSION What is Kubernetes-specific vs. general. Two of our findings are general to LLM serving regardless of orchestrator: token weight dominates request count (Table II), and queue buildup is gated by KV-cache pressure (Section IV-E). One is specific to the Kubernetes actuation path: the multi-minute startup delay is what makes lookahead the dominant factor (Table I, Fig. 1). A serving stack with fast actuation would shift the balance away from lookahead and toward demand estimation.
Practical recommendations. (1) Deploy EWMA-UCB first: token-aware tracking + delay lookahead + a bounded margin (β=1.5, γ=0.8) cut violations from 53% to 0.5%; lookahead alone is 14×, the margin a further 3×. It needs only token-rate metrics and a margin scaled by ∆. (2) Monitor token throughput and gpu cache usage perc together, not QPS. (3) Budget for a ∼1.5-replica (∼19%) adaptation premium over hindsight-optimal static; static is cheaper only when load is predictable. Limitations. The simulator simplifies scheduling: absolute TTFT is ∼2× off Vidur, so we restrict simulator claims to relative comparisons and validate the absolute timing mechanism on hardware (Section V). The hardware experiment is intentionally small-scale and validates the timing mechanism, not the full token-decomposed policy or its statistics. All experiments use a single 7B model and colocated prefill/decode; larger models and disaggregated serving may exhibit different dynamics. Results use ServeGen-generated workloads; proprietary production traces would add validation. Conclusion. We decomposed predictive LLM autoscaling into four factors and found that delay-aware lookahead is dominant (14×), the UCB margin adds robustness at no cost, and more complex predictors yield diminishing or negative returns. Controlled experiments show token granularity is necessary because context-length-driven cache pressure—not request count—dominates TTFT degradation. A real-Kubernetes validation confirmed the mechanism: predictive lookahead cut TTFT violations from 63.5% to 3.7% versus reactive KEDA. The residual gap to oracle static provisioning is a scale-down control problem, not a prediction problem, motivating modelpredictive control over identified plant models as future work. We will release the simulator, controllers, workloads, and plotting scripts upon publication. R EFERENCES [1] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with PagedAttention,” in Proc. ACM SOSP, 2023. [2] L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, C. Barrett, and Y. Sheng, “SGLang: Efficient execution of structured language model programs,” in Proc. NeurIPS, 2024. [3] G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, “Orca: A distributed serving system for transformer-based generative models,” in Proc. USENIX OSDI, 2022. [4] The Kubernetes Authors, “Horizontal pod autoscaling,” https: //kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/, 2024. [5] The KEDA Authors, “KEDA: Kubernetes event-driven autoscaling,” https://keda.sh, 2024. [6] Y. Zhong, S. Liu, J. Chen, J. Hu, Y. Zhu, X. Liu, X. Jin, and H. Zhang, “DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving,” in Proc. USENIX OSDI, 2024. [7] P. Patel, E. Choukse, C. Zhang, A. Shah, I. Goiri, S. Maleki, and R. Bianchini, “Splitwise: Efficient generative LLM inference using phase splitting,” in Proc. ACM/IEEE ISCA, 2024. [8] A. Agrawal, N. Kedia, J. Mohan et al., “Vidur: A large-scale simulation framework for LLM inference,” in Proc. MLSys, 2024. [9] Y. Bar-Shalom, X. R. Li, and T. Kirubarajan, Estimation with Applications to Tracking and Navigation. Wiley, 2004.
[10] A. Patke, D. Reddy, S. Jha, C. Narayanaswami, Z. Kalbarczyk, and R. Iyer, “Hierarchical autoscaling for large language model serving with Chiron,” arXiv preprint arXiv:2501.08090, 2025. [11] Y. Xiang, X. Li, K. Qian, W. Yu, E. Zhai, and X. Jin, “ServeGen: Workload characterization and generation of large language model serving in production,” arXiv preprint arXiv:2505.09999, 2025. [12] A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, A. Tumanov, and R. Ramjee, “Sarathi-Serve: Efficient LLM serving with chunked prefills,” in Proc. USENIX OSDI, 2024. [13] Y. Fu, L. Xue, Y. Huang, A.-O. Brabete, D. Ustiugov, Y. Patel, and L. Mai, “ServerlessLLM: Low-latency serverless inference for large language models,” in Proc. USENIX OSDI, 2024. [14] Red Hat and Google and IBM, “llm-d: Kubernetes-native distributed LLM inference,” https://github.com/llm-d/llm-d, 2025. [15] H. Zhang, Y. Tang, A. Khandelwal, and I. Stoica, “SHEPHERD: Serving DNNs in the wild,” in Proc. USENIX NSDI, 2023. [16] K. Rzadca, P. Findeisen, J. Swiderski, P. Zych, P. Broniek, J. Kusmierek, P. Nowak, B. Strack, P. Witusowski, S. Hand, and J. Wilkes, “Autopilot: Workload autoscaling at Google scale,” in Proc. ACM EuroSys, 2020. [17] J. L. Hellerstein, Y. Diao, S. Parekh, and D. M. Tilbury, Feedback Control of Computing Systems. Wiley-IEEE Press, 2004.