Slant: Shaping Traffic for Resource-Efficient Microservices

Slant: Shaping Traffic for Resource-Efficient Microservices Congwei Zhang1

Chieh-Jan Mike Liang2

Mengbai Xiao1

Yifei Zou1

xczhw@mail.sdu.edu.cn

liang.mike@microsoft.com

xiaomb@sdu.edu.cn

yfzou@sdu.edu.cn

Shulun Wu1

Xiangyu Gao3

Xiuzhen Cheng1

Dongxiao Yu1

202335232@mail.sdu.edu.cn

gao-xy24@mails.tsinghua.edu.cn

xzcheng@sdu.edu.cn

dxyu@sdu.edu.cn

1 Shandong University

2 Microsoft Research

CPU in use

Microservice architectures rely on load balancers to distribute requests across service replicas, yet existing designs largely optimize for balanced routing without considering how routing shapes autoscaling signals and resource efficiency. We present Slant, a resource-aware load balancing framework that intentionally creates safe traffic imbalance to improve resource utilization. Slant profiles service-specific critical points, computes conservative multi-dimensional capacity, derives imbalance-aware routing weights that maintain hot and cold pods, and falls back to safer behavior under anomalies. By exposing clearer utilization signals, Slant works with existing vertical pod autoscalers (VPAs) without modification. Experiments on three microservice benchmarks show that Slant reduces allocated CPU cores by up to 26% while lowering P99 latency by up to 18% compared to Round-Robin. Under dynamic workloads, Slant triggers VPA scale-up 45% faster and completes scale-in 40% faster. Under anomalies, it recovers 40% faster from pod failures and limits latency amplification to 1.8× versus 2.6× during downstream slowdowns.

1

CPU utilization(%)

arXiv:2610.38778v1 [cs.DC] 28 Oct 2026

Abstract

3 Tsinghua University

Provisioned headroom

100 Scaled down by VPA

75 50 25 0

P1 P2 P3 P4 P5 P6 P7

P1 P2 P3 P4 P5 P6 P7

Figure 1: Balanced traffic obscures scaling signals and fragments reserved capacity, while intentional imbalance preserves similar workload with substantially lower total allocation.

shape the resource signals observed by downstream managers. Load balancers and resource management systems operate largely independently, lacking coordination. However, there exists an inherent coupling between these two systems that cannot be ignored. Auto-scalers, from threshold-based designs to learning-based systems, make decisions based on resource utilization metrics observed from each Pod [39–44], and these metrics are directly shaped by the traffic distribution determined by the load balancer. In other words, the load balancing strategy directly influences the input signals that auto-scalers rely upon. This is not an optional association but an architectural inevitability: traffic distribution determines the load on each Pod, which in turn determines the metrics observed by the scaler, ultimately affecting scaling decisions. The problem is that existing load balancing strategies are designed without any consideration of this downstream impact. This disconnect leads to tangible efficiency losses. When a load balancer evenly distributes traffic across all instances, each Pod operates at an ambiguous utilization level, typically hovering around 50–60%. For auto-scalers, this represents the most challenging input scenario: scaling up appears unnecessary while scaling down seems risky. The result is conservative inaction. Figure 1 illustrates this effect using seven Pods (P1–P7): in both bar charts, the orange portion represents used CPU and the gray portion represents allocated but unused CPU. Under balanced traffic distribution, all Pods

Introduction

Microservice architecture has become the dominant paradigm for cloud-native applications [1–3]. In such systems, a single user request often traverses dozens of services before completion, supported by modern deployment and observability infrastructures such as Docker, gRPC, Istio, Prometheus, and Jaeger [4–11]. The load balancer serves as a critical component in this path, determining which backend instance handles each incoming request. The effectiveness of load balancing strategies directly impacts system latency, throughput, and resource utilization. Existing load balancing and overload control strategies [12– 25], whether consistent-hashing-based designs [26–29], production reverse proxies [30–35], or more sophisticated application-aware variants [36–38], still largely optimize for balanced or responsiveness-oriented dispatch. While this seems reasonable, these approaches share a fundamental blind spot: they focus on how traffic should be distributed at the moment of routing, without considering how routing decisions 1

maintain similar moderate utilization and retain similar CPU allocation, providing no clear signal for scaling decisions. Meanwhile, each instance must reserve headroom to handle potential traffic bursts, but this headroom is scattered across the entire fleet and cannot be consolidated, leading to systematic resource fragmentation. In contrast, under skewed traffic, a few Pods run hot while several cold Pods remain lightly loaded and can be assigned much smaller CPU limits, allowing the system to sustain a similar amount of useful work with substantially less total allocated CPU. The deeper issue is that load balancing and auto-scaling pursue objectives that are not naturally aligned. Load balancing seeks instantaneous balance across instances, while vertical auto-scaling seeks efficient resource utilization over time [42, 43]. These goals can conflict. Consider a scenario where the VPA determines that a Pod’s CPU limit can be safely reduced to save resources. The ideal condition for this operation is a Pod with consistently low load that clearly indicates over-provisioning. However, a balance-oriented load balancing strategy continuously assigns traffic to this Pod, keeping it in an ambiguous state where scaling down never becomes safe to execute. Our key insight is that balanced traffic distribution does not equate to efficient resource utilization. In certain conditions, intentionally creating imbalance yields better outcomes. By directing traffic such that some instances operate at higher utilization while others remain lightly loaded, we achieve several benefits: heavily loaded instances emit clear signals for resource demand, and lightly loaded instances can have their resources scaled down safely by VPA. The critical question becomes: how much imbalance is safe? The answer depends on service-specific performance characteristics and resource sensitivity, which are known to vary substantially across systems and operating conditions [36, 42, 45]. Different services exhibit vastly different tolerance to resource pressure, and even within a single service, different resource dimensions such as CPU, memory, and I/O have different safety boundaries. Through our investigation, we make the following observations that guide our design. First, balanced traffic distribution obscures scaling signals, making it difficult for both threshold-based and learning-based auto-scalers to reach confident decisions. Second, performance critical points vary significantly across services due to differences in their computational characteristics and resource dependencies. Third, different resource dimensions exhibit different tolerance characteristics, with CPU allowing graceful degradation while memory exhibits cliff behavior upon exceeding limits. We elaborate on these observations with detailed measurements in Section 2. Based on these observations, we propose Slant, a resourceaware load balancing framework that leverages intentional traffic imbalance to maximize resource efficiency while satisfying SLO requirements and coordinating with existing auto-

scalers. At a high level, Slant combines a Service Profiler to learn service-specific performance boundaries, a Multidimensional Constraint Solver to estimate safe remaining capacity across resource dimensions, an Imbalance-aware Routing Optimizer to periodically compute routing weights that concentrate traffic on selected hot pods while preserving cold pods, and a Fallback Controller to detect anomalies and revert to safer behavior when needed. These control-plane decisions are continuously fed by monitoring signals and executed by a lightweight data plane, enabling Slant to improve scaling signals without modifying the underlying auto-scaler. Our main contributions are as follows: • We identify and analyze the overlooked coupling between load balancing and auto-scaling, demonstrating that existing balance-oriented strategies inadvertently create difficulties for resource management. • We propose the concept of intentional imbalance and develop Slant, a complete framework that profiles service critical points, enforces multi-dimensional safety constraints, computes imbalance-aware routing weights, and provides fallback control under anomalies. • We show how Slant works with existing vertical autoscalers by creating clearer scale-out and scale-in signals, enabling faster and more decisive resource adjustment without changing the auto-scaler itself. On Social Network, Slant triggers the first VPA scale-up 45% faster and completes scale-in 40% faster than conventional balanced routing. • We evaluate Slant on multiple microservice benchmarks and anomaly scenarios. Slant reduces allocated CPU cores by up to 26% while also lowering P99 latency by up to 18% and reducing SLO violation minutes by 60%. Under failures and downstream slowdowns, it recovers 40% faster from pod failure and limits latency amplification to 1.8× versus 2.6× for Round-Robin.

2

Background and Motivation

Our goal is to improve resource efficiency for microservice applications while satisfying end-to-end latency SLOs. The motivating measurements in this section are conducted on Social-Network from DeathStarBench [46], deployed on a smaller Kubernetes cluster with 160 CPU cores across five nodes. We use Locust to generate workloads following realworld diurnal patterns, with a 200ms P99 latency SLO. Our full end-to-end evaluation in Section 5 is performed on a larger 20-node cluster.

2.1 Balanced Distribution Obscures Scaling Signals Auto-scalers rely on resource utilization metrics to make scaling decisions. Learning-based Vertical Pod Autoscaler 2

Ambiguous Zone VPA Threshold

Pod 1 Pod 2

Pod 3 Pod 4

600

80

Ambiguous Signal

60 40 20 0

user-timeline compose-post media-filter nginx-thrift

Scale-up Signal

P99 Latency (ms)

CPU Utilization (%)

100

Consolidation Opportunity 0

50

100

150 Time (s)

200

250

300

0

50

100

150 Time (s)

200

250

300

400

Critical Zone

200

Figure 2: CPU utilization under balanced versus imbalanced distribution. Balanced distribution obscures scaling signals.

SLO Threshold

0

(VPA) approaches like Autothrottle [42] learn the relationship between CPU throttle ratios and latency to adjust resource limits. They assume that observed metrics accurately reflect true resource pressure. However, balanced load distribution causes all replicas to operate at similar, moderate utilization levels, creating ambiguity for auto-scalers. Consider a service with four replicas each at 55% CPU utilization. This presents a weak signal: the auto-scaler cannot easily distinguish genuine resource pressure from transient fluctuations, leading to conservative resource allocation. Figure 2 illustrates this phenomenon for the compose-post service. Under Round-Robin distribution, all replicas hover around 50–60% utilization, providing no clear scaling signal for VPA to reduce allocations. Under imbalanced distribution (70% traffic to two replicas, 30% to others), utilization divergence becomes apparent: heavily loaded replicas signal actual capacity needs, while lightly loaded ones clearly indicate safe scale-down opportunities for the VPA.

20

40 60 CPU Utilization (%)

80

Figure 3: P99 latency versus CPU utilization for four services. Critical points range from 60% to 85%. 2000

OOM Killed

P99 Latency (ms)

1000 500 200 100 50 20 10

20

40 60 CPU Utilization (%)

80

100

2048

1536 1024 512 Available Memory (MB)

64

Figure 4: Latency under CPU (left) versus memory (right) pressure. CPU degrades gradually; memory exhibits cliff behavior.

2.3

Resource Dimensions Differ in Tolerance

CPU and memory exhibit fundamentally different behavior under pressure. CPU causes gradual degradation, meaning the CFS scheduler throttles processes and increases latency predictably. Memory pressure remains stable until the limit is reached, then triggers GC pauses or OOM kills with catastrophic latency impact. Figure 4 shows this for post-storage-mongodb. CPU pressure (left) increases latency smoothly from 15ms to 120ms. Memory pressure (right) maintains 20ms latency until dropping below 512MB, then spikes to over 2 seconds.

Observation 2.1. Balanced traffic distribution causes similar utilization across replicas, preventing auto-scalers from identifying clear scaling opportunities.

2.2 Performance Critical Points Vary Across Services Different microservices exhibit different performance degradation patterns under resource pressure. Some degrade gracefully until roughly 85% utilization; others experience sharp latency increases at 60–70% due to contention or queuing effects. Figure 3 profiles the latency-utilization relationship for four services. The user-timeline service shows gentle degradation, while compose-post has a sharp inflection at 65% utilization. The media-filter ML service displays cliff behavior at 75%. These differences arise from varying compute patterns, synchronization requirements, and external dependencies.

Observation 2.3. CPU allows gradual degradation suitable for reactive scaling; memory exhibits cliff behavior requiring larger safety margins.

2.4

Design Implications

Our observations reveal that balanced distribution obscures scaling signals, fragments headroom, and ignores heterogeneous service characteristics. This motivates intentional imbalance, which is carefully calibrated to per-service critical points and multi-dimensional constraints. Such an approach provides clearer signals, consolidates headroom, and maintains SLO compliance.

Observation 2.2. Microservices exhibit different degradation patterns, with performance critical points varying from 60% to 85% utilization. 3

ally kept lightweight. It performs no complex computation, relying solely on table lookups and random number generation. This design ensures minimal latency overhead to request processing. Monitoring Plane. The monitoring plane continuously collects resource metrics (CPU, memory, I/O, network) and service-level metrics (latency percentiles, error rates) from all pods. These metrics serve two purposes. First, they feed into the control plane to inform routing decisions. Second, they enable real-time anomaly detection. When the anomaly detector identifies signs of overload or degradation, it triggers the Fallback Controller to take corrective action.

Figure 5: Architecture overview of Slant. The system consists of three planes: the control plane computes routing decisions, the data plane executes request forwarding, and the monitoring plane collects metrics and detects anomalies.

3 3.1

Interface with Vertical Auto-scalers. A key design goal of Slant is to work seamlessly with existing auto-scaling infrastructure. Rather than replacing the auto-scaler, we enhance the signals it receives. By intentionally creating load imbalance, Slant causes some pods to exhibit high utilization (clearly signaling the need for scaling CPU limits up) while others remain near-idle (clearly eligible for scaling CPU limits down). Standard vertical auto-scalers and learningbased resource managers such as Autothrottle and Sinan can then make more decisive scaling decisions based on these clearer signals, without any modification to the auto-scaler itself [42, 43].

System Design System Overview

Figure 5 illustrates the overall architecture of Slant. The system is organized into three planes that work together to achieve imbalance-aware load balancing. Control Plane. The control plane serves as the brain of Slant, responsible for computing and coordinating routing decisions. It consists of four core components. The Service Profiler builds and maintains a multi-dimensional profile for each service, characterizing its critical points across different resource dimensions. The Multi-dimensional Constraint Solver takes the profiled critical points and real-time resource metrics as input, computing the remaining capacity of each pod while ensuring safety across all resource dimensions. The Imbalance-aware Routing Optimizer uses these capacity estimates to compute optimal routing weights that intentionally create load imbalance by concentrating traffic on some pods while keeping others lightly loaded. The Fallback Controller monitors for anomalies and triggers corrective actions when the system deviates from expected behavior. The control plane operates on a periodic basis (every 5 seconds by default) to update routing weights. This design choice balances responsiveness with stability: frequent updates enable quick adaptation to load changes, while avoiding the oscillation that per-request decisions might cause.

3.2

Service Profiler

The Service Profiler addresses a fundamental challenge: determining how much load each service can safely handle. Different services have vastly different critical points, which denote the utilization level beyond which performance degrades unacceptably. A stateless computation service might tolerate 85% CPU utilization without issues, while a service with complex lock contention might start suffering at 50%. The Profiler’s job is to learn these critical points for each service across multiple resource dimensions. 3.2.1

Critical Point Modeling

We model the critical point not as a single threshold but as a probability distribution. This reflects the reality that performance degradation is not a sharp cliff but a gradual slope, and that the exact point of degradation varies with workload characteristics, time of day, and other factors. Formally, for service s and resource dimension d (e.g., CPU, memory), we model the critical point as a random variable Cs,d following a distribution parameterized by (µs,d , σs,d ). The probability that utilization level u is safe can be expressed as:

Data Plane. The data plane handles the actual request forwarding. We implement the data plane logic as a sidecar proxy co-located with each service instance, following the service-mesh deployment model popularized by systems such as Istio and Envoy [7, 31]. Upon receiving an incoming request, the sidecar consults a local weight table (periodically synchronized from the control plane) and selects a target pod using weighted random selection. The data plane is intention-

 P(safe | u, s, d) = P(Cs,d > u) = 1 − Φ

4

u − µs,d σs,d

 ,

(1)

3.3

where Φ is the cumulative distribution function of the standard normal distribution. When making routing decisions, we operate at a conservative percentile (e.g., the 10th percentile of the distribution), ensuring high confidence that we remain within safe bounds. 3.2.2

The Multi-dimensional Constraint Solver addresses a critical insight: different resource dimensions have fundamentally different characteristics and failure modes. CPU overload causes slowdown but is recoverable; memory exhaustion causes crashes with no graceful degradation. A routing decision that looks safe from a CPU perspective might be dangerous from a memory perspective. The Solver’s role is to compute a unified “remaining capacity” score for each pod that respects constraints across all dimensions.

Hybrid Profiling Methodology

The Profiler employs three complementary methods to estimate and update critical point distributions, as each method has distinct strengths and limitations.

3.3.1

Offline Stress Testing. Before a service enters production, we conduct controlled stress tests in a staging environment. We gradually increase the load while monitoring latency percentiles (P50, P99, P999) and error rates. The critical point is identified as the utilization level where latency exceeds acceptable thresholds (e.g., P99 latency increases by more than 50% from baseline) or errors begin to appear. This method provides accurate initial estimates but cannot capture production-specific behaviors or long-term drift.

Per-dimension Capacity Modeling

For each pod p and resource dimension d, we compute the remaining capacity as the gap between current utilization and the conservative critical point: conservative Cap p,d = Cs(p),d −U p,d ,

(2)

where s(p) denotes the service that pod p belongs to, conservative is the conservative (e.g., 10th percentile) critiCs,d cal point from the Profiler, and U p,d is the current utilization of pod p on dimension d. Different dimensions require different safety margins based on their failure characteristics. For CPU, we allow aggressive utilization up to the critical point (typically 75–85%), since CPU overload manifests as increased latency but recovers quickly when load decreases. For memory, we maintain a substantial safety margin (typically 25–35% below the limit), because memory pressure can trigger garbage collection storms, and exceeding limits causes immediate OOM kills with no opportunity for graceful handling. For I/O, safety margins depend on the storage medium, such that SSDs tolerate higher utilization than HDDs. We also account for I/O queue depth, as high queue depths cause latency spikes even at moderate throughput. For network, we consider both bandwidth utilization and connection counts, since high connection counts can exhaust file descriptors or cause connection tracking table overflow.

Online Probing. In production, we occasionally and carefully probe the boundaries of known-safe regions. When a pod’s utilization approaches the estimated critical point with high headroom in other pods (ensuring we can absorb traffic if problems arise), we allow slightly higher utilization and observe the outcome. Successful probes (no degradation observed) cause us to slightly increase our estimate of µs,d ; degradation events cause us to decrease it. We implement this as a conservative bandit algorithm that prioritizes safety over exploration. Historical Analysis. We continuously analyze historical data correlating resource utilization with service-level indicators (latency, errors). Using changepoint detection algorithms, we identify instances where performance degraded and correlate them with resource metrics at that time. This passive observation complements active probing by capturing degradation events that occur naturally, including rare edge cases that deliberate probing might miss. 3.2.3

Multi-dimensional Constraint Solver

3.3.2

Unified Capacity Score

The key insight of the Solver is that a pod’s true remaining capacity is determined by its most constrained dimension. We compute a unified capacity score as:

Handling Profile Drift

Service characteristics change over time due to code updates, data growth, and shifting traffic patterns. The Profiler addresses this through continuous adaptation and explicit invalidation. The online probing and historical analysis methods continuously refine estimates, allowing gradual drift to be tracked automatically. For abrupt changes (e.g., a new deployment), we detect the event through integration with the CI/CD pipeline and temporarily widen the confidence interval (increase σs,d ) until sufficient new observations accumulate.

Cap p = min wd · Cap p,d , d∈D

(3)

where D is the set of resource dimensions and wd is a normalization weight that accounts for different scales across dimensions. This min-based aggregation ensures that routing decisions respect all constraints simultaneously, treating a pod with 50% CPU headroom but only 5% memory headroom as nearly full. 5

3.3.3

Algorithm 1 Imbalance-aware Weight Computation

Request Impact Estimation

Require: Pods {p1 , . . . , pN }, capacity scores {Cap1 , . . . , CapN }, target cold pod ratio γ Ensure: Routing weights {w1 , . . . , wN } 1: Sort pods by current utilization: U(1) ≤ U(2) ≤ · · · ≤ U(N) 2: Ncold ← ⌈γ · N⌉ commentNumber of cold pods to maintain 3: C ← {p(1) , . . . , p(Ncold ) } commentCold pod set 4: H ← {p(Ncold +1) , . . . , p(N) } commentHot pod candidates 5: for pi ∈ C do 6: wi ← ε commentMinimal weight for cold pods 7: end for 8: Wremaining ← 1 − |C | · ε 9: for pi ∈ H do Capi 10: wi ← Wremaining · ∑ Cap

A remaining challenge is that the resource impact of an incoming request is unknown before it executes. We address this through statistical modeling of request costs. For each service, we maintain a distribution of per-request resource consumption across dimensions, estimated from historical observations. In the current system, this model is aggregated into the periodic capacity estimates used by the routing optimizer; when constructing these estimates, we consider the expected resource impact plus a safety margin: Impactd = E[costd ] + α · Std[costd ],

(4)

where α controls the conservativeness (typically 1–2 standard deviations). These impact estimates are folded into the per-pod capacity scores used for periodic weight computation, ensuring that routing weights are derived from conservative multi-dimensional feasibility estimates rather than raw utilization alone. For services with heterogeneous request types, Slant can be extended to be request-type-aware. If request metadata (e.g., API endpoint or query parameters) is available, the control plane can maintain separate cost distributions per request class and derive class-specific capacity estimates and weight tables. We leave this request-type-aware execution path as an extension rather than a required component of the current implementation.

3.4

p j ∈H

weight return {w1 , . . . , wN }

where Ui (w) = wi · λ · costavg is the expected utilization of pod i given routing weights w, τcold is the threshold for considering a pod “cold” (e.g., 10% utilization), and β is a reward for maintaining cold pods to facilitate scale-in.

Imbalance-aware Routing Optimizer

3.4.2

The Routing Optimizer is the core component that realizes our key insight: intentionally creating load imbalance to improve resource efficiency and scaling clarity. Unlike traditional load balancers that aim to equalize utilization across pods, our optimizer deliberately concentrates load on some pods while keeping others lightly loaded. 3.4.1

We formulate routing weight computation as an optimization problem. Let N be the number of pods for a service, λ be the total incoming request rate, and wi be the routing weight for pod i (where ∑i wi = 1). Let Capi denote the unified capacity score from the Constraint Solver. The objective function balances utilization maximization with maintaining cold pods:

3.4.3

N w

(5)

subject to: ∀i

(6)

N

∑ wi = 1,

wi ≥ 0,

Coordination with Auto-scaler

The routing strategy creates utilization patterns that naturally enhance auto-scaler effectiveness. When incoming traffic increases, the hot pods approach their capacity limits. Because we concentrate load on fewer pods (rather than spreading thinly across all), these pods generate clear resource-pressure signals earlier and more decisively. The VPA therefore observes stronger evidence that CPU limits should be increased.

i=1

Ui (w) ≤ Ciconservative ,

Practical Algorithm

Solving the full optimization problem at each decision epoch can be computationally expensive. We employ a practical heuristic algorithm that achieves near-optimal results efficiently. Algorithm 1 describes our weight computation procedure. The key idea is to partition pods into two groups: cold pods that should receive minimal traffic, and hot pods that should absorb the remaining load. Among hot pods, traffic is distributed proportionally to their remaining capacity, ensuring that pods with more headroom receive more traffic. The algorithm runs periodically (every 5 seconds) rather than per-request, striking a balance between adaptiveness and stability. Per-request decisions would enable finer-grained control but risk oscillation and add latency overhead.

Optimization Formulation

max ∑ (Ui (w) + β · 1[Ui (w) < τcold ])

j

11: end for 12: Verify: Ensure Ui (wi ) ≤ Ciconservative for all i 13: Adjust: If any constraint violated, redistribute excess

(7)

i=1

6

When traffic decreases, our algorithm prioritizes draining specific pods to near-zero utilization while maintaining reasonable load on others. The VPA then observes clear scaledown candidates whose CPU limits can be reduced safely, rather than a fleet of moderately utilized pods that all look ambiguous.

3.5

breathing room by spreading load evenly, reducing the risk of any single pod being overwhelmed. We maintain this fallback state until metrics stabilize, then cautiously re-engage imbalanced routing. At Level 3 (Emergency Escalation), for severe situations such as high error rates across multiple pods or rapid resource exhaustion, we take aggressive action. First, we enable rate limiting at the service entry point to cap incoming traffic at a sustainable level. Second, we send an explicit signal to the auto-scaler requesting emergency scale-out, bypassing normal threshold-based triggers. Third, we alert operators for manual investigation.

Fallback Controller

The Fallback Controller provides safety guarantees when things go wrong. Despite careful profiling and conservative decision-making, prediction errors are inevitable. Traffic patterns may shift unexpectedly, service behavior may change due to bugs, or cascading failures may occur in downstream dependencies. The Controller’s role is to detect problems quickly and take corrective action before they escalate. 3.5.1

3.5.3

After a fallback event, the system does not simply resume normal operation; instead, it learns from the incident. The Profiler receives feedback about the conditions that led to the anomaly, allowing it to adjust critical point estimates. If a pod showed distress at 75% CPU utilization when our estimate predicted safety up to 80%, we revise µs,CPU downward accordingly. We also implement a “cooling off” period after incidents. The system operates more conservatively (lower target utilization, wider safety margins) for a configurable duration, ensuring stability before returning to efficiency-optimized operation.

Anomaly Detection

The Controller continuously monitors service-level indicators to detect signs of distress. We track three primary signals. For latency deviation, we maintain an exponentially weighted moving average (EWMA) of P99 latency for each pod; when current P99 exceeds the EWMA by more than a threshold (e.g., 50%), we flag the pod as potentially overloaded. We use relative deviation rather than absolute thresholds to handle services with naturally different latency profiles. For error rate spikes, a sudden increase in error rate (e.g., HTTP 5xx responses, timeout errors) indicates severe problems; we monitor the error rate over a sliding window and trigger alerts when it exceeds baseline by a significant margin. For resource exhaustion warnings, even before latency degrades, we may observe leading indicators of impending problems: rapid memory growth approaching limits, CPU throttling events, or I/O queue depth spikes. These early warnings enable proactive intervention. 3.5.2

Recovery and Learning

4

Implementation

We implement Slant as a standalone system that integrates with Kubernetes-based microservice deployments. The current prototype consists of approximately 8,500 lines of Go code for the control plane and 1,200 lines of Lua code for the data plane extensions. The control plane runs as a Kubernetes service that periodically computes routing weights from monitored resource signals, while the data plane is implemented through Envoy sidecars that execute lightweight weighted forwarding decisions. Detailed descriptions of the experimental testbed, workloads, and metrics are deferred to Section 5.1.

Graduated Response

When anomalies are detected, the Controller responds with graduated actions based on severity. This design prevents overreaction to transient issues while ensuring serious problems are addressed promptly. At Level 1 (Targeted Traffic Shift), for mild anomalies such as latency slightly elevated on one pod, we reduce the routing weight for the affected pod while redistributing its share to healthy pods. This surgical intervention often resolves the issue without broader disruption. If metrics recover within a timeout period (e.g., 30 seconds), we gradually restore normal weights. At Level 2 (Fallback to Balanced), if Level 1 intervention fails to resolve the issue, or if multiple pods show signs of distress simultaneously, we temporarily abandon the imbalanced strategy and revert to round-robin distribution. This provides

5 5.1

Evaluation Methodology

Benchmark Applications. We evaluate Slant using three diverse microservice applications that represent different complexity levels and architectural patterns commonly found in production environments. Online Boutique is a lightweight e-commerce application from Google comprising 10 microservices with simple gRPC communications [47]. Most request flows involve only 2 to 7

Experiment Setup. We conduct experiments on a Kubernetes cluster with 20 worker nodes, each equipped with 32 CPU cores (Intel Xeon Gold 6326 @ 2.90GHz), 128GB RAM, and 1TB NVMe SSD, connected via 25Gbps Ethernet. We use Kubernetes 1.28 and deploy Autothrottle [42, 49] as our VPA, dynamically adjusting CPU limits to meet P99 latency SLOs. For load generation, we use a distributed setup with dedicated client machines running customized wrk2 [50] instances. Unless otherwise specified, we generate load following an open-loop model to accurately measure latency under varying conditions. Each experiment runs for at least 10 minutes after a 2-minute warmup period, and we report the average of three independent runs. We define an SLO violation as any request with end-toend latency exceeding the 99th percentile target: 100ms for Online Boutique, 200ms for Social Network, and 500ms for Train Ticket. These thresholds are based on typical production requirements for similar applications.

Comparison Baselines. We compare Slant against four baseline load balancing strategies: • Round-Robin (RR): Distributes requests equally across all available pods in a circular fashion. This is the default strategy in many production deployments due to its simplicity. • Least-Connections (LC): Routes each request to the pod with the fewest active connections, providing basic load awareness without resource monitoring. • Power of Two Choices (P2C): A classic randomized load balancing strategy that randomly selects two candidate pods and routes the request to the one with fewer active connections, avoiding the herd effect of LC.

CPU Cores

SLO Viol.

Boutique

Trace-Driven Workload. Beyond the benchmark applications, we also evaluate Slant under a 24-hour production trace derived from an anonymized large-scale web service. This trace-driven experiment is used to study long-running VPA behavior under diurnal dynamics and bursty demand, complementing the controlled benchmark-based comparisons above.

P99 (ms)

Round-Robin Least-Conn P2C Prequal Slant

52 49 48 46 45

48.5 48.0 47.5 47.0 43.5

0.02% 0.01% 0.01% 0.01% 0.00%

Social

Table 1: End-to-end performance summary at 60% load. Slant achieves the best resource efficiency (lowest CPU cores allocated) while maintaining competitive latency and zero SLO violations.

Round-Robin Least-Conn P2C Prequal Slant

156 148 142 139 128

115.0 113.5 112.0 110.5 81.5

0.15% 0.12% 0.08% 0.05% 0.02%

Train

3 services with minimal fan-out, making it ideal for measuring baseline performance and algorithmic overhead in low-complexity environments. Social Network, adapted from DeathStarBench [46], represents moderate complexity with 28 microservices including both stateless and stateful components. It features two MLbased services that introduce computational intensity and hardware acceleration dependencies. With typical request paths spanning 4 to 6 services, this workload effectively tests adaptive algorithms under conditions of service heterogeneity and mixed resource demands. Train Ticket, developed by Fudan University [48], simulates a comprehensive ticket booking system with 68 microservices. This compute-intensive application includes complex dependencies, request chains exceeding 10 services, and mixed RPC communications. Its large scale and varying latency profiles make it particularly valuable for stress-testing load balancing policies under high concurrency. All three applications run on Kubernetes within an Istio service mesh, ensuring a consistent environment that allows for direct comparison of load balancing behaviors across different workloads.

App

System

Round-Robin Least-Conn P2C Prequal Slant

412 395 378 362 338

285.5 282.0 278.5 275.0 210.5

0.28% 0.22% 0.18% 0.12% 0.05%

5.2

• Prequal [36]: A state-of-the-art adaptive load balancer that selects servers based on estimated latency and active requests-in-flight (RIF). It actively probes server load and leverages the power-of-d-choices paradigm, extended with asynchronous reusable probes, representing the best existing approach for microservice environments.

End-to-End Performance

We first evaluate the overall performance of Slant compared to baselines under steady-state conditions. For each benchmark application, we generate constant load at 60% of the maximum sustainable throughput and measure latency distribution, throughput, and resource utilization. Table 1 summarizes the end-to-end performance across all benchmark applications. On Online Boutique, all systems perform similarly due to the application’s simplicity, but Slant still delivers the lowest P99 latency (45ms) and the fewest

All baselines use the same Envoy-based data plane infrastructure to ensure fair comparison, differing only in their weight computation logic. 8

allocated CPU cores (43.5). On Social Network, Slant reduces P99 latency by 12% relative to Round-Robin and 8% relative to Prequal, while cutting allocated CPU cores from 115.0 to 81.5. On Train Ticket, the most complex benchmark, Slant lowers P99 latency from 412ms to 338ms and reduces allocated CPU cores from 285.5 to 210.5. Overall, Slant consistently requires the fewest allocated CPU cores, saving up to 26% compared to baselines, while also delivering the lowest P99 latency and fewest SLO violations. These results show that intentional load imbalance does not sacrifice service quality; instead, it improves both resource efficiency and performance by operating pods near their efficient region and consolidating scattered headroom.

5.3

sume cluster resources. 5.3.2

We then evaluate auto-scaling performance under a 24-hour diurnal trace from a production environment, applying continuous, dynamic load. Figure 6 shows the CPU cores allocated over time. Because Slant continuously creates clear resource pressure signals for VPA scale-up and clear under-utilization signals for VPA scale-in, its resource allocation curve tracks the actual load curve much more tightly than baselines. Over the 24-hour period, Slant achieves a 22% reduction in average CPU cores allocated compared to Prequal, and 28% compared to Round-Robin. Despite using fewer resources, Slant experiences 60% fewer SLO violation minutes than Round-Robin, demonstrating that tighter VPA tracking improves both efficiency and reliability.

Auto-scaling Effectiveness

A key hypothesis of Slant is that intentional load imbalance produces clearer signals for auto-scalers like VPA, enabling faster and more accurate resource adjustments. We evaluate this hypothesis using dynamic workloads that require the system to scale resource limits up and down. 5.3.1

Dynamic Workload Trace

5.4

Resource Efficiency under Varying Load

We now examine how resource efficiency varies across different load levels. To make the tradeoff clearer, we separate resource efficiency into two views: the number of pods required to sustain each load level, and the resulting average CPU utilization. Figure 8 summarizes how different policies trade off provisioned capacity and achieved utilization on Social Network. The left subfigure reports the average number of pods needed at each load level, while the right subfigure reports the corresponding average CPU utilization. First, Slant consistently requires the fewest pods across the entire load range. At 60% load, Slant uses 18 pods, compared to 23 for Round-Robin, 22 for Least-Conn, 21 for P2C, and 20 for Prequal. This corresponds to a 22% reduction relative to Round-Robin and still a 10% reduction relative to the strongest baseline, Prequal. Second, the CPU utilization plot shows the complementary effect of the same behavior. Baselines such as Round-Robin and Least-Conn spread requests more evenly, which leaves more partially utilized pods and keeps average utilization in the 40–56% range. P2C and Prequal improve this trend, but Slant drives utilization substantially higher, reaching 58–70% and staying close to the 70% target over the typical operating region. Third, the two subfigures together highlight the efficiency ordering among methods: Round-Robin is the least efficient, Least-Conn improves modestly, P2C provides a stronger baseline, Prequal performs better still, and Slant is best overall. The advantage is most visible in the moderate load regime (40–70%), where production services spend much of their time. At very low load, minimum pod-count constraints limit the gap, while at very high load all methods begin to converge because sufficient total capacity must still be provisioned to avoid SLO violations.

Response to Step-load Variations

We first evaluate how quickly each system triggers resource adjustments under sudden load shifts. Starting from a steady state at 40% capacity, we instantaneously increase load to 80% capacity at t = 0, hold it, and then drop it back to 40% at t = 300 seconds. This allows us to measure both scale-out speed (provisioning sufficient CPU cores to prevent SLO violations) and scale-in efficiency (reclaiming over-provisioned cores to save resources). Figure 7 shows the resource adjustment timeline on Social Network. During the scale-out phase (t = 0), Slant triggers the first VPA scale-up event within 18 seconds, while Round-Robin takes 33 seconds. This 45% improvement occurs because Slant concentrates load on hot pods, causing them to exhibit high latency and CPU throttling much faster. Round-Robin, conversely, spreads load evenly, so all pods slowly drift toward saturation without clearly signaling urgent demand. Slant reaches the target CPU core allocation in 52 seconds, compared to 78 seconds for Round-Robin. During this scaling period, Slant experiences only 0.8% SLO violations (vs. 2.3% for Round-Robin) because cold pods provide immediate overflow capacity. During the scale-in phase (t = 300), traditional load balancers face a fundamental challenge: balanced distribution causes all pods to drop to moderate utilization, leaving the VPA uncertain about how much resource can be safely reclaimed. Slant addresses this by actively draining specific pods to near-zero utilization. These pods become unambiguous scale-in candidates, allowing the VPA to act decisively. Consequently, Slant completes scale-in 40% faster than baselines, significantly reducing the time excess CPU cores con9

Allocated CPU Cores

Round-Robin

Least-Conn

P2C

Prequal

Slant

125 100 75 50 25 0:00

3:00

6:00

9:00

12:00 15:00 Time of Day (Hours)

18:00

21:00

24:00

Figure 6: Resource allocation under a 24-hour production trace. Slant enables the VPA to track load tightly, significantly reducing over-provisioned cores while minimizing SLO violations.

Request Rate (req/s)

Least-Conn P2C

Prequal Slant

800

100

600

80

400

60

200 0

120

40 0

100

200

300 Time (s)

400

500

imal. The system simply promotes another lightly-loaded pod to cold status and continues operation with negligible disruption.

Allocated CPU Cores

Request Rate (Load) Round-Robin

1000

5.5.2

Microservices often depend on external services whose performance may degrade unexpectedly. We inject a 2× latency increase in a downstream database service and observe the cascade effects. When downstream latency increases, requests take longer to complete, causing connection and thread pools to fill up. Traditional load balancers, unaware of this internal state, continue routing requests uniformly, potentially overwhelming all pods. Slant’s multi-dimensional monitoring detects the increased queue depth and connection count, triggering proactive traffic reduction to affected pods before they become completely unresponsive. In our experiment, Slant limits the latency increase to 1.8× (compared to 2.6× for Round-Robin) and prevents the cascading failure that causes 5% error rate in baseline systems.

Figure 7: VPA response to step-load variations. Slant accelerates both scale-out and scale-in by creating clear resource pressure and under-utilization signals.

Figure 9 shows that improved efficiency does not come at the cost of latency. Slant maintains equal or lower P99 latency across all load levels. The latency advantage is most pronounced at high load (80–100%), where Slant’s P99 latency is 15–20% lower than baselines. This occurs because Slant keeps cold pods genuinely cold, allowing them to absorb burst traffic without the queuing delays that occur when all pods are moderately loaded.

5.5

Robustness under Anomalies

5.5.1

Pod Failure Handling

Downstream Latency Increase

600

5.6

We simulate pod failures by terminating pods handling active traffic and measure recovery behavior. Figure 10 shows the recovery timeline. When a hot pod fails, Slant detects the failure within 2 seconds (via health check failure and connection errors) and redistributes its traffic to other pods. Because cold pods have substantial spare capacity, they absorb the additional load without significant latency increase. Full recovery (latency returning to baseline) takes 8 seconds for Slant compared to 14 seconds for Round-Robin, a 40% improvement. Notably, when a cold pod fails in Slant, the impact is min-

System Overhead

Table 2 summarizes the overhead. The control plane consumes 0.3 CPU cores and 256MB memory, which is negligible for a cluster managing thousands of pods. The data plane adds only 0.08ms per-request latency (measured as the difference between Slant and a pass-through proxy), which is less than 0.1% of typical request latency. The additional memory per sidecar (12MB for weight tables and routing logic) is minimal compared to Envoy’s base footprint. Network overhead for metrics collection and weight synchronization totals less than 3KB/s per pod, insignificant compared to application traffic. 10

P2C Prequal Typical Operating Range

40

20

0

300

Slant

P99 Latency (ms)

Average Pod Count

RR Least-Conn

22% fewer than RR

200

30%

40%

50% 60% 70% Load Level (%)

80%

90%

RR Least-Conn

80

P2C Prequal

High Load Zone

100

20% 30% 40% 50% 60% 70% 80% 90% 100% Load Level (%)

100%

Figure 9: P99 latency across load levels. Slant maintains lower latency than baselines, with the gap widening at high load due to better hot-cold pod management.

Slant

70

Prequal Slant

SLO Threshold: 150ms

0 20%

(a) Average pod count.

Target: 70%

60

P99 Latency (ms)

CPU Utilization (%)

Round-Robin Least-Conn P2C

50 40 30

20%

30%

40%

50% 60% 70% Load Level (%)

80%

90%

100%

(b) Average CPU utilization.

Round-Robin Slant

Pod Failure Injected

100

0

Figure 8: Resource efficiency across load levels on Social Network. Slant consistently requires fewer pods while maintaining higher CPU utilization. P2C improves over LeastConn, but remains less efficient than Prequal and Slant.

5.7

200

0

10

20 Time (s)

30

40

Figure 10: Recovery from pod failure. Slant recovers 40% faster because cold pods can immediately absorb the failed pod’s traffic without becoming overloaded.

Ablation Study lack of fallback mechanisms allows problems to escalate before the regular control loop can react.

To understand the contribution of each component, we evaluate Slant variants with specific features disabled. Table 3 presents the ablation study results on Social Network. Without Imbalanced Routing: Replacing our imbalance-aware optimizer with capacity-proportional balanced routing eliminates the core contribution; allocated CPU cores increase significantly from 81.5 to 110.0, and P99 latency degrades by 11%, confirming that intentional imbalance is essential for high resource efficiency. Without Multidimensional Constraints: Using only CPU utilization for capacity estimation increases SLO violations to 0.15%. While average CPU core allocation remains low (82.5), the system operates unsafely during memory-intensive request bursts, leading to 8× more OOM events and 5× more timeout errors in I/O operations because the single-dimensional model fails to predict non-CPU capacity exhaustion. Without Service Profiler: Using fixed, conservative critical points (70% CPU for all services) instead of learned profiles increases the required CPU cores to 96.0 and degrades P99 latency by 8%, as fixed thresholds are too conservative for some services and too aggressive for others. Without Fallback Controller: Disabling anomaly detection and graduated fallback increases SLO violations to 0.18%; under transient anomalies (e.g., brief load spikes or temporary downstream slowdowns), the

6

Related Work

Microservice performance optimization and servicegraph-aware systems. Several recent systems improve microservice performance by exploiting application structure or stronger resource control [51–66]. Autothrottle [42] studies SLO-oriented microservice resource management and uses a bi-level design to adjust CPU allocation based on the relationship between throttling and latency. MuCache [67] improves performance through a general caching framework over microservice graphs. LuoShen [68] highlights the importance of efficient traffic processing and consolidation in multi-tenant, multi-service edge clouds. These studies show that microservice performance depends on more than local request dispatch. However, they focus on resource allocation, caching, or gateway consolidation, rather than on how the load balancing policy itself shapes the utilization patterns seen by downstream resource managers. In contrast, our work focuses on the coupling between traffic distribution and resource management: instead of pursuing balance by default, we intentionally create safe traffic imbalance so that hot repli11

Table 2: System overhead breakdown. Slant introduces minimal overhead that is negligible compared to typical request processing times. Component

Metric

Overhead

Control Plane

CPU usage Memory usage

0.3 cores 256 MB

Data Plane

Per-request latency Memory per sidecar

0.08 ms 12 MB

Monitoring

Metrics bandwidth Weight sync bandwidth

2.1 KB/s/pod 0.5 KB/s/pod

CoreSync [83] explicitly studies the interaction between scheduling and overload control for latency-sensitive workloads. Beaver [84] addresses practical observability for distributed cloud services through efficient partial snapshots, improving the ability to reason about system behavior under dynamic conditions. These studies are complementary to ours in that they all recognize the importance of system interactions, feedback loops, and performance-aware control. Our work differs in both layer and objective. We focus on servicelevel request routing inside microservice deployments, and we show that the conventional goal of balanced traffic distribution may conflict with efficient vertical resource management. Building on this insight, our design combines service profiling, multi-dimensional safety constraints, imbalance-aware routing, and fallback control to improve both resource efficiency and tail latency.

Table 3: Ablation study results on Social Network at 60% load. Each row disables one component, showing its impact on key metrics. Config

P99 (ms) CPU Cores SLO Viol.

Slant (full)

128

81.5

0.02%

w/o Imbalance w/o Multi-dim w/o Profiler w/o Fallback

142 135 138 131

110 82.5 96 81.5

0.08% 0.15% 0.12% 0.18%

7

Conclusion

In this paper, we argued that balanced request distribution is often misaligned with efficient vertical resource management in microservice systems. To address this mismatch, we designed Slant, a resource-aware load balancing framework that profiles service-specific performance boundaries, computes conservative multi-dimensional capacity estimates, periodically applies intentional traffic imbalance through routing weights, and coordinates safely with existing VPA mechanisms through fallback control. Across three benchmark applications, a production trace-driven workload, and multiple anomaly scenarios, Slant reduces allocated CPU cores by up to 26%, lowers P99 latency by up to 18%, triggers VPA scale-up 45% faster, completes scale-in 40% faster, and improves robustness under pod failures and downstream slowdowns, without requiring changes to application code or the underlying autoscaler in production deployments and remains easy to adopt in Kubernetes clusters.

cas expose clearer scale-up signals and cold replicas become clearer scale-in targets. Learning-based and performance-aware systems optimization. Another closely related direction is learningbased or data-driven performance optimization for complex systems [69, 70]. Namyar et al. [71] highlight that learningenabled systems should be analyzed end-to-end rather than at the model component alone, emphasizing the importance of system-level performance reasoning. Zeng et al. [45] demonstrate in a production Alibaba Cloud CDN deployment that different operating conditions require different congestioncontrol choices, and that learned selection can outperform onesize-fits-all policies. These works reinforce a key lesson also reflected in our setting: under heterogeneous workloads and operating conditions, static policies are often suboptimal. Our approach shares this perspective, but targets a different control point. Rather than selecting transport protocols or analyzing learning-enabled pipelines, we learn service-specific critical points and use them to drive online, imbalance-aware routing decisions under multi-dimensional resource constraints. Performance-aware control, mitigation, and coupled system dynamics. Recent work has also shown that control policies in large-scale systems cannot be designed independently from the performance signals they induce [72–81]. Namyar et al. [82] improve failure mitigation by ranking actions using end-to-end performance information, while 12

References

[12] Yichuan Jiang. A survey of task allocation and load balancing in distributed systems. IEEE Trans. Parallel Distributed Syst., 27(2):585–599, 2016.

[1] Darby Huye, Yuri Shkuro, and Raja R. Sambasivan. Lifting the veil on meta’s microservice architecture: Analyses of topology and request workflows. In Proceedings of the 2023 USENIX Annual Technical Conference, USENIX ATC 2023, Boston, MA, USA, July 10-12, 2023, pages 419–432, 2023.

[13] Swarup Acharya, Michael J. Franklin, and Stanley B. Zdonik. Balancing push and pull for data broadcast. In SIGMOD 1997, Proceedings ACM SIGMOD International Conference on Management of Data, May 13-15, 1997, Tucson, Arizona, USA, pages 183–194, 1997.

[2] Shutian Luo, Huanle Xu, Chengzhi Lu, Kejiang Ye, Guoyao Xu, Liping Zhang, Yu Ding, Jian He, and Chengzhong Xu. Characterizing microservice dependency and performance: Alibaba trace analysis. In SoCC ’21: ACM Symposium on Cloud Computing, Seattle, WA, USA, November 1 - 4, 2021, pages 412–426, 2021.

[14] Inho Cho, Ahmed Saeed, Joshua Fried, Seo Jin Park, Mohammad Alizadeh, and Adam Belay. Overload control for µs-scale rpcs with breakwater. In 14th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2020, Virtual Event, November 4-6, 2020, pages 299–314, 2020. [15] David Chou, Tianyin Xu, Kaushik Veeraraghavan, Andrew Newell, Sonia Margulis, Lin Xiao, Pol Mauri Ruiz, Justin Meza, Kiryong Ha, Shruti Padmanabha, Kevin Cole, and Dmitri Perelman. Taiji: managing global user traffic for large-scale internet services at the edge. In Proceedings of the 27th ACM Symposium on Operating Systems Principles, SOSP 2019, Huntsville, ON, Canada, October 27-30, 2019, pages 430–446, 2019.

[3] Xuejian Chi, Yifei Zou, Congwei Zhang, Yapu Zhang, Jiguo Yu, Xiuzhen Cheng, Falko Dressler, and Dongxiao Yu. Efficient and green service function chain deployment and scheduling for mobile edge computing. IEEE Transactions on Mobile Computing, 2026. [4] Dirk Merkel. Docker: lightweight linux containers for consistent development and deployment. Linux journal, 2014(239):2, 2014.

[16] Sangmin Lee, Zhenhua Guo, Omer Sunercan, Jun Ying, Thawan Kooburat, Suryadeep Biswal, Jun Chen, Kun Huang, Yatpang Cheung, Yiding Zhou, Kaushik Veeraraghavan, Biren Damani, Pol Mauri Ruiz, Vikas Mehta, and Chunqiang Tang. Shard manager: A generic shard management framework for geo-distributed applications. In SOSP ’21: ACM SIGOPS 28th Symposium on Operating Systems Principles, Virtual Event / Koblenz, Germany, October 26-29, 2021, pages 553–569, 2021.

[5] Varun Talwar. gRPC: A True Internet-Scale RPC Framework is Now 1.0 and Ready for Production Deployments. https://cloud.google.com/blog/products/gcp /grpc-a-true-internetscale-rpc-framework-i s-now-1-and-readyfor-production-deploymen ts, 2016. Accessed: 2023-09-21. [6] Roy Thomas Fielding. Architectural styles and the design of network-based software architectures. University of California, Irvine, 2000.

[17] Parveen Patel, Deepak Bansal, Lihua Yuan, Ashwin Murthy, Albert G. Greenberg, David A. Maltz, Randy Kern, Hemant Kumar, Marios Zikos, Hongyu Wu, Changhoon Kim, and Naveen Karri. Ananta: cloud scale load balancing. In ACM SIGCOMM 2013 Conference, SIGCOMM 2013, Hong Kong, August 12-16, 2013, pages 207–218, 2013.

[7] Istio Authors. Istio: Connect, secure, control, and observe services, 2025. Accessed: 2025-05-09. [8] Bjorn Rabenstein and Julius Volz. Prometheus: A nextgeneration monitoring system (talk). USENIX Association, 2015.

[18] Harshit Saokar, Soteris Demetriou, Nick Magerko, Max Kontorovich, Josh Kirstein, Margot Leibold, Dimitrios Skarlatos, Hitesh Khandelwal, and Chunqiang Tang. Servicerouter: Hyperscale and minimal cost service mesh at meta. In 17th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2023, Boston, MA, USA, July 10-12, 2023, pages 969–985, 2023.

[9] Grafana Labs. Grafana: The open-source observability platform. https://grafana.com/, 2025. Accessed: 2025-05-11. [10] Jaeger Authors. Jaeger: Open source distributed tracing. https://www.jaegertracing.io/, 2025. Accessed: 2025-05-11. [11] Congwei Zhang, Yifei Zou, Zuyuan Zhang, Dongxiao Yu, Jorge Torres Gómez, Tian Lan, Falko Dressler, and Xiuzhen Cheng. Distributed age-of-information scheduling with noma via deep reinforcement learning. IEEE Transactions on Mobile Computing, 24(1):30–44, 2024.

[19] Hao Zhou, Ming Chen, Qian Lin, Yong Wang, Xiaobin She, Sifan Liu, Rui Gu, Beng Chin Ooi, and Junfeng Yang. Overload control for scaling wechat microservices. In Proceedings of the ACM Symposium on Cloud 13

ipv6-based iot networks. In Proceedings of the 20th Workshop on Mobility in the Evolving Internet Architecture, pages 25–31, 2025.

Computing, SoCC 2018, Carlsbad, CA, USA, October 11-13, 2018, pages 149–161, 2018. [20] Shutian Luo, Jianxiong Liao, Chenyu Lin, Huanle Xu, Zhi Zhou, and Chengzhong Xu. Embracing imbalance: Dynamic load shifting among microservice containers in shared clusters. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ASPLOS 2025, Rotterdam, Netherlands, 30 March 2025 - 3 April 2025, pages 309–324, 2025.

[29] Congwei Zhang, Youming Tao, Yifei Zou, Yu Yang, Ruirui Zhang, Xiaoyu Zhang, Falko Dressler, Xiuzhen Cheng, and Dongxiao Yu. Crisp: Co-designing pruning and scheduling for efficient moe inference on edge servers. In Proceedings of the 32nd Annual International Conference on Mobile Computing and Networking, pages 1067–1082, 2026.

[21] Justine Sherry, Ranysha Ware, Isabel Suizo, Srinivasan Seshan, and Adithya Abraham Philip. Congestion safety audits for internet services. Available at SSRN 5383658, 2025.

[30] Igor Sysoev. Nginx: High performance web server. https://nginx.org/, 2025. Accessed: 2025-05-16. [31] Envoy Proxy Authors. Envoy: Cloud-native highperformance edge/middle/service proxy, 2025. Accessed: 2025-05-09.

[22] Umesh Krishnaswamy, Rachee Singh, Nikolaj S. Bjørner, and Himanshu Raj. Decentralized cloud widearea network traffic engineering with BLASTSHIELD. In Amar Phanishayee and Vyas Sekar, editors, 19th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2022, Renton, WA, USA, April 4-6, 2022, pages 325–338. USENIX Association, 2022.

[32] Traefik Labs. Traefik: The cloud native application proxy. https://traefik.io/, 2025. Accessed: 202505-16. [33] Microsoft. YARP: Yet Another Reverse Proxy. https: //microsoft.github.io/reverse-proxy/, 2023. Accessed: 2023-09-21.

[23] Vamsi Addanki, Oliver Michel, and Stefan Schmid. {PowerTCP}: Pushing the performance limits of datacenter networks. In 19th USENIX symposium on networked systems design and implementation (NSDI 22), pages 51–70, 2022.

[34] Linux Virtual Server Project. Ipvs - ip virtual server. https://www.linuxvirtualserver.org/softwar e/ipvs.html, 2025. Accessed: 2025-05-16.

[24] Youming Tao, Yaowu Yang, Shuzhen Chen, Huiqun Li, Congwei Zhang, and Falko Dressler. Distributed pac learning from quantum data with efficient communication.

[35] Congwei Zhang, Huiqun Li, Pengfei Song, Yanwei Zheng, Yiqian Sun, Pengpeng Kang, Xiaoyu Tian, Jiongcheng Yan, and Dongxiao Yu. A unified robust learning framework for power system operation with noise-agnostic generalization. High-Confidence Computing, page 100382, 2026.

[25] Yifei Zou, Zuyuan Zhang, Congwei Zhang, Yanwei Zheng, Dongxiao Yu, and Jiguo Yu. A distributed abstract mac layer for cooperative learning on internet of vehicles. IEEE Transactions on Intelligent Transportation Systems, 25(8):8972–8983, 2024.

[36] Bartek Wydrowski, Robert Kleinberg, Stephen M. Rumble, and Aaron Archer. Load is not what you should balance: Introducing prequal. In 21st USENIX Symposium on Networked Systems Design and Implementation, NSDI 2024, Santa Clara, CA, April 15-17, 2024, 2024.

[26] Daniel E. Eisenbud, Cheng Yi, Carlo Contavalli, Cody Smith, Roman Kononov, Eric Mann-Hielscher, Ardas Cilingiroglu, Bin Cheyney, Wentao Shang, and Jinnah Dylan Hosein. Maglev: A fast and reliable software network load balancer. In 13th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2016, Santa Clara, CA, USA, March 16-18, 2016, pages 523–535, 2016.

[37] P. Lalith Suresh, Marco Canini, Stefan Schmid, and Anja Feldmann. C3: cutting tail latency in cloud data stores via adaptive replica selection. In 12th USENIX Symposium on Networked Systems Design and Implementation, NSDI 15, Oakland, CA, USA, May 4-6, 2015, pages 513–527, 2015.

[27] John Lamping and Eric Veach. A fast, minimal memory, consistent hash algorithm. CoRR, abs/1406.2294, 2014.

[38] Congwei Zhang, Guangpeng Qi, Zhen Ma, Long Ma, Xuejian Chi, Huiqun Li, Dongxiao Yu, and Xiuzhen Cheng. Balancing safety and learning in iot: A riskaware reinforcement learning approach. IEEE Internet Computing, 2026.

[28] Shulun Wu, Congwei Zhang, Yifei Zou, Guangyong Shang, Zhen Ma, Xiaoming Wang, Xuejian Chi, and Dongxiao Yu. An adaptive backbone construction for 14

[39] Kubernetes horizontal pod autoscaling, 2024. https: //kubernetes.io/docs/tasks/run-application /horizontal-pod-autoscale/.

[48] Xiang Zhou, Xin Peng, Tao Xie, Jun Sun, Chao Ji, Wenhai Li, and Dan Ding. Fault analysis and debugging of microservice systems: Industrial survey, benchmark system, and empirical study. IEEE Trans. Software Eng., 47(2):243–260, 2021.

[40] Amazon Web Services. Network load balancers. https: //docs.aws.amazon.com/elasticloadbalancing /latest/network/network-load-balancers.htm l, 2025. Accessed: 2025-05-16.

[49] Dmitry Duplyakin, Robert Ricci, Aleksander Maricq, Gary Wong, Jonathon Duerig, Eric Eide, Leigh Stoller, Mike Hibler, David Johnson, Kirk Webb, Aditya Akella, Kuangching Wang, Glenn Ricart, Larry Landweber, Chip Elliott, Michael Zink, Emmanuel Cecchet, Snigdhaswin Kar, and Prabodh Mishra. The design and operation of CloudLab. In Proceedings of the USENIX Annual Technical Conference (ATC), pages 1–14, 2019.

[41] Aws ec2, 2024. https://aws.amazon.com/cn/ec2/. [42] Zibo Wang, Pinghe Li, Chieh-Jan Mike Liang, Feng Wu, and Francis Y Yan. Autothrottle: A practical {Bi-Level} approach to resource management for {SLO-Targeted} microservices. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), pages 149–165, 2024.

[50] Gil Tene. wrk2: A constant throughput, correct latency recording variant of wrk. https://github.com/gil tene/wrk2, 2023.

[43] Yanqi Zhang, Weizhe Hua, Zhuangzhuang Zhou, G. Edward Suh, and Christina Delimitrou. Sinan: Ml-based and qos-aware resource management for cloud microservices. In ASPLOS ’21: 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Virtual Event, USA, April 19-23, 2021, pages 167–181, 2021.

[51] Chunqiang Tang, Kenny Yu, Kaushik Veeraraghavan, Jonathan Kaldor, Scott Michelson, Thawan Kooburat, Aravind Anbudurai, Matthew Clark, Kabir Gogia, Long Cheng, Ben Christensen, Alex Gartrell, Maxim Khutornenko, Sachin Kulkarni, Marcin Pawlowski, Tuomas Pelkonen, Andre Rodrigues, Rounak Tibrewal, Vaishnavi Venkatesan, and Peter Zhang. Twine: A unified cluster management system for shared infrastructure. In 14th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2020, Virtual Event, November 4-6, 2020, pages 787–803, 2020.

[44] Peng Wang, Senmao Qi, Jiahao Yu, Congwei Zhang, Li Yang, Yifei Zou, and Dongxiao Yu. Distributed ageof-information scheduling with noma in mobile edge computing. Tsinghua Science and Technology, 2026. [45] Xuan Zeng, Haoran Xu, Chen Chen, Xumiao Zhang, Xiaoxi Zhang, Xu Chen, Guihai Chen, Yubing Qiu, Yiping Zhang, Chong Hao, et al. Learning {ProductionOptimized} congestion control selection for alibaba cloud {CDN}. In 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25), pages 827–845, 2025.

[52] Vivek M. Bhasi, Jashwant Raj Gunasekaran, Prashanth Thinakaran, Cyan Subhra Mishra, Mahmut Taylan Kandemir, and Chita R. Das. Kraken: Adaptive container provisioning for deploying dynamic dags in serverless platforms. In SoCC ’21: ACM Symposium on Cloud Computing, Seattle, WA, USA, November 1 - 4, 2021, pages 153–167, 2021.

[46] Yu Gan, Yanqi Zhang, Dailun Cheng, Ankitha Shetty, Priyal Rathi, Nayan Katarki, Ariana Bruno, Justin Hu, Brian Ritchken, Brendon Jackson, Kelvin Hu, Meghna Pancholi, Yuan He, Brett Clancy, Chris Colen, Fukang Wen, Catherine Leung, Siyuan Wang, Leon Zaruvinsky, Mateo Espinosa, Rick Lin, Zhongling Liu, Jake Padilla, and Christina Delimitrou. An open-source benchmark suite for microservices and their hardware-software implications for cloud & edge systems. In Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS 2019, Providence, RI, USA, April 13-17, 2019, pages 3–18, 2019.

[53] Shuang Chen, Christina Delimitrou, and José F. Martínez. PARTIES: qos-aware resource partitioning for multiple interactive services. In Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS 2019, Providence, RI, USA, April 13-17, 2019, pages 107–120, 2019. [54] Ka Ho Chow, Umesh Deshpande, Sangeetha Seshadri, and Ling Liu. Deeprest: deep resource estimation for interactive microservices. In EuroSys ’22: Seventeenth European Conference on Computer Systems, Rennes, France, April 5 - 8, 2022, pages 181–198, 2022.

[47] GoogleCloudPlatform. Online boutique (microservicesdemo). https://github.com/GoogleCloudPlatfo rm/microservices-demo, 2025. Accessed: 2025-0418.

[55] Yu Gan, Mingyu Liang, Sundar Dev, David Lo, and Christina Delimitrou. Sage: practical and scalable mldriven performance debugging in microservices. In ASPLOS ’21: 26th ACM International Conference on 15

an intelligent fine-grained resource management framework for slo-oriented microservices. In 14th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2020, Virtual Event, November 4-6, 2020, pages 805–825, 2020.

Architectural Support for Programming Languages and Operating Systems, Virtual Event, USA, April 19-23, 2021, pages 135–151, 2021. [56] Alim Ul Gias, Giuliano Casale, and C. Murray Woodside. ATOM: model-driven autoscaling for microservices. In 39th IEEE International Conference on Distributed Computing Systems, ICDCS 2019, Dallas, TX, USA, July 7-10, 2019, pages 1994–2004, 2019.

[64] Zhe Yang, Phuong Nguyen, Haiming Jin, and Klara Nahrstedt. MIRAS: model-based reinforcement learning for microservice resource allocation over scientific workflows. In 39th IEEE International Conference on Distributed Computing Systems, ICDCS 2019, Dallas, TX, USA, July 7-10, 2019, pages 122–132, 2019.

[57] Ram Srivatsa Kannan, Lavanya Subramanian, Ashwin Raju, Jeongseob Ahn, Jason Mars, and Lingjia Tang. Grandslam: Guaranteeing slas for jobs in microservices execution frameworks. In Proceedings of the Fourteenth EuroSys Conference 2019, Dresden, Germany, March 25-28, 2019, pages 34:1–34:16, 2019.

[65] Guangba Yu, Pengfei Chen, and Zibin Zheng. Microscaler: Automatic scaling for microservices with an online learning approach. In 2019 IEEE International Conference on Web Services, ICWS 2019, Milan, Italy, July 8-13, 2019, pages 68–75, 2019.

[58] Shutian Luo, Huanle Xu, Kejiang Ye, Guoyao Xu, Liping Zhang, Jian He, Guodong Yang, and Chengzhong Xu. Erms: Efficient resource management for shared microservices with SLA guarantees. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, ASPLOS 2023, Vancouver, BC, Canada, March 25-29, 2023, pages 62–77, 2023.

[66] Kaushik Veeraraghavan, Justin Meza, David Chou, Wonho Kim, Sonia Margulis, Scott Michelson, Rajesh Nishtala, Daniel Obenshain, Dmitri Perelman, and Yee Jiun Song. Kraken: Leveraging live traffic tests to identify and resolve resource utilization bottlenecks in large scale web services. In 12th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2016, Savannah, GA, USA, November 2-4, 2016, pages 635–651, 2016.

[59] Amirhossein Mirhosseini, Sameh Elnikety, and Thomas F. Wenisch. Parslo: A gradient descent-based approach for near-optimal partial SLO allotment in microservices. In SoCC ’21: ACM Symposium on Cloud Computing, Seattle, WA, USA, November 1 - 4, 2021, pages 442–457, 2021.

[67] Haoran Zhang, Konstantinos Kallas, Spyros Pavlatos, Rajeev Alur, Sebastian Angel, and Vincent Liu. {MuCache}: A general framework for caching in microservice graphs. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), pages 221–238, 2024.

[60] Amirhossein Mirhosseini and Thomas F. Wenisch. µsteal: a theory-backed framework for preemptive work and resource stealing in mixed-criticality microservices. In ICS ’21: 2021 International Conference on Supercomputing, Virtual Event, USA, June 14-17, 2021, pages 102–114, 2021.

[68] Tian Pan, Kun Liu, Xionglie Wei, Yisong Qiao, Jun Hu, Zhiguo Li, Jun Liang, Tiesheng Cheng, Wenqiang Su, Jie Lu, et al. {LuoShen}: A {Hyper-Converged} programmable gateway for {Multi-Tenant}{MultiService} edge clouds. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), pages 877–892, 2024.

[61] Amirhossein Mirhosseini, Brendan L. West, Geoffrey W. Blake, and Thomas F. Wenisch. Q-zilla: A scheduling framework and core microarchitecture for tail-tolerant microservices. In IEEE International Symposium on High Performance Computer Architecture, HPCA 2020, San Diego, CA, USA, February 22-26, 2020, pages 207– 219, 2020.

[69] Hongzi Mao, Mohammad Alizadeh, Ishai Menache, and Srikanth Kandula. Resource management with deep reinforcement learning. In Proceedings of the 15th ACM Workshop on Hot Topics in Networks, HotNets ’16, pages 50–56, New York, NY, USA, 2016. Association for Computing Machinery.

[62] Jennifer Ortiz, Brendan Lee, Magdalena Balazinska, Johannes Gehrke, and Joseph L. Hellerstein. Slaorchestrator: Reducing the cost of performance slas for cloud data analytics. In Proceedings of the 2018 USENIX Annual Technical Conference, USENIX ATC 2018, Boston, MA, USA, July 11-13, 2018, pages 547–560, 2018.

[70] Muhammad Bilal, Marco Serafini, Marco Canini, and Rodrigo Rodrigues. Do the best cloud configurations grow on trees? an experimental evaluation of black box algorithms for optimizing cloud workloads. 2020.

[63] Haoran Qiu, Subho S. Banerjee, Saurabh Jha, Zbigniew T. Kalbarczyk, and Ravishankar K. Iyer. FIRM: 16

[71] Pooria Namyar, Michael Schapira, Ramesh Govindan, Santiago Segarra, Ryan Beckett, Siva Kesava Reddy Kakarla, and Behnaz Arzani. End-to-end performance analysis of learning-enabled systems. In Proceedings of the 23rd ACM Workshop on Hot Topics in Networks, pages 86–94, 2024.

Symposium on Networked Systems Design and Implementation, NSDI 2023, Boston, MA, April 17-19, 2023, pages 515–529. USENIX Association, 2023. [78] Weitao Wang, Dingming Wu, Sushovan Das, Afsaneh Rahbar, Ang Chen, and TS Eugene Ng. {RDC}:{Energy-Efficient} data center network congestion relief with topological reconfigurability at the edge. In 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22), pages 1267–1288, 2022.

[72] Shinae Woo, Justine Sherry, Sangjin Han, Sue Moon, Sylvia Ratnasamy, and Scott Shenker. Elastic scaling of stateful network functions. In Sujata Banerjee and Srinivasan Seshan, editors, 15th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2018, Renton, WA, USA, April 9-11, 2018, pages 299– 312. USENIX Association, 2018.

[79] Miguel Alves Ferreira, Nirav Atre, Justine Sherry, and João Luís Sobrinho. Impossibility results for data-center routing with congestion control and unsplittable flows. In Proceedings of the 43rd ACM Symposium on Principles of Distributed Computing, pages 358–368, 2024.

[73] Antonis Manousis, Rahul Anand Sharma, Vyas Sekar, and Justine Sherry. Contention-aware performance prediction for virtualized network functions. In Henning Schulzrinne and Vishal Misra, editors, SIGCOMM ’20: Proceedings of the 2020 Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication, Virtual Event, USA, August 10-14, 2020, pages 270–282. ACM, 2020.

[80] Ennan Zhai, Ang Chen, Ruzica Piskac, Mahesh Balakrishnan, Bingchuan Tian, Bo Song, and Haoliang Zhang. Check before you change: Preventing correlated failures in service updates. In 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20), pages 575–589, 2020. [81] Justine Sherry, Chang Lan, Raluca Ada Popa, and Sylvia Ratnasamy. Blindbox: Deep packet inspection over encrypted traffic. In Proceedings of the 2015 ACM conference on special interest group on data communication, pages 213–226, 2015.

[74] Yiran Lei, Francisco Pereira, Arvind Krishnamurthy, and Justine Sherry. Switchnic: An hybrid architecture for network functions with fast and consistent shared state. Proceedings of the ACM on Networking, 3(CoNEXT4):1–21, 2025.

[82] Pooria Namyar, Arvin Ghavidel, Daniel Crankshaw, Daniel S Berger, Kevin Hsieh, Srikanth Kandula, Ramesh Govindan, and Behnaz Arzani. Enhancing network failure mitigation with {Performance-Aware} ranking. In 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25), pages 335–357, 2025.

[75] Myung-Ki Shin, Ki-Hyuk Nam, and Hyoung-Jun Kim. Software-defined networking (SDN): A reference architecture and open apis. In International Conference on Information and Communication Technology Convergence, ICTC 2012, "Global Open Innovation Summit for Smart ICT Convergence", Jeju Island, Korea, October 15-17, 2012, pages 360–361, 2012.

[83] Bhaskar Pardeshi, Eric Stuhr, and Ahmed Saeed. Coresync: A protocol for joint core scheduling and overload control of µs-scale tasks. In 2025 IEEE 33rd International Conference on Network Protocols (ICNP), pages 1–11. IEEE, 2025.

[76] Harsha V. Madhyastha, Tomas Isdal, Michael Piatek, Colin Dixon, Thomas E. Anderson, Arvind Krishnamurthy, and Arun Venkataramani. iplane: An information plane for distributed services. In Brian N. Bershad and Jeffrey C. Mogul, editors, 7th Symposium on Operating Systems Design and Implementation (OSDI ’06), November 6-8, Seattle, WA, USA, pages 367–380. USENIX Association, 2006.

[84] Liangcheng Yu, Xiao Zhang, Haoran Zhang, John Sonchack, Dan Ports, and Vincent Liu. Beaver: Practical partial snapshots for distributed cloud services. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 233–249, 2024.

[77] Umesh Krishnaswamy, Rachee Singh, Paul Mattes, Paul-Andre C. Bissonnette, Nikolaj S. Bjørner, Zahira Nasrin, Sonal Kothari, Prabhakar Reddy, John Abeln, Srikanth Kandula, Himanshu Raj, Luis Irún-Briz, Jamie Gaudette, and Erica Lan. Onewan is better than two: Unifying a split WAN architecture. In Mahesh Balakrishnan and Manya Ghobadi, editors, 20th USENIX 17

Related documents

• Slant: Shaping Traffic for Resource-Efficient Microservices#1489848
• PackServe: SLO-Aware Request Scheduling for Agentic LLM Serving at Scale#1117936
• PackServe: SLO-Aware Request Scheduling for Agentic LLM Serving at Scale#1108733
• HAServe: Horizon-Aware Scheduling and Memory Co-Design for Efficient LLM Serving#1355574
• ConfluxServe: Reconciling Resource Utilization and Parallelism Efficiency across Prefill and Decode in LLM Serving#1488602