HiLO: Towards Product-Level Graceful Degradation with a Quality-Aware, Load-Aware Control Framework

HiLO: Towards Product-Level Graceful Degradation with a Quality-Aware, Load-Aware Control Framework

arXiv:2610.34844v1 [cs.DC] 26 Oct 2026

Knut Magne Risvik Microsoft

Zhuowei Li Microsoft

Nirupam Purushothama Microsoft

Hunter Su Microsoft

Greg Dunham Microsoft

Yan Reznikov Microsoft Abstract

This sustained growth in traffic, coupled with transient capacity losses from machine failures, new feature deployments, and faulty services keep product availability at risk during overload. Moreover, search engines and similar products increasingly leverage LLMs [35], creating significant strain on GPU capacity. This requires all services dependent on them to frequently cope with increased latencies and bottlenecks. These fluctuations routinely occur on time scales faster than human intervention can adapt [9,28], necessitating automated policies that react quickly to prevent overload conditions from spiraling into outages. To mitigate this risk, most online services rely on two broad strategies: overprovisioning and service degradation. Overprovisioning maintains significant headroom to absorb unexpected traffic increases but is expensive and inefficient at scale. In practice, capacity constraints (particularly for specialized resources such as GPUs) are persistent [2, 14]. Moreover, the scale and timing of demand spikes remain difficult to predict [7], thereby reducing the efficiency of overprovisioning. Hence, many systems leverage graceful degradation policies to ensure their survival under overload conditions [18, 22, 30]. Especially in Bing’s case, additional headroom is infeasible, making graceful degradation essential. Commonly, service-level protections such as load shedding, throttling, and circuit breakers [3, 21] are employed to preserve availability. While effective at protecting individual services, these mechanisms cannot promise a baseline product quality because they operate locally and independently. This is because, in products composed of many interdependent services, independent degradation decisions interact in unexpected ways, causing sharp shifts in latency and quality that can far outlast the triggering event, a metastable dynamic [4, 11, 12, 20] (e.g., capacity challenged services causing major dips in quality and service health; §2.2). Similarly, approaches that rely on manual knobs introduce a large interaction space across features and services, requiring continual maintenance [18]. In this context, we introduce HiLO, an automated, qualityaware and load-aware product-level degradation system. Each service reports a load-factor, a normalized measure of its cur-

Graceful degradation is a key mechanism employed by largescale systems to handle overload conditions. Its importance has grown further as GPU-intensive LLM inference introduces persistent capacity constraints that cannot be addressed through provisioning alone. Common degradation approaches rely on service-level mechanisms such as load shedding and circuit breakers, or on feature-level switches (some manually controlled) to preserve availability and quality. While effective at protecting services, such approaches lead to untestable product states and thereby unpredictable product quality. We present HiLO (High Load Optimizer), an automated product-level degradation system designed to handle graceful degradation at scale. HiLO maps observed load to a product state lattice: a small set of pre-validated product states, partially ordered by degradation severity. Dependencies are resolved at design time enabling deterministic and continuous control without requiring runtime dependency discovery. The same abstraction applies unchanged across heterogeneous services, including GPU-intensive deep models, LLMs and their serving layers. At Microsoft, HiLO is deployed in Bing (serving over a billion monthly active users), where it manages on average two traffic surges per week across all datacenters, bounding OneDCG (relevance) drop to under 0.5 points (the operational target) even under sustained load exceeding 2× baseline peak. HiLO complements existing service-level protections with product-level coordination and makes product quality measurable, testable and predictable.

1

Introduction

Web products such as Bing serve over a billion monthly active users [27]. Online traffic has grown significantly over the past several years [6, 8, 10]. In parallel, AI-integrated search and chat systems have seen rapid growth in adoption, contributing additional demand [19]. Additionally, web services face increased volumes of DDoS attacks, automated bot traffic and large-scale scraping activity [8, 29]. 1

rent load. HiLO maps the aggregate load-factor to a product state lattice: a small, finite set of pre-validated product states, partially ordered by degradation severity. Each state deterministically maps to coordinated service levels (each service’s degradation level) across all participating services. States are created through a formal construction process by addressing dependencies at design time, enabling continuous and autonomous control without runtime dependency discovery. This abstraction applies unchanged across heterogeneous services, including GPU-intensive models and LLMs. Drawing on over six years of operating graceful degradation across Bing’s datacenters, with HiLO in production since 2025, we share the design insights and operational lessons, and evaluate HiLO across multiple production scenarios.

2

event, and hence the time it takes for underlying services to return to normalcy. Capacity savings: This metric quantifies the amount of capacity reclaimed during the graceful degradation time. We use “load-factor” (LF) (§3.1) to measure the capacity utilization level of a service during runtime.

2.2 Evolution from Basic Graceful Degradation to HiLO - Quality Awareness At Bing, for many years, the degradation policy focused on ensuring high availability by protecting individual services during overload conditions. Services performed two primary activities: (1) Monitor health: Individual services self-monitored local load or health signals such as CPU utilization, request queue depth, latency, or error rates. (2) Adjust service level: Based on the current load situation, services independently applied predefined degradation actions when thresholds were crossed. Common mechanisms included disabling non-critical features, reducing computation fidelity, limiting request fanout, increasing cache reuse, or shedding low priority traffic. This allowed Bing to maintain availability well above 99.95%. However, this policy introduced three fundamental challenges. (1) The current product state became difficult to reason about, (2) it became difficult to provide consistent baseline quality guarantees to the users and (3) inter-service dependencies led to localized orchestration and control loops. Some components attempted to directly control their immediate dependencies using feedback-based mechanisms (e.g., PID-style control [24]), which in practice complicated testing, debuggability, measurability and experimentation at the product level. The following incident illustrates how a local degradation policy fails to protect overall system health and cannot hold baseline quality.

Background and Motivation

As described in the introduction, demand capacity mismatch is a persistent challenge. Here we discuss past learnings, the metrics used to evaluate the implications of graceful degradation, and design gaps.

2.1

Five Metrics to Watch

The following metrics are important to evaluate the implications of a graceful degradation policy. Availability: This metric captures whether the graceful degradation policy preserves basic functionality and prevents service outages [3]. Quality: This is a combination of several signals. The primary one at Bing, OneDCG is a normalized variant of discounted cumulative gain [13], used to measure relevance of results. In addition, error signals such as no-result rates and user-facing experiment metrics such as content freshness and engagement (e.g., reformulation rates) are measured to validate the quality under each product state (§4.3.2).

0.25

OneDCG (Δ)

Stability(%) =

Other DCs

0.00

Stability: This metric determines the percentage of queries that receive consistent responses across repeated executions under identical conditions, capturing the predictability of product behavior: |{(qi , q j ) : ri = r j }| × 100 |Qpairs |

−0.25 −0.50 −0.75

Affected DC

−1.00

where Qpairs is the set of query pairs (qi , q j ) issued under identical observable context (same query, same time window), and ri , r j are their respective responses. A pair is counted as stable if the responses are identical. Higher instability typically indicates underlying systemic issues (i.e., abnormal service behavior) and hence is closely monitored.

−1.25 14:00

18:00

22:00 02:00 Time (PST)

06:00

10:00

Figure 1: Per-datacenter OneDCG (relevance) on the day of incident. Overload in one datacenter causes OneDCG to drop ~1.11 points. No coordinated mechanism existed to handle this gracefully.

Recovery velocity: This indicates the time taken for stability and quality to return to pre-incident levels after an overload 2

Figure 1 depicts the movement in result quality after a feature deployment increased the call volume to a modelserving service shared by multiple workflows. The change passed flight testing at fractional traffic, where capacity was sufficient. After full deployment, one datacenter was already running near its capacity limits when a transient load spike (under 10 minutes) pushed it past its capacity limit. Following its own service-local policy, the already pressured shared model-serving service degraded to protect itself and stayed available. But its callers kept calling under their own policies with no coordinated way to reduce demand, so that the bottleneck surfaced upstream as elevated latencies and cascading errors, causing a OneDCG decline of 1.11 points; the service survived, but quality suffered significantly. Figure 2, related to the same incident, depicts how the product stability fell from ~90% to ~63% and took over four hours to recover (indicating prolonged abnormality with underlying services even after the spike receded). This incident triggered a severity-2 escalation. OneDCG fluctuated for another two days (whenever load increased) and recovered to its pre-incident baseline after a fix to reduce the additional dependency calls (by the initial deployment) was deployed.

on local protection to handle overflows. When the system reached its limit, no mechanism existed to reduce demand on the shared service before the spike or to coordinate recovery afterward, and debugging was hard because the product-level state was unclear. Ultimately, quality (OneDCG) during this event was an emergent behavior rather than a controlled outcome. This calls for a fast-acting, product-level policy that acts before overload by formalizing load signals, dependencies, and their coordination, which is what HiLO provides.

2.3

For graceful degradation many systems either adopt featurecentric degradation or admission control mechanisms or a combination of techniques to preserve service availability. Feature-centric control is achieved by degrading optional product functionality in order to reclaim resources while preserving core behavior. A canonical work is Brownout [15]. It suggests splitting an application into components where optional components can be isolated and degraded. A dimmer at application level indicates how frequently these optional components are deactivated and a controller controls the dimmer based on measured latency. Meta’s DefCon system [18] is another representative example of feature-level degradation at scale. It exposes binary degradation knobs that allow operators to selectively and manually control feature execution under overload. It assumes strong isolation between feature execution paths, noting that features at Meta are typically implemented as separate RPCs with minimal control flow interaction. It achieves control through config-driven operations by allowing operators to utilize a relational database to determine the outcome of choosing a set of control knobs. Designed for infrequent site emergencies with human driven operation, Meta also notes the operational cost of maintaining hundreds of knobs that grow in number over time. Google’s SRE guidance [3] discusses overload handling around load shedding (per-task throttling based on CPU, memory, queue length or request priorities) and graceful degradation. It emphasizes the importance of testing degraded modes regularly and maintaining operational visibility through monitoring and alerts. SEDA [33] is another canonical line of work that structures applications into explicit event-driven stages and queues preventing resources from getting overloaded. Use of dynamic resource controllers is suggested at each stage to keep it within control limits. The controller observes performance and load, automatically adjusting the stage’s resource usage. Its extension presents an adaptive admission control [32] mechanism that attempts to meet the P90 response time target by filtering requests at each stage. Instead of CPU utilization and fraction of users, this work focuses on response time as the primary metric. Based on the calculated P90 latency at this moment, the controller adjusts the admission rate.

100 95 Stability (%)

90 85 80 Asia-Pacific DC

~4h recovery

75 70 65 60 55 14:00

18:00

~63% 22:00 02:00 Time (PST)

06:00

Related Work

10:00

Figure 2: Product stability in the affected DC on the day of incident. A transient load spike (under 10 minutes) causes system-wide instability. Stability drops from ~90% to ~63% and takes over four hours to recover. Performance isolation guarantees for individual services always existed at Bing (each service owns its capacity, so one service’s overload cannot starve another), and service-local degradation (at least throttling) was the norm. This particular incident highlights the risk of an uncoordinated degradation policy, which could result in metastable failures [4] and prolonged quality loss, even when underlying service availability is maintained. This dynamic is not specific to model serving. It can arise at any shared, capacity-constrained service. At Bing this is an increasingly common issue with models and their serving layers given persistent GPU capacity constraints. The investigation exposed several structural gaps. Services were allowed to operate near their capacity limits, relying 3

3.1

Dagor [37] from WeChat decouples overload control from service logic, making it service agnostic. Each server monitors the average waiting time of requests in its pending queue (window based on time and request count) to detect overload. On detection, it performs admission control using business priority (e.g., login and payment requests are higher priority than instant messaging). Dagor introduces user priority to provide finer-grained shedding within each business priority tier. Business and user priorities are composed into a compound admission level that is adjusted adaptively. Overloaded servers also piggyback their admission thresholds to upstream callers, enabling collaborative early shedding. Breakwater [5] targets overload control for microsecond scale RPCs, where the cost of dropping a request is comparable to processing it. It presents a server driven admission control scheme that prevents overload through the use of server issued credits. Here, a credit represents availability at the server to process a single request by the client that receives the credit. Server load is effectively managed through the distribution of total credits and the rate of distribution. Rajomon [34] extends this line of work with market-based approach where services set prices for their APIs and clients attach tokens to requests, enabling decentralized and coordinated rate limiting across the call graph. In practice, comparable strategies often combined with circuit breakers [21, 25] are reported across industry. Sometimes feature degradation is achieved through admission control. Netflix describes selectively throttling lower priority requests so that playback experience remains uninterrupted [22,23]. It describes the need to continuously prioritize requests (including finer-grained prioritization at service layer), progressively throttle based on priority and validate through chaos testing. Similarly, Uber’s QALM framework sheds non-critical requests based on criticality tiers during overload. It creates separate queues for endpoints each with its own overload detector (that monitors latency). Then using request criticality drops requests only in the affected queue thereby isolating the impact and preserving core trip and infra requests [30]. AWS [1] documents graceful degradation as a reliability pattern, recommending that services transform hard dependencies into soft ones to preserve availability under failure. When compared to the above methods, HiLO targets systems where services are tightly interdependent, capacity is heterogeneous, and overload requires continuous autonomous management rather than manual intervention or binary feature toggling. For example, large search and LLM-serving stacks such as Bing serve queries that span across multiple CPU-bound services and GPU-bound ranking/LLM models with complex dependencies. The design is detailed in §3.

3

Load-Factor and Product State Lattice

HiLO is built on two core abstractions: a Load-factor that summarizes service-level capacity pressure, and a product state lattice that defines the space of valid, coordinated degradation behaviors. Figure 3 shows HiLO’s internal pipeline. Each service emits a Load-Factor (LF), in real time, a normalized scalar signal reflecting its current service load. LFs are derived locally from service-specific indicators such as CPU or GPU utilization, request queue depth, tail latency, or dependency backpressure. How each service chooses this signal, and adapts it as its workload evolves, is discussed in §5. HiLO operates outside the critical path (Figure 4), requires no global capacity model or comparable units across services; it assumes only monotonicity (higher LF values indicate higher overload risk) and treats aggregated LFs as ordinal congestion signals rather than precise optimization inputs. In Bing, LFs are reported on a bounded scale and aggregated per service using robust sampling (e.g., reservoir sampling [31]) to dampen noise and short-lived spikes. A product state represents a coordinated operating mode of the product under overload. Each product state deterministically maps to a predefined service level for every participating service (e.g., disabled features, simplified execution paths, reduced fan-out, or lower cost model variants). Product states are partially ordered by degradation severity, forming a lattice in which higher states dominate lower ones in terms of capacity reclamation while incurring bounded, pre-validated quality loss. In Bing, we use 6 product states per scenario (§3.4), with each service implementing 2-6 service levels depending on its flexibility. The product state lattice is constructed at design time. Service owners define candidate service levels, quantify their capacity impact, and validate quality effects locally. Dependency effects are resolved during composition. Figure 5 illustrates this: a product state switch moves individual services to new levels (indicated by arrows), changing not only the service’s own behavior but also how it invokes its callees. For example, both how often a caller calls a callee and which lower-cost variant it selects are independently controlled. Product states are finalized after validating them using end-to-end test and experimentation hooks (§3.5). At runtime, HiLO selects a single active product state per scenario, transitions monotonically under increasing load, and applies controlled recovery to avoid unnecessary oscillations. No per-request decisions, runtime dependency discovery, or dynamic call-graph inference are performed. In practice we label these states as L0 through L5 , in order of increasing degradation severity. Positioning: Unlike DefCon [18] which exposes a large number of feature-level knobs whose interactions determine product behavior implicitly, and Brownout [15] which controls a single application without cross-service coordination, the lattice constrains online behavior to a small set of vali-

HiLO (High Load Optimizer)

Guided by the gaps identified in §2.2, we now describe HiLO’s core abstractions and architecture. 4

Figure 3: HiLO internal decision pipeline. Services periodically report load signals, which are sampled and aggregated in HiLO. This is used to determine the load at product level and the corresponding product state (Lk ), which determines the service levels for participating services. HiLO disseminates this back to the services which enforce them.

Figure 4: System overview showcasing HiLO’s position in the stack. HiLO operates out of the request path, aggregates load signals reported by individual services, selects a product state (Lk ), and disseminates coordinated configuration decisions to participating services.

Figure 5: Dependency handling under coordination at product level. HiLO determines service levels based on the active product state. This determines predefined execution logic and downstream service variants, enforcing consistent dependency resolution across services.

dated, coordinated product states. Unlike admission control and circuit breakers, which act locally to protect service SLOs, the lattice provides an explicit product-level quality baseline under overload. HiLO deliberately trades fine-grained optimality for predictability, testability, and operational clarity, which we find essential in large, dependency-rich systems combining services with GPU-constrained model and LLM pipelines.

capacity constraints. An alternative approach is to use perrequest classifiers to route queries between expensive and cheap model variants at runtime. HiLO avoids this complexity. The expensive model is always used under normal conditions, and degradation to lower-cost variants is applied uniformly only when overload is detected. This ensures best OneDCG by default, at no additional quality loss (unlike classifiers). Table 1 summarizes the methods by which we bring heavily used models or their model-serving layers (at Bing) into HiLO’s fold. The specific application of these strategies is discussed in the case studies section (§4.2). To illustrate how these strategies work together, consider a production ranking model whose degradation is coordinated

3.2

Deep Model & LLM Specific Strategies

As discussed in §2.2, applying HiLO to GPU-intensive deep models and their serving layers is critical for Bing due to

5

across two services. At all degradation levels, capacity is reclaimed by progressively reducing multi-queries (MQ) and documents processed. At deeper levels, model layer reduction is additionally applied (see §4.2.1 for details). Less disruptive upstream actions are applied first, with model-level actions added at deeper levels. The sequence is determined and validated at design time. No additional architectural changes are required.

on the shared service. After that, service levels and product-state config are determined in three phases. Phase 1 - Service local degradation validation: Service owners first identify a set of local tasks that could be degraded and estimate the impact. For each task they estimate the load-factor improvement and quality drop based on the current load-factor utilization, its trigger rate (i.e., execution frequency), the proposed alternative action, and its relative priority. If needed, these estimates are discussed with product teams to finalize the candidates. Then these actions are grouped together to a level based on their relative-priority and estimated overall service load-factor improvement. Service teams then conduct tests and experiments (A/B that are service local) to validate the estimates and then proceed to finalize the service levels. For example, this pertains to model layer reduction (§3.2) where a model’s inference cost can be reduced at the model service itself. Phase 2 - Dependency resolution: In this phase, callees that require upstream changes to reduce load coordinate with their callers to integrate cross-service degradation actions (e.g., those listed in Table 1) into the caller’s service levels. If this introduces new sub-levels, they are incorporated into the caller’s level definitions. The same estimation, testing and experimentation steps from Phase 1 are repeated jointly by callee and caller to validate levels against metric movement, priorities and reclamation targets. Phase 3 - Product-states creation: In this phase, productstate config is finally created in HiLO using the service levels created in the previous two phases. Product (Scenario - §3.4) owners now use the existing set of service levels and frame them into product states where every higher product state indicates further degradation in service levels. These candidate states are then subjected to end-to-end testing and experimentation (using end-to-end test-hooks and experimentationhooks, described in §3.5). Candidate states that either cause unintended experiences or degrade user-metrics beyond targets are discarded or adjusted as needed. Finally, end-to-end capacity tests help understand if the desired effect (survival, quality maintenance and rapid stability recovery) is obtained when the entire system is subject to the peak survival goal.

Table 1: Strategies utilized by Bing to balance degradation and OneDCG for GPU constrained models, LLMs and their serving layers. #

Strategy

Severity

a. b. c. d. e.

Reduce multi-queries (MQ) Reduce # docs evaluated Model layer reduction Skip model Increase cache expiration

Mild to Moderate Mild to Moderate Moderate to High High Mild to High

3.3

Product and Service-Level Construction

This section details the three phase construction process outlined in §3.1. Table 2 is a representative example of the product-state config. Each state represents a coordinated service-level combination. We present case studies of how levels and targets are realized for deep models and services (§4.2.1) and an LLM product (Copilot Search, §4.2.2). Table 2: Product state lattice: Product state and service-level mapping. Higher number indicates greater degradation. Product State L0 L1 L2 L3 L4 L5

Service-1 Level

...

Service-X Level

0 1 2 3 3 4

... ... ... ... ... ...

0 0 1 1 2 2

Capacity reclamation targets: Each service’s per-level reclamation targets are derived from a stack-wide survival goal. For example, if a product sets a target of 2× baselinepeak requests per second (RPS) survivability without throttling, each service then determines the number of levels required to support that goal and estimates the target loadfactor reduction for each level at constant RPS. Headroom creation strategies may include reducing local compute, reducing demand from upstream services, or both. When a service is shared across products, the reclamation targets per product/scenario (§3.4) are determined independently. Runtime product-state selection, however, is relative. For example, a scenario’s degradation can be driven by other scenarios’ load

3.4

Further Controls in HiLO

Production deployment revealed the need for additional controls beyond product-state selection. Distinct product verticals demanded independent degradation policies, and business stakeholders requested value-aware degradation. HiLO incorporates both without altering its core abstraction. Route Context (Scenario Identifier): A scenario identifier represents a distinct operating context in which a product state applies, such as different request routes or entry points that share a common call graph (e.g., product verticals such as Web Search, Image Search, Video Search etc.). This allows

6

multiple routes to operate under different product states concurrently, even when backed by the same set of services. HiLO separates policies between scenarios and relative priorities between routes. This enables coordinated but differentiated degradation behavior across different contexts. Request Priority: Request priority is optional metadata that classifies traffic within a given route into priority buckets (e.g., high, medium, low). This enables finer-grained degradation by allowing lower-priority traffic to be degraded before higher-value requests [23]. HiLO publishes its policy to the services by request-priority. This ensures that degradation is enforced gradually without impacting valuable traffic first. Priority assignment (on the request path) is external to HiLO and propagated through the request path as metadata. The frontend service (Service A in Figure 4) is responsible for identifying the route context and assigning request priority, ensuring that downstream services operate on consistent, endto-end metadata. By incorporating these dimensions, HiLO enables controlled, quality-aware degradation across contexts.

3.5

states are applied to a fraction of user-traffic (typically 0.5% over 1 or 3 days and then gradually promoted to 1% and 5%) and business metric (e.g., revenue/user, DAU, sessionengagement/user) movement is noted. This makes state selection and product configuration a data-driven process. This methodical approach to identifying states and monitoring them in real-time ensures a high level of predictability in the platform state at any given time.

3.6

Safety and Operational Properties

Some salient properties and design considerations worth understanding are as follows. Eliminating untoward interactions: A single active state per scenario eliminates ambiguity in product behavior and prevents untoward interactions between independently applied controls. Tuning oscillations: When load increases, HiLO transitions monotonically toward more restrictive product states. Recovery transitions are damped by a configurable factor, allowing operators to tune the tradeoff between fast recovery and sustained stability. Failure isolation: Each service enforces the more restrictive of HiLO’s assigned level and its own locally determined level. When HiLO is unavailable, services continue to apply their self-determined service levels independently, preserving availability without requiring HiLO’s coordination. Once HiLO recovers, coordination at product level is reintroduced without disrupting ongoing service execution.

Monitoring, Testing & Experimentation

Monitoring: HiLO and rest of the services are seamlessly integrated into the logging and monitoring framework of the platform, requiring no additional architectural changes. For example, in Bing, on the server side, all impressions including headers and response details are logged at individual services and at Frontend. A dedicated logging pipeline stitches these logs and directs them to Magnetar (MT), an internal largescale offline data processing platform built on Kafka [17], Spark [36] and HDFS [26] with Titan as interactive visualization (Microsoft internal) and querying layer. For real-time reporting and operational monitoring, services upload metrics and counters to Geneva (similar to Azure monitor), which provides near-real-time operational visibility. This infrastructure is simply extended by logging three details at HiLO: the aggregated load-factors from participating services, the selected product state, and the corresponding service levels disseminated to each service. Services log the HiLO-assigned state alongside each impression. Existing dashboards consume this additional metadata to report degradation behavior at product level as needed. Test hooks: Before deploying new product states or service levels, to user traffic, those need to be tested. HiLO exposes test-hooks that allow operators and service owners to validate behavior in isolation. A new config is first configured as a test-scenario in HiLO. Frontend service is then directed to tag either test traffic or a fraction of forked production traffic with the test scenario identifier. HiLO applies the candidate states only to this tagged traffic, enabling measurement of OneDCG, latency, and capacity-utilization without affecting real users. Experiment hooks: After test validation, acceptable states are promoted to experiments (via the existing A/B testing framework [16]) using experimentation hooks. Here new

4

Evaluation and Results

HiLO’s performance is evaluated on the following dimensions: operational response to real overload conditions, comparative performance without HiLO, load-factor and capacity improvements during degradation for services and GPUintensive models, and user-facing impact through A/B experiments and quality metrics.

4.1 HiLO’s Overload Response Evaluated Against Key Metrics 4.1.1

HiLO’s Response to Overload

Figure 6 presents HiLO’s response to one overload event in production. The top chart showcases the increase in traffic (RPS) experienced by a DC in North America. Traffic reached 1.93× its baseline peak and subsequently stayed in the overload territory for another 3 hours. The bottom chart shows product load-factor as computed and seen by HiLO and its product-state transitions. Quality remained within target limits and stability recovered quickly for this case; discussed further in §4.1.2. In a separate model-induced overload (APAC DC, 0.94× peak, ∼6 hours daily over two days), HiLO held

7

OneDCG impact to 0.14 points, versus the 1.11-point drop and severity-2 escalation of the §2.2 incident. Note that productstate transitions follow load-factor, not the traffic curve. 1× baseline peak does not necessarily map to 100% load-factor (Figure 6) due to headroom and the current query-mix considerations.

that crossed the overload threshold that month, alongside 16 planned capacity tests (a total of 25 events). Figure 7 depicts these 9 user traffic overload events managed by HiLO across three regions varying in intensity (1.02× to 2.38× baseline peak) and duration (∼10 min to ∼180 min). The x-axis shows overload duration (first to last minute of user traffic exceeding 1.0× baseline peak) and the y-axis shows active degradation time (total minutes the product was held at a degraded level, L1 or higher). Baseline peak denotes each datacenter’s representative peak traffic level. As noted, some events are fleeting in nature ruling out manual intervention entirely. The figure confirms that active degradation time is always shorter than the overload duration (all points below the diagonal): HiLO degrades only while load-factor exceeds the threshold and recovers promptly. During the rest of the overload, capacity headroom keeps load-factor below the threshold (Figure 6, bottom panel), so the product runs at full quality (L0 ).

HiLO Response — North America DC, Jan 23 2026

User Traffic (× baseline peak)

2.5

Peak: 1.93×

2.0 1.5

1×

1.0 0.5

LF = 100

75

3 2

50

1 25

5

5

5

5

5

5

5

5

5

5

5

5

:1

:4

:1

:4

:1

:4

:1

:4

:1

:4

:1

:4

05

06

06

07

07

08

08

09

09

10

10

0

HiLO Product State

100

05

Product Load Factor

0.0

Availability: All DCs maintained availability above 99.95% (target objective) through these events. Quality Impact: 0.1

Time (Jan 23, 2026)

HiLO’s One Month Operational Snapshot

Datacenter

175 Active Degradation Time (minutes)

OneDCG Drop

Figure 6: HiLO’s response to a sustained traffic surge in a North America DC (Jan 23, 2026). Top: user traffic normalized to baseline peak. Bottom: Computed product load-factor and HiLO’s product-state selection. 4.1.2

125 100 75 50

-0.6

North America Asia-Pacific 01/09 Europe (L3, 2.28x) 01/29 (L2, 1.03x) 01/26 (L5, 1.22x)

150

01/26 (L1, 1.02x)

01/20 (L3, 1.96x)

01/23 (L3, 1.93x)

0 0

25

50 75 100 125 Overload Duration (minutes)

Figure 8 shows that even the worst-case drop for OneDCG (0.425 points at 2.38× peak load) remains below the 0.5-point target threshold. No-result rates also remained within normal limits. This demonstrates HiLO’s ability to maintain quality under overload.

01/23 (L5, 2.38x) 150

EU EU EU NA APAC APAC NA NA APAC 01/23 01/09 01/20 01/23 01/12 01/23 01/26 01/29 01/26 Ordered by peak traffic surge

Figure 8: OneDCG drop during user traffic surges (Jan 2026), ordered by peak intensity. Annotations show peak traffic (×baseline peak) and degradation duration.

01/12 (L3, 1.43x)

01/23 (L5, 1.4x)

25

2.38× 2.28× 1.96× 1.93× 1.43× 1.40× 1.22× 1.03× 1.02× 9m 114m 31m 97m 30m 10m 17m 10m 8m no impact no impact −0.065 −0.067 -0.1 −0.090 −0.120 Europe -0.2 −0.201 N. America -0.3 Asia-Pacific −0.334 -0.4 −0.425 -0.5 Target: −0.5 0.0

175

Recovery velocity: Figure 9 shows metric recovery times. OneDCG recovery is defined as the 5-min rolling avg returning within 0.05 of the pre-event baseline; stability recovery as the first 5-min avg reaching 90%. Duration from overload end to the first minute of these windows is reported. Metric recovery as noted in the chart (both max and median) confirms HiLO’s ability to keep underlying services healthy during overload enabling rapid recovery once load subsides.

Figure 7: HiLO managed degradation during user traffic surge conditions in January 2026. (Lk ) indicates the max product state reached during degradation. During Jan 2026, with HiLO operational across all Bing datacenters (DCs), it actively managed all 9 user-traffic surges 8

N/A N/A

EU EU EU NA APAC APAC NA NA APAC 01/23 01/09 01/20 01/23 01/12 01/23 01/26 01/29 01/26 Ordered by peak traffic surge

Capacity Test: Traffic Ramp + Product-State Response 2.0 Total Traffic (× baseline peak)

Figure 9: Metric recovery after overload ends, measured from the last minute of traffic exceeding 1.0× baseline peak. Adaptive response to varying surge profiles: Figure 10 compares two APAC DC events with similar peak traffic but different proportional surges (+45% from 1× baseline peak vs. +203% from 0.5×). HiLO’s response is not determined by peak traffic alone; it adapts to load-factor dynamics, producing different escalation depths for the same peak.

0.5

2

2

0.5

1 0 −10

0 :0

Time (Jan 12 — Asia-Pacific DC)

APAC EU

4 3

HiLO Level

2

7 :2 07

2 :1 07

7 :5 06

2 :4 06

7 :2 06

2 :1 06

7 :5 05

:4

2

0

05

7 :2 05

34 35

2.13× 2.06×

-0.05 -0.02

35 36

2.06× 2.16×

0 0

Both tests targeted ∼2× baseline peak load. With HiLO, no OneDCG impact is observed. That said, capacity tests are short-duration events where load ramps gradually; the OneDCG movements discussed in §4.1.2 provide a more representative picture of impact under production overload. Figure 12 shows the difference in No-result error rates with and without HiLO. No-result error rate measures the fraction of queries for which backend services fail to produce results, indicating capacity-induced failures. Without HiLO, these error rates are consistently higher or increase as the test progresses. HiLO now allows for better coordination and lower error rates. With HiLO, operators regularly stress DCs to ∼2× baseline with confidence. This section highlights HiLO’s ability to not only provide testability and observability improvements but also to coordinate degradation more effectively under overload conditions.

1 0.0 Time (Jan 23 — Asia-Pacific DC)

Figure 10: HiLO’s adaptive response to two different spike profiles. 4.1.3

With HiLO

5 HiLO Level

User Traffic (× baseline peak)

(b) +203% spike → L5

0.5

40

Dur. Peak OneDCG Dur. Peak OneDCG (min) Load ∆ (min) Load ∆

00

5 :4 23

0 :3 23

5 23

:1

0 :0

5

23

:4

0

22

22

:3

5 :1 22

Without HiLO

0

Peak: 1.40x 1.5 Spike: +203% Max level: L5 Active: 10 min 1.0

10 20 30 Minutes from test start

Table 3: Capacity test comparison: two DCs, without and with HiLO. Peak load vs. OneDCG drop compared.

1 0.0

0

Table 3 summarizes the test parameters for capacity tests on two datacenters, without and with HiLO deployment.

HiLO Level

User Traffic (× baseline peak)

3

HiLO Level

3

1.0

Figure 11: Capacity test with HiLO on a Europe DC: normalized traffic (synthetic + organic) and HiLO’s response.

5 4

4

1.5

0.0

(a) +45% spike → L3 Peak: 1.43x 1.5 Spike: +45% Max level: L3 Active: 29 min 1.0

5

2.16×

HiLO Product State

N/A 1m

4m

10m 1m

21m

36m

45m

62m

0

6m

9m 5m

20

17m

26m 19m

40

3m

Recovery Time After Overload Ended (minutes)

60

degradation. In the baseline (“Without HiLO”), services still degrade under service-local policies (uncoordinated). Bing conducts capacity tests by slowly injecting synthetic traffic to desired level and stopping it. This gradual rampup and HiLO response are shown in Figure 11. Apparent timing differences between traffic and HiLO’s response, including recovery lag after ramp-down and state changes before peak traffic reflect HiLO’s decision interval (§4.3.1), LF signal delays, and degradation actions reclaiming capacity. Without HiLO this product-level behavior was not observable.

OneDCG Stability

Comparative Performance: Capacity Tests

In this section we present capacity test results demonstrating how HiLO improves the system’s coordination during 9

No-Result Rate During 2× Capacity Tests

(previously impossible). OneDCG drop (even during degradation times) was within the target (max drop of 0.38 points). Stability bounced back swiftly once the load normalized. In DC-C, the redirected traffic exposed insufficient capacity for a GPU-intensive model, causing states to oscillate (Figure 14b). Degradation and recovery (§3.6) are configurable and reflect business priorities. Under sustained overload, holding a degraded state too long has larger revenue implications (at least for Bing) than those of re-attempting recovery once loadfactor falls below the overload threshold. Consequently, in this case, the system oscillated between degradation and recovery until the deficit was resolved (hours later) by provisioning additional model capacity. But unlike §2.2 where a similar bottleneck caused a major OneDCG drop and a severity-2 escalation, here HiLO held quality within limits and kept underlying services healthy throughout, requiring no severity-2 interventions. Hence, the key win here is the ability to autonomously maintain the system health and quality under overload, rather than the ability to avoid oscillations entirely during the overload window.

No-Result Rate (% Users)

(a) Asia-Pacific DC 0.010

Without HiLO With HiLO

0.005

No-Result Rate (% Users)

0.000 0

5

10

15

20

25

30

10 15 20 25 Minutes from test start

30

(b) Europe DC 0.02 0.01 0.00 0

5

35

Figure 12: No-result error rate (3-min moving avg) with vs. without HiLO, across two DCs over the test duration. HiLO’s coordinated degradation shows reduced errors. In APAC DC the increase towards the end without HiLO is dramatic.

0.2

30+ Hours, 3 Datacenters and Autonomous Overload Management

OneDCG (Δ)

4.1.4

Recently HiLO autonomously managed states when a major US DC (DC-A) was taken offline for ∼30+ hours for maintenance. Traffic was redistributed by CDN and two other DCs absorbed the load (Figure 13). Availability remained above target SLO with degradation applied only as necessary. DC-B operated at L1 -L2 for 83% of its 16.7 degradation hours, while DC-C’s L5 spikes were brief (0.4 hours).

(a) Mar 16 02:00

Mar 16 14:00 L1

DC-B DC-C

Mar 17 02:00

L1

0.0 L1 (mild)

Mar 17 14:00

Mar 18 02:00

L2

L2

L3

L4

2.5

5.0 7.5 10.0 12.5 Hours at Degradation State L2

6.8h

L3

L4

Mar 17 02:00

Mar 17 14:00

Mar 18 02:00

(b) Mar 16 14:00 L1

Mar 17 02:00 L2

L3

Mar 17 14:00 L4

Mar 18 02:00 L5

90%

Figure 14: (a) OneDCG values for DC-B and DC-C during this period. DCs remain within normal variance throughout. (b) Stability at DC-C, colored by HiLO states, recovers to the baseline immediately after load subsides.

Mar 18 14:00 L3

Mar 16 14:00

50

L0

0.25

(a)

75

Mar 16 02:00

0.50

0.00

DC-B DC-C

Mar 16 02:00

DC-A DC-B DC-C

0.75

−0.2

Stability (%)

Relative Traffic

1.00

0.0

16.7h

(b) 15.0 L5 (severe)

4.2

Workload Case Studies

4.2.1

Deep Models and Services Evaluation

Here we present the levels and corresponding capacity and quality measurements for an expensive deep ranking model and a couple of core services.

Figure 13: (a) Relative organic traffic across three DCs during planned maintenance. DC-A is taken offline (shaded region), and its load is redistributed to DC-B and DC-C. (b) HiLO state distribution during the maintenance window.

Deep Model: Capacity Gains and Quality Tradeoffs. Table 4 provides the degradation modes for a sample ranking model, driven by two independent mechanisms. First,

Figure 14 demonstrates the ability to track stability by state 10

upstream Multi-Queries (MQ) and document reduction progressively reduces inference calls at the model (up to 29% at L5 ), creating significant headroom while the model itself operates unchanged. Second, at deeper levels (L4 –L5 ), model layer reduction reduces per-inference latency and compute cost. These metrics are reported independently: call volume reduction acts on demand (fewer calls), while model layer reduction acts on supply (cheaper per call). For comparison, when this model is throttled or skipped entirely, the OneDCG impact far exceeds the Severe threshold (≫0.5), underscoring the value of graduated degradation over binary model disabling. Note: model layer reduction is currently validated and pending production deployment. Further degradation levels are being explored simultaneously. OneDCG is Bing’s primary, business-critical quality metric, so service owners carefully manage its drop across levels to keep it within target during degradation. It is not the only signal we track. Broader user-facing online metrics such as content freshness and engagement (§4.3.2) can degrade more at deeper states even when OneDCG is held.

service graph and one degraded level), while ReRanking uses six levels (five of them degraded). 4.2.2

HiLO applies beyond Web Search and is deployed for Copilot Search, an LLM-centric conversational search experience within Bing. Copilot Search uses 5 product states (L0 –L4 ), with load-factors derived from underlying model services. Figure 15 shows HiLO’s product states across three regions during a production overload event. Unlike Web Search, where degradation adjusts backend processing, Copilot Search degrades by progressively disabling product features that trigger model calls: repositioning the Copilot Search tab rightward (L1 ), disabling additional suggestions (L2 ), disabling positional features (L3 ), and hiding the tab entirely (L4 ). Bing Web Search and Copilot Search represent two distinct execution regimes: fan-out-heavy low-latency services and LLM-centric pipelines with high and variable inference latency. Notably, Copilot Search degradation actions are enforced entirely at the frontend layer (e.g., tab repositioning, feature hiding), whereas Web Search degradation is enforced by backend services. Despite these differences in execution regime and degradation mechanics, HiLO’s framework applies unchanged.

Table 4: Ranking model: Two independent capacity mechanisms and their OneDCG impact. Call vol. ∆ = inference call change via upstream MQ/doc reduction; Lat. ∆ = latency reduction via model layer reduction at constant RPS. OneDCG impact categories: Min. (<0.1), Mod. (0.1–0.25), High (0.25– 0.5), Sev. (>0.5). Call Vol. ∆

L1 L2 L3 L4 L5

∼−3% N/A ∼−7% N/A ∼−13% N/A ∼−22% ∼−7% ∼−29% ∼−12%

MQ + doc reduc. MQ + doc reduc. MQ + doc reduc. Above + layer reduc. Above + layer reduc.

Region-US

HiLO Product State

State Action

Case Study: HiLO for Copilot Search

Lat. ∆ OneDCG vs L0 Imp. Min. Min. Min. Mod. Mod.

L4 L3 L2 L1 L0

12:00

13:00

14:00

Region-EU

15:00 16:00 Time (UTC)

Region-Asia

17:00

18:00

19:00

Figure 15: HiLO & Copilot Search during an overload event.

4.3

Operational and User-Facing Impact

4.3.1

HiLO Operational Overhead

Services: Capacity Gains and Quality Tradeoffs. Table 5: Balance between capacity gains and OneDCG impact for two services across product states. Service

State

CPU ∆ (vs L0 )

Latency ∆ (vs L0 )

OneDCG Imp.

Exec. Engine ReRanking ReRanking ReRanking ReRanking ReRanking

L3 + L1 L2 L3 L4 L5

−40% −5% −10% −23% −33% −36%

N/A −4% −4% −14% −25% −25%

Mod. Min. Min. Min. Mod. Mod.

HiLO operates outside the request path and adds no execution path latency. Table 6 summarizes its operational footprint. Table 6: HiLO operational overhead in production. Metric Decision period HiLO internal latency (P95) # CPU Cores per region 4.3.2

Table 5 reports the CPU and latency reductions (capacity gains) and the corresponding OneDCG impact across levels for two services. Exec. Engine operates with two levels (full

Value 2 minutes <1 ms 64

User-Facing Impact of Product States

HiLO’s product states are validated through user-facing A/B experiments before deployment. Each candidate state is ac11

Load signals are service-specific. In our experience, no single indicator works for every service. The right one reflects where a service actually bottlenecks. A network-bound service leans on queue length, since requests pile up when the network is the constraint even while CPU stays low. By contrast, a CPU-bound service, such as ranking, is better captured by CPU utilization. GPU-bound services, such as models and their serving layers, rely on GPU utilization. Some services combine these into a hybrid indicator, such as CPU utilization with queue length, to capture their bottleneck better. The right indicator can also shift as a service’s workload evolves. This is precisely where a single load-factor abstraction helps. Each service chooses the indicator, or blend, that matches its own bottleneck, and HiLO adapts automatically because all it requires is a normalized load-factor. Design-time burden. Defining product states requires each participating service to identify degradation actions, quantify their capacity impact, and compose them into coherent product states. This often takes weeks to months of cross-team coordination. Identifying degradable actions is often non-trivial in services with complex call graphs and dependencies. For example, a service with 90k+ sub-workflows, each using a tiny fraction of CPU, required three months of analysis. Ownership can also be ambiguous: platform teams optimize for load and capacity while feature teams own workflows and userfacing quality, and it is often unclear whether a degradation action belongs in the leaf service or the calling layer. Callgraph analysis reduces manual effort by surfacing degradable actions, but analysis time still grows with service complexity. Full automation of construction remains open, with adaptive, learned control (§6) an active direction. Much of this cost for Bing reflects retrofitting HiLO onto large, pre-existing services as their RPS grew. The framework itself is general and applies to new and existing services alike. A service that treats degradation as a first-class concern from the start onboards with considerably less effort, though some overhead remains for defining and validating degradation options. Staged adoption. Given this burden, HiLO adoption is better done in stages, and at Bing it proceeded in phases. Services first emitted load-factor signals; coordinated degradation was then applied to the most critical, capacity-constrained backend services and models, and validated end-to-end. Then gradually HiLO coverage was expanded across the stack. Services not yet onboarded continued under their existing service-local degradation until then. At this scale, adoption also requires executive sponsorship and a dedicated central team to drive cross-team coordination. Applicability to decoupled architectures. A natural question is whether product-level coordination scales to a product built from many small, independently owned services. Managing many decoupled services from a central control plane, out of band, is an established pattern in service meshes. HiLO aggregates per-service status, selects a product state, and distributes coordinated service levels, while each service

tivated on a controlled fraction of production traffic, and its impact on key business metrics (e.g., revenue, DAU, session engagement) is measured against baseline (L0 ). This helps measure the quality tradeoffs at each state and makes state selection a data-driven process. Table 7 summarizes user-facing impact of product states L1 and L2 vs. baseline (L0 ). Retention is unaffected. Content freshness degrades monotonically, confirming expected tradeoffs under degradation. Distinct Search Query (DSQ) reformulation rate, indicating that users could not find satisfactory results with their initial query, increases consistently across both surfaces and states (+0.14% to +3.49%), with the mobile L1 increase (+3.49%) being statistically significant. This directionally consistent trend confirms a mild but measurable user-facing quality tradeoff under degradation. Tail latency improves on both surfaces, confirming capacity reclamation as an aggregate effect of coordinated degradation across all participating services. Other metrics (e.g., revenue) exhibit similar directional patterns but are omitted here. Table 7: User-facing metric impact by product state. Each cell shows the relative change (∆%) vs. baseline (L0 ). Metrics marked with ∗ are statistically significant at p < 0.05. Metric

Surface

L1 ∆%

L2 ∆%

Retention Short-Term DAU

Both

≈0%

≈0%

Content Freshness SuperFresh Rate SuperFresh Rate

Desktop Mobile

−11.02%∗ −10.70%∗

−15.21%∗ −14.41%∗

Engagement DSQ Reformulation Rate DSQ Reformulation Rate

Desktop Mobile

+0.14% +3.49%∗

+0.36% +1.93%

Latency (Server-side) WebAnswer Latency P95 WebAnswer Latency P95

Desktop Mobile

−1.65%∗ −1.69%∗

−3.30%∗ −3.37%∗

5

Learnings and Limitations

HiLO’s design involves deliberate tradeoffs that bound its applicability. This section draws on Bing’s experience and guides adoption decisions. Notably, degradation changes are carefully validated and rolled out, reducing the risk of live incidents. Coarse-grained control. At Bing, HiLO operates over a small number of product states (6 in number), each representing a discrete operating state. This coarse granularity simplifies validation and operational reasoning but limits the system’s ability to make fine-grained, per-feature adjustments. In our experience, coarse-grained control is sufficient: finer granularity neither reclaims meaningfully more capacity per step nor accelerates reclamation.

12

enforces its assigned level in its own request path. Because HiLO stays off the request path, allows services to fall back to local degradation when unavailable (§3.6), and partitions coordination by scenario (§3.4), the central aggregator is not a choke point. Unlike a service mesh, HiLO coordinates qualityaware degradation and adds no per-request latency (§4.3.1) as enforcement is by the service itself. For the configuration itself, HiLO coordination is generally applied to the few services on the capacity-constrained critical path, identified at design time (§3.3). The rest of the services can continue under their own service-local degradation, backed by performance isolation guarantees. In Bing, a given scenario coordinates on the order of a dozen major services and models, so the design cost typically scales with the number of bottleneck services and not the total count. Quality vs. stability tension. Stability is a choice. Recovery from degradation is governed by a configurable damping factor (§3.6). We can achieve a highly stable outcome by enforcing degradation for longer periods. Bing’s policy attempts to strike a balance and at times accepts reduced stability during degradation. For example, under 2× load, aggregate stability drops while OneDCG is preserved (Figure 14). Policy is optimized for business metrics (including revenue) rather than treating stability as a goal in itself. Track degradation itself for operational monitoring. Effective degradation can obscure underlying issues. In one instance, insufficient model capacity caused daily degradation across multiple DCs for over two weeks. OneDCG and error rates remained within limits, and stability recovered quickly after each incident, masking the underlying issue and making it appear less severe than it was. This led us to treat degradation itself as a first-class signal, tracking its frequency, depth, and duration independently. For example, if a single model causes deeper or longer degradation, then it is escalated as a severity-2 incident. Metric latency and tuning risk. While availability and OneDCG are measurable in near real-time, business-critical metrics such as revenue and daily active users arrive with multi-hour or next-day latency. A misconfigured product state may appear healthy on fast-loop signals but cause business impact that surfaces only hours later, necessitating both realtime signals and A/B experimentation (§4.3.2). Load-spreading first, degradation next. Overload handling happens at two layers. First, increased load in a region is spread globally via CDN routing. Aggregated backend load-factor is also one of the quality signals used for routing at CDN. Graceful degradation at DC level kicks in only if load-spreading proves inadequate. This is why HiLO is scoped per datacenter rather than globally. This mechanism assumes parity across datacenters, without which spreading is infeasible (as features become unavailable). Because the two layers apply independent thresholds and timing to the shared signal, misalignment can still cause feedback. In one observed event, a double traffic spike pointed to such interac-

tion between CDN redistribution and HiLO’s response, reinforcing the need for threshold alignment. Bing manages this via appropriate damping factors for failaways and DC-level degradation to prevent rapid oscillations. Continuous investment and optimization. Periodic validation of states through local tests, capacity tests and A/B experiments is essential. For example, initial A/B experiments revealed that certain mobile surface metrics diverged from expectations, prompting recalibration actions. Based on demonstrated capacity gains, model layer reduction investments are scaled up: exploring deeper degradation levels and expanding coverage to additional models. Also, initially model refresh cycles outpaced degradation validation, prompting us to incorporate degraded-mode design and validation in the regular model development cycle. Similarly, the LFs of services and models outside HiLO’s purview are monitored regularly for capacity and quality impact and onboarded to HiLO as needed. Finally, recovery times, particularly stability, vary across DCs and are tracked and studied for optimization opportunities.

6

Conclusion and Future Work

Graceful degradation at Bing has evolved over six years. Early approaches prioritized service-local protections and maintained high availability but left product quality as an emergent behavior. HiLO emerged from our experience managing degradation across dependent services and the need to balance availability and quality. HiLO makes quality a controlled outcome by resolving dependencies at design time and autonomously operating over a small lattice of pre-validated product states. Deployed since 2025, HiLO continuously manages overload events across all Bing datacenters (serving over a billion monthly active users [27]), bounding quality loss within operational targets across Web Search and LLM pipelines. Our evaluation demonstrates that HiLO preserves availability, bounds quality loss, and produces monotonically ordered user-facing impact validated through A/B experiments, transforming degradation into a predictable and well-controlled process at product level. Future directions include integrating product-level awareness into cross-region traffic management and exploring learned control policies for HiLO that discover optimal product states using historical load patterns and quality outcomes.

Acknowledgments We thank the teams across Search and AI Platform whose contributions enable coordinated graceful degradation across Bing. This includes frontend, serving platform, relevance, live site engineering, and Bing metrics. We also thank our leadership for their continued support. We finally thank the NSDI review team and our shepherd for their thorough reviews and careful feedback, which materially improved this paper.

13

References

Wagner, Matthias Wichtlhuber, Juan Tapiador, Narseo Vallina-Rodriguez, Oliver Hohlfeld, and Georgios Smaragdakis. A Year in Lockdown: How the Waves of COVID-19 Impact Internet Traffic. Communications of the ACM, 64(7):101–108, 2021.

Implement Grace[1] Amazon Web Services. ful Degradation to Transform Applicable Hard Dependencies into Soft Dependencies. AWS Well-Architected Framework, https: //docs.aws.amazon.com/wellarchitected/202203-31/framework/rel_mitigate_interaction_ failure_graceful_degradation.html, 2022.

[11] Lexiang Huang, Matthew Magnusson, Abishek Bangalore Muralikrishna, Salman Estyak, Rebecca Isaacs, Abutalib Aghayev, Timothy Zhu, and Aleksey Charapko. Metastable Failures in the Wild. In Proceedings of OSDI, pages 73–90, 2022.

[2] Amazon Web Services. Navigating GPU Challenges: Cost Optimizing AI Workloads on AWS. https: //aws.amazon.com/blogs/aws-cloud-financialmanagement/navigating-gpu-challenges-costAcoptimizing-ai-workloads-on-aws/, 2025. cessed: 2026-02-20.

[12] Peng Huang, Chuanxiong Guo, Lidong Zhou, Jacob R. Lorch, Yingnong Dang, Murali Chintalapati, and Randolph Yao. Gray Failure: The Achilles’ Heel of CloudScale Systems. In Proceedings of HotOS, pages 150– 155, 2017.

[3] Betsy Beyer, Chris Jones, Jennifer Petoff, and Niall Richard Murphy. Site Reliability Engineering: How Google Runs Production Systems. O’Reilly Media, 2016. Chapters 21 (Handling Overload) and 22 (Addressing Cascading Failures).

[13] Kalervo Järvelin and Jaana Kekäläinen. Cumulated Gain-Based Evaluation of IR Techniques. ACM Transactions on Information Systems, 20(4):422–446, 2002. [14] Arjun Kharpal. A ‘Seismic’ Nvidia Shift, AI Chip Shortages and How It’s Threatening to Hike Gadget Prices. https://www.cnbc.com/2025/12/02/nvidiashift-ai-chip-shortages-threatening-tohike-gadget-prices.html, 2025. Accessed: 2026-02-20.

[4] Nathan Bronson, Abutalib Aghayev, Aleksey Charapko, and Timothy Zhu. Metastable Failures in Distributed Systems. In Proceedings of HotOS, pages 221–227, 2021. [5] Inho Cho, Ahmed Saeed, Joshua Fried, Seo Jin Park, Mohammad Alizadeh, and Adam Belay. Overload Control for µs-Scale RPCs with Breakwater. In Proceedings of OSDI, pages 299–314, 2020.

[15] Cristian Klein, Martina Maggio, Karl-Erik Årzén, and Francisco Hernández-Rodriguez. Brownout: Building More Robust Cloud Applications. In Proceedings of ICSE, pages 700–711, 2014.

[6] Cisco. Cisco Annual Internet Report (2018– 2023) White Paper. https://www.cisco.com/ c/en/us/solutions/collateral/executiveperspectives/annual-internet-report/whitepaper-c11-741490.html, 2020. Accessed: March 2026.

[16] Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M. Henne. Controlled Experiments on the Web: Survey and Practical Guide. Data Mining and Knowledge Discovery, 18(1):140–181, 2009. [17] Jay Kreps, Neha Narkhede, and Jun Rao. Kafka: a Distributed Messaging System for Log Processing. In Proceedings of the NetDB, 2011.

[7] Cloudflare. Training a Million Models Per Day to Save Customers of All Sizes from DDoS Attacks. https://blog.cloudflare.com/training-amillion-models-per-day-to-save-customersof-all-sizes-from-ddos/, 2024. Accessed: 2026-02-20.

[18] Justin J. Meza, Thote Gowda, Ahmed Eid, et al. Defcon: Preventing Overload with Graceful Feature Degradation. In Proceedings of the 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI), pages 607–622, 2023.

[8] Cloudflare. The 2025 Cloudflare Radar Year in Review. https://blog.cloudflare.com/radar2025-year-in-review/, 2025. Accessed: February 2026.

[19] Microsoft. Announcing the Next Wave of AI Innovation with Microsoft Bing and Edge. https://blogs.microsoft.com/blog/2023/ 05/04/announcing-the-next-wave-of-aiinnovation-with-microsoft-bing-and-edge/, 2023. Bing surpassed 100 million daily active users. Accessed: March 2026.

[9] Jeffrey Dean and Luiz André Barroso. The Tail at Scale. Communications of the ACM, 56(2):74–80, 2013. [10] Anja Feldmann, Oliver Gasser, Franziska Lichtblau, Enric Pujol, Ingmar Poese, Christoph Dietzel, Daniel

14

[20] Jeffrey C. Mogul. Emergent (Mis)behavior vs. Complex Software Systems. In Proceedings of EuroSys, pages 293–304, 2006.

[31] Jeffrey S. Vitter. Random Sampling with a Reservoir. ACM Transactions on Mathematical Software, 11(1):37– 57, 1985.

[21] Netflix. Hystrix: Latency and Fault Tolerance for Distributed Systems. https://github.com/Netflix/ Hystrix, 2013.

[32] Matt Welsh and David Culler. Adaptive Overload Control for Busy Internet Servers. In Proceedings of USITS, pages 43–56, 2003.

[22] Netflix. Keeping Netflix Reliable Using Prioritized Load Shedding. Netflix Tech Blog, https://netflixtechblog.com/keepingnetflix-reliable-using-prioritized-loadshedding-6cc827b02f94, 2020.

[33] Matt Welsh, David Culler, and Eric Brewer. SEDA: An Architecture for Well-Conditioned, Scalable Internet Services. In Proceedings of SOSP, pages 230–243, 2001. [34] Jiali Xing, Akis Giannoukos, Paul Loh, Shuyue Wang, Justin Qiu, Henri Maxime Demoulin, Konstantinos Kallas, and Benjamin C. Lee. Rajomon: Decentralized and Coordinated Overload Control for LatencySensitive Microservices. In Proceedings of the 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI), pages 21–36, 2025.

[23] Netflix. Enhancing Netflix Reliability with ServiceLevel Prioritized Load Shedding. Netflix Tech Blog, https://netflixtechblog.com/enhancingnetflix-reliability-with-service-levelprioritized-load-shedding-e735e6ce8f7d, 2024. [24] Karl Johan Åström and Tore Hägglund. PID Controllers: Theory, Design, and Tuning. Instrument Society of America, Research Triangle Park, NC, 2nd edition, 1995.

[35] Haoyi Xiong, Jiang Bian, Yuchen Li, Xuhong Li, Mengnan Du, Shuaiqiang Wang, Dawei Yin, and Sumi Helal. When Search Engine Services Meet Large Language Models: Visions and Challenges. IEEE Transactions on Services Computing, 17(6):4558–4577, 2024.

[25] Resilience4j Contributors. Resilience4j: Fault Tolerance Library. https://resilience4j.readme.io/, 2023.

[36] Matei Zaharia, Reynold S. Xin, Patrick Wendell, Tathagata Das, Michael Armbrust, Ankur Dave, Xiangrui Meng, Josh Rosen, Shivaram Venkatesh, Michael J. Franklin, Ali Ghodsi, Joseph Gonzalez, Scott Shenker, and Ion Stoica. Apache Spark: A Unified Engine for Big Data Processing. Communications of the ACM, 59(11):56–65, 2016.

[26] Konstantin Shvachko, Hairong Kuang, Sanjay Radia, and Robert Chansler. The Hadoop Distributed File System. In 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), pages 1–10. IEEE, 2010. https: [27] Microsoft FY26 Q3 Earnings. //www.microsoft.com/en-us/investor/events/ fy-2026/earnings-fy-2026-q3, 2026. Bing monthly active users reached one billion. Accessed: October 2026.

[37] Hao Zhou, Ming Chen, Qian Lin, Yong Wang, Xiaobin She, Sifan Liu, Rui Gu, Beng Chin Ooi, and Junfeng Yang. Overload Control for Scaling WeChat Microservices. In Proceedings of SoCC, pages 149–161, 2018.

[28] Lalith Suresh, Marco Canini, Stefan Schmid, and Anja Feldmann. C3: Cutting Tail Latency in Cloud Data Stores via Adaptive Replica Selection. In Proceedings of NSDI, pages 513–527, 2015. [29] Thales/Imperva. 2025 Imperva Bad Bot Report: AI Fuels Rise of Hard-to-Detect Bots. https:// cpl.thalesgroup.com/about-us/newsroom/2025imperva-bad-bot-report-ai-internet-traffic, 2025. Accessed: February 2026. [30] Uber. Introducing QALM, Uber’s QoS Load Management Framework. Uber Engineering Blog, https://www.uber.com/blog/qalm-qos-loadmanagement-framework/, 2018.

15

Related documents

• HiLO: Towards Product-Level Graceful Degradation with a Quality-Aware, Load-Aware Control Framework#1457998
• The benefits and challenges of explicit memory management in OpenMP Target GPU offloading#610057
• The benefits and challenges of explicit memory management in OpenMP Target GPU offloading#608995
• Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Staging#1142386
• Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Staging#1122291