FIRE: Failure Isolation and Reliability for Production-Connected Experiments at Scale

arXiv:2611.14962v1 [cs.SE] 11 Nov 2026

FIRE: Failure Isolation and Reliability for Production-Connected Experiments at Scale Shripad Nadgowda

Gaurav Mitra

Sing Sing Ma

Vipul Chaskar

Meta, Inc. USA shri.nadgowda@gmail.com

Meta, Inc. USA gmitra@meta.com

Meta, Inc. USA singsing@meta.com

Meta, Inc. USA vchaskar@meta.com

Pradipta Ghosh

Canturk Isci

Onkar Deshpande

Meta, Inc. USA iampradipta@meta.com

Meta, Inc. USA cisci@meta.com

Meta, Inc. USA odeshpande@meta.com

Edgardo Aviles-Lopez

Ryan Feng

Ján Ondrušek

Meta, Inc. USA zhengliangfeng@meta.com

Meta, Inc. USA ondrusek@meta.com

Abstract Pre-production changes are usually tested in isolated environments to prevent their side effects from reaching production. Some record-and-replay platforms allow a productionconnected mode: candidate changes receive recorded production requests while selected downstream calls reach production dependencies. This improves fidelity before deployment, but it also introduces reliability risks. At hyperscale, experimental traffic can fan out across shared services, trigger latent defects, or send malformed input to systems serving user requests. In this paper, we show how production-connected experimentation can operate reliably in a large-scale recommendation serving system. To contain downstream effects without sacrificing realistic dependency behavior, the platform uses three complementary guardrails. Isolated tiers redirect replay traffic away from shared dependencies where replication is practical. Dependency monitors observe downstream health and experiment-attributed errors and terminate experiments after sustained violations. Traffic pinning limits experimental calls to a designated subset of downstream workers when full replication would require scarce capacity. Isolated ranking tiers process billions of replayed requests per day using under 1% of the corresponding serving capacity. Over 90 days, these tiers experienced more than 200,000 abnormal worker exits without directly affecting productionserving workers. During a separate three-month window, dependency monitors cancelled approximately 5% of experiments for the measured ranking service and accounted for 87% of its cancellations. Dependency-aware triggering also

Meta, Inc. USA edgardo@meta.com

reduced code-review experiment executions by 25%. These results show that high-fidelity, production-connected experimentation can operate at scale when downstream effects are isolated where possible, bounded where necessary, and continuously monitored and attributed.

1

Introduction

Testing candidate changes before deployment is an established part of software delivery. These tests are typically run in environments isolated from production so that crashes, malformed requests, and excess resource use cannot affect production systems. Isolation, however, can weaken the resulting signal. Mocked dependencies may omit important cross-service behavior, while complete replicas require current data, configuration, and specialized hardware. Canary testing provides realistic dependency behavior, but only after a change has entered the release path. Record-and-replay platforms improve pre-deployment fidelity by exercising candidate code with requests captured from production [28, 35]. Some platforms also permit selected calls from the experimental deployment to reach production dependencies [9]. We call this model productionconnected experimentation. It combines an isolated service under test (SUT) with realistic downstream behavior before the candidate change lands. Operating this model for a hyperscale recommendation serving system creates a separate reliability problem. A recommendation request can cross query, aggregation, ranking, feature, inference-gateway, and prediction services and fan out to many downstream workers. Experimental traffic can

2 System Context 2.1 Recommendation serving stack

therefore consume shared capacity, exercise latent dependency defects, or carry malformed input beyond the SUT. Failures may appear several service boundaries away from the experiment that caused them, making containment and attribution difficult. Observed failures illustrate this problem. In one case, replayed requests repeatedly exercised a latent timeout defect in a shared ranking service and caused workers to restart. In another, a candidate ranking change sent malformed tensor metadata to a shared prediction service and left acceleratorbacked workers unable to serve requests. Other effects included delayed resource cleanup and contamination of downstream telemetry. These cases show that protecting the SUT is insufficient when its outgoing calls remain connected to shared services. The platform combines three guardrails. Isolated tiers redirect replay traffic away from shared dependencies where a representative replica can be maintained. Dependency monitors observe downstream health and experiment-attributed errors, then terminate experiments after sustained violations. Traffic pinning limits experimental calls to a designated subset of downstream workers when full replication would require scarce capacity. The mechanisms provide different guarantees and are selected according to each dependency’s state, capacity, and failure modes. Scale also makes unnecessary experiments a reliability concern because every run consumes platform resources and may add load to shared dependencies. The platform therefore uses build-graph reachability for source changes and observed runtime consumption for configuration changes. This dependency-aware triggering reduced code-review experiment executions by 25%. This paper makes three contributions:

In the recommendation serving stack, each request traverses a graph of cooperating services rather than a single binary. Figure 1 shows the representative part of that graph that is in scope for this paper. A query and aggregation service receives the product request, resolves viewer and surface context, assembles the candidate set to be scored, and combines ranked results into the response returned to the product. A ranking service scores and orders those candidates, retrieving model features from feature services and sending model-evaluation requests through an inference gateway. The gateway routes those requests to a prediction service, which executes machinelearned models on accelerator-backed workers. The request graph widens at ranking. Scoring one candidate set can require many calls to feature services and the inference gateway. The prediction layer hosts many models because different request classes and stages of the ranking funnel use different model sets. Query and aggregation, ranking, and the inference gateway are independently deployed services and can each be selected as the SUT. Two properties matter for production-connected experimentation. A failure can surface several service boundaries away from the component that caused it, and one ranking request can issue many downstream calls. Replaying a small request stream can therefore create meaningful demand on a shared dependency. We use product surface to mean a user-facing feature that requests recommendations, such as a feed or short-form video view. Recommendation services commonly run as multiple independent tiers. A tier is an instance group for a particular product surface or request path. Tiers may share code while differing in request mix, candidate-set size, dependency fan-out, model set, and latency budget. A change that is neutral for one tier can therefore regress another, so validation is organized around the tier being exercised rather than only around the service binary. The dependency classes resist isolation for different reasons. These differences determine which containment mechanism applies in Section 4. Ranking and inference-gateway tiers can often be replicated as isolated tiers because their serving state and configuration can be kept close enough to the corresponding shared tier for replay comparisons to remain meaningful. Feature services are harder to replicate because they depend on large, continuously updated stores; stale or divergent state can cause the same replayed request to produce different results for reasons unrelated to the candidate change. Prediction services are constrained by accelerator-backed capacity and by the number of distinct

• We characterize the reliability risks created when predeployment experiments retain access to shared production dependencies. • We present a guardrail design that combines selective isolation, continuous downstream monitoring and attribution, and bounded worker exposure. • We evaluate these mechanisms at scale. Isolated ranking tiers process billions of replayed requests per day using under 1% of the corresponding serving capacity and experienced more than 200,000 abnormal worker exits over 90 days without directly affecting workers serving production traffic. Dependency monitors accounted for 87% of experiment cancellations in a separate three-month window. These results show how production-connected experimentation can provide realistic pre-deployment signals while keeping downstream effects observable, attributable, and contained. 2

task still held references to request-scoped objects, so after the request was torn down it could de-reference freed state and crash the worker. The change passed pre-submit validation because the timeout path depended on a new caller pattern that was rarely exercised by ordinary request traffic at the time of submission. A record-and-replay experiment later drove recorded requests through that path at experiment rate. The replay did not introduce the defect, but it concentrated enough matching requests to exercise the latent path repeatedly. Ranking workers receiving the experiment’s replayed traffic began crashing and restarting. Automated health signals detected the failures quickly, but the first diagnostic step was attribution: the restart signature identified the failing service, not the upstream experiment whose replayed requests were triggering the failure. Containment therefore required separating experiment-originated traffic from ordinary requests and then stopping the relevant experiments. Stopping the experiments removed the replay pressure, while the timeout path was fixed separately so the asynchronous task could not outlive the request state it accessed. This scenario illustrates why reproducing downstream behavior is different from containing downstream effects. The same mechanism that creates risk also creates signal: replayed requests exercise real downstream behavior, but any effect of those requests can also reach the shared dependency. The replayed requests were sampled from ordinary serving traffic, but the defect was timing-dependent rather than content-dependent. It appeared only when a request timed out after asynchronous feature-extraction work had started but before that work finished. Replay can change that timing by repeating sampled requests and driving the SUT at a controlled utilization, which changes queuing, cache behavior, and downstream response timing. As a result, requests that completed normally when originally served can exercise a cancellation interleaving during replay. The reliability concern arose because those replayed requests reached a shared downstream dependency rather than an isolated equivalent; where such isolation is impractical, the platform needs monitoring and attribution at the downstream dependency.

Product Clients

Query and aggregate

Ranking

Inference Gateway

Feature service

prediction service

Content and feature store

Accelerator fleet (many models)

Figure 1: Representative serving graph scoped to this paper. A request enters query and aggregation, passes to ranking, and fans out to a feature service and an inference gateway. The gateway routes model requests to a prediction service backed by an accelerator fleet. Stacked outlines indicate independent tiers. model instances. As a result, no single containment strategy applies uniformly across the graph. Experiment model. The platform selects one service tier as the SUT, runs control and treatment versions in comparable environments, and replays the same sampled requests against both. A workload binds that tier to its request corpus, resource reservation, and counters. The SUT receives no user requests, but downstream calls that are not redirected can still reach shared dependencies.

3

Real-World Failure Modes

Production-connected experimentation creates failure modes that do not appear in fully isolated testing. Replayed requests can exercise shared downstream dependencies, so the visible failure may occur outside the SUT that originated the traffic. The two scenarios below show complementary causes: replay can expose a latent defect in a dependency, or a candidate change can send invalid input to that dependency. Table 1 summarizes the broader risks and the platform response to each.

3.1

3.2

Injecting invalid input

A contrasting scenario starts with the candidate change rather than with a latent defect in the dependency. A ranking change altered how features were converted into tensors before calling a shared prediction service. For one model configuration, the change treated a per-candidate feature as though it were request-level state, collapsing the batch dimension. The resulting tensor had shape [1, features] where the model contract required [batch, features].

Exercising a latent dependency defect

One scenario began with a change to the ranking service’s request-timeout handling. On timeout, the service canceled the request but allowed an asynchronous feature-extraction task within the request handler to continue running. That 3

Risk

Consequence

Platform response

Replay reaches a replicable dependency Shared serving capacity can be affected Failure appears outside the SUT Its cause is difficult to identify

Isolated tier Dependency monitor and request identity Fault reaches many workers Small traffic volume can cause broad impact Traffic pinning Old replay keeps dependency state ac- Cleanup or model reaping is delayed Bounded replay lookback tive Unrelated workloads are executed Resource use and downstream exposure in- Dependency-aware triggering crease

Table 1: Risks in production-connected experimentation and the platform response to each. The malformed tensor crossed the SUT boundary and failed inside the prediction service. The failure was not a local ranking error: the downstream service received input that violated a model contract, triggered an accelerator assertion, and left affected workers unable to continue serving requests until they were restarted. The visible symptoms therefore appeared in the shared dependency rather than in the deployment that produced the malformed input. Diagnosis required correlating evidence from multiple services: the ranking experiment that constructed the tensor, the inference gateway that routed the request, and the model whose input contract was violated. The first response was to stop the experiment and recover the affected workers. The same failure mode later appeared through other request paths to the shared dependency. The difficult part was identifying all upstream services and request paths that could emit the malformed input. This recurrence showed that coverage is part of the design problem. A mechanism protects only the request paths and dependencies to which it is applied.

3.3

workers reachable by replayed traffic, not only the fraction of requests the experiment contributes. SUT-local validation is insufficient. Pre-submit checks can correctly evaluate the SUT while missing effects that appear only after the SUT calls a shared downstream dependency. The failure may depend on request timing, downstream batching, model contracts, or dependency state outside the SUT’s local counters. The platform therefore needs guardrails that observe downstream dependencies directly and connect their signals back to the originating experiment. Together, these requirements define where each guardrail applies. Isolation removes a shared edge where a representative tier is practical. Monitoring covers dependencies that remain shared. Pinning limits fan-out for failures whose impact depends on the number of workers reached. Section 4 describes these choices and their remaining limits.

4

Downstream Guardrails

This section presents three guardrails for containing downstream effects of production-connected experiments: isolated tiers, dependency monitors, and traffic pinning. Figure 2 places these mechanisms on the representative serving graph and shows where each acts on the downstream path.

What the pattern requires

The scenarios above share three properties, each of which maps to a guardrail in Section 4. Detection must be paired with attribution. Health signals can show that a downstream dependency is failing, but they do not automatically identify which experiment caused the failure. When routing, error, or restart records do not carry experiment identity, separating experiment-originated requests from other requests becomes a diagnostic step rather than a query. A guardrail that preserves caller identity is therefore the precondition for stopping one faulty experiment without interrupting unrelated validation. Containment must account for fan-out. Experimental traffic may be small relative to overall dependency volume, but crash-class failures are not necessarily proportional to traffic share. If an experiment fans out across many workers, one malformed request per worker can make the affected set large even when the experiment’s request volume is modest. For these failures, the relevant quantity is the number of

4.1

Selective isolation

The strongest guardrail is to route replayed downstream calls away from the shared dependency entirely. We use isolated tiers for dependencies that can be replicated with enough fidelity for replay comparisons to remain meaningful. When such a tier exists, experiment traffic reaches the isolated tier rather than the shared one, so failures in that dependency class are contained within the experimental path. Isolated tiers are provisioned selectively because replicating the full serving graph is not practical. Three criteria determine where they are useful. First, the dependency’s serving state and configuration must be reproducible enough that the same replayed request produces comparable behavior; a stale or divergent replica would make the controltreatment comparison inconclusive. Second, the tier must be maintainable as the corresponding shared tier changes, 4

(a) Query SUT

(b) Ranking SUT

(c) Inferencegateway SUT

Replay workload (Query service)

Replay workload (Ranking service)

Replay workload (Inference Gateway)

Query and aggregate (SUT)

Ranking (Isolated Tier)

Feature service

Ranking (SUT)

Inference Gateway (Isolated Tier)

Inference Gateway (SUT)

Content and feature store Prediction service

Accelerator fleet Pinned worker subset

Dependency monitors

Figure 2: Guardrail placement as the SUT moves downstream. (a) A query experiment routes calls through isolated ranking and inference-gateway tiers. (b) A ranking experiment uses an isolated inference-gateway tier. (c) An inference-gateway experiment reaches the shared prediction service directly. Dependency monitors observe the prediction service in all three examples, and traffic pinning limits experimental calls to the hatched worker subset. Solid arrows show requests; dotted arrows show monitoring. including software, configuration, scaling, and health checks. Third, the required capacity must be practical to reserve. This is why isolated tiers are more suitable for ranking than for accelerator-backed prediction models or stateful feature stores. Selected ranking and inference-gateway deployments satisfy these criteria and can be maintained as isolated tiers. Replayed calls that would otherwise reach their shared counterparts are routed to the corresponding isolated tier. For the ranking service measured in Section 7, these isolated tiers use, in aggregate, less than one percent of the worker capacity of the corresponding shared ranking tiers. Isolation removes that downstream exposure rather than merely bounding it. The tradeoff is resource demand: it is preferred where a representative tier can be kept current, but it is not a general solution for every dependency in the serving graph.

4.2

for each detector 𝑑 configured for the workload: 𝑑.prime(); 𝑐𝑑 ← 0 while the experiment is running: sleep until the next detector is due for each due detector 𝑑, with 𝑡 the elapsed time: if 𝑡 < warmup𝑑 : continue if 𝑡 > warmup𝑑 + maxDur 𝑑 : retire 𝑑 𝑟 ← 𝑑.check() 𝑐𝑑 ← 𝑟 .triggered ? 𝑐𝑑 + 1 : 0 if 𝑐𝑑 ≥ consecutive𝑑 : cancel; return

Figure 3: Monitor loop. Each detector has its own warmup, poll interval, consecutive-violation threshold, and monitoring duration. Clear checks reset the violation count, so only sustained violations terminate an experiment.

detectors, and terminates the experiment when a detector reports a sustained violation. Figure 3 shows the loop. Detectors are skipped during warm-up, while deployment and replay are still settling, and retired after their monitoring window so long-running experiments do not keep unnecessary checks active. Requiring

Continuous detection: dependency monitors

When a downstream dependency remains shared, the platform monitors it during the experiment. A dependency monitor runs alongside the experiment, polls a configured set of 5

consecutive violations avoids terminating experiments on single transient excursions. The platform uses two detector classes. Dependency-health detectors observe whether a shared downstream dependency remains healthy enough for the experiment to continue. They read the dependency’s own counters and compare them against a baseline captured at the start of the monitoring window, as well as against absolute limits. For example, one detector tracks unexpected worker restarts in the prediction service; another tracks downstream exception and invalidresult counts. These detectors prevent an experiment from continuing when the downstream environment is unstable, but they do not by themselves prove that the experiment caused the instability. Experiment-attributed detectors provide that narrower signal when the downstream records carry enough caller identity. The malformed-request detector resolves the control and treatment jobs for the running experiment, then queries the prediction service’s error records for requests originating from those jobs. A violation is therefore evidence about the monitored experiment rather than about the shared dependency as a whole. This allows the platform to terminate one faulty experiment without interrupting unrelated validation. Two limits follow from this design. First, polling introduces detection latency: a condition must occur after warmup, be observed on a poll, and persist for the configured number of consecutive checks. Monitors therefore bound how long a condition continues; they do not prevent instantaneous failures, such as an invalid request that leaves an accelerator-backed worker unusable. Preventing that class belongs at the dependency boundary, through input validation before a request is accepted. Second, attribution is only as precise as the identity available in downstream records. If a downstream service records errors at batch granularity, a malformed request can implicate the jobs that contributed to the batch rather than only the single originating request. Section 7 accounts for this when reporting monitor terminations, and Section 9 states the general limit: attribution is strongest at the immediate caller and weakens with depth.

4.3

across every shard serving the model. A malformed request that crashes a worker or corrupts device state is therefore confined to the designated shard. Other shards remain available for serving traffic. Pinning trades breadth of exposure for concentrated load. It is useful for crash-class failures, where impact depends on how many workers receive a malformed request. It is less suitable for resource contention because concentrating replay traffic can make the designated shard a straggler. Dependency-health monitoring therefore remains necessary alongside pinning.

4.4

Applying the guardrails

The mechanisms are selected from properties of each downstream dependency. When a representative copy of the dependency can be maintained with practical resource demand, the workload routes replay to an isolated tier. When a dependency must remain shared, the workload attaches health monitors wherever suitable downstream signals are available. If the dependency is partitioned and its main failure mode is local to a worker or shard, traffic pinning adds a bound on fan-out. These choices address different failure classes. Isolation removes experimental traffic from a shared request path. Monitoring bounds how long a visible violation can continue. Pinning bounds the workers reached before detection. Lifecycle effects and writes to shared state require separate controls, such as suppressing the operation, tagging experimental traffic, or limiting corpus age. No one guardrail covers every effect described in Section 5.5. Coverage is the combination of these choices rather than a property of one mechanism. The guardrails also do not make experiments independent. Concurrent experiments can share isolated tiers, pinned workers, and dependency-health signals. Section 9 returns to the resulting interference and attribution limits.

5

Experimentation Platform

This section describes the platform’s experiment topologies, execution model, change inputs, lifecycle triggers, production connectivity, and replay behavior.

Bounded exposure: traffic pinning 5.1

Prediction is difficult to isolate because the service hosts many model deployments on accelerator-backed workers. Maintaining a representative isolated tier for every model and variant would require reserved accelerator capacity. Monitoring still applies, but it can react only after a fault becomes visible. Traffic pinning limits what an experiment can reach before detection. Under normal routing, calls from an experiment are sent to one designated shard instead of being distributed

Experiment topologies

The platform supports two complementary topologies for collecting and evaluating experiment evidence. They differ in how checks are defined and when they run; neither changes which downstream dependencies the service under test (SUT) can reach. Figure 4(a) shows the counter-aggregation topology. Control and treatment versions of the SUT run as paired sides. A host-local collector reads service-specific counters from 6

endpoints exposed by each side and resource-usage counters from each host. The samples are written to a time-series store, aggregated across trials, and converted into a regression report. This topology supports broad comparisons of resource use, latency, reliability, and service-specific counters. Figure 4(b) shows the harness-driven topology. A programmable harness can evaluate either one SUT instance or paired control and treatment instances. Single-instance tests include fuzzing and other checks that determine whether an input violates a required property. Paired tests can compare service-specific outputs directly, such as ranking order or normalized discounted cumulative gain (NDCG). The harness can evaluate conditions during execution, perform final checks when the test ends, and terminate the test after a sustained violation. The two topologies can also be combined. The dependency monitors in Section 4.2 read counters from shared downstream dependencies and use the programmable harness to evaluate them while an experiment runs. Counter collection supplies the health signals; the harness supplies the polling, threshold, and termination logic. In either topology, calls leaving the SUT may still reach shared production dependencies, so the guardrails in Section 4 apply independently of how evidence is evaluated.

5.2

Trigger

Diff-time policy gate Land-time detection Periodic calibration Release promotion Other tooling

one change 62% range 10% n/a 9% release 18% ad hoc 1%

62% 12% 13% 12% <1%

Table 2: Lifecycle triggers for one ranking service over a representative measurement window.

dependencies drawn from that snapshot. Source control is provided by Sapling [18].

5.4

Lifecycle triggers

The platform is invoked by several lifecycle workflows. These workflows differ in which revision they test, how precisely a regression can be attributed, how much latency a developer or release process can tolerate, and how much shared downstream exposure they create. Figure 5 shows how source, configuration, periodic, and release triggers feed the same experiment executor. When a later-stage run detects a regression, bisection can select intermediate revisions from the candidate range and submit additional runs. In Figure 5, curved arrows represent proposed source or configuration changes joining their mainlines. Dash-dotted paths create a revision-scoped code snapshot, and dashed paths submit that snapshot for experiment execution. Colored markers distinguish periodic, release, and manually initiated bisection runs; 𝐶 and 𝑅 labels denote successive changes and releases. A bisection range query identifies the revisions between a known-good and a known-bad result; the manual trigger then selects intermediate revisions from that range for additional experiments. The vertical direction follows the progression from code to an executable container. Diff-time validation runs before submission, when a regression usually maps to one proposed change. Land-time validation re-tests the mainline revision, which may contain a range of changes. Release validation tests the artifact prepared for promotion, providing the strongest release fidelity but the weakest attribution. Periodic A/A experiments run independently of a source or configuration change. They remain in Table 2 to account for platform volume. Execution share counts experiment runs, while trial-time share sums their trial duration. The similar diff-time shares indicate near-average duration. Land-time and periodic executions run longer on average, while release executions run for less time. Counts describe trigger volume; trial time approximates platform demand.

Experiment execution

Each workload runs as several separately allocated paired trials. Every trial has a warm-up period followed by a measurement window. Control and treatment are matched by logical host type, processor family, kernel version, operating system, hardware fingerprint, and region. Trials that fail for infrastructure reasons may be replaced up to a configured limit, while attempted, replaced, and excluded trials remain recorded for diagnosis.

5.3

Blocking role Attribution Exec. share Trial-time

Change delivery

Two versioned stores matter. The source monorepo contains service code, libraries, build definitions, and shared infrastructure. A separate repository contains runtime configuration, including service tunables and traffic controls. This creates two classes of change with different dependency signals. A source change can alter executable behavior, dependencies, or resource use, and its declared reach is visible in the build graph. A configuration change can alter runtime behavior without changing a build target, so build-graph reachability does not apply. Section 6 describes how trigger selection handles this difference. Builds are revision-scoped. A package is produced from a target graph evaluated at one source revision, with first-party 7

Replay Workload Service Under Test (SUT)

Build

Side-A counters

Deploy

Side-B counters

Service Under Test (SUT)

host-a collector

host-b

TimeSeries DB

Aggr

Build

Report

Deploy

Side-A Side-B At the end

collector

Harness

(a) Counter aggregation

Harness

During replay

(b) Harness-driven checks

Figure 4: Experiment topologies. (a) Collectors aggregate counters from paired SUT instances and their hosts. (b) A programmable harness checks one SUT instance or compares paired instances; the paired form is shown. Manual Bisect Trigger

Periodic Trigger

Release Trigger

Code to Container

C1

Code Snapshot C3

C2

C5

Experiment Trigger

Bisect range query

• Crash or corruption. A candidate change can emit input that a shared dependency mishandles, or replay can exercise a latent dependency defect. One malformed request can leave a worker unavailable, so impact need not be proportional to traffic share. • Lifecycle side effects. Some downstream interactions affect resource accounting even when they do not write user-visible state. For example, replaying captured traffic for a stale model refreshed its usage reference count and delayed reaping. Such calls may not alter a user response, but they can change resource lifecycle decisions. Workloads must suppress, redirect, tag, or account for these interactions. • Observability. Downstream services may count replayed requests in load, autoscaling, dashboard, or alerting signals unless caller identity distinguishes experiment traffic. Concurrent experiments can therefore also perturb a dependency-health baseline. Experiment-local monitoring observes the control and treatment deployments. It does not, by itself, capture these downstream effects, which is why the platform needs the guardrails in Section 4.

Diff / Pull Request

C6

src mainline

C4

config mainline periodic

R1

R2

R3

release Manual bisection

Experiment Execution

Figure 5: Experiment triggers across the change lifecycle, including bisection of a candidate revision range.

5.5

What reaches production

Two boundaries have to be distinguished. The first is the boundary around the SUT itself. On the ingress side, the SUT serves replayed requests rather than user requests, and no product surface depends on the availability of the experimental deployment. A crash or local performance regression in the SUT therefore affects the experiment, not the user-facing request path. The second boundary is the SUT’s egress. A downstream call leaving the SUT can be handled in three ways: redirected to an isolated tier reserved for experimental traffic; confined to a bounded subset of workers behind the dependency; or sent to the shared downstream dependency without redirection. Section 4 describes the first two mechanisms and when each applies. The third case is what makes productionconnected experimentation different from fully isolated preproduction testing. When downstream calls remain connected, an experiment can affect shared dependencies in several ways. • Capacity. Replayed requests consume downstream capacity. Because ranking requests fan out, downstream demand can exceed the experiment’s request rate and concurrent experiments compound it.

5.6

Replay fidelity and its limits

Three properties of the replay path shape what the measurements mean: sampled recording, paired replay, and bounded recency. For each tier, the platform records traffic from a small number of serving tasks. Those tasks sample incoming requests uniformly. In the workloads we study, sampling rates range from 10−5 to 6.74 × 10−5 , corresponding to approximately one in 100,000 to one in 14,800 eligible requests. The recorded corpus is therefore a sample of recent tier traffic, not a complete trace. During an experiment, both control and treatment receive the same recorded requests. For each request, the load generator sends the request to both sides, randomizing which side 8

starts first. Any unusually heavy or light request therefore appears in both arms, reducing request-mix noise. The corpus is drawn from a recent lookback window, typically one to four hours. This bounds how stale replayed accesses can be, reducing lifecycle side effects such as those described in Section 5.5. Replay remains an approximation: it preserves sampled request contents, not full serving concurrency or causal user sessions. Replay rates are set to exercise the SUT at a high but controlled utilization. This leaves headroom against overload while making resource regressions visible in latency. The measured latency change is useful for deciding whether the candidate regresses the service, but it is not a claim that the same latency change would appear under every serving load. Pairing does not make the two sides independent. Both can call the same downstream dependency, so a treatment-side effect can also influence measurements observed by control. Dependency monitors limit harmful interference, but they do not remove this statistical coupling.

6

The platform also supports priority sampling across workloads. This mechanism is different from reachability filtering. Reachability attempts to exclude only workloads that the change cannot affect; sampling deliberately runs only a fraction of otherwise relevant diff-time workloads for lowerpriority tiers, with later lifecycle stages providing additional coverage. This trades earlier coverage for lower resource demand and less exposure to shared downstream dependencies. In the measured rollout, source dependency filtering reduced diff-time workload executions by 25% without changing the artifacts evaluated by the remaining experiments.

7

Evaluation

We first evaluate the guardrails in aggregate and then report measurements for each mechanism. We then evaluate dependency-aware triggering. Each result states its measurement scope and distinguishes observed outcomes from configuration-level guarantees.

Dependency-Aware Triggering

7.1

The platform validates two broad classes of changes: sourcecode changes that affect built artifacts, and configuration changes that affect runtime behavior without necessarily changing the binary. Every experiment consumes capacity and can add load to shared downstream dependencies. Trigger selection is therefore a containment mechanism, not only an efficiency optimization: a workload should run for a change only when the change can affect the artifact or runtime behavior that workload validates. For source-code changes, the platform uses dependency reachability. A workload is eligible only when the changed files can reach the package target used by that workload through the declared build graph. This avoids running experiments for services or tiers outside the change’s dependency closure. Configuration changes require a different signal. Prefixpath filters provide a coarse first pass, but a prefix match can include stale, unused, or unrelated configuration. It also cannot tell which tiers actually consume a setting at runtime. The platform therefore maintains a runtime consumption index keyed by service and configuration name. A configuration change triggers a workload when that service has recently read the changed configuration at runtime. This ties experiment selection to observed use, rather than to repository layout alone. Because the consumption index operates at configurationfile granularity, we also evaluate a setting-level refinement that uses recent per-setting reads and keeps the file-level decision when evidence is missing.

Aggregate guardrail outcomes

Figure 6 aggregates the failure observations used in this study by position relative to the combined guardrail rollout. We use a derived resource-burden measure associated with each observation. The figure compares the combined burden from malformed-input and latent downstream-defect observations targeted by the guardrails. Absolute values and per-class shares are not reported, and records without a reported value are excluded. The guardrails were introduced incrementally rather than enabled as one atomic change. We therefore divide the measurement window into three periods. Pre-rollout precedes the mechanisms evaluated here, rollout contains experiments with partial guardrail coverage, and follow-up begins after all three mechanisms were active on the covered paths. This avoids assigning experiments run under partial coverage to either endpoint. The replay volume is nearly balanced across the endpoint periods: 46% of the measured experiments ran before rollout, 8% during rollout, and 46% during follow-up. Faults continued to be observed during the 81-day follow-up. The guardrails do not stop a candidate from producing faulty behavior or prevent a latent fault in a production dependency from being exercised. Instead, isolation redirected the effects, pinning bounded the workers they could reach, and monitors stopped experiments after violations became visible. The derived resource-burden measure recorded no burden during follow-up. This is an observed containment outcome, not a causal estimate that the guardrails reduced the underlying fault rate. 9

0.5

None recorded

0.0 Pre-rollout

Fatal signal

10000

Baseline

Worker exits/day

Relative resource burden (pre-rollout = 1)

(a) Recorded resource burden 1.0

Follow-up

100

10

1

Replay experiments (%)

3

guardrails active

4

5

6

7

8

9

Ten-day interval >200,000 abnormal exits contained

46%

3% OOM

96% fatal signal 1% other

Figure 7: Abnormal worker exits in isolated ranking tiers over 90 days. The upper panel reports exits per day across nine ten-day intervals on a logarithmic scale; the lower bar summarizes the total as 96% fatal-signal exits, 3% OOM kills, and 1% other exits. Planned updates and preemptions are excluded.

25

8%

Faults observed

Rollout

Follow-up

0 Pre-rollout

Figure 6: Aggregate outcomes across the incremental guardrail rollout. (a) Resource burden from targeted failure observations is normalized to the pre-rollout reference. Faults were still observed after rollout, but no burden was recorded during follow-up. (b) Prerollout and follow-up each contain 46% of measured replay experiments, while the rollout period contains the 8% run under partial coverage.

7.2

2

(b) Replay experiments by rollout phase 46%

Other

1000

1

50

OOM kill

Detector Dependency health Experiment-attributed

𝑛

Median

p90

Maximum

534 14

62 s 121 s

64 s 121 s

69 s 364 s

Table 3: Interval from a detector’s first observed violation to experiment termination. a shared prediction service by matching downstream errors to the originating experiment. Over a separate three-month observation window, dependency monitors cancelled approximately 5% of the service’s experiments and accounted for about 87% of all experiment cancellations. Of these monitor-initiated cancellations, approximately 81% came from the dependency-health detector for prediction-worker crashes and 19% from the experimentattributed malformed-request detector. For response time, Table 3 uses a separate 16-day window and reports the interval between a detector first observing a violation and the experiment being terminated. For both detector classes, the median response is approximately one configured polling interval after the first observation. The interval begins when the detector first observes the violation, not at fault onset. Dependency-health terminations are also not independent: one disturbance can terminate several experiments, so their count measures exposed experiments rather than distinct faults.

Per-guardrail effectiveness

Selective isolation. Over a representative multi-day window, isolated ranking tiers handled billions of replayed requests per day. Without this redirection, those requests would have reached the corresponding shared ranking tiers. The isolated tiers require less than one percent of their provisioned worker capacity. Request volume is estimated from sampled routing records; the capacity comparison uses worker counts. The result applies to the covered ranking paths. To measure the failures contained by this isolation, we counted abnormal worker exits in the isolated ranking tiers over 90 days. We excluded planned updates and infrastructure preemptions and deduplicated reports by task instance. As Figure 7 shows, the tiers contained more than 200,000 abnormal worker exits: 96% were fatal-signal exits, 3% were OOM kills, and 1% were other exits. Routing replay traffic to these tiers prevented these failures from directly affecting production-serving workers. Many exits were repeated instances of the same crash loops, so the count measures contained worker failures rather than distinct defects.

Traffic pinning. Pinning directs experiment requests to one designated shard rather than every shard serving the model. This bounds task-local failures to that shard, but not effects propagated through shared state. Because telemetry cannot

Dependency monitoring. Experiment-attributed detectors terminated experiments that emitted malformed requests to 10

Isolated crash

61%

Crash-like exits

0

25

50

11% 75

100

Share within each measurement unit (%)

Figure 8: Crash concentration on pinned prediction tasks over 60 days. Crashes on the same task separated by at most ten minutes form one episode. Repeated bursts account for 39% of crash-like exits but 11% of failure episodes.

Record-and-replay validation. Record and replay improves input fidelity but does not determine how dependencies are handled. Liu et al. identify mocking points so replayed microservice requests execute against isolated dependencies [28]; AREX, Keploy, and GoReplay make different choices about capture scope and dependency stubbing [2, 7, 22]. Diffy sends the same requests to candidate and known-good instances and compares responses [35], but response comparison does not itself isolate their downstream calls. Preproduction environments provide controlled connectivity from candidate deployments to live services [8]. ServiceLab is a published third-party system that provides paired replay, matched execution environments, counter comparison, and baseline-based regression detection [9]. It also permits selected calls from the SUT to reach dependencies that handle production traffic, while leaving sideeffect prevention to service owners. Our work addresses the complementary platform problem: systematic containment, monitoring, and attribution when experiments are not fully isolated.

establish how many workers would otherwise have failed, we report the routing bound rather than avoided impact. We also measured how crashes recur on pinned prediction tasks. We identified the designated tasks from a representative one-day routing snapshot and examined their task-exit telemetry over the preceding 60 days. We excluded planned updates and infrastructure preemptions, deduplicated reports by task instance, and treated segmentation faults and OOM kills as crash-like exits. Consecutive crashes on the same task, separated by at most ten minutes, form one failure episode. As Figure 8 shows, 39% of crash-like exits occurred in repeated bursts. After each burst is counted once, repeated bursts account for 11% of failure episodes. This grouping prevents a crash loop from being counted as a sequence of independent containment events. The measurement characterizes recurrence on pinned targets; it does not estimate how many production workers would have failed without pinning.

7.3

Related Work

Canary deployment. Canary and progressive delivery systems expose a change to production traffic only after earlier validation. Canary Analysis Service, Kayenta, Gandalf, and Conveyor compare candidate and baseline behavior during limited rollout and control promotion [10, 19, 21, 27]. Open systems package the same pattern as automated progressive delivery [3, 14, 23], while learned risk models increasingly inform promotion decisions [1]. These systems limit exposure during rollout. Our setting instead reaches shared dependencies while a candidate is still under review, before rollout can provide feedback.

39% 89%

Failure episodes

8

Repeated crash burst

Dependency-aware triggering

Source-code selection uses build-graph reachability; configuration selection uses observed runtime consumption. In the measured rollout, source dependency filtering reduced diff-time workload executions by 25%. The artifact, replay corpus, and trial configuration of the remaining experiments were unchanged. The reduction comes from avoiding workloads outside the expected dependency closure, not from shorter runs. An offline replay of 775 changed settings found that this refinement would suppress 72.3% of the changes selected by file-level consumption. In a held-out final week, 1.7% of suppressed settings were later observed as read by the SUT. These results evaluate the selection rule offline; the refinement is not enabled in the measurements reported here. The first result counts workload executions; the second counts changed settings, not executions or compute saved. Section 9 discusses the coverage limits of both selectors.

Production-scale testing and benchmarks. Kraken drives live traffic to find resource bottlenecks, WSMeter evaluates warehouse-scale hardware with production workloads, and Treadmill provides precise open-loop load generation and tail-latency attribution [26, 36, 39]. DeathStarBench and TeaStore provide reproducible microservice topologies [15, 38]. These systems evaluate an existing deployment or a controlled benchmark. Our concern is a candidate change whose replayed calls converge with production traffic at shared dependencies. Regression-test selection. Test selection uses observed dependencies, commit pruning, or learned prediction [17, 30, 31]. Our source-change selector uses build-graph reachability, which is sound only relative to the declared dependency closure; runtime loading, generated code, and undeclared inputs require conservative fallback. Configuration state is absent from that graph, so we instead use observed runtime 11

consumption. In our setting, selection reduces both experiment capacity and calls to shared dependencies, making it a reliability mechanism as well as a scheduling optimization.

effects propagated through shared state. Isolated tiers provide a stronger boundary where faithful replication is practical; monitors act only after a violation becomes visible.

Containment and chaos engineering. Chaos engineering perturbs production within a controlled blast radius and aborts when guardrail signals degrade [4]. ChAP is the closest published analogue to our guardrail layer: it routes a fraction of production traffic to instrumented control and experiment clusters and terminates a run on metric deviation [5]. A chaos experiment specifies its perturbation, even when the resulting behavior is uncertain. Candidate code has a broader failure space, so our guardrails instead combine downstream health, experiment-attributed errors, selective isolation, and bounded fan-out.

Concurrent experiments. Experiments sharing an isolated tier, pinned shard, or health signal remain coupled through contention and common disturbances. Detectors distinguish them only when downstream records preserve caller identity. Replay and selection coverage. Replay preserves sampled requests, not the full distribution of concurrency or causal sessions, and selected recording tasks can omit traffic patterns (Section 5.6). Build-graph filtering covers declared dependencies; runtime filtering can miss settings not read during observation. Priority sampling also defers eligible workloads. Section 7.3 therefore measures executions avoided, not regressions missed.

Failure attribution and tracing. Distributed tracing systems propagate causal context across service boundaries, and dynamic tracing systems evaluate cross-service queries [20, 29, 33]. Our monitors separate health detection from attribution. A health detector can observe a dependency at depth, while error attribution requires caller identity to survive the request path and is strongest at the immediate dependency. This distinction determines which failures can be assigned automatically to one experiment.

Generality. Measurements focus primarily on one family of recommendation services. The design assumes placement constraints, service introspection, and downstream caller identity; other systems require equivalent scheduling, monitoring, and attribution.

10

Candidate changes receive realistic traffic and dependency behavior before they enter the mainline, but productionconnected execution also extends their effects beyond the SUT. Some dependencies can be isolated, some can only be monitored, and some require a bound on the workers that replayed traffic can reach. The experimentation platform studied here combines isolated tiers, dependency monitoring, and traffic pinning for these cases. Dependency-aware triggering supports the same operating model by reducing experiment volume and downstream exposure. The broader lesson is that early validation against shared serving dependencies requires downstream effects to remain observable, attributable, and contained.

Experiment analysis. Online controlled experiments address related statistical questions at a different operating point: randomized users observed over long periods, rather than a small number of paired host trials [24, 25]. Our replay uses common random numbers [16]: both sides receive the same corpus, removing request-mix differences by construction rather than estimating them afterward, as in covariatebased variance reduction [12]. Evaluating many counters raises the multiple-comparison problem [6], while sensitivity to tail latency motivates precise paired measurements [11]. Matched placement. Allocating control and treatment together is related to gang scheduling, where several tasks are useful only when placed as a unit [13, 32]. Cluster managers expose placement constraints and multi-job allocation mechanisms [34, 37]. The matched environments in our platform use this scheduling capability as substrate; scheduling itself is not a contribution of this paper.

9

Conclusion

References [1] Rui Abreu, Vijayaraghavan Murali, Peter C. Rigby, Chandra Maddila, Weiyan Sun, Jun Ge, Kaavya Chinniah, Audris Mockus, Megh Mehta, and Nachiappan Nagappan. 2025. Moving Faster and Reducing Risk: Using LLMs in Release Deployment. In Proceedings of the 47th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice. 448–457. doi:10.1109/ICSE-SEIP66354.2025.00045 [2] AREX Project. [n. d.]. AREX: Automated Record-and-Replay Regression Testing. https://github.com/arextest. Accessed 2026-08-02. [3] Argo Project. [n. d.]. Argo Rollouts: Progressive Delivery for Kubernetes. https://github.com/argoproj/argo-rollouts. Accessed 2026-0802. [4] Ali Basiri, Niosha Behnam, Ruud de Rooij, Lorin Hochstein, Luke Kosewski, Justin Reynolds, and Casey Rosenthal. 2016. Chaos Engineering. IEEE Software 33, 3 (2016), 35–41. doi:10.1109/MS.2016.60

Limitations

Attribution and containment boundaries. Health monitors can observe configured dependencies, but attribution requires preserved caller identity or propagated experiment context [20]. Traffic pinning confines task-local failures, not 12

[5] Ali Basiri, Lorin Hochstein, Nora Jones, and Haley Tucker. 2019. Automating Chaos Experiments in Production. In 41st International Conference on Software Engineering: Software Engineering in Practice (ICSESEIP). IEEE/ACM, 31–40. doi:10.1109/ICSE-SEIP.2019.00012 [6] Yoav Benjamini and Yosef Hochberg. 1995. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society: Series B (Methodological) 57, 1 (1995), 289–300. doi:10.1111/j.2517-6161.1995.tb02031.x [7] Leonid Bugaev and GoReplay Contributors. [n. d.]. GoReplay: Traffic Recording and Replay. https://github.com/buger/goreplay. Accessed 2026-08-02. [8] Jeremy J. Carroll, Pankaj Anand, and David Guo. 2021. Preproduction Deploys: Cloud-Native Integration Testing. arXiv:2110.08588. doi:10. 48550/arXiv.2110.08588 [9] Mike Chow, Yang Wang, William Wang, Ayichew Hailu, Rohan Bopardikar, Bin Zhang, Jialiang Qu, David Meisner, Santosh Sonawane, Yunqi Zhang, Rodrigo Paim, Mack Ward, Ivor Huang, Matt McNally, Daniel Hodges, Zoltan Farkas, Caner Gocmen, Elvis Huang, and Chunqiang Tang. 2024. ServiceLab: Preventing Tiny Performance Regressions at Hyperscale through Pre-Production Testing. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, 545–562. https://www.usenix.org/conference/ osdi24/presentation/chow Best Paper Award. [10] Štěpán Davidovič and Betsy Beyer. 2018. Canary Analysis Service. Commun. ACM 61, 5 (2018), 54–62. doi:10.1145/3190566 [11] Jeffrey Dean and Luiz André Barroso. 2013. The Tail at Scale. Commun. ACM 56, 2 (2013), 74–80. doi:10.1145/2408776.2408794 [12] Alex Deng, Ya Xu, Ron Kohavi, and Toby Walker. 2013. Improving the Sensitivity of Online Controlled Experiments by Utilizing PreExperiment Data. In 6th ACM International Conference on Web Search and Data Mining (WSDM). ACM, 123–132. doi:10.1145/2433396.2433413 [13] Dror G. Feitelson and Larry Rudolph. 1992. Gang Scheduling Performance Benefits for Fine-Grain Synchronization. J. Parallel and Distrib. Comput. 16, 4 (1992), 306–318. doi:10.1016/0743-7315(92)90014-E [14] FluxCD Project. [n. d.]. Flagger: Progressive Delivery for Kubernetes. https://github.com/fluxcd/flagger. Accessed 2026-08-02. [15] Yu Gan et al. 2019. An Open-Source Benchmark Suite for Microservices and Their Hardware-Software Implications for Cloud and Edge Systems. In Proceedings of the 24th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). 3–18. doi:10.1145/3297858.3304013 [16] Paul Glasserman and David D. Yao. 1992. Some Guidelines and Guarantees for Common Random Numbers. Management Science 38, 6 (1992), 884–908. doi:10.1287/mnsc.38.6.884 [17] Milos Gligoric, Lamyaa Eloussi, and Darko Marinov. 2015. Practical Regression Test Selection with Dynamic File Dependencies. In 2015 International Symposium on Software Testing and Analysis (ISSTA). ACM, 211–222. doi:10.1145/2771783.2771784 [18] Durham Goode. 2022. Sapling: Source Control That’s User-Friendly and Scalable. Engineering at Meta. https://engineering.fb.com/2022/ 11/15/open-source/sapling-source-control-scalable/ Accessed 202608-03. [19] Boris Grubic, Yang Wang, Tyler Petrochko, Ran Yaniv, Brad Jones, David Callies, Matt Clarke-Lauer, Dan Kelley, Soteris Demetriou, Kenny Yu, and Chunqiang Tang. 2023. Conveyor: One-Tool-Fits-All Continuous Software Deployment at Meta. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23). USENIX Association, 325–342. https://www.usenix.org/conference/osdi23/ presentation/grubic [20] Jonathan Kaldor, Jonathan Mace, Michał Bejda, Edison Gao, Wiktor Kuropatwa, Joe O’Neill, Kian Win Ong, Bill Schaller, Pingjia Shan, Brendan Viscomi, Vinod Venkataraman, Kaushik Veeraraghavan, and

Yee Jiun Song. 2017. Canopy: An End-to-End Performance Tracing and Analysis System. In 26th Symposium on Operating Systems Principles (SOSP). ACM, 34–50. doi:10.1145/3132747.3132749 [21] Nikhil Kaul and Andrew Phillips. 2018. Introducing Kayenta: An Open Automated Canary Analysis Tool from Google and Netflix. Google Cloud Blog. https://cloud.google.com/blog/products/gcp/introducingkayenta-an-open-automated-canary-analysis-tool-from-googleand-netflix Accessed 2026-08-02. [22] Keploy Project. [n. d.]. Keploy: Record-and-Replay Integration Testing. https://github.com/keploy/keploy. Accessed 2026-08-02. [23] Keptn Project. [n. d.]. Quality Gates. https://v1.keptn.sh/docs/0.19.x/ define/quality-gates/. Accessed 2026-08-02; Keptn V1 is end-of-life. [24] Ron Kohavi, Alex Deng, Brian Frasca, Toby Walker, Ya Xu, and Nils Pohlmann. 2013. Online Controlled Experiments at Large Scale. In 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). ACM, 1168–1176. doi:10.1145/2487575.2488217 [25] Ron Kohavi, Diane Tang, and Ya Xu. 2020. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press. doi:10.1017/9781108653985 [26] Jaewon Lee, Changkyu Kim, Kun Lin, Liqun Cheng, Rama Govindaraju, and Jangwoo Kim. 2018. WSMeter: A Performance Evaluation Methodology for Google’s Production Warehouse-Scale Computers. In 23rd ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). 549–563. doi:10.1145/3173162.3173196 [27] Ze Li, Qian Cheng, Ken Hsieh, Yingnong Dang, Peng Huang, Pankaj Singh, Xinsheng Yang, Qingwei Lin, Youjiang Wu, Sebastien Levy, and Murali Chintalapati. 2020. Gandalf: An Intelligent, End-ToEnd Analytics Service for Safe Deployment in Large-Scale Cloud Infrastructure. In 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20). USENIX Association, 389–402. https://www.usenix.org/conference/nsdi20/presentation/li [28] Jiangchao Liu, Jierui Liu, Peng Di, Alex X. Liu, and Zexin Zhong. 2022. Record and Replay of Online Traffic for Microservices with Automatic Mocking Point Identification. In Proceedings of the 44th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP ’22). Association for Computing Machinery, 221– 230. doi:10.1145/3510457.3513029 [29] Jonathan Mace, Ryan Roelke, and Rodrigo Fonseca. 2015. Pivot Tracing: Dynamic Causal Monitoring for Distributed Systems. In 25th Symposium on Operating Systems Principles (SOSP). ACM, 378–393. doi:10.1145/2815400.2815415 [30] Mateusz Machalica, Alex Samylkin, Meredith Porth, and Satish Chandra. 2019. Predictive Test Selection. In 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE/ACM, 91–100. doi:10.1109/ICSE-SEIP.2019.00018 [31] Atif Memon, Zebao Gao, Bao Nguyen, Sanjeev Dhanda, Eric Nickell, Rob Siemborski, and John Micco. 2017. Taming Google-Scale Continuous Testing. In Proceedings of the 39th International Conference on Software Engineering: Software Engineering in Practice Track. 233–242. doi:10.1109/ICSE-SEIP.2017.16 [32] John K. Ousterhout. 1982. Scheduling Techniques for Concurrent Systems. In 3rd International Conference on Distributed Computing Systems (ICDCS). IEEE, 22–30. [33] Benjamin H. Sigelman, Luiz André Barroso, Mike Burrows, Pat Stephenson, Manoj Plakal, Donald Beaver, Saul Jaspan, and Chandan Shanbhag. 2010. Dapper, a Large-Scale Distributed Systems Tracing Infrastructure. Technical Report dapper-2010-1. Google, Inc. https://research.google/pubs/pub36356/ [34] Chunqiang Tang, Kenny Yu, Kaushik Veeraraghavan, Jonathan Kaldor, Scott Michelson, Thawan Kooburat, Aravind Anbudurai, Matthew Clark, Kabir Gogia, Long Cheng, Ben Christensen, Alex Gartrell, 13

[37] Abhishek Verma, Luis Pedrosa, Madhukar Korupolu, David Oppenheimer, Eric Tune, and John Wilkes. 2015. Large-Scale Cluster Management at Google with Borg. In 10th European Conference on Computer Systems (EuroSys). ACM. doi:10.1145/2741948.2741964 [38] Joakim von Kistowski, Simon Eismann, Norbert Schmitt, Andre Bauer, Johannes Grohmann, and Samuel Kounev. 2018. TeaStore: A MicroService Reference Application for Benchmarking, Modeling and Resource Management Research. In 2018 IEEE 26th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS). 223–236. doi:10.1109/MASCOTS.2018. 00030 [39] Yunqi Zhang, David Meisner, Jason Mars, and Lingjia Tang. 2016. Treadmill: Attributing the Source of Tail Latency through Precise Load Testing and Statistical Inference. In 43rd Annual IEEE/ACM International Symposium on Computer Architecture (ISCA). 456–468. doi:10.1109/ISCA.2016.47

Maxim Khutornenko, Sachin Kulkarni, Marcin Pawlowski, Tuomas Pelkonen, Andre Rodrigues, Rounak Tibrewal, Vaishnavi Venkatesan, and Peter Zhang. 2020. Twine: A Unified Cluster Management System for Shared Infrastructure. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20). USENIX Association, 787– 803. https://www.usenix.org/conference/osdi20/presentation/tang [35] Twitter Engineering. [n. d.]. Diffy: Testing Services by Comparing Candidate and Known-Good Instances. https://github.com/twitterarchive/diffy. Accessed 2026-08-02; archived project. [36] Kaushik Veeraraghavan, Justin Meza, David Chou, Wonho Kim, Sonia Margulis, Scott Michelson, Rajesh Nishtala, Daniel Obenshain, Dmitri Perelman, and Yee Jiun Song. 2016. Kraken: Leveraging Live Traffic Tests to Identify and Resolve Resource Utilization Bottlenecks in Large Scale Web Services. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16). USENIX Association, 635–651. https://www.usenix.org/conference/osdi16/technicalsessions/presentation/veeraraghavan

14

Related documents

• FIRE: Failure Isolation and Reliability for Production-Connected Experiments at Scale#1648992
• Randomization Tests in Switchback Experiments#1131473
• Serving a Revisable World: Versioned Execution for Interruptible Agents#1160977
• Serving a Revisable World: Versioned Execution for Interruptible Agents#1156095
• Bounded Retained History in a Proxy Flow Replay: A Mixed Predictive Pilot and an Allocation Study#1585850