arXiv:2607.20050v1 [cs.SE] 22 Jul 2026
SequenceFI: Non-intrusive Temporal Fault Injection for Microservice Systems Yuzhen Tan
Jian Wang
Bing Li
Shaolin Tan
School of Computer Science Wuhan University Wuhan, Hubei, China [email protected]
School of Computer Science Wuhan University Zhongguancun Laboratory Wuhan, Hubei, China [email protected]
School of Computer Science Wuhan University Zhongguancun Laboratory Wuhan, Hubei, China [email protected]
Zhongguancun Laboratory Beijing, China [email protected]
Abstract—Fault injection is widely used to evaluate the resilience of microservice systems, where client requests often span multiple services and execution stages. Existing request-level techniques usually control where and what faults are injected, but not when they are activated within a distributed execution. This limitation makes it difficult to reproduce timing-dependent failures, such as failures after state-changing side effects, ordersensitive concurrent responses, and partial failures among repeated downstream calls. This paper presents SequenceFI, a nonintrusive framework for temporal fault injection in microservice systems. SequenceFI observes message-level send and receive events, propagates compact temporal evidence along request executions, and triggers faults only when occurrence-sensitive temporal guards are satisfied. It further synthesizes temporal guards from traces, reducing the need for exhaustive enumeration of temporal fault-injection configurations, while requiring no modifications to application code or serialization libraries. We implement SequenceFI on Kubernetes and evaluate it on four widely used microservice benchmarks. Across nine temporalfault scenarios and 450 valid trials, SequenceFI achieves 100.0% temporal success without premature or multiple injections, finds effective configurations in one attempt on average, and reduces aggregate end-to-end search time by 95.91% compared with HRandom. Index Terms—Microservices, temporal fault injection, sidecar proxy
I. I NTRODUCTION Microservice architectures improve scalability, deployment flexibility, and development agility by decomposing applications into independently deployable services. However, a single user request often traverses multiple services through dynamic inter-service interactions, making resilience depend on the behavior of the entire distributed execution. Chaos engineering is widely used to assess such resilience by deliberately introducing adverse conditions and observing whether systems maintain or recover expected behavior [1]–[5]. Fault injection (FI) provides a key experimental mechanism for chaos engineering and resilience testing by perturbing requests, services, networks, or containers. Existing request-level FI techniques [2], [6] can specify where a fault is injected and what fault is applied, such as failing an API request or response. However, many microservice resilience failures also depend on when the fault occurs within a distributed execution. For example, dropping a payment
response after the payment has been charged creates a different recovery problem from failing the payment request before the charge is issued; a risk-check failure may expose a bug only if it arrives after another concurrent response has already caused the caller to commit success; and a product-page aggregation bug may appear only when a specific subset of homogeneous downstream responses fails. These examples capture three representative temporal constraints: execution phase, relative response order, and occurrence or cardinality among repeated calls. We formalize these constraints in Section III. Without temporal control, an FI experiment may trigger too early, affect the wrong occurrence, or collapse a partial failure into an all-or-nothing failure, thereby missing the intended resilience defect and creating false confidence in recovery behavior. Testing these scenarios requires temporal fault injection (TFI), where a fault is activated only when specified execution events or occurrence conditions have been satisfied. As discussed in Section II-B1, the need for temporal control is not limited to rare corner cases. Under a method-level approximation, our analysis of eight public API specification corpora identifies 19,701 of 38,631 operations (51.0%) as potentially state-changing. For these operations, a failure before a downstream side effect and the same failure after the side effect can impose fundamentally different recovery obligations, especially under retries, compensation, and partial-failure handling. More generally, distributed workflows involving concurrent or repeated downstream calls can exhibit order-sensitive and occurrence-sensitive behavior. These observations make temporal position an essential dimension of practical microservice fault injection. Prior work has explored TFI for microservices. 3MileBeach [7], the most directly related TFI system we are aware of, attaches temporal prerequisites to temporal fault-injection configurations (TFICs). However, it achieves temporal control via serialization-layer instrumentation and explores TFICs via randomized enumeration. As a result, practical cloud-native TFI still faces two challenges. (1) Deployment bottleneck of serialization-layer TFI. Serialization-layer TFI couples the injector to language-specific libraries and generated code; Section II-B2 reports 170 serialization libraries across nine language families. (2) TFIC-generation bottleneck. TFIC
generation must jointly search static targets and temporal guards; Section II-B3 shows a three-event guard space reaches 4.04M candidates. Although lineage-driven FI techniques such as LDFI and FastFI reduce search space by leveraging request lineages and static targets [8], [9], they do not synthesize temporal guards. This paper presents SequenceFI, a lightweight non-intrusive framework for practical TFI in microservice systems. SequenceFI places the interposition point on the communication path rather than inside application code or serialization libraries. Its TFIProxy sidecars observe message-level send/receive events, propagate compact temporal evidence along request executions, and inject faults only when both the static target and the required temporal context are satisfied. To avoid exhaustive TFIC enumeration, SequenceFI separates static target selection from temporal guard generation: it derives occurrence-aware targets from request lineages and generates compact After guards from traces that distinguish the intended execution context from earlier matching points. We implement SequenceFI as a Kubernetes-based prototype for HTTP and gRPC microservices and evaluate it on four benchmarks. Across nine temporal-fault scenarios, SequenceFI achieves 100.0% temporal success across all valid trials without premature or multiple injections. The post-effect experiments expose benchmark-level recovery weaknesses in state-changing workflows, where repeated client actions after an uncertain outcome can lead to duplicate business operations. For temporal guard generation, SequenceFI identifies an effective configuration in one attempt in all cases and reduces aggregate end-to-end search time by 95.91% compared with H-Random, a trace-guided randomized baseline. The main contributions of the paper are summarized as follows: • We propose SequenceFI, a sidecar-based TFI framework that observes message-level events, propagates compact temporal evidence, and triggers faults only when occurrence-aware static targets and execution-contextsensitive temporal guards are jointly satisfied. To the best of our knowledge, SequenceFI is the first non-intrusive TFI framework for microservice systems without modifying application code. • We design a novel trace-guided TFIC generation algorithm that separates static target selection from temporal guard synthesis and derives compact After predicates from distinguishing temporal evidence, thereby avoiding exhaustive or randomized enumeration of temporal configurations. • We evaluate SequenceFI on four microservice benchmarks, showing precise temporal triggering, efficient TFIC generation, and practical deployment with low runtime overhead. II. BACKGROUND AND M OTIVATION A. Background 1) Microservice Request Executions: A user-visible operation in a microservice application may invoke a chain
Time
Payment Service
Checkout Service
send(req)
Phase 1: Request Sending
Request
recv(req)
Phase 2: Request Receiving
Execute Process send(resp)
recv(resp)
Response
Phase 3: Request Receiving
Phase 4: Response Receiving
A microservice request execution can be viewed as a trace of observable message-level events.
Fig. 1. Message-level view of a microservice request execution.
or fan-out of calls across independently deployed services. We represent such an execution as a trace of observable message-level boundary-event occurrences. For each interservice HTTP or RPC call, the trace records four event types: request send, request receive, response send, and response receive, as illustrated in Fig. 1. These events provide a communication-level view of the distributed execution without requiring access to service-internal logic. For an individual API call, the four boundary events follow a causal order from request send to response receive. Events belonging to concurrent branches, however, may not have a unique global order. Temporal conditions can therefore be expressed in terms of whether particular boundary events have occurred before a selected injection point, rather than assuming a total order over all events in the distributed execution. 2) Request-Level and Temporal Fault Injection: Requestlevel fault injection (RLFI) confines a perturbation to selected request executions by specifying a request selector, an injection point or phase, and a fault type. For example, an RLFI configuration may inject an error into requests to, or responses from, a selected service endpoint. Temporal fault injection (TFI) extends RLFI with a temporal predicate over events observed within the same distributed request execution. This predicate determines whether the fault is eligible to be triggered at its static target; in this paper, we refer to it as a temporal guard. Thus, a fault is injected only when both the static target is matched, and the temporal guard is satisfied. 3MileBeach realizes such temporal control by attaching temporal prerequisites to fault-injection configurations [7]. Accordingly, a temporal fault-injection configuration specifies not only where and what fault to inject, but also the temporal execution context in which the fault becomes eligible. 3) State-Changing Operations and Outcome Uncertainty: Retries are a common recovery mechanism in microservice systems, but their correctness depends on whether an operation can modify durable state or produce externally visible side
TABLE I O PERATION - LEVEL PREVALENCE OF STATE - CHANGING OPERATIONS IN EIGHT PUBLIC API CORPORA . Corpus Microsoft Graph v1.0 [11] Azure REST API Specs [12] Google APIs Discovery [13] Cloudflare API [14] GitHub REST API [15] DigitalOcean API v2 [16] Stripe API [17] Jira Cloud REST API v3 [18] Total
Total 16422 8083 8016 3051 1186 635 619 619 38631
Safe 8634 4090 3231 1475 620 331 274 275 18930
StateChanging 7788 3993 4785 1576 566 304 345 344 19701
Percent(%) 47.4 49.4 59.7 51.7 47.7 47.9 55.7 55.6 51.0
effects. For the purposes of this paper, we distinguish safe operations, whose intended semantics do not request such state changes, from state-changing operations, such as creating an order, reserving inventory, charging a payment method, updating a profile, deleting a resource, or publishing an event. Safety should not be conflated with idempotence. An operation is idempotent if multiple identical requests have the same intended effect as a single request. Therefore, an operation may be state-changing while remaining idempotent; PUT and DELETE, for example, are idempotent under HTTP semantics even though they can modify server-side state [10]. For state-changing operations, an important temporal fault window is the outcome-uncertain window: the downstream side effect may already have occurred, while the upstream caller has not yet observed its result. A fault in this window can leave the caller unable to determine whether retrying, compensating, or reporting failure is safe, potentially causing duplicate effects, lost updates, or an inconsistent business state. B. Motivation 1) Temporal Faults in State-Changing Operations: Following the HTTP method semantics specified in RFC 9110 [10], we classify GET, HEAD, OPTIONS, and TRACE as safe methods. As a method-level approximation, we classify POST, PUT, PATCH, and DELETE as potentially statechanging because they may modify application state or produce externally visible side effects. As summarized in Table I, our analysis covers 38,631 unique operations from eight public API specification corpora. Of these operations, 19,701 (51.0%) use potentially state-changing methods, with the proportion in each corpus ranging from 47.4% to 59.7%. Such operations can create an outcome-uncertain window when a downstream side effect has occurred, but a failure prevents the caller from observing its result. In this window, the caller cannot determine whether retrying, compensating, or reporting failure is safe; an incorrect recovery action may duplicate, lose, or corrupt business state. A conventional injector that triggers before the side effect cannot reproduce this state, even when it targets the same API. Testing such recovery behavior, therefore, requires FI to control not only where a fault occurs, but also when it occurs relative to the state-changing execution.
TABLE II L ANGUAGE - LEVEL DIVERSITY OF SERIALIZATION / DESERIALIZATION LIBRARIES AND THEIR COVERED DATA FORMATS . Language C/C++†
Libs 55
C# / .NET
34
Java
21
Python
14
JavaScript / Node.js
12
Kotlin
11
Go
10
PHP TypeScript Total
8 5 170
Formats JSON, YAML, Protobuf, MessagePack, FlatBuffers, Cap’n Proto, Thrift, binary JSON, XML, YAML, Protobuf, MessagePack, BSON, binary JSON, XML, YAML, Protobuf, Avro, MessagePack, Hessian, binary JSON, YAML, MessagePack, Protobuf, CBOR, Avro, Thrift, Pickle JSON, YAML, XML, Protobuf, CBOR, MessagePack, Avro JSON, Protobuf, CBOR, YAML, HOCON, Properties JSON, YAML, Protobuf, CBOR, MessagePack, Avro JSON, XML, YAML, Protobuf, MessagePack JSON, Protobuf, CBOR, MessagePack, Avro JSON, XML, YAML, Protobuf, MessagePack, CBOR, Avro, Thrift, binary
† C/C++ combines the C++ and C topic slices reported by GitHub.
Insight 1: Potentially state-changing methods constitute 51.0% of the operations in eight public API specification corpora. Because failures before and after a side effect impose different recovery obligations, practical FI should distinguish the temporal window in which a fault occurs. 2) Limitations of Serialization-Layer TFI: The closest existing microservice TFI approach, 3MileBeach, achieves temporal control by instrumenting the message serialization/deserialization layer [7]. Although this design avoids modifying application logic, it couples the fault injector to concrete serialization stacks, including language-specific libraries, generatedcode frameworks, encoders, and wire formats. In polyglot microservice systems, these stacks can differ across services and evolve independently, making it difficult to identify and maintain uniform instrumentation points. To characterize this heterogeneity, we use repositories listed under GitHub’s serialization-library topic [19] as a conservative public indicator of the serialization ecosystem. As shown in Table II, the major language slices contain 170 repositories across nine language families and support diverse formats, including JSON, XML, YAML, Protobuf, Avro, Thrift, and custom binary encodings. This evidence does not imply that every library must be individually supported, but it illustrates the recurring adaptation and validation burden of library-level instrumentation as serialization libraries, generated code, runtime behavior, and data formats evolve. These limitations motivate moving the interposition point to a more implementationindependent communication layer.
(a) Post-effect Failure Window
(b) Order-sensitive Concurrent Responses
(c) k-of-n Partial Failures
Fault after a committed side effect
Fault depends on the relative order of responses
Fault targets a specific subset among concurrent requests
Checkout Service
Payment Service
Charge
1
Charge OK
2.1
OR 2.2
Time Out Error
1
Guard: after commit (Charge OK sent)
Fault: drop / delay response
Request (send)
Risk Service
User Service
Sign Up Service
Client
Frontend Service
Product Service GetProduct #1
CreateUser
1
RiskCheck 2
Guard: recv(User OK) first
OK 3
Fault: delay FAIL
FAIL
5
2
4
...
Guard event (event observed)
Response (recv)
Product OK #2 GetProduct #3
3
SUCCESS
4
Product OK #1 GetProduct #2
Fault injection point (action triggered)
Product #3 (dropped) GetProduct #4
Product OK #4
Guard: after #2 OK Fault: drop #3 response
...
Dropped message
Fig. 2. Three representative temporal fault patterns in microservices. Event-guarded fault injection uses send/receive events as temporal evidence to target post-effect, order-sensitive, and occurrence-specific failure windows that are difficult to distinguish using static request-level or phase-level fault injection. TABLE III NAIVE TFIC ENUMERATION FOR REPRESENTATIVE BENCHMARK REQUESTS . Benchmark
Request
NR
r=1
r=2
r=3
Online Boutique Hotel Reservation Sock Shop Train Ticket
POST /checkout GET /hotels POST /orders POST /preserve
12 5 9 25
576 100 324 2.5K
13.5K 950 5.7K 123.8K
207.6K 5.7K 64.3K 4.04M
K and M denote 103 and 106 .
Insight 2: Serialization-layer TFI is coupled to heterogeneous and evolving language-specific stacks. A communication-layer proxy can provide temporal interception without modifying application code or serialization libraries. 3) Configuration-Space Explosion in TFIC Generation: TFI introduces a configuration burden because a TFIC must specify both a static injection target and a temporal guard. We illustrate this burden using four microservice benchmarks. For each benchmark, we select the request trace containing the largest number of inter-service API invocations, thereby examining the most temporally complex observed request. For a request R invoking NR APIs, each API exposes up to four boundary events, yielding 4NR candidate temporal events. Assuming an r-event guard is a conjunction of r distinct events whose order within the guard is irrelevant, a naive enumerator explores NR 4Nr R configurations, where NR accounts for static targets and 4Nr R for temporal guards. This estimate remains conservative because it ignores faulttype multiplicity, repeated occurrences, request/response phase choices, retries, and concurrent fan-out. Even under this simplified model, Table III shows that naive enumeration reaches 207.6K candidates for Online Boutique and 4.04M for Train Ticket with only three temporal events. Thus, practical TFI requires algorithmic generation of static targets and temporal
guards rather than relying on randomized TFIC enumeration. Insight 3: Even under a simplified distinct-event model, the TFIC search space grows to millions of candidates for a three-event guard. This motivates algorithmic static-target selection and temporal-guard generation. III. T EMPORAL FAULT PATTERNS IN M ICROSERVICES Fig. 2 illustrates three temporal fault scenarios in microservice executions. From these scenarios, we identify three representative temporal fault patterns that require event-guarded injection to reproduce timing-dependent failure conditions. A. Post-Effect Failure Window In the checkout-payment scenario in Fig. 2(a), Payment may charge the user, but its response is lost before reaching Checkout. Checkout observes a timeout despite the completed charge, creating an outcome-uncertain state in which an unsafe retry may produce a duplicate payment. This scenario instantiates a post-effect failure pattern, where a failure occurs after a state-changing operation has taken effect but before its caller observes the result. Static request-level FI cannot construct this window: failing Checkout prevents the charge, while failing Payment before completion models an unsuccessful charge rather than a successful charge with a lost result. Static phase matching may be insufficient when repeated calls or intervening events make the occurrence of the response ambiguous. TFI instead conditions activation on message events, allowing Payment to complete before faulting its response. B. Order-Sensitive Concurrent Responses In the signup scenario in Fig. 2(b), UserService may return a successful CreateUser response before RiskService returns a failure. If the caller commits success after CreateUser, the later RiskCheck failure may arrive too late to prevent an externally visible success. This scenario instantiates an order-sensitive concurrent-response pattern, where application
Client
IV. M ETHOD
Microservice Application Checkout Pod
Payment Pod
A. Approach Overview
APP Container
Figure 3 presents the architecture of SequenceFI, which comprises four components: TFIProxy sidecars deployed alongside microservice pods, a trace collector, a TFIC generator, and an injection controller. A temporal fault-injection configuration (TFIC) consists of two parts: a static configuration and a temporal guard. The static configuration specifies where and what to inject, including the target API, the occurrence index, the injection phase, and the fault type. The temporal guard specifies when the fault should become active by constraining the execution context through previously observed events. A fault is injected only when both the static target and temporal guard are satisfied. SequenceFI observes only message-level send/receive events through TFIProxy sidecars, requiring no modification to application code or serialization libraries. During execution, each TFIProxy records message events, propagates compact binary-encoded temporal evidence across service boundaries, evaluates temporal guards online, and triggers the configured fault only when the required temporal context has been reached. SequenceFI operates through two interacting paths. Along the execution and generation path (black arrows), client requests are executed, traces and injection outcomes are collected, and the TFIC generator derives candidate configurations. Along the injection control path (blue arrows), the injection controller distributes the generated TFICs to the corresponding TFIProxy sidecars for the next injection round. Injection results are continuously fed back to the TFIC generator, allowing SequenceFI to iteratively refine configurations until effective temporal fault injections are identified.
APP Container
TFIProxy
TFIProxy ①Record send/recv events ②Binary-encode ③Propagate Event evidence ④Evaluate guard ⑤Inject faults when satisfied
Request
… APP Container
TFIProxy
Trace Collector
Injection Controller
①Execution traces ③Send/recv event evidence ②Temporal event information ④Injection results
①Select next TFIC ②Dispatch to target sidecar proxy ③Coordinate iniection round
TFIC Generator Static Configuration Solver
Execution and Generation Path
Temporal Guard Solver
Injection Control Path
Fig. 3. Overview of SequenceFI.
behavior depends on the arrival order of concurrent responses. Static FI can make RiskCheck fail, but cannot ensure that the failure occurs after CreateUser has influenced the caller in the intended interleaving. TFI uses ordering evidence to activate the RiskCheck failure after the CreateUser success has been observed, making the relevant interleaving reproducible.
C. k-of-n Partial Failures In the aggregation scenario in Fig. 2(c), a frontend may issue multiple homogeneous GetProduct requests while rendering a page. Its aggregation logic may behave differently when all calls succeed, all fail, or only selected responses fail. This scenario instantiates a k-of-n partial-failure pattern, where correctness depends on exactly which subset of homogeneous or concurrent requests fails. Route-based static FI is too coarsegrained because it may affect every matching GetProduct call rather than a designated occurrence. TFI uses occurrence counts, request identities, or trace-level evidence to target selected calls, enabling precise k-of-n failure states. The three patterns are not intended to form an exhaustive or mutually exclusive taxonomy of microservice failures. Instead, they characterize three primitive temporal constraints needed to reproduce the motivating scenarios: phase constraints, ordering constraints, and cardinality constraints. More complex temporal fault scenarios can be modeled by composing these constraints across multiple events and requests. This observation motivates our method: observing send/receive events, evaluating temporal guards over these events, and triggering faults only when the corresponding guard is satisfied.
B. TFIC Generation SequenceFI generates TFICs in a feedback-driven manner. Given a workload, it first executes the request without fault injection and collects the corresponding trace. Each trace is represented as an ordered sequence of observable send/receive events. Based on the collected traces and previous injection results, SequenceFI generates a TFIC in two steps: static configuration solving and temporal guard generation. The former determines the target API, the injection phase, and the fault type, while the latter determines the temporal condition under which the fault should be triggered. Static configuration solving. A TFIC is decomposed into a static configuration and a temporal guard. The static configuration is defined as: σ = ⟨api, k, phase, f aultT ype⟩,
(1)
where api denotes the target service endpoint or RPC method to be fault-injected, k denotes the local occurrence index of the specified (api, phase) operation within a request execution, phase ∈ {req, resp} specifies whether the fault is injected at the request or response side, and f aultT ype denotes the fault action, such as delay, drop, error, or abort. Here, k is counted only among runtime operations that match the same api and
phase; it does not encode a global temporal order over all events in the trace. SequenceFI adapts FastFI [9], which builds on LDFI, to derive candidate static configurations from request lineages. Each candidate variable represents a possible occurrencespecific static injection point, identified by the target API, the local occurrence index of the corresponding API-phase operation, the injection phase, and the fault type. The solver searches for a minimal set of candidates that can affect the client-level request outcome. Temporal guard generating. SequenceFI restricts guards to a uniform After form over conjunctions of occurrencecount atoms. This form covers three temporal fault patterns in Section III because each can be expressed as prerequisite evidence before a selected target occurrence: post-effect faults require a success-response event, order-sensitive faults require an observed competing response, and k-of-n faults require preceding homogeneous occurrences. The static configuration selects the target API, phase, and occurrence, while the After guard checks event-count evidence. This monotonic design enables compact evidence propagation and distinguishes the target occurrence from premature trigger points. For each static configuration σ derived for a client-level request r, SequenceFI generates an After guard that enables the fault after the required temporal evidence has been observed. Let τnew denote the trace from the latest execution of request r, and let Tr denote the set of observed traces for the same request, including τnew . We model each trace τ ∈ Tr as a time-ordered sequence of observed events, τ = ⟨e1 , . . . , em ⟩. For an event e, its prefix count at position i is: cntτ (e, i) = |{j < i | ej = e}|.
(2)
An atomic guard condition is written as (e, t), meaning that event e has occurred at least t times when the guard is evaluated at the current injection point. Let iσ (τ ) denote the position in trace τ of the k-th runtime occurrence that matches the api and phase specified by σ. SequenceFI constructs candidate atoms that are true at the target occurrence across all traces in Tr : Ur,σ = {(e, te ) | te = min cntτ (e, iσ (τ )), te > 0}. τ ∈Tr
(3)
Since σ includes the local occurrence index k, it identifies the target position iσ (τ ) in each trace τ , i.e., the k-th occurrence of the specified (api, phase) operation. However, the same (api, phase) operation may occur before iσ (τ ); we denote each earlier occurrence position by p, with p < iσ (τ ). These earlier occurrences are potential premature trigger points: if a guard does not exclude them, the fault may be enabled before the execution reaches the temporal context. To prevent such triggering, SequenceFI compares the target position with each earlier occurrence position p in each trace τ ∈ Tr and computes a distinguishing set: Dτ (p, iσ (τ )) = {(e, t) ∈ Ur,σ | cntτ (e, p) < t}.
(4)
Each atom in Dτ (p, iσ (τ )) represents temporal evidence that is satisfied at the target position iσ (τ ) but not at the earlier
occurrence p. Therefore, choosing at least one atom from this set ensures that the guard remains false at p. By selecting atoms that distinguish every earlier occurrence of the same (api, phase) operation from the target position across the observed traces, SequenceFI constructs an After guard that prevents premature triggering and enables the fault only after the required temporal evidence has been observed. SequenceFI then reduces guard generation to a minimum hitting set problem: G∗ = arg min |G| G⊆U
s.t.
∀D ∈ F, G ∩ D ̸= ∅,
(5)
where F is the collection of all distinguishing sets across traces. The resulting temporal guard is: ! ^ a . g = After (6) a∈G∗
Feedback-driven refinement. After a TFIC ⟨σ, g⟩ is generated, the injection controller dispatches it to the target sidecar proxy for the next injection round. The proxy injects the specified fault only when the request reaches the static configuration σ in Equation 1, and the guard g in Equation 6 is satisfied. SequenceFI then observes the execution result. If the fault exposes a failure, the TFIC is retained as an effective configuration. Otherwise, the new trace and injection result are fed back to the generator, which refines either the static configuration or the temporal guard. This process repeats until no new effective TFICs can be derived under the current workload and exploration budget. V. I MPLEMENTATION We implemented SequenceFI as a Kubernetes-based prototype that realizes non-intrusive temporal fault injection through a lightweight sidecar runtime, TFIProxy. TFIProxy interposes on application-layer inter-service communication and performs temporal fault injection without modifying application code or language-specific libraries. A Kubernetes mutating admission webhook [20] automatically injects an init container for traffic redirection and the TFIProxy sidecar for request and response interception. TFIProxy supports HTTP/1.x and gRPC over HTTP/2 and transparently forwards traffic that does not match any deployed TFIC to preserve normal service execution. As shown in Fig. 4, TFIProxy maintains rule-based runtime configurations derived from TFICs. Each rule specifies the static target, injection phase, fault type, and temporal guard. During execution, TFIProxy observes request/response boundary events and encodes only FIC-relevant events into a compact binary evidence vector. The vector is propagated through the custom x-fic-evidence header in both request and response directions. To preserve causal relationships across service boundaries, W3C Trace Context is used solely for call association: TFIProxy stores a lightweight token in the tracestate field to associate each outbound call with its corresponding inbound parent context [21], [22]. Each proxy maintains a local token-to-evidence map, allowing evidence returned from downstream services to be merged into the
Request:
Checkout Service Pod
x-fic-evidence: Ein tracestate: hash(req)
Response: Upstream Frontend Service x-fic-evidence: Eout ∪ send(resp)
Checkout Service Application Container Temporal Fault Injection Proxy 1 On Inbound Request
Req
E
x-fic-evidence tracestate
Ein ∪ recv(req)
2 Local State
Key
Value
hash(req)
E
Bind parent request to evidence
Request: x-fic-evidence: Ein ∪ send(req) tracestate: hash(req)
Downstream Payment Service
3 On Downstream Response
Resp
4 Local Decision
Match <api, k, phase> x-fic-evidence
Match After(E)
Response: x-fic-evidence: Eout
E
Eout ∪ recv(resp)
Inject / Forward
Fig. 4. Implementation overview of SequenceFI’s TFIProxy sidecar.
correct parent request context while avoiding propagation of complete execution histories. At each interception point, TFIProxy first checks whether the current message matches the static target of the deployed TFIC. If so, it evaluates the temporal guard over the current evidence vector. The configured fault is triggered only when both the static target and temporal guard are satisfied; otherwise, the message is forwarded normally. Our prototype supports delay, drop, connection-reset, and abort-style faults. For aborts, TFIProxy returns HTTP 503 for HTTP traffic and gRPC UNAVAILABLE for gRPC calls. TFIProxy records execution traces, event evidence, and injection outcomes, which are fed back to the TFIC generator for subsequent injection rounds. VI. E VALUATION We evaluate SequenceFI through the following research questions, covering temporal triggering correctness, TFIC generation efficiency, and runtime deployment overhead. • RQ1: How effectively can SequenceFI trigger faults at the intended temporal windows of representative temporal fault scenarios while avoiding premature triggering? • RQ2: How efficiently can SequenceFI generate effective TFICs compared with trace-guided and unguided randomized approaches? • RQ3: What runtime overhead does SequenceFI introduce in terms of throughput and sidecar resource consumption? RQ1 evaluates whether SequenceFI can correctly trigger faults at the intended temporal windows of representative temporal fault scenarios without premature triggering. RQ2 evaluates the efficiency of the proposed TFIC generation algorithm by comparing it with a trace-guided randomized baseline and a 3MileBeach-style unguided random enumeration baseline. RQ3 evaluates the runtime overhead of the sidecar-based implementation in terms of throughput and sidecar CPU and memory usage. A. Evaluation Design 1) Benchmarks: We use four widely used microservice benchmarks: Online Boutique [23], Hotel Reservation [24], Sock Shop [25], and Train Ticket [26]. These benchmarks are well-suited for evaluating temporal fault injection because
client-level requests typically span multiple services, comprise multiple execution stages, and often involve concurrent or repeated downstream calls, thereby providing representative execution contexts for temporal fault scenarios in this paper. For each benchmark, we select representative business requests whose executions can instantiate the temporal fault patterns discussed in Section III. We use these requests to evaluate whether our approach can trigger the intended temporal fault scenarios without premature activation, to collect traces for temporal guard generation, and to measure the runtime overhead of sidecar-based fault injection under realistic workloads. 2) Evaluation Environment: All experiments were conducted on a six-node Kubernetes v1.29.15 cluster comprising one control-plane node (Intel Xeon E3-1240 v5, 16 GB RAM) and five worker nodes (Intel Core i7-4790, 16 GB RAM each), all running Ubuntu 22.04. Evaluation scripts used Python 3.9. B. RQ1: Effectiveness of Temporal Fault Triggering Experimental Setup. RQ1 evaluates whether SequenceFI can trigger representative temporal fault patterns at their intended execution windows while avoiding premature triggering at earlier temporal positions or homogeneous occurrences. We study nine scenarios across four benchmarks, covering posteffect, order-sensitive, and k-of-n homogeneous failures. For each scenario, we first collect 10 fault-free traces for temporal guard generation and then perform 50 independent injection trials per method. We compare SequenceFI with two static baselines: Static-Req, which matches only the target API, and Static-Phase, which additionally distinguishes request and response phases but does not support temporal guards. Overall, each method contributes 450 valid trials. Metrics. Each metric is defined as a Boolean condition for every valid trial and reported as the percentage of valid trials satisfying that condition. The primary metric is temporal success (TS), which is satisfied if and only if a fault is triggered exactly once within the intended temporal window of the target scenario: TS = CW ∧ PS ∧ ¬Prem ∧ ¬Miss ∧ ¬Mult, where correct-window (CW) is satisfied if at least one injection reaches the intended API, phase, and occurrence; patternsatisfaction (PS) is satisfied if the injected fault satisfies the pattern-specific temporal semantics; premature (Prem) is true if at least one injection occurs before the intended temporal window; missed (Miss) is true if no injection occurs within the intended temporal window when that window remains observable or can be aligned with a clean execution; and multiple (Mult) is true if more than one injection occurs within the same client request. These metrics are not mutually exclusive and are reported to explain why temporal triggering succeeds or fails. For the post-effect pattern, PS is satisfied only if the downstream state change occurs before the caller observes the injected failure. For the order-sensitive pattern, PS is satisfied only if the required ordering evidence has been established before fault activation. For the k-of-n pattern, PS is satisfied only if the fault is injected at the intended homogeneous occurrence without activation at any earlier occurrence. Unlike TS, PS does not require uniqueness;
TABLE IV T EMPORAL - WINDOW TRIGGERING BY REPRESENTATIVE PATTERN . VALUES ARE PERCENTAGES OVER VALID TRIALS . SequenceFI
Representative Pattern
#Scenarios
Post-effect Order-sensitive k-of-n
3 3 3
Overall
9
Static-Req
Static-Phase
#Trials TS
TS
CW
PS
Prem
Miss
Mult
TS
CW
PS
Prem
Miss
Mult
150 150 150
100.0 100.0 100.0
0.0 0.0 0.0
0.0 0.0 0.0
0.0 0.0 0.0
100.0 100.0 100.0
100.0 100.0 100.0
0.0 0.0 33.3
100.0 66.7 0.0
100.0 66.7 32.7
100.0 66.7 28.7
0.0 33.3 71.3
0.0 20.0 34.0
0.0 0.0 33.3
450
100.0
0.0
0.0
0.0
100.0
100.0
11.1
55.6
66.4
65.1
34.9
18.0
11.1
TABLE V T EMPORAL - WINDOW TRIGGERING FOR INDIVIDUAL SCENARIOS . VALUES ARE PERCENTAGES OVER VALID TRIALS . Benchmark
Scenario
Temporal Pattern
Online Boutique
Payment post-effect
Post-effect
Online Boutique
Payment after shipping quote
Order-sensitive
Online Boutique
Cart GetProduct response occurrence
k-of-n
Hotel Reservation
Reservation after user check
Order-sensitive
Sock Shop
Payment post-effect
Post-effect
Sock Shop
User subresource response
k-of-n
Train Ticket
Trip-search post-effect
Post-effect
Train Ticket
Route after first route response
Order-sensitive
Train Ticket
Route response occurrence
k-of-n
SequenceFI k 1 1 2 1 1 2 1 2 3
Static-Req
Static-Phase
#Trials 50 50 50 50 50 50 50 50 50
TS
TS
CW
PS
Prem
Miss
Mult
TS
CW
PS
Prem
Miss
Mult
100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0
100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0
0.0 0.0 0.0 0.0 0.0 100.0 0.0 0.0 0.0
100.0 100.0 0.0 100.0 100.0 0.0 100.0 0.0 0.0
100.0 100.0 0.0 100.0 100.0 98.0 100.0 0.0 0.0
100.0 100.0 0.0 100.0 100.0 86.0 100.0 0.0 0.0
0.0 0.0 100.0 0.0 0.0 14.0 0.0 100.0 100.0
0.0 0.0 34.0 0.0 0.0 2.0 0.0 60.0 66.0
0.0 0.0 0.0 0.0 0.0 100.0 0.0 0.0 0.0
therefore, additional injections after the intended occurrence preserve PS but are reflected by Mult.
Results. Tables IV and V report aggregate and specific instance results. SequenceFI achieves 100.0% TS across all trials and does not produce premature or multiple injections. This shows that the synthesized temporal guards enable faults only at intended temporal windows. Static-Req fails to trigger any intended temporal window. Its injections occur at request-side events, before post-effect, response-ordering, or occurrence-specific evidence can be held. As a result, StaticReq obtains 0.0% TS for three representative patterns, and Static-Req trials are premature and missed with respect to the intended reference window. This confirms that request-only static matching cannot express the temporal windows.
Static-Phase is more capable but still insufficient without event-level temporal guards. It succeeds in all post-effect trials and in two order-sensitive scenarios where the configured response phase already coincides with the required temporal evidence. However, this success occurs only when the temporal condition degenerates to a static phase match. When ordering evidence or occurrence-level discrimination is required, Static-Phase becomes unreliable: it succeeds in only 66.7% of order-sensitive trials and 0.0% of k-of-n trials. In Sock Shop, it reaches the user subresource response but injects into multiple homogeneous responses in every trial; in Train Ticket, it fires at earlier route responses and misses the intended occurrence. Overall, Static-Phase improves over Static-Req but still achieves only 55.6% aggregate TS, because phase information alone cannot distinguish execution contexts that require temporal evidence or occurrence-level discrimination.
Answer to RQ1. SequenceFI achieves 100.0% temporal triggering across all evaluated scenarios by combining occurrence-aware static targets with synthesized temporal guards. In contrast, static baselines fail when triggering depends on temporal evidence or occurrence discrimination, confirming that SequenceFI precisely targets post-effect, ordering-dependent, and occurrence-specific windows. C. RQ2: Performance of TFIC Generation Experimental Setup. RQ2 evaluates whether SequenceFI can synthesize effective temporal guards with fewer faultinjection attempts than randomized TFIC generation. An effective TFIC triggers the fault at the intended API occurrence and temporal window, without premature, missed, or multiple injections under the RQ1 oracle. Because directly comparable microservice TFI systems are limited, RQ2 focuses on randomized TFIC generation, including the closest prior strategy and a stronger trace-guided variant. We compare SequenceFI with 3MileBeach-Random, an unguided enumeration baseline following 3MileBeach [7], and H-Random, which uses the same clean traces as SequenceFI but samples guard atoms heuristically instead of solving the hitting-set problem. All methods share the same benchmark request, static target, phase, fault type, deployment procedure, workload, and oracle. Metrics. Fault Injection Attempts counts the number of fault-injection attempts required to identify the first effective TFIC. Solving Time measures the cumulative local time spent generating candidate TFICs before success. End-to-end Time measures the wall-clock search time until the first effective TFIC is confirmed, including guard generation, deployment, execution, trace collection, and oracle checking. Results. Table VI shows that SequenceFI finds an effective TFIC in one attempt for every benchmark, while H-Random and 3MileBeach-Random require 23 and 726 attempts on
TABLE VI C OMPARISON OF TFIC GENERATION EFFICIENCY AMONG S EQUENCE FI, H-R ANDOM , AND 3M ILE B EACH -R ANDOM . Benchmark / Request
Method
Fault Injection Solving End-to-end Attempts Time (ms) Time (s)
Online Boutique / Cart workflow
SequenceFI H-Rand 3MB-Rand
1 3 14
18.40 61.19 5.58
17.27 56.23 253.08
Hotel Reservation / Repeated recommendations
SequenceFI H-Rand 3MB-Rand
1 2 3
2.64 13.07 0.63
16.92 38.68 57.01
Sock Shop / Order flow
SequenceFI H-Rand 3MB-Rand
1 39 1078
49.70 8114.92 283.47
17.02 782.75 18242.15
Train Ticket / Left-ticket request
SequenceFI H-Rand 3MB-Rand
1 48 1807
1.61 1594.85 441.88
17.52 804.67 34873.18
average, respectively. H-Random reduces exploration with trace-derived heuristics, but still requires many executions in occurrence-sensitive cases such as Sock Shop and Train Ticket. In contrast, SequenceFI distinguishes the intended occurrence from premature trigger points before deployment. This attempt reduction directly improves end-to-end efficiency. SequenceFI takes 17.18s on average, compared with 420.58s for H-Random and 13,356.36s for 3MileBeachRandom. In aggregate across benchmarks, SequenceFI reduces end-to-end temporal-guard search time by 95.91% over HRandom and 99.86% over 3MileBeach-Random. These results show that deterministic guard generation outperforms both trace-guided and unguided random enumeration. Answer to RQ2. SequenceFI identifies effective TFICs in a single fault injection attempt across all evaluated benchmarks. Compared with H-Random and 3MileBeachRandom, it reduces the average number of fault injection attempts from 23 and 726 to 1, respectively, while shortening aggregate end-to-end search time by 95.91% and 99.86%.
tively. All resource measurements are container-specific: SequenceFI reports only the TFIProxy sidecar, and Istio reports only the Envoy sidecar, excluding application containers. Throughput overhead. Figure 5 shows that SequenceFI consistently preserves throughput close to the NoProxy deployment while outperforming the Istio sidecar across all benchmarks and levels of concurrent clients. NoProxy achieves the highest RPS in all 12 workload cases, while SequenceFI consistently ranks between NoProxy and Istio. On average, SequenceFI retains 97.3% of NoProxy throughput, corresponding to a 2.7% throughput loss. In comparison, Istio retains 91.5% of NoProxy throughput, corresponding to an 8.5% loss. SequenceFI also outperforms Istio in every workload case, improving RPS by 6.6% on average and up to 12.9%. Sidecar resource overhead. Figure 6 compares the sidecar CPU and memory usage of SequenceFI and Istio. SequenceFI uses substantially fewer sidecar resources than Istio across most workload cases. Averaged over the 12 cases, SequenceFI consumes 308.20 mCPU, while Istio consumes 743.39 mCPU. Thus, SequenceFI uses only 41.5% of the CPU used by the Istio sidecar on average. The memory difference is even larger: SequenceFI uses 11.87 MiB on average, while Istio uses 507.95 MiB. Thus, SequenceFI uses only 2.3% of the memory used by the Istio sidecar on average. This gap reflects the narrower scope of TFIProxy, which performs temporal event tracking and guard evaluation rather than providing a full service-mesh data plane. Answer to RQ3. SequenceFI introduces low runtime overhead. It preserves 97.3% of NoProxy throughput on average, consistently outperforms the Istio sidecar baseline, and uses substantially less sidecar CPU and memory. These results show that SequenceFI can provide temporal fault-triggering capability with modest throughput cost and lightweight sidecar resource usage. VII. D ISCUSSION A. Limitations
D. RQ3: Overhead of Sidecar-Based TFI Experimental Setup. RQ3 evaluates the runtime overhead of SequenceFI when temporal events are observed and guards are evaluated, but no faults are injected. Since SequenceFI adopts a sidecar architecture, we compare it with a widely adopted production-grade sidecar. We compare three deployment modes for throughput: NoProxy, which runs benchmarks without any sidecar; SequenceFI, which deploys the TFIProxy sidecar; and Istio, which deploys an Envoy sidecar. For resource overhead, we compare only the sidecar containers of SequenceFI and Istio because NoProxy has no sidecar. We conduct experiments on four benchmarks with 4, 8, and 16 concurrent clients, yielding 12 workload configurations. Each configuration is repeated at least six times, resulting in 216 measurement trials without request failures. Metrics. Throughput is measured in requests per second (RPS). Resource overhead is measured by average sidecar CPU and memory consumption in mCPU and MiB, respec-
SequenceFI uses OpenTelemetry-compatible W3C Trace Context for cross-service call association and a custom x-fic-evidence header for temporal evidence during request execution [21], [22]. Deployments without compatible context propagation require an equivalent call-association mechanism. The current prototype supports HTTP/1.x and gRPC over HTTP/2. Extending SequenceFI to additional communication protocols requires protocol-specific interception, event extraction, and metadata propagation. SequenceFI currently supports only After guards over positive occurrence-count evidence. This design targets the representative temporal fault patterns studied in this paper, all of which are governed by monotonic enabling conditions: once the required temporal evidence has been observed, the fault remains eligible for injection. Richer operators, such as Before and Until, and finer-grained predicates, such as bounded intervals, event absence, and deadline conditions, could support additional temporal fault scenarios but would
100
20
80
15
40
5 0
4
8 Concurrent Clients (a) Online Boutique
16
15.0
50
60
10
17.5
60
12.5
40
RPS
25
70
RPS
120
RPS
RPS
30
30
10.0 7.5
20
5.0
20
10
2.5
0
0
4
8 Concurrent Clients (b) Hotel Reservation
16
SequenceFI
4
8 Concurrent Clients (c) Sock Shop
NoProxy
0.0
16
4
8 Concurrent Clients (d) Train Ticket
16
Istio
2000
700 2000
700 2000
700 2000
700
1500
600 1500
600 1500
600 1500
600
500
500
500
500
1000
1000
400 500 300
20
200
10
100 0
4
8 Concurrent Clients (a) Online Boutique
16
0
1000
400 500 300
20
200
10
100 0
4
8 Concurrent Clients
16
0
(b) Hotel Reservation SequenceFI CPU
Istio CPU
1000
400 500 300
20
200
10
100 0
4
8 Concurrent Clients
16
(c) Sock Shop SequenceFI Memory
0
400 500 300
20
200
10
100 0
Memory (MiB)
CPU (mCPU)
Fig. 5. Throughput under different deployment modes.
4
8 Concurrent Clients
16
0
(d) Train Ticket
Istio Memory
Fig. 6. Sidecar CPU and memory usage.
require reasoning about non-monotonic temporal conditions. Future work will extend SequenceFI with richer temporal operators, event types, and temporal predicates. B. Threats to Validity 1) Internal Threats to Validity: Internal threats concern baseline fairness and implementation artifacts. Because no prior system simultaneously supports sidecar-based deployment and temporal triggering, we use capability-focused baselines: Static-Req and Static-Phase for temporal triggering, and 3MileBeach-Random and H-Random for TFIC generation. We mitigate baseline-related threats by using the same requests, workloads, targets, phases, fault types, deployments, trial settings, and oracles across compared methods. We record traces, propagated evidence, and injection outcomes to validate trials. 2) External Threats to Validity: External threats concern whether our findings generalize beyond the evaluated systems and scenarios. We mitigate this threat by evaluating four microservice benchmarks, Online Boutique, Hotel Reservation, Sock Shop, and Train Ticket [23]–[26], which cover diverse service topologies, request workflows, and HTTP- and gRPCbased communication. We also study nine scenarios spanning three representative temporal fault patterns. 3) Construct Threats to Validity: Construct threats concern whether our metrics capture temporal-triggering effectiveness. We mitigate this threat by reporting CW, PS, Prem, Miss, and Mult to separate failure modes and by applying benchmarkspecific validation rules for post-effect scenarios across re-
peated trials. These metrics assess injection precision but do not directly quantify user or business impact. VIII. R ELATED W ORK Chaos engineering commonly relies on fault injection to expose latent resilience bugs in distributed systems and cloudnative microservices [1]–[5]. Previous work has advanced FI through scenario selection and failure analysis [27]– [30], reproducing realistic distributed failures [31]–[34], and microservice-oriented testing based on lineage-driven search, fitness-guided prioritization, request-level injection, recovery testing, and resilience profiling [6], [8], [9], [35]–[38]. However, these approaches primarily determine where and what faults to inject, rather than when faults should become active during a distributed request execution. Proxy-based FI reduces application-level instrumentation and supports resilience testing in microservice deployments, as demonstrated by Gremlin [39]. To the best of our knowledge, 3MileBeach [7] is the only prior work that explicitly models temporal prerequisites for fault activation in microservice fault injection. It introduces temporal fault injection by guarding fault activation with temporal prerequisites over interservice message flows, enabling timing-sensitive resilience bugs. In contrast, SequenceFI targets practical deployment in cloud-native microservices through non-intrusive sidecar enforcement, lightweight temporal-evidence propagation, and occurrence-aware temporal fault injection.
IX. C ONCLUSION This paper presents SequenceFI, a non-intrusive framework for temporal fault injection in microservice systems. By observing message-level send/receive events, propagating compact temporal evidence, and synthesizing occurrence-sensitive After guards from traces, SequenceFI enables faults to be triggered at precise temporal windows without modifying application code or serialization libraries. Evaluations on four benchmarks demonstrate that SequenceFI makes temporal fault injection practical, precise, and lightweight for microservice resilience testing. Future work will extend SequenceFI to support richer temporal predicates and protocols. R EFERENCES [1] A. Basiri, N. Behnam, R. de Rooij, L. Hochstein, L. Kosewski, J. Reynolds, and C. Rosenthal, “Chaos engineering,” IEEE Software, vol. 33, no. 3, pp. 35–41, 2016. [2] A. Basiri, L. Hochstein, N. Jones, and H. Tucker, “Automating chaos experiments in production,” in 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 2019, pp. 31–40. [3] Netflix, “Chaos monkey,” https://github.com/Netflix/chaosmonkey/, 2025. [4] C. N. C. Foundation, “Chaos mesh,” https://chaos-mesh.org/, 2025. [5] C. N. C. Foundation, “Chaos blade,” https://chaosblade.io/, 2025. [6] H. Chen, P. Chen, G. Yu, X. Li, and Z. He, “Microfi: Non-intrusive and prioritized request-level fault injection for microservice applications,” IEEE Transactions on Dependable and Secure Computing, vol. 21, no. 5, pp. 4921–4938, 2024. [7] J. Zhang, R. Ferydouni, A. Montana, D. Bittman, and P. Alvaro, “3milebeach: A tracer with teeth,” in Proceedings of the ACM Symposium on Cloud Computing, ser. SoCC ’21. New York, NY, USA: Association for Computing Machinery, 2021, p. 458–472. [Online]. Available: https://doi.org/10.1145/3472883.3486986 [8] P. Alvaro, J. Rosen, and J. M. Hellerstein, “Lineage-driven fault injection,” in Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, ser. SIGMOD ’15. New York, NY, USA: Association for Computing Machinery, 2015, p. 331–346. [Online]. Available: https://doi.org/10.1145/2723372.2723711 [9] Y. Tan, J. Wang, S. Xie, B. Li, Y. Yong, N. Zhang, and S. Tan, “Fastfi: Enhancing api call-site robustness in microservice-based systems with fault injection,” ACM Trans. Softw. Eng. Methodol., May 2026, just Accepted. [Online]. Available: https://doi.org/10.1145/3813806 [10] R. T. Fielding, M. Nottingham, and J. Reschke, “RFC 9110: HTTP Semantics,” Internet Engineering Task Force, Request for Comments 9110, 2022, accessed: 2026-06-01. [Online]. Available: https://www.rfc-editor.org/rfc/rfc9110.html [11] Microsoft, “Microsoft Graph OpenAPI Metadata,” GitHub repository, 2026, accessed: 2026-06-01. [Online]. Available: https://github.com/ microsoftgraph/msgraph-metadata [12] Microsoft Azure, “Azure REST API Specifications,” GitHub repository, 2026, accessed: 2026-06-01. [Online]. Available: https://github.com/ Azure/azure-rest-api-specs [13] Google, “Google APIs Discovery Artifact Manager,” GitHub repository, 2026, accessed: 2026-06-01. [Online]. Available: https://github.com/ googleapis/discovery-artifact-manager [14] Cloudflare, “Cloudflare API Schemas,” GitHub repository, 2026, accessed: 2026-06-01. [Online]. Available: https://github.com/cloudflare/ api-schemas [15] GitHub, “GitHub REST API OpenAPI Description,” GitHub repository, 2026, accessed: 2026-06-01. [Online]. Available: https://github.com/ github/rest-api-description [16] DigitalOcean, “DigitalOcean API v2 OpenAPI Specification,” GitHub repository, 2026, accessed: 2026-06-01. [Online]. Available: https: //github.com/digitalocean/openapi [17] Stripe, “Stripe OpenAPI Specification,” GitHub repository, 2026, accessed: 2026-06-01. [Online]. Available: https://github.com/stripe/ openapi
[18] Atlassian, “Jira Cloud Platform REST API v3,” Atlassian Developer Documentation, 2026, accessed: 2026-06-01. [Online]. Available: https://developer.atlassian.com/cloud/jira/platform/rest/v3/intro/ [19] GitHub, “GitHub Topics: Serialization Library,” https://github.com/ topics/serialization-library, accessed: 2026-06-01. [20] C. N. C. Foundation, “Kubernetes webhook,” https://kubernetes.io/docs/ reference/access-authn-authz/webhook/, 2026. [21] C. N. C. Foundation, “Opentelemetry,” https://opentelemetry.io/, 2026. [22] W. W. W. Consortium, “Trace context,” https://www.w3.org/TR/ trace-context/, 2021. [23] G. C. Platform, “Online boutique,” https://github.com/ GoogleCloudPlatform/microservices-demo/, 2025. [24] Y. Gan, Y. Zhang, D. Cheng, A. Shetty, P. Rathi, N. Katarki, A. Bruno, J. Hu, B. Ritchken, B. Jackson, K. Hu, M. Pancholi, Y. He, B. Clancy, C. Colen, F. Wen, C. Leung, S. Wang, L. Zaruvinsky, M. Espinosa, R. Lin, Z. Liu, J. Padilla, and C. Delimitrou, “An open-source benchmark suite for microservices and their hardwaresoftware implications for cloud & edge systems,” in Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems, ser. ASPLOS ’19. New York, NY, USA: Association for Computing Machinery, 2019, p. 3–18. [Online]. Available: https://doi.org/10.1145/3297858.3304013 [25] microservices demo, “Sock shop,” https://github.com/ microservices-demo/microservices-demo/, 2025. [26] X. Zhou, X. Peng, T. Xie, J. Sun, C. Xu, C. Ji, and W. Zhao, “Benchmarking microservice systems for software engineering research,” in Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings, ser. ICSE ’18. New York, NY, USA: Association for Computing Machinery, 2018, p. 323–324. [Online]. Available: https://doi.org/10.1145/3183440.3194991 [27] H. Ikeuchi, A. Watanabe, and Y. Takahashi, “Coverage based failure injection toward efficient chaos engineering,” in ICC 2023 - IEEE International Conference on Communications, 2023, pp. 4571–4577. [28] D. Cotroneo, L. De Simone, P. Liguori, and R. Natella, “Fault injection analytics: A novel approach to discover failure modes in cloudcomputing systems,” IEEE Transactions on Dependable and Secure Computing, vol. 19, no. 3, pp. 1476–1491, 2022. [29] Q. Wang, J. Rios, S. Jha, K. Shanmugam, F. Bagehorn, X. Yang, R. Filepp, N. Abe, and L. Shwartz, “Fault injection based interventional causal learning for distributed applications,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 13, pp. 15 738–15 744, 2024. [30] D. Kesim, A. van Hoorn, S. Frank, and M. Häussler, “Identifying and prioritizing chaos experiments by using established risk analysis techniques,” in 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE), 2020, pp. 229–240. [31] L. Zhang, B. Morin, B. Baudry, and M. Monperrus, “Maximizing error injection realism for chaos engineering with system calls,” IEEE Transactions on Dependable and Secure Computing, vol. 19, no. 4, pp. 2695–2708, 2022. [32] Y. Chen, X. Sun, S. Nath, Z. Yang, and T. Xu, “Push-Button reliability testing for Cloud-Backed applications with rainmaker,” in 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). Boston, MA: USENIX Association, 2023, pp. 1701–1716. [Online]. Available: https://www.usenix.org/conference/ nsdi23/presentation/chen-yinfang [33] J. Pan, H. Wu, T. Leesatapornwongsa, S. Nath, and P. Huang, “Efficient reproduction of fault-induced failures in distributed systems with feedback-driven fault injection,” in Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles, ser. SOSP ’24. New York, NY, USA: Association for Computing Machinery, 2024, pp. 46–62. [Online]. Available: https://doi.org/10.1145/3694715.3695979 [34] R. Lu, Y. Lu, Y. Jiang, G. Xue, and P. Huang, “One-Size-FitsNone: Understanding and enhancing Slow-Fault tolerance in modern distributed systems,” in 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25). Philadelphia, PA: USENIX Association, 2025, pp. 359–378. [Online]. Available: https://www.usenix.org/conference/nsdi25/presentation/lu [35] P. Alvaro, K. Andrus, C. Sanden, C. Rosenthal, A. Basiri, and L. Hochstein, “Automating failure testing research at internet scale,” in Proceedings of the Seventh ACM Symposium on Cloud Computing, ser. SoCC ’16. New York, NY, USA: Association for Computing Machinery, 2016, pp. 17–28. [Online]. Available: https://doi.org/10.1145/2987550.2987555
[36] Z. Long, G. Wu, X. Chen, C. Cui, W. Chen, and J. Wei, “Fitnessguided resilience testing of microservice-based applications,” in 2020 IEEE International Conference on Web Services (ICWS), 2020, pp. 151– 158. [37] C. S. Meiklejohn, A. Estrada, Y. Song, H. Miller, and R. Padhye, “Service-level fault injection testing,” in Proceedings of the ACM Symposium on Cloud Computing, ser. SoCC ’21. New York, NY, USA: Association for Computing Machinery, 2021, pp. 388–402. [Online]. Available: https://doi.org/10.1145/3472883.3487005 [38] T. Yang, C. Lee, J. Shen, Y. Su, C. Feng, Y. Yang, and M. R. Lyu, “Microres: Versatile resilience profiling in microservices via degradation dissemination indexing,” in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ser. ISSTA 2024. New York, NY, USA: Association for Computing Machinery, 2024, pp. 325–337. [Online]. Available: https://doi.org/10.1145/3650212.3652131 [39] V. Heorhiadi, S. Rajagopalan, H. Jamjoom, M. K. Reiter, and V. Sekar, “Gremlin: Systematic resilience testing of microservices,” in 2016 IEEE 36th International Conference on Distributed Computing Systems (ICDCS), 2016, pp. 57–66.