arXiv CSopen access

TORAI: Unsupervised Fine-grained RCA using Multi-Source Telemetry Data

TORAI: Multi-source Root Cause Analysis for Blind Spots in Microservice Service Call Graph

arXiv:2604.13522v1 [cs.SE] 15 Apr 2026

LUAN PHAM, RMIT University, Australia HUONG HA, RMIT University, Australia XIUZHEN ZHANG, RMIT University, Australia HONGYU ZHANG, Chongqing University, China Existing multi-source root cause analysis (RCA) methods for microservice systems assume all services have traces to construct a service call graph. However, this assumption is not practical as microservice systems evolve rapidly and may contain blackbox services without traces, such as compiled software or unsupported services. We refer to these services as blind spots. In the presence of blind spots, the performance of existing multi-source RCA methods may be affected, as they only diagnose visible services on the call graph. To overcome this limitation, we propose TORAI, a novel unsupervised approach that effectively pinpoints finegrained root causes without relying on the service call graph. Instead, TORAI first measures anomaly severity using available multi-source telemetry data. It then performs clustering to group services based on their severity symptoms and conducts causal analysis to rank services within each severity cluster. Finally, TORAI aggregates the cluster rankings and uses hypothesis testing to identify fine-grained root causes. TORAI provides an unsupervised approach that leverages available multi-source telemetry data for RCA without requiring a constructed service call graph or further intrusive actions, thus addressing the limitations of existing methods. Our experiments on three benchmark systems demonstrate that TORAI outperforms stateof-the-art baselines remarkably in the presence of blind spots. Performance on real-world failures further shows that TORAI can accurately pinpoint the root causes in top-3 recommendations. CCS Concepts: • Software and its engineering → Software reliability; Software performance. Additional Key Words and Phrases: Root Cause Analysis, Microservice Systems, Telemetry Data ACM Reference Format: Luan Pham, Huong Ha, Xiuzhen Zhang, and Hongyu Zhang. 2026. TORAI: Multi-source Root Cause Analysis for Blind Spots in Microservice Service Call Graph. Proc. ACM Softw. Eng. 3, FSE, Article FSE130 (July 2026), 23 pages. https://doi.org/10.1145/3808137

1

Introduction

Microservices have emerged as a popular form of software systems because of their advantages such as loose coupling, resource flexibility, and simplified deployment processes. However, the inherent complexity of microservice systems often leads to failures, affecting user experience, and causing substantial economic losses. For example, a one-hour downtime on Amazon may cost up to 100 million dollars [42, 44, 46]. Therefore, effectively and efficiently diagnosing the root causes of failures in microservice systems is crucial to quickly mitigate the failure and minimize its impact. To effectively conduct root cause analysis (RCA), system operators typically gather three primary sources of telemetry data: metrics, logs, and traces [28, 62, 65]. Metrics, typically presented as time Authors’ Contact Information: Luan Pham, RMIT University, Melbourne, Australia, [email protected]; Huong Ha, RMIT University, Melbourne, Australia, [email protected]; Xiuzhen Zhang, RMIT University, Melbourne, Australia, [email protected]; Hongyu Zhang, Chongqing University, Chongqing, China, [email protected]. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). © 2026 Copyright held by the owner/author(s). ACM 2994-970X/2026/7-ARTFSE130 https://doi.org/10.1145/3808137

Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE130. Publication date: July 2026.

FSE130:2

Luan Pham, Huong Ha, Xiuzhen Zhang, and Hongyu Zhang

series, provide insights into service status, including system-level and application-level metrics like CPU usage and average response time. Logs, comprising semi-structured text, record events that occur at runtime, including request handling events, state changes. Traces, structured in a tree format, record the sequence of user request invocations. Unlike metrics and logs, traces usually require heavy instrumentation with distributed tracing, often involving modifications to microservices’ source code [16, 27, 52, 62]. Figure 2A illustrates the multi-source telemetry data. We have identified three key limitations of services with traces existing multi-source RCA methods [28, 48, 59, Frontservices w/o traces 62, 65]: they are either (1) approaches that asend (blind spots) sume the system has full trace coverage, allowing them to construct a full service call graph from traces, or (2) supervised approaches requiring a large volume of labeled data, or (3) intrusive RecomAds Checkout Cart mendation approaches requiring heavy instrumentation of the microservices’ source code. First, most multisource RCA methods [28, 48, 59, 62, 65], except Cart those omitting trace information [66, 67], as- Shipping Payment Product Email Currency Catalog Database sume the system has full trace coverage, i.e., all services in the system are instrumented with Fig. 1. Service Call Graph of the Online Boutique midistributed tracing, because they rely on traces croservice system containing blind spots (i.e., services to construct the service call graph required in without distributed tracing instrumentation). their RCA methods. This requirement limits their applicability, especially in large and evolving microservice systems where new services or versions are introduced all the time, the engineers may not have enough effort to implement the distributed tracing for newly introduced services [16, 52], resulting in many blind spots (i.e., services are not in the service call graph), that directly impact RCA performance, as shown in Figure 1. Second, many multi-source RCA approaches [28, 34, 59, 65] are supervised approaches, requiring a large volume of training data to train their models. It is generally impractical to expect such a large volume of training data, as microservice systems are typically large and rapidly developed with many updates [15, 52]. Third, a recent multi-source RCA method [62] requires a tight integration between traces and logs (i.e., each log line must contain the trace_id), which demands extensive effort to implement the necessary Q: How do we address the mentioned limitations? changes in the microservices’ source We aim to propose a novel multi-source RCA method code and may become infeasible when there are closed-source components. that satisfies three criteria: (1) avoiding using the call graph constructed from traces; (2) relying on unsupervised Consequently, these three limitations make existing multi-source RCA meth- techniques; and (3) leveraging available telemetry data ods inflexible and challenging to im- without requiring further updates to the source code. plement in real-world scenarios. In this study, we introduce TORAI, a novel multi-source RCA method for identifying fine-grained root causes of microservice failures. TORAI employs unsupervised techniques intuitively. First, it measures the anomaly severity from multi-source telemetry data. Second, it clusters services with similar severity symptoms. Next, it applies causal analysis to identify coarse-grained root causes (i.e., root cause services). Finally, it uses hypothesis testing to derive fine-grained root causes indicators (e.g., metrics or logs indicating the underlying root cause). TORAI offers several advantages over existing multi-source RCA methods. First, it does not rely on the service call graph (see Figure 1), eliminating the need for full trace coverage. Second, it uses unsupervised techniques such as clustering, and causal analysis, removing the need for labeled data and enabling Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE130. Publication date: July 2026.

TORAI: Multi-source Root Cause Analysis for Blind Spots in Microservice Service Call Graph

FSE130:3

direct application to microservice systems without requiring training. Third, it effectively leverages multi-source telemetry data, allowing for the root cause diagnosis in services not present in the call graph, without the need for additional intrusive instrumentation to achieve full trace coverage. To evaluate TORAI, we conduct extensive experiments on 270 failure cases collected from three benchmark microservice systems and 10 real-world incidents, comparing TORAI against nine state-of-the-art RCA methods. The experimental results demonstrate that TORAI outperforms all baselines in localizing both coarse-grained and fine-grained root causes in terms of effectiveness and efficiency. We further evaluate TORAI on real-world failures from a major internet provider. The results show that TORAI outperforms state-of-the-art baselines and achieves 100% accuracy in ranking the root cause within the top three recommendations. In summary, our main contributions are as follows: • We identify three key limitations of existing multi-source RCA methods, including the assumption of full trace coverage (i.e., assuming the constructed service call graph is complete and contains no blind spots), the reliance on labeled data, and the requirement for intrusive instrumentation. These limitations motivate our work. • We introduce TORAI, a novel RCA method that addresses the mentioned limitations effectively. TORAI pinpoints fine-grained root causes by diagnosing available multi-source telemetry without requiring a constructed call graph or labeled data, making it adaptable to a wide range of microservice systems. • We conduct extensive experiments to evaluate TORAI and the experimental results demonstrate that TORAI outperforms existing methods in both effectiveness and efficiency. Additionally, we evaluate TORAI with real-world failures and the results confirm its potential. 2 2.1

Background Key Terminologies

Failures denote the incapacity of a service to perform its functions [46, 53]. Faults correspond to the root causes of such failures (e.g., CPU overload, memory leaks, or network disconnections) [17, 53]. Anomalies are defined as observable symptoms of failures [44, 46]. Root cause analysis (RCA) is the process of identifying why a failure has occurred [28], i.e., pinpointing the failure’s root causes. RCA entails a comprehensive examination of multi-source telemetry data (i.e., metrics, logs, and traces) to derive coarse-grained root causes (i.e., root cause services), and fine-grained root causes (i.e., root cause indicators). The system operators can use these suggested root cause services and indicators (e.g., specific metrics, logs, or traces) to identify the underlying root cause of the failures. The use of terminologies aligns with existing RCA works [25, 29, 44, 60]. 2.2

Problem Formulation

𝑁 . Multi-source telemetry data, Consider a microservice system S comprising 𝑁 services {𝑠 𝑖 }𝑖=1 including metrics, logs, and traces, are collected for each service, if available. At each time step 𝑡, we obtain multi-source telemetry data D𝑡𝑖 = {M𝑡(𝑖,𝑚) , L𝑡(𝑖,𝑙 ) , T C𝑡(𝑖,𝑠 ) } for each service 𝑠 𝑖 , where M (𝑖,𝑚) is a set of 𝑚 metrics, L𝑡(𝑖,𝑙 ) represents a set of 𝑙 logs, and T C𝑡(𝑖,𝑠 ) denotes a set of 𝑠 traces. Given the anomaly detection time 𝑡ˆ𝐴 (i.e., the start timestamp of the abnormal window detected by the anomaly detection module), let us denote D𝑡0 ≤𝑡 <𝑡ˆ𝐴 as the multi-source telemetry data collected during normal periods and D𝑡ˆ𝐴 ≤𝑡 <𝑡ˆ𝐴 +𝑇 as the data collected during the failure occurrence period, our objective is to develop a method that identifies the root causes of the failure using these datasets. The expected RCA output is a ranked list of root cause services and their corresponding root cause indicators (e.g., specific metrics, logs, or traces that indicate the root causes).

Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE130. Publication date: July 2026.

FSE130:4

2.3

Luan Pham, Huong Ha, Xiuzhen Zhang, and Hongyu Zhang

Related Work and Motivation

2.3.1 The Blind Spots of Existing Multi-Source RCA. Existing multi-source RCA methods [24, 28, 48, 59, 62, 65] typically construct a service call graph from traces, assuming the system has full trace coverage, i.e., 100% of services are instrumented with distributed tracing [16, 52]. They then integrate metrics and logs into the service call graph to perform RCA. This approach is only effective when the system has no blind spots, i.e., services without traces or closed-source components [16, 28, 52, 62]. In this paper, we refer to services or components not detected by the traces but that should be as blind spots. In the presence of blind spots, existing multi-source RCA methods [20, 28, 59, 62, 65] may not perform RCA effectively, as the constructed service call graph is incomplete. Consequently, key metrics and logs may be overlooked, and their behavior may not be analyzed when diagnosing the root cause of failure in microservice systems. Recent studies [16, 52] and industry reports [38] show that requiring the microservice systems to be fully instrumented is ineffective and inefficient. In reality, microservice systems are often developed in many different languages, and the call graph can become exceedingly intricate, with some systems comprising up to 1,500 services [35]. Shen et al. [52] show that engineers spend hours to instrument mere tens of lines of code for a single component. Hence, they may not have sufficient time to instrument every component before deployment, leading to numerous blind spots in the constructed call graph. TraceWeaver [16] explicitly acknowledges that eBPF is “insufficient to solve the tracing challenge” as “there is no guarantee that the application will propagate these headers to related outgoing backend requests.” Industry reports [38] confirm that blind spots remain prevalent in production systems despite advances in eBPF-based tracing. In this paper, we aim to bypass the assumption of full traces coverage and develop a multi-source RCA method to localize root causes effectively, even when some or all traces are missing. It is important to note that metrics and logs are relatively easy to collect, as they do not require source code modification, unlike traces. Metrics such as response time and error rates can be collected without traces by monitoring agents or service mesh without requiring any source code modification [44, 52]. This is standard practice in industry [4] and is how existing metric-based RCA works collect data [25, 29, 44]. 2.3.2 Supervised RCA Approaches In A Dynamic World. Supervised multi-source RCA methods [28, 59, 65, 66] rely on large volumes of manually labeled training data to train their models, which often incorporate graph neural networks or convolutional layers. However, requiring such large amounts of labeled data is often impractical or challenging in real-world scenarios. First, microservice systems are highly dynamic, with old services being updated and new services being released frequently. Second, historical faults are often resolved, while new faults are introduced over time [52]. Additionally, future edge-case faults, which could cause significant losses, are unlikely to be present in historical failure datasets [64]. Third, microservice systems are typically large in scale [15], making it costly and impractical to manually label data to cover all services and fault types. Therefore, our aim is to develop an RCA method that leverages unsupervised techniques to reduce the dependency on labeled data. 2.3.3 Tight Integration between Multi-source Data. Recent multi-source RCA studies [28, 62, 65–67] require multi-source telemetry data to be tightly integrated. For example, Nezha [62] requires every log line in the microservices to be modified to contain a trace_id. To achieve this, engineers must manually modify every log or print statement in the microservices’ source code, which is time-consuming and costly. This task becomes impossible when dealing with black box components, third-party services, or new frameworks that do not yet support distributed tracing [19, 52]. In addition, Eadro [28] and DiagFusion [65] extract features from metrics and logs, integrate these features into a graph constructed using traces. These methods struggle in the presence of blind spots, as many metrics and logs may not correspond to any location on the graph, leading to the Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE130. Publication date: July 2026.

TORAI: Multi-source Root Cause Analysis for Blind Spots in Microservice Service Call Graph

FSE130:5

failure to use these valuable data for failure diagnosis. In summary, most existing multi-source RCA approaches may be impractical, as their assumptions often might not be fully met in realworld scenarios. Some other multi-source RCA methods do not require this tight integration, but interestingly, they completely omit trace information [55, 66, 67] or log information [69]. In other words, they fail to fully take advantage of all available telemetry data. 2.3.4 Summary. These motivations drive the design of TORAI, a novel multi-source RCA method that overcomes the existing limitations. Firstly, our method leverages multi-source telemetry data without requiring full trace coverage [28, 62, 65], allowing it to diagnose the fine-grained root causes of failures with high performance even in the absence of some or all traces. We do not attempt to reconstruct the call graph, instead, we design TORAI to perform RCA effectively without requiring a complete call graph. This design is inspired by recent theoretical work [39], which proves that identifying root causes based on anomaly scores is causally justified. Recent studies [45] also show that call graph reconstruction can be ineffective for RCA due to limitations of causal discovery algorithms. Secondly, our method can be applied without modifying the source code of microservices for tight integration between traces and logs [62]. Thirdly, our method does not rely on labeled training data [28, 59, 65, 66], making it adaptable to a wide range of systems. The details of our proposed RCA method, TORAI, are presented in the next section. 3

TORAI: Unsupervised Fine-grained RCA using Multi-Source Telemetry Data

When an anomaly is detected, TORAI is triggered to perform RCA as follows. First, TORAI collects the multi-source telemetry data and transforms them into time series (Sec. 3.1). Second, it measures the anomaly severity of each data source for all services using SeverityScorer (Sec. 3.2). Third, it uses SymptomCluster to group services exhibiting similar severity symptoms (Sec. 3.3). Fourth, it conducts causal analysis to rank the root causes within each severity group using CausalRanker (Sec. 3.4). Then, it performs RankAggregation to aggregate the results from the SymptomCluster and CausalRanker steps, yielding a root cause service ranked list (Sec. 3.5). Finally, TORAI uses FineGrainer to perform hypothesis testing and derive fine-grained root cause indicators for the corresponding root cause services (Sec. 3.6). The overview of TORAI is shown in Fig. 2. 3.1

Transform Multi-source Telemetry Data

TORAI transforms multi-source telemetry data into time series for RCA. It collects data during normal periods to learn expected behaviors and during abnormal periods to analyze and identify root causes. The processing steps for each data source are as follows: 3.1.1 Metrics. TORAI collects four metrics types: Traffic (e.g., requests per minute), Saturation (e.g., CPU/memory utilization), Latency (e.g., average response time), and Errors (e.g., the rate of failed requests) [44], known as the four golden signals in site reliability engineering [4], during 𝑖 to both normal and abnormal periods. For each service 𝑠 𝑖 , TORAI compiles a set of time series XM represent the metrics data. 3.1.2 Logs. We focus on log occurrences rather than log semantics, as empirical studies have shown that the quality of log semantics cannot be guaranteed [22, 28] and semantic extraction requires significant computational resources. First, TORAI parses logs into log templates using Drain [21] by removing variables from the log messages. Next, TORAI bins the occurrence of log templates based on their timestamps to obtain time series. This process generates a set of time 𝑖 , representing the log occurrences of each service 𝑠 𝑖 . series XL Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE130. Publication date: July 2026.

FSE130:6

Luan Pham, Huong Ha, Xiuzhen Zhang, and Hongyu Zhang

A) Collect and Transform Telemetry Data Latency CPU Mem Metrics Service 1: ... ... Service 5: Logs

D) CausalRanker

Service 3

Service 4 Service 6 ...

B) SeverityScorer

C) SymptomCluster

Time- Seveseries rity

Service 5 Service 2

...

...

Service 1: Service 2: Service 3: Service 4: ... Severity Vector

CPU 0.73 TraceLat 0.12

Service 6

Service 2

Critical

Service 1

Service 4

Traces Service 5: /call-s3, HTTP 500 ...

Normal

F) FineGrainer

Service 1 Service 5

Service 1: "failed to handle" ... Service 5: "request to s3 sent"

E) RankAggregation

G) Fine-grained Root Causes

Service 1 Service 2 Service 5

... ...

Service 3 Service 4 Service 6

Latency 0.12 ErrLog 0.11 ... Root Cause Score Indicator

...

Root causes Service 1-CPU Service 5-Latency Service 2-ErrLog ...

Score 0.73 0.12 0.11 ...

Fig. 2. Overview of TORAI. (A) TORAI transforms telemetry data into time series. (B) It computes anomaly 𝑖 , 𝜌 𝑖 , 𝜌 𝑖 ] for each service 𝑠 𝑖 (with missing data sources denoted by ⊏ severity scores, producing vectors [𝜌𝑚 ⊐; 𝑙 𝑡𝑐 e.g., Service 2 is a blind spot). (C) TORAI clusters services based on their severity symptoms. (D) Within each cluster, it applies causal inference-based RCA to rank potential root causes. (E) TORAI aggregates rankings from steps C and D to generate a coarse-grained root cause list. (F) It then conducts fine-grained RCA using hypothesis testing to produce (G) a fine-grained ranked list of root causes.

3.1.3 Traces. We extract available information from traces, but do not construct the call graph as it may be incomplete (see Sec. 2.3). TORAI transforms latency and status code in traces into time series. Each time series presents the frequency of latency and status code for each trace operation. We refer to time series data derived from multi-source telemetry data as multi-source time 𝑖 , X𝑖 , X𝑖 } series data. Each service 𝑠 𝑖 has a set of multi-source time series X𝑖 = {XM L T C 𝑡 0 ≤𝑡 <𝑡ˆ𝐴 +𝑇 𝑖 𝑖 𝑖 where XM , XL , and XT C denote the set of time series for metrics, logs, and traces. The data is split at the anomaly detection time 𝑡ˆ𝐴 into normal (X𝑡𝑖 ≤𝑡 <𝑡ˆ ) and abnormal (X𝑡ˆ𝑖 ≤𝑡 <𝑡ˆ +𝑇 ) sets. 0

3.2

𝐴

𝐴

𝐴

SeverityScorer

SeverityScorer measures anomaly severity scores for each time series and generates a severity vector for each service. These severity vectors effectively assist SymptomCluster in grouping services based on their severity symptoms, thereby quickly separating abnormal services from normal ones (see Sec. 3.3). This also reduces overhead for CausalRanker, which further analyses the multi-source time series to derive the root causes (see Sec. 3.4). It is important to note that SeverityScorer does not aim to identify the root causes directly but instead focuses on quickly assessing the severity of each service based on its multi-source telemetry data after a failure occurs, thereby helping subsequent components of TORAI concentrate their focus on analysing the abnormal services. In particular, for each time series 𝑥 (𝑖,𝑗 ) , SeverityScorer learns the mean 𝜇 (𝑖,𝑗 ) and standard deviation 𝜎 (𝑖,𝑗 ) during the normal period. Then during the abnormal period, for each data point 𝑥𝑡(𝑖,𝑗 ) of the time series 𝑥 (𝑖,𝑗 ) , SeverityScorer measures how far it diverges from the expected value. This deviation is denoted as 𝑎𝑡(𝑖,𝑗 ) and is computed as 𝑎𝑡(𝑖,𝑗 ) = 𝑥𝑡(𝑖,𝑗 ) − 𝜇 (𝑖,𝑗 ) /𝜎 (𝑖,𝑗 ) . SeverityScorer then aggregates 𝑎𝑡(𝑖,𝑗 ) for all the available data during the abnormal period, yielding the anomaly severity score 𝜌 (𝑖,𝑗 ) = max𝑡ˆ𝐴 ≤𝑡 <𝑡ˆ𝐴 +𝑇 𝑎𝑡(𝑖,𝑗 ) . Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE130. Publication date: July 2026.

TORAI: Multi-source Root Cause Analysis for Blind Spots in Microservice Service Call Graph

FSE130:7

Using the anomaly severity scores 𝜌 (𝑖,𝑗 ) across all time series and data sources, SeverityScorer 𝑖 , 𝜌 𝑖 , 𝜌 𝑖 ], which represents the severity status of assigns each service 𝑠 𝑖 a severity vector [𝜌𝑚 𝑡𝑐 𝑙 each service as observed through its multi-source telemetry data. If a data source is missing, SeverityScorer imputes the corresponding 𝜌 value as 0, indicating no anomalies were detected from that source. This flexible design enables TORAI to use available data sources, such as metrics and 𝑖 and 𝜌 𝑖 are obtained by integrating logs, without requiring the presence of traces. Specifically, 𝜌𝑚 𝑡𝑐 the severity scores of the time series derived from metrics and traces associated with service 𝑠 𝑖 , (𝑖,𝑗 ) 𝑖 =Í 𝑖 =Í 𝑖 𝜌𝑚 𝑖 i.e., 𝜌𝑚 and 𝜌𝑡𝑐 𝜌𝑡𝑐(𝑖,𝑗 ) . For logs, 𝜌𝑙𝑖 is determined as the score of the 𝑥 (𝑖,𝑗 ) ∈ XM 𝑥 (𝑖,𝑗 ) ∈ XTC most anomalous log templates for 𝑠 𝑖 , i.e., 𝜌𝑙𝑖 = max𝑥 (𝑖,𝑗 ) ∈ X𝑖 𝜌𝑙(𝑖,𝑗 ) , since the number of log templates L between services varies significantly. 3.3

SymptomCluster

SymptomCluster performs clustering to group services with similar severity symptoms. The goal is to effectively separate abnormal services from normal ones, reducing the overhead for CausalRanker when conducting causal analysis, inspired by recent research [45] that highlights how including all services in the causal analysis can be inefficient and ineffective. Specifically, SymptomCluster proposes using Gaussian Mixture Model (GMM) [3] for this clustering task. GMM is a probabilistic model that assumes the data come from a mixture of several Gaussian distributions, each representing a cluster. GMM is known for its efficiency and has been successfully applied in various domains [40]. We first use the Bayesian Information Criterion (BIC) [50] to identify the optimal number of clusters (i.e., the number of Gaussian distributions). Specifically, we iterate through all possible values for the number of clusters. For each iteration, 𝑖 , 𝜌 𝑖 , 𝜌 𝑖 ]} 𝑁 (𝑁 is the number of SymptomCluster runs GMM on the set of severity vectors {[𝜌𝑚 𝑙 𝑡𝑐 𝑖=1 services) and calculates the corresponding BIC score. The optimal number of clusters is chosen as the number that yields the lowest BIC score [3, 45]. CPU ErrLog TraceLat Once the optimal number of clusters is determined, SympService 1: tomCluster performs clustering again using this value. For each cluster, SymptomCluster measures the cluster severity Service 2: score as the mean severity score of all services within that Service 5: cluster (i.e., the mean of the Gaussian distribution). Finally, SymptomCluster generates a ranked list of clusters (a clusterRoot cause level ranked list), where the highest-ranked clusters are the Service 1 Service 2 most likely to contain the root cause. This approach aligns ErrLog TraceLat Service 1 with the intuition that operators should prioritize higher CPU severity services for failure diagnosis, while setting lower Service 1 Service 5 TraceLat TraceLat priority to services with less severity and normal services. In addition, GMM allows soft clustering, meaning a service Fig. 3. CausalRanker analyses the may be probabilistically assigned to multiple clusters. Con- multi-source time series data of all sersequently, a “weak” anomalous service can be included in vices within each severity group to conmultiple clusters for causal analysis, enabling the discovery struct a causal graph and identify the of cross-cluster causal relationships. root causes. (ErrLog: Error Logs, TraceLat: Latency extracted from Traces).

3.4

CausalRanker: Causal-based Root Cause Analysis

CausalRanker is designed to identify the root causes within each severity group obtained from the previous step. It operates on the belief that the root cause services may not exhibit the strongest anomaly severity but will have cause-effect relationships with affected services. These causal relationships are reflected in their time series data. CausalRanker performs causal inference-based Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE130. Publication date: July 2026.

FSE130:8

Luan Pham, Huong Ha, Xiuzhen Zhang, and Hongyu Zhang

RCA, analysing the multi-source time series of all services in each severity group by examining their anomalous patterns (e.g., periodicity, causality) to rank the root causes in each group. Our CausalRanker ranks services within each cluster via divide-and-conquer causal inference using multi-source time series data. A causal graph is a directed acyclic graph (DAG) where nodes represent time series (e.g., CpuUsage, ErrLog) and directed edges represent causal relationships (e.g., CpuUsage → ErrLog indicates that CPU anomaly causes error log anomaly). This is a standard representation in causal discovery literature [26, 54]. A root cause is identified as a node that has no parents in the causal graph (i.e., an interventional target). Specifically, CausalRanker randomly partitions the given set of time series into smaller groups called chunks. The time series are randomly partitioned into smaller chunks for efficient causal analysis, as analyzing all time series simultaneously is computationally expensive and has been shown to degrade performance [45]. It then applies the Ψ-PC [26], an advanced causal discovery method in the presence of interventions (i.e., failures), to construct causal graphs and identify root causes, referred to as interventional targets, within each chunk. Recursively, CausalRanker groups these root causes and repeats the process until only one chunk remains. Finally, the root causes are ranked based on their (conditional) independence test scores. Lower independence test scores indicate stronger causal influence, suggesting higher likelihood of being the root cause. This provides a ranked list of root cause services for each cluster. Consider the example in Figure 2C and Figure 3, “Service 1", “Service 2", and “Service 5" are grouped in the same severity group. In this example, the root cause is “Service 1 - CPU", which causes an increase in error logs (Service 1-ErrLog) and latency (Service 1 - TraceLat). Therefore, we design CausalRanker to construct the causal graph and identify “Service 1" as the root cause service within this severity group. Our CausalRanker offers two key advancements over existing methods [25, 60]. First, prior studies [25, 60] use all available time series for causal analysis, which has recently been shown to degrade the performance of causal inference [45]. In contrast, CausalRanker leverages SymptomCluster, which groups services with similar severity levels, ensuring that only relevant time series are analyzed for causal relationships to derive the root cause. Second, previous studies rely solely on causal discovery algorithms to identify fine-grained root causes (e.g., RCD [25] employs the Ψ-PC algorithm, while CausalRCA [60] uses DAG-GNN [63]), an approach shown to be ineffective [45]. In TORAI, we introduce FineGrainer, which applies a robust hypothesis testing method to produce more accurate fine-grained root cause rankings within our framework (see Sec. 3.6). 3.5

RankAggregation

In this phase, we combine the ranked lists from SymptomCluster and CausalRanker to produce an aggregated ranked list of coarse-grained root cause services. We first prioritize the clusterlevel ranking provided by SymptomCluster (Sec. 3.3), meaning that services in clusters with higher severity are ranked higher. Second, within each cluster, we follow the ranking provided by CausalRanker (Sec. 3.4), meaning that if Service A is ranked higher than Service B within a cluster, then Service A will also be ranked higher than Service B in the final aggregated list. For instance, given a cluster-level ranking [Group A, Group B], where the internal ranking for Group A is [a,c] and for Group B is [e,p,t], our RankAggregation produces the corresponding aggregated ranking [a,c,e,p,t]. The highest-ranked items in the aggregated list have the highest probability of being the root cause service of the failure. The pseudo code for RankAggregation is presented in Algorithm 1. From our empirical observations, the top severity cluster may contain a single or multiple services. When the top severity cluster contains only one service, it is possible that the root cause service belongs to the second-highest severity cluster. Therefore, we apply CausalRanker to all clusters, not just the top severity cluster, and aggregate the results through this Rank Aggregation step. Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE130. Publication date: July 2026.

TORAI: Multi-source Root Cause Analysis for Blind Spots in Microservice Service Call Graph

CPU tA TraceLat tA = 0.05, tA = 0.8 medtA = 0.04, IQRtA = 1.14

tA

tA = 1.29, tA = 3.09 medtA = 0.37, IQRtA = 1.61

FSE130:9

Fig. 4. The Robustness of FineGrainer to Imprecise Anomaly Detection. At time 𝑡𝐴 , a failure occurs, causing a spike in CPU that eventually leads to increased latency (TraceLat). At 𝑡ˆ𝐴 , TraceLat surpasses the anomaly detection threshold, triggering an anomaly detection. This delayed detection introduces abnormal data (outliers) into the normal period of the CPU time series. The median and interquartile range (IQR) demonstrate greater robustness to these outliers compared to the 𝜇 and 𝜎.

In cases where the root cause does not exhibit strong anomalies (e.g., a code defect that only manifests in downstream services), TORAI ranks the affected services at the top. This allows quicker identification of actual root causes, instead of troubleshooting all possible services. For example, if a code defect causes correlated failures in services A and B, TORAI ranks A and B as top candidates, enabling operators to inspect these services and trace back to the true root cause promptly. Algorithm 1 Pseudo-code of RankAggregation Require: a ranked list 𝑅𝑐𝑙𝑢𝑠𝑡𝑒𝑟 from SymptomCluster (Sec. 3.3), function CausalRanker (Sec. 3.4) Return: a ranked list of coarse-grained root cause services 𝑅𝑠 1: function RankAggregation(𝑅𝑐𝑙𝑢𝑠𝑡𝑒𝑟 , CausalRanker) 2: 𝑅𝑠 ← empty list 3: for 𝑐𝑙𝑢𝑠𝑡𝑒𝑟 ∈ 𝑅𝑐𝑙𝑢𝑠𝑡𝑒𝑟 do // loop from the highest to lowest ranked cluster // get the corresponding ranked list from CausalRanker of this cluster 4: 𝑅𝑖 ← CausalRanker(cluster) 5: for 𝑠 ∈ 𝑅𝑖 do 6: 𝑅𝑠 ← 𝑅𝑠 appends 𝑠 7: return 𝑅𝑠 3.6

FineGrainer

After identifying the coarse-grained root cause services, we propose FineGrainer to determine fine-grained root cause indicators (e.g., specific metrics or logs indicating the root cause of failure) for each service, which better assists operators in diagnosing underlying issues [29, 33, 44]. FineGrainer performs hypothesis testing on time series indicators to detect significant distribution changes after the anomaly detection time [29, 44]. Our key insight is that root cause indicators exhibit significant distributional changes after a failure occurs. We perform hypothesis testing where the null hypothesis 𝐻 0 states that the time series distribution remains unchanged after 𝑡ˆ𝐴 . Specifically, for the time series 𝑥 of each indicator (i.e., the time series data corresponding to each metric, log template, or trace), FineGrainer learns the median (𝑚𝑒𝑑) and interquartile range (𝐼𝑄𝑅) of the time series 𝑥 prior to the anomaly, spanning from 𝑡 0 to 𝑡ˆ𝐴 . It then measures how significantly each time series 𝑥 deviates from its expected central tendency during the failure period. This deviation is quantified as 𝑎𝑡 = 𝑥𝑡 − 𝑚𝑒𝑑 /𝐼𝑄𝑅. All values of 𝑎𝑡 across the time step 𝑡 during the failure period are consolidated to yield 𝛾 = max𝑡ˆ𝐴 ≤𝑡 <𝑡ˆ𝐴 +𝑇 𝑎𝑡 . Higher 𝛾 values provide stronger evidence against 𝐻 0 , indicating greater likelihood that the indicator is the root cause. We prefer median and IQR over mean and standard deviation for analysing fine-grained root cause indicators. Prior work using mean and standard deviation [29] can degrade when anomaly detection times 𝑡ˆ𝐴 are imprecise, as outliers from the actual failure period contaminate the normal period statistics. Median and IQR are more robust to such outliers (see Figure 4). 4

Experimental Results

This section addresses the following research questions: Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE130. Publication date: July 2026.

FSE130:10

Luan Pham, Huong Ha, Xiuzhen Zhang, and Hongyu Zhang

Table 1. Properties of collected datasets(#service, #metric, #log, #trace: number of services, metrics, log templates, and trace operations, per case. #fault: number of fault types).

Name #service #metric #log #trace #fault #cases Online Boutique 11 77 33±9 17 6 90 Sock Shop 11 74 67±53 6 90 Train Ticket 64 376 163±44 148.3±26 6 90

• RQ1: How effective is TORAI in coarse-grained RCA? • RQ2: How effective is TORAI in fine-grained RCA? • RQ3: How efficient is TORAI in performing RCA? • RQ4: How do TORAI’s core components contribute to its overall performance? • RQ5: How does TORAI perform in real-world scenarios? 4.1

Datasets

We deploy three widely used microservice benchmark systems, namely Online Boutique [9], Sock Shop [11], and Train Ticket [14], on a Kubernetes cluster featuring five worker nodes, all configured with their default settings. Online Boutique is an e-commerce platform comprising 11 services that facilitate tasks such as browsing items, adding items to a user’s cart, and making orders. Sock Shop, another online shopping system, consists of 11 services communicating via HTTP. Train Ticket is a ticket booking system that has 64 services, making it one of the largest benchmark microservice systems. Compared to Online Boutique and Sock Shop, Train Ticket has a complex design, various types of invocations and many log templates. The three benchmark microservice systems are widely used to evaluate RCA performance in existing literature [23, 25, 30, 56, 58, 60, 61, 68]. To simulate user interactions, we Kubernetes Cluster customize the provided load generservice 3 ators of these systems to emulate a Front-end Load Generators service 2 load of 40-50 requests per second Data Hog CPU across all services. To gather mety leak Collector or em M service3Multi-source Fault rics, we employ Prometheus [10], service 4 service 1 database Temeletry Data Injectors along with cAdvisor [1] and the Istio service mesh [6], to monitor Fig. 5. Illustration of our setup for the microservice systems and and collect application-level and the multi-source telemetry data collection. resource-level metrics. For log collection, we deploy Datadog Vector [8] and Loki [5] to gather and aggregate logs from all service instances, storing them in Elasticsearch [2]. Traces are gathered using Jaeger [7], with data sent to Elasticsearch for storage. Our telemetry data collection setup is depicted in Fig. 5, similar to existing works [23, 25, 28, 30, 32, 56–58, 60–62, 68]. We inject six common faults: CPU hog (CPU), memory leak (MEM), disk IO stress (DISK), socket stress (SOCKET), network delay (DELAY), and packet loss (LOSS) into key services of each benchmark system. In particular, we inject faults into five Sock Shop services (user, catalogue, orders, payment, and carts), five Online Boutique services (email, currency, recommendation, product, checkout), and five Train Ticket services (order, route, auth, train, travel). These services play an important role in their respective systems, as issues with their performance quickly impact the overall health of the system [25, 28, 60]. Firstly, we let the systems run normally for ten minutes to gather normal metrics, logs, and traces. Then, we follow the existing practice [25, 28, 58, 60, 61] to inject faults into the running services. For CPU, MEM, DISK, and SOCKET, we use stress-ng [12] to stress the container resource. For DELAY and LOSS, we use tc [13] to manipulate the container traffic. For each combination of fault type and targeted service, we repeat the fault injections and Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE130. Publication date: July 2026.

TORAI: Multi-source Root Cause Analysis for Blind Spots in Microservice Service Call Graph

FSE130:11

data collections three times, resulting in a total of 90 collected failure cases for each benchmark microservice system. Table 1 presents the statistics summarizing the collected data. Blind spots in the benchmark systems. Despite their widespread use for benchmarking, these systems have several blind spots (i.e., services without traces) by default. In the Online Boutique system, 7 out of 11 services are instrumented with tracing, leaving 4 blind spots. The Sock Shop system is not instrumented at all, meaning no traces are available to construct a service call graph. In the Train Ticket system, 27 out of 64 services are instrumented with tracing, resulting in 37 blind spots. Existing RCA methods that rely on call graphs [28, 62, 65] will be unable to diagnose root causes in these blind spots, as they do not appear in the call graph. 4.2

Evaluation Metrics

Following existing works [25, 44, 45], we use two standard evaluation metrics: 𝐴𝐶@𝑘 and 𝐴𝑣𝑔@𝑘 to measure the RCA performance. Given a set of failure cases A, 𝐴𝐶@𝑘 is calculated as follows, Í 𝑎 𝑎 1 ∑︁ 𝑖<𝑘 𝑅 [𝑖] ∈ 𝑉𝑟𝑐 𝐴𝐶@𝑘 = , (1) 𝑎 𝑎∈𝐴 |𝐴| 𝑚𝑖𝑛(𝑘, |𝑉𝑟𝑐 |) where 𝑅𝑎 [𝑖] is the 𝑖th ranking result for the failure case 𝑎 by an RCA method, and 𝑉𝑟𝑐𝑎 is the true root cause set of case 𝑎. 𝐴𝐶@𝑘 represents the probability the top 𝑘 results of the given method include the true root causes. Its values range from 0 to 1, with higher values indicating better performance. Í 𝐴𝑣𝑔@𝑘, which shows the overall RCA performance, is measured as 𝐴𝑣𝑔@𝑘 = 𝑘1 𝑘𝑗=1 𝐴𝐶@𝑗. Due to space constraints, we also refer to 𝐴𝐶@1, 𝐴𝐶@3, and 𝐴𝑣𝑔@5 as T1, T3, and A5, respectively. We run all experiments on Linux servers each with 8 CPUs and 16GB RAM. In addition, we repeat each experiment five times and report the average results to minimize the impact of randomness. We use one-way ANOVA to assess overall differences among methods and pairwise t-tests for comparing individual method pairs. We report results as statistically significant when 𝑝 < 0.05. 4.3

Baselines

We select nine RCA baselines from previous studies for performance comparison with our proposed multi-source RCA method, namely: PDiagnose [24], HeMiRCA [69], CausalRCA [60], MicroCause [37], RCD [25], CIRCA [29], BARO [44], MicroRank [61], and TraceRCA [30]. Detailed information of these methods is as follows: • PDiagnose [24]: PDiagnose is a multi-source RCA method that transforms metrics, logs, and traces into time series and determines root causes through voting. It relies on traces to determine the root cause services, and metrics to derive fine-grained root causes. • HeMiRCA [69]: HeMiRCA is a multi-source RCA method, which relies on the monotonic correlation between metrics and trace-based anomaly scores. HeMiRCA first measures trace-based anomaly scores and then exploits the correlations between metrics and the trace anomaly scores to rank the suspicious metrics and microservices. However, HeMiRCA does not use logs. • CausalRCA [60]: CausalRCA constructs the causal graph from time series derived from metrics data using DAG-GNN [63], a gradient-based causal structure learning method. Then it employs PageRank to rank the root causes from the estimated graph. • MicroCause [37]: MicroCause uses PCMCI [49] to construct the causal graph. Then, it applies temporal cause-oriented random walk to rank the root causes from the estimated causal graph. • RCD [25]: RCD adopts a divide-and-conquer strategy to partition time series into chunks. Then, it uses the Ψ-PC [26] to build causal graphs and identify root causes within each chunk. Recursively, it combines these root causes and iterates until only one chunk remains. • CIRCA [29]: CIRCA relies on a provided call graph to construct a causal graph. It then uses regression-based hypothesis testing analysis to identify root causes. Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE130. Publication date: July 2026.

FSE130:12

Luan Pham, Huong Ha, Xiuzhen Zhang, and Hongyu Zhang

• BARO [44]: BARO uses a nonparametric hypothesis testing technique based on median and IQR to measure the change of metrics time series after the failure time and rank the root causes. • MicroRank [61]: MicroRank is a trace-based RCA approach that combines personalized PageRank and Spectrum method to identify suspicious root causes from the collected trace data. • TraceRCA [30]: TraceRCA uses spectrum analysis to identify the root cause services, based on the insight that a service with more abnormal and fewer normal traces passing through it is more likely to be the root cause. For the baselines HeMiRCA, CausalRCA, MicroCause, RCD, CIRCA, BARO, MicroRank, and TraceRCA, we use their publicly available implementation and default hyperparameter settings suggested in their respective papers. We verified their correctness of the obtained source code by reproducing the presented results in the original and related papers. For PDiagnose, we follow previous works [24, 62, 65] to implement it since its source code is unavailable. Furthermore, recent multi-source RCA methods [28, 31, 62, 65, 66] exhibit limitations that prevent us from adopting them as baselines. Specifically, some methods [28, 31, 65, 66] require labeled training data, which is unavailable to us, while [62] requires manual effort to integrate trace_id into every log line. In contrast, our method does not require labeled data or heavy instrumentation into the systems. It is important to note that four of our selected baselines (CausalRCA, MicroCause, RCD, BARO) are metric-based methods that do not require call graphs or trace data. They take time series as input and perform RCA using causal discovery or statistical analysis. These methods represent state-of-the-art metric-based RCA approaches in the literature. 4.4

RQ1. Effectiveness in Coarse-grained Root Cause Analysis

In this section, we evaluate the coarse-grained RCA performance of our proposed TORAI and the RCA baseline methods on all three datasets. Tables 2, 3, and 4 report the overall performance of all methods on the Online Boutique, Sock Shop, and Train Ticket datasets at coarse-grained level, respectively. We calculate the accuracy for each type of fault: CPU hog (CPU), memory leak (MEM), disk IO stress (DISK), socket stress (SOCKET), network delay (DELAY), and packet loss (LOSS). Additionally, we report the AVERAGE scores to present the overall performance across fault types and data sources. We perform statistical analysis on the results using the t-tests to check the pairwise differences among all RCA methods. We bold the best result iff the statistical tests report a significant difference (𝑝 < 0.05) compared to others. In the tables, the first column indicates the data sources used for the methods in the second column. We draw the following observations: (1) TORAI performs the best on all three datasets. For example, on the Online Boutique dataset, TORAI achieves an average T1, T3, and A5 score of 0.83, 0.95, and 0.93 when diagnosing root cause service, while metric-based CausalRCA achieves 0.21, 0.68, and 0.6, respectively. On the Sock Shop dataset, TORAI achieves the averages of 0.84, 0.96, and 0.94 for T1, T3, and A5, respectively, while the multi-source version of RCD reaches 0.39, 0.73, and 0.66. On the Train Ticket dataset, the best average A5 scores of CausalRCA, RCD, and CIRCA are 0.43, 0.64, and 0.46, respectively. Our TORAI beats them with a large margin, achieving an A5 of 0.89. (2) TORAI effectively integrates multi-source data, surpassing most baselines. For example, on the Train Ticket dataset, when incorporating metrics, logs, and traces, TORAI achieves the T1, T3, and A5 scores of 0.77, 0.92, and 0.89, respectively. In contrast, other RCA methods like CausalRCA, RCD, CIRCA, and MicroCause perform worse because their designs are less effective. These methods typically attempt to build causal graphs from all time series data and rely on scoring techniques like PageRank, random walk, or hypothesis testing, without leveraging clustering or severity scoring as TORAI does. Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE130. Publication date: July 2026.

TORAI: Multi-source Root Cause Analysis for Blind Spots in Microservice Service Call Graph

FSE130:13

Table 2. RCA performance of TORAI and baselines on the Online Boutique dataset, across six fault types. The best results are in bold iff the t-test reported a significant difference compared to other baselines (𝑝 < 0.05). CPU MEM DISK SOCKET DELAY LOSS AVERAGE T1 T3 A5 T1 T3 A5 T1 T3 A5 T1 T3 A5 T1 T3 A5 T1 T3 A5 T1 T3 A5 BARO 0.47 0.80 0.72 0.93 1.00 0.99 1.00 1.00 1.00 0.60 0.87 0.83 0.47 0.67 0.63 0.53 0.60 0.64 0.67 0.82 0.80 CausalRCA 0.20 0.60 0.53 0.33 0.87 0.77 0.07 0.67 0.52 0.27 0.67 0.61 0.20 0.73 0.61 0.20 0.53 0.53 0.21 0.68 0.60 CIRCA 0.73 0.93 0.88 0.67 0.93 0.89 0.80 0.87 0.85 0.67 0.87 0.83 0.47 0.80 0.75 0.67 0.93 0.88 0.67 0.89 0.85 Metric MicroCause 0.20 0.33 0.33 0.07 0.20 0.23 0.27 0.40 0.37 0.27 0.40 0.37 0.00 0.07 0.07 0.00 0.07 0.11 0.14 0.25 0.25 RCD 0.87 1.00 0.94 0.67 0.87 0.81 0.73 0.87 0.81 0.80 0.93 0.91 0.27 0.60 0.52 0.20 0.53 0.44 0.59 0.80 0.74 BARO 0.00 0.00 0.00 0.00 0.07 0.07 0.07 0.13 0.12 0.00 0.07 0.09 0.00 0.00 0.00 0.07 0.13 0.11 0.02 0.07 0.06 CausalRCA 0.00 0.13 0.15 0.07 0.13 0.16 0.00 0.00 0.12 0.00 0.13 0.13 0.00 0.27 0.25 0.07 0.67 0.52 0.02 0.22 0.22 CIRCA 0.00 0.20 0.24 0.07 0.27 0.27 0.13 0.27 0.32 0.13 0.27 0.32 0.00 0.07 0.13 0.07 0.13 0.19 0.07 0.20 0.25 Log MicroCause 0.27 0.67 0.57 0.13 0.40 0.39 0.31 0.69 0.68 0.13 0.47 0.47 0.00 0.07 0.11 0.33 0.47 0.51 0.20 0.46 0.46 RCD 0.00 0.07 0.05 0.00 0.07 0.05 0.00 0.07 0.04 0.00 0.13 0.11 0.07 0.13 0.12 0.27 0.33 0.32 0.06 0.13 0.12 BARO 0.40 0.80 0.72 0.07 0.33 0.28 0.27 0.47 0.49 0.53 0.60 0.61 0.00 0.07 0.07 0.27 0.60 0.51 0.26 0.48 0.45 CausalRCA 0.30 0.37 0.36 0.23 0.33 0.36 0.07 0.20 0.19 0.20 0.27 0.27 0.00 0.03 0.03 0.03 0.30 0.30 0.14 0.25 0.25 CIRCA 0.40 0.60 0.55 0.27 0.53 0.53 0.33 0.67 0.59 0.33 0.53 0.52 0.07 0.13 0.12 0.73 0.80 0.83 0.36 0.54 0.52 MicroCause 0.17 0.33 0.27 0.00 0.00 0.04 0.10 0.40 0.40 0.00 0.20 0.32 0.00 0.00 0.00 0.00 0.00 0.00 0.05 0.16 0.17 Trace RCD 0.47 0.49 0.49 0.31 0.38 0.38 0.17 0.43 0.37 0.24 0.52 0.46 0.00 0.04 0.03 0.04 0.06 0.06 0.21 0.32 0.30 MicroRank 0.00 0.40 0.36 0.00 0.40 0.36 0.00 0.40 0.36 0.00 0.40 0.36 0.00 0.40 0.36 0.00 0.27 0.19 0.00 0.38 0.33 TraceRCA 0.07 0.80 0.69 0.00 0.53 0.57 0.20 0.80 0.72 0.07 0.73 0.67 0.20 0.53 0.57 0.07 0.53 0.49 0.10 0.65 0.62 BARO 0.47 0.80 0.75 0.93 1.00 0.99 1.00 1.00 1.00 0.60 0.80 0.79 0.47 0.67 0.61 0.67 0.67 0.71 0.69 0.82 0.81 CausalRCA 0.13 0.30 0.30 0.12 0.24 0.25 0.00 0.07 0.10 0.00 0.13 0.15 0.00 0.30 0.28 0.07 0.50 0.41 0.09 0.24 0.22 Metric CIRCA 0.00 0.07 0.04 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.07 0.07 0.00 0.02 0.02 + MicroCause 0.29 0.50 0.46 0.07 0.33 0.31 0.21 0.43 0.43 0.27 0.67 0.61 0.07 0.20 0.19 0.13 0.50 0.48 0.17 0.44 0.41 Log RCD 0.79 0.95 0.91 0.45 0.79 0.71 0.77 0.84 0.84 0.91 0.96 0.95 0.36 0.67 0.60 0.29 0.67 0.59 0.60 0.81 0.77 TORAI 0.67 0.80 0.81 0.87 1.00 0.97 0.93 1.00 0.95 0.80 0.87 0.84 0.73 1.00 0.92 0.77 1.00 0.93 0.80 0.95 0.91 BARO 0.47 0.80 0.75 0.93 1.00 0.99 1.00 1.00 1.00 0.60 0.80 0.79 0.47 0.67 0.61 0.67 0.67 0.71 0.69 0.82 0.81 CausalRCA 0.20 0.27 0.27 0.13 0.27 0.29 0.07 0.07 0.09 0.07 0.20 0.23 0.07 0.33 0.31 0.07 0.27 0.28 0.10 0.24 0.25 Metric CIRCA 0.00 0.07 0.04 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.03 0.00 0.01 0.01 + MicroCause 0.50 0.67 0.60 0.25 0.25 0.25 0.33 0.78 0.76 0.50 0.50 0.50 0.00 0.33 0.30 0.14 0.29 0.37 0.29 0.47 0.46 Log PDiagnose 0.00 0.80 0.60 0.00 0.40 0.51 0.00 0.73 0.59 0.00 0.60 0.52 0.20 0.40 0.55 0.00 0.47 0.47 0.03 0.57 0.54 + HeMiRCA 0.43 0.77 0.72 0.23 0.36 0.37 0.89 0.95 0.94 0.20 0.40 0.43 0.33 0.53 0.55 0.40 0.40 0.40 0.41 0.57 0.57 Trace RCD 0.80 1.00 0.94 0.47 0.80 0.71 0.80 1.00 0.95 1.00 1.00 1.00 0.33 0.40 0.51 0.40 0.60 0.60 0.63 0.80 0.79 TORAI 0.92 1.00 0.98 0.67 1.00 0.93 1.00 1.00 1.00 0.87 0.93 0.95 0.67 0.73 0.76 0.87 1.00 0.96 0.83 0.95 0.93

Data Source

Method

(*) T1, T3, and A5 denote AC@1, AC@3, and Avg@5, respectively.

Table 3. RCA performance of TORAI and baselines on the Sock Shop dataset, across six fault types. The best results are in bold iff the t-test reported a significant difference compared to other baselines (𝑝 < 0.05). CPU MEM DISK SOCKET DELAY LOSS AVERAGE T1 T3 A5 T1 T3 A5 T1 T3 A5 T1 T3 A5 T1 T3 A5 T1 T3 A5 T1 T3 A5 BARO 0.00 1.00 0.80 0.20 1.00 0.83 0.00 0.93 0.77 0.00 0.93 0.71 0.00 0.87 0.68 0.20 1.00 0.80 0.07 0.96 0.76 CausalRCA 0.20 0.60 0.55 0.40 0.80 0.75 0.20 0.60 0.55 0.33 0.60 0.60 0.27 0.40 0.43 0.00 0.33 0.32 0.23 0.56 0.53 Metric CIRCA 0.87 1.00 0.97 0.87 0.93 0.95 0.87 0.87 0.89 0.67 1.00 0.92 0.67 0.87 0.85 0.47 0.87 0.81 0.74 0.92 0.90 MicroCause 0.07 0.13 0.16 0.00 0.20 0.23 0.00 0.13 0.12 0.33 0.40 0.44 0.13 0.27 0.28 0.07 0.27 0.24 0.10 0.23 0.25 RCD 0.47 0.73 0.68 0.27 0.40 0.36 0.47 0.67 0.61 0.47 0.87 0.77 0.40 0.73 0.64 0.20 0.40 0.35 0.38 0.63 0.57 BARO 0.20 0.47 0.51 0.20 0.47 0.48 0.13 0.33 0.39 0.13 0.33 0.39 0.20 0.33 0.40 0.13 0.60 0.52 0.17 0.42 0.45 CIRCA 0.13 0.53 0.48 0.00 0.33 0.29 0.07 0.47 0.41 0.00 0.40 0.36 0.07 0.33 0.36 0.20 0.47 0.52 0.08 0.42 0.40 Log CausalRCA 0.10 0.33 0.37 0.10 0.53 0.47 0.23 0.43 0.42 0.20 0.50 0.47 0.10 0.20 0.21 0.47 0.80 0.76 0.20 0.47 0.45 MicroCause 0.50 0.75 0.70 0.29 0.71 0.60 0.00 0.33 0.27 0.13 0.38 0.38 0.00 0.17 0.13 0.00 0.43 0.34 0.15 0.46 0.40 RCD 0.07 0.21 0.18 0.09 0.16 0.14 0.12 0.33 0.28 0.09 0.19 0.17 0.08 0.15 0.13 0.37 0.48 0.46 0.14 0.25 0.23 BARO 0.00 1.00 0.80 0.20 1.00 0.83 0.00 1.00 0.79 0.00 0.93 0.71 0.00 0.80 0.65 0.20 1.00 0.80 0.07 0.96 0.76 CausalRCA 0.20 0.33 0.33 0.47 0.73 0.73 0.20 0.40 0.44 0.07 0.40 0.35 0.20 0.53 0.48 0.20 0.47 0.51 0.22 0.48 0.47 CIRCA 0.00 0.00 0.00 0.07 0.07 0.08 0.00 0.00 0.05 0.00 0.00 0.00 0.00 0.07 0.07 0.00 0.20 0.19 0.01 0.06 0.07 Metric MicroCause 0.33 0.33 0.40 0.17 0.33 0.37 0.00 0.67 0.53 0.25 0.75 0.63 0.20 0.80 0.72 0.00 0.40 0.36 0.16 0.55 0.50 + RCD 0.61 0.77 0.74 0.20 0.49 0.42 0.36 0.64 0.59 0.67 0.89 0.85 0.28 0.67 0.58 0.19 0.92 0.76 0.39 0.73 0.66 Log TORAI 0.93 1.00 0.98 1.00 1.00 1.00 0.93 0.93 0.95 0.84 0.93 0.94 0.75 0.88 0.86 0.60 1.00 0.92 0.84 0.96 0.94

Data Source

Method

(*) T1, T3, and A5 denote AC@1, AC@3, and Avg@5, respectively.

(3) TORAI can diagnose failures in blind spots. PDiagnose and HeMiRCA are multi-source RCA methods but they rely heavily on trace data to construct the service call graph. Consequently, Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE130. Publication date: July 2026.

FSE130:14

Luan Pham, Huong Ha, Xiuzhen Zhang, and Hongyu Zhang

Table 4. RCA performance of TORAI and baselines on the Train Ticket dataset, across six fault types. The best results are in bold iff the t-test reported a significant difference compared to other baselines (𝑝 < 0.05). CPU MEM DISK SOCKET DELAY LOSS AVERAGE T1 T3 A5 T1 T3 A5 T1 T3 A5 T1 T3 A5 T1 T3 A5 T1 T3 A5 T1 T3 A5 BARO 0.47 0.80 0.72 0.93 1.00 0.99 1.00 1.00 1.00 0.60 0.87 0.83 0.47 0.67 0.63 0.53 0.60 0.64 0.67 0.82 0.80 CausalRCA 0.40 0.63 0.59 0.10 0.27 0.24 0.43 0.83 0.75 0.23 0.50 0.45 0.13 0.23 0.21 0.03 0.37 0.33 0.22 0.47 0.43 Metric CIRCA 0.27 0.27 0.28 0.47 0.73 0.68 0.53 0.67 0.64 0.27 0.53 0.52 0.20 0.27 0.28 0.20 0.33 0.35 0.32 0.47 0.46 MicroCause 0.19 0.44 0.40 0.00 0.09 0.07 0.40 0.40 0.40 0.00 0.17 0.15 0.00 0.22 0.13 0.00 0.00 0.07 0.10 0.22 0.20 RCD 0.13 0.13 0.16 0.07 0.07 0.07 0.00 0.07 0.05 0.13 0.33 0.29 0.13 0.13 0.15 0.07 0.07 0.07 0.09 0.13 0.13 BARO 0.00 0.00 0.00 0.00 0.07 0.07 0.07 0.13 0.12 0.00 0.07 0.09 0.00 0.00 0.00 0.07 0.13 0.11 0.02 0.07 0.06 CausalRCA 0.07 0.20 0.22 0.07 0.10 0.09 0.07 0.10 0.12 0.00 0.07 0.08 0.03 0.20 0.15 0.10 0.20 0.20 0.06 0.15 0.14 Log CIRCA 0.13 0.20 0.27 0.07 0.33 0.27 0.07 0.07 0.12 0.07 0.20 0.23 0.13 0.33 0.33 0.13 0.33 0.33 0.10 0.24 0.26 MicroCause 0.00 0.00 0.10 0.20 0.40 0.40 0.14 0.14 0.14 0.13 0.38 0.28 0.11 0.11 0.16 0.11 0.22 0.20 0.12 0.21 0.21 RCD 0.07 0.16 0.14 0.12 0.15 0.14 0.07 0.15 0.13 0.06 0.19 0.17 0.11 0.24 0.21 0.12 0.16 0.15 0.09 0.18 0.16 BARO 0.40 0.80 0.72 0.07 0.33 0.28 0.27 0.47 0.49 0.53 0.60 0.61 0.00 0.07 0.07 0.27 0.60 0.51 0.26 0.48 0.45 CausalRCA 0.07 0.33 0.29 0.07 0.20 0.19 0.20 0.27 0.25 0.00 0.00 0.03 0.00 0.07 0.05 0.27 0.47 0.44 0.10 0.22 0.21 CIRCA 0.13 0.20 0.23 0.20 0.27 0.25 0.20 0.20 0.21 0.13 0.13 0.15 0.20 0.33 0.35 0.13 0.27 0.27 0.17 0.23 0.24 MicroCause 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Trace MicroRank 0.21 0.43 0.34 0.25 0.38 0.33 0.00 0.36 0.27 0.30 0.40 0.36 0.08 0.31 0.23 0.14 0.36 0.30 0.16 0.37 0.31 RCD 0.53 0.87 0.79 0.53 0.67 0.63 0.67 0.67 0.69 0.73 0.80 0.79 0.13 0.27 0.21 0.60 0.73 0.71 0.53 0.67 0.64 TraceRCA 0.64 0.79 0.74 0.63 0.88 0.83 0.64 0.71 0.74 0.60 0.80 0.76 0.85 0.85 0.88 0.57 0.71 0.67 0.66 0.79 0.77 BARO 0.47 0.80 0.75 0.93 1.00 0.99 1.00 1.00 1.00 0.60 0.80 0.79 0.47 0.67 0.61 0.67 0.67 0.71 0.69 0.82 0.81 Metric CIRCA 0.00 0.13 0.09 0.00 0.07 0.08 0.07 0.07 0.09 0.00 0.13 0.15 0.00 0.07 0.05 0.07 0.07 0.09 0.02 0.09 0.09 + MicroCause 0.00 0.00 0.00 0.07 0.07 0.07 0.00 0.00 0.00 0.07 0.07 0.08 0.00 0.00 0.00 0.07 0.13 0.13 0.04 0.05 0.05 Log RCD 0.13 0.27 0.23 0.00 0.07 0.08 0.13 0.27 0.21 0.20 0.33 0.29 0.00 0.13 0.12 0.07 0.13 0.11 0.09 0.20 0.17 TORAI 0.53 0.87 0.80 1.00 1.00 1.00 1.00 1.00 1.00 0.80 0.87 0.87 0.53 0.67 0.64 0.57 0.63 0.67 0.74 0.84 0.83 BARO 0.47 0.80 0.75 0.93 1.00 0.99 1.00 1.00 1.00 0.60 0.80 0.79 0.47 0.67 0.61 0.67 0.67 0.71 0.69 0.82 0.81 Metric CIRCA 0.00 0.07 0.09 0.07 0.13 0.21 0.00 0.07 0.09 0.07 0.13 0.16 0.07 0.07 0.07 0.13 0.20 0.17 0.06 0.11 0.13 + HeMiRCA 0.00 0.07 0.15 0.13 0.27 0.27 0.13 0.20 0.19 0.20 0.47 0.43 0.07 0.33 0.32 0.07 0.13 0.15 0.10 0.25 0.25 Log MicroCause 0.07 0.13 0.11 0.07 0.07 0.07 0.00 0.07 0.11 0.00 0.00 0.03 0.00 0.00 0.00 0.00 0.13 0.16 0.02 0.07 0.08 + PDiagnose 0.60 0.87 0.81 0.40 0.47 0.48 0.33 0.73 0.69 0.33 0.67 0.60 0.87 0.87 0.87 0.33 0.60 0.57 0.48 0.70 0.67 Trace RCD 0.17 0.84 0.72 0.00 0.44 0.39 0.07 0.76 0.62 0.21 0.77 0.66 0.05 0.28 0.25 0.07 0.75 0.60 0.10 0.64 0.54 TORAI 0.60 0.87 0.83 1.00 1.00 1.00 1.00 1.00 1.00 0.73 1.00 0.93 0.56 0.67 0.66 0.73 1.00 0.92 0.77 0.92 0.89

Data Source

Method

(*) CausalRCA exceeds the limit of 2 hours per case. T1, T3, and A5 denote AC@1, AC@3, and Avg@5, respectively.

they cannot perform RCA for the Sock Shop dataset, where there is no trace data to construct the call graph (i.e., all services are blind spots). Meanwhile, TORAI can still perform RCA effectively in the presence of blind spots using metrics and logs without requiring traces to construct a call graph (see Table 3). It is worth noting that metrics and logs are easier to collect as developers do not need to spend much effort to obtain them. Metrics can be automatically obtained by a monitoring system, and logs are naturally produced by the systems for troubleshooting purposes. In contrast, developers need to spend tremendous effort to instrument the system with distributed tracing [52]. (4) On the large Train Ticket system, TORAI outperforms other methods by a significant margin. For example, our method achieves an Avg@5 score of 0.89, while CausalRCA, CIRCA, and RCD achieve their best scores of 0.43, 0.46, and 0.64, respectively. Across all faults, TORAI achieves Avg@5 scores of 0.83, 1, 1, 0.93, 0.66, and 0.92 for CPU, MEM, DISK, SOCKET, DELAY, and LOSS, respectively. This demonstrates the effectiveness of our proposed TORAI, which works not only on small demo systems but also on large systems with many services. 4.5

RQ2. Effectiveness in Fine-grained Root Cause Analysis

In this section, we evaluate the fine-grained RCA performance of our proposed TORAI alongside baseline RCA methods across all three datasets. Tables 5, 6, and 7 report the fine-grained RCA performance of all methods on the Online Boutique, Sock Shop, and Train Ticket datasets. The fine-grained ground truths are derived from the indicators linked to the fault injection operations. For instance, when a CPU hog fault is injected into the order service, the fine-grained ground truth indicator is the "order-cpu" metric. We measure and report the accuracy for each fault type. Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE130. Publication date: July 2026.

TORAI: Multi-source Root Cause Analysis for Blind Spots in Microservice Service Call Graph

FSE130:15

Table 5. Fine-grained RCA performance of TORAI and baselines on the Online Boutique dataset, across six fault types. The best results are in bold iff the t-test reported a significant difference compared to other baselines (𝑝 < 0.05). For all baselines, we select their best setup when taking different data sources. CPU MEM DISK SOCKET DELAY LOSS AVERAGE T1 T3 A5 T1 T3 A5 T1 T3 A5 T1 T3 A5 T1 T3 A5 T1 T3 A5 T1 T3 A5 BARO 0 0.33 0.43 0.2 0.8 0.67 0 0 0 0 0 0 0 0.73 0.6 0.33 0.73 0.65 0.09 0.43 0.39 CausalRCA 0.2 0.53 0.45 0.47 0.67 0.65 0.07 0.27 0.32 0 0.4 0.35 0.27 0.67 0.61 0.13 0.27 0.23 0.19 0.47 0.44 0.13 0.47 0.44 0.47 0.67 0.64 0.87 0.87 0.87 0 0.07 0.08 0.47 0.73 0.67 0.33 0.6 0.61 0.38 0.57 0.55 CIRCA MicroCause 0.13 0.4 0.32 0 0 0.04 0.04 0.31 0.29 0 0.07 0.09 0 0 0 0 0 0 0.03 0.13 0.12 HeMiRCA 0.25 0.27 0.29 0.1 0.15 0.15 0.77 0.82 0.82 0.07 0.07 0.1 0.13 0.23 0.25 0.4 0.4 0.4 0.29 0.32 0.34 0.07 0.07 0.07 0.27 0.33 0.31 0 0 0 0 0 0 0.33 0.33 0.4 0.2 0.2 0.2 0.15 0.16 0.16 RCD TORAI 0.13 0.73 0.63 0.4 0.67 0.6 1 1 1 0 0.07 0.35 0.6 0.67 0.65 0.8 0.87 0.85 0.49 0.67 0.68 Method

Table 6. Fine-grained RCA performance of TORAI and baselines on the Sock Shop dataset, across six fault types. The best results are in bold iff the t-test reported a significant difference compared to others (𝑝 < 0.05). For all baselines, we select their best setup when taking different data sources. CPU MEM DISK SOCKET DELAY LOSS AVERAGE T1 T3 A5 T1 T3 A5 T1 T3 A5 T1 T3 A5 T1 T3 A5 T1 T3 A5 T1 T3 A5 BARO 0 0.87 0.68 0.2 1 0.8 0 0 0 0 0 0.01 0 0.87 0.68 0.2 1 0.8 0.07 0.62 0.50 CausalRCA 0.27 0.63 0.58 0.3 0.67 0.59 0.07 0.17 0.17 0 0 0.03 0.33 0.57 0.56 0.23 0.37 0.37 0.2 0.37 0.36 CIRCA 0.4 0.8 0.72 0.8 1 0.95 0.67 0.67 0.67 0 0.2 0.16 0.53 0.8 0.79 0.6 0.87 0.85 0.5 0.69 0.67 MicroCause 0 0.09 0.07 0 0.08 0.05 0.09 0.18 0.18 0.07 0.2 0.23 0 0 0 0 0 0 0.03 0.08 0.08 RCD 0.11 0.15 0.15 0.05 0.16 0.15 0 0 0 0.13 0.19 0.2 0.24 0.27 0.27 0.03 0.03 0.03 0.09 0.13 0.13 TORAI 0.6 0.93 0.85 0.93 1 0.99 0.67 0.67 0.67 0 0.13 0.32 0.73 0.73 0.73 0.6 0.6 0.6 0.59 0.68 0.69 Method

Table 7. Fine-grained RCA performance of TORAI and baselines on the Train Ticket dataset, across six fault types. The best results are in bold iff the t-test reported a significant difference compared to other baselines (𝑝 < 0.05). For all baselines, we select their best setup when taking different data sources. CPU MEM DISK SOCKET DELAY LOSS AVERAGE T1 T3 A5 T1 T3 A5 T1 T3 A5 T1 T3 A5 T1 T3 A5 T1 T3 A5 T1 T3 A5 BARO 0.07 0.33 0.32 0.33 0.8 0.75 1 1 1 0 0 0.01 0.47 0.67 0.63 0.53 0.53 0.53 0.4 0.56 0.54 CausalRCA 0.33 0.47 0.49 0 0.03 0.02 0.43 0.73 0.68 0 0 0 0.13 0.13 0.13 0.03 0.27 0.22 0.15 0.27 0.26 CIRCA 0.01 0.05 0.05 0 0.23 0.15 0.05 0.16 0.12 0.02 0.12 0.07 0 0 0.09 0 0.09 0.11 0.01 0.11 0.10 HeMiRCA 0 0.07 0.05 0.07 0.07 0.07 0 0 0 0 0 0 0.07 0.13 0.12 0 0 0 0.02 0.05 0.04 MicroCause 0.06 0.11 0.11 0 0.02 0.02 0.12 0.12 0.12 0 0.01 0.04 0 0 0.03 0 0 0 0.03 0.04 0.05 RCD 0.03 0.07 0.05 0 0 0 0 0 0 0.2 0.23 0.23 0 0 0 0 0 0 0.04 0.05 0.05 TORAI 0.2 0.6 0.51 0.27 1 0.85 1 1 1 0.07 0.47 0.43 0.6 0.6 0.6 0.6 0.73 0.68 0.46 0.73 0.68 Method

Additionally, we perform statistical analysis using t-tests to check for pairwise differences among all RCA methods. (1) TORAI outperforms all baselines in fine-grained RCA. In the Online Boutique dataset (Table 5), TORAI achieves an average accuracy of 0.68 when diagnosing the root cause indicators. Meanwhile, BARO, the second-best method, achieves the average accuracy of 0.39. This is because BARO only uses hypothesis testing to infer the root cause while our TORAI also uses causal information and cluster the anomaly severity behaviour. (2) Hypothesis testing-based and causal inference-based methods deliver competitive fine-grained RCA performance. BARO, CausalRCA, and CIRCA demonstrate strong fine-grained RCA capabilities. On the Train Ticket dataset, BARO achieves an average A5 score of 0.54 (the second highest), while our TORAI achieves a score of 0.68. Notably, BARO relies solely on hypothesis testing without considering causal relationships between time series and their anomaly severity. CausalRCA consistently performs well in fine-grained RCA, ranking among the top methods (TORAI, CIRCA, BARO). It constructs a causal graph using DAG-GNN and applies PageRank to identify root causes. CIRCA achieves an average A5 score of 0.67 on the Sock Shop dataset, while TORAI achieves a slightly higher score of 0.69. CIRCA combines causal inference with regression-based hypothesis testing. Notably, TORAI integrates causal inference in its CausalRanker, hypothesis testing in FineGrainer, and clustering in SymptomCluster, enabling it to achieve the best overall performance. Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE130. Publication date: July 2026.

FSE130:16

4.6

Luan Pham, Huong Ha, Xiuzhen Zhang, and Hongyu Zhang

RQ3. Efficiency in Root Cause Analysis

Table 8. Efficiency comparison.

(1) TORAI analyzes the root cause of failOnline Sock Train Method ures in seconds. TORAI takes an average of 12.9, Boutique Shop Ticket 15.63, and 20.59 seconds to perform RCA on the BARO 0.01 0.01 0.01 Online Boutique, Sock Shop, and Train Ticket 2.55 14.75 TraceRCA datasets, respectively. In comparison, MicroCause RCD 3.91 2.96 20.87 takes 177.09, 129.13, and 3935.85 seconds while 3.24 1.26 61.53 PDiagnose RCD, completes RCA in 3.91, 2.96, and 20.87 secMicroRank 77.56 91.04 onds, respectively. 62.08 55.38 206.02 HeMiRCA (2) TORAI’s efficiency remains stable reCIRCA 4.65 3.5 312.77 gardless of the system’s scale. TORAI takes an MicroCause 177.09 129.13 3935.85 average of 12.9 seconds to perform RCA on the CausalRCA 179.45 397.12 (*) Online Boutique system (11 services), and 20.59 TORAI 12.9 15.63 20.59 seconds on the Train Ticket system (64 services). In contrast, most baselines tend to slow down con- (*) Exceeding the limit of 2 hours per case. siderably when applied to larger systems. For in- (-) Sock Shop’s trace data is unavailable. stance, CausalRCA takes 180 seconds to handle Online Boutique but 20200 seconds to handle a case Train Ticket (100 times slower). This is because CausalRCA use deep neural network to perform causal discovery on all input time series, leading to a huge number of possible edges to analyse. BARO is the fastest method since it uses simple statistical method applying on time series data individually. In summary, our TORAI can efficiently analyze root causes using multi-source telemetry data with minimal overhead, providing results in seconds even for microservice systems with a large number of services. Table 9. Ablation Study of TORAI Online Boutiq Sock Shop Train Ticket T1 T3 A5 T1 T3 A5 T1 T3 A5 CausalRanker 0.6 0.81 0.77 0.39 0.73 0.66 0.09 0.2 0.17 Metric, SeverityScorer 0.8 0.93 0.91 0.8 0.95 0.93 0.72 0.83 0.81 Log FineGrainer 0.69 0.82 0.81 0.07 0.96 0.76 0.69 0.82 0.81 TORAI 0.8 0.95 0.91 0.84 0.96 0.94 0.74 0.84 0.83 Metric, CausalRanker 0.63 0.8 0.79 0.1 0.64 0.54 Log, SeverityScorer 0.82 0.94 0.92 - 0.71 0.87 0.85 Trace FineGrainer 0.69 0.82 0.81 - 0.69 0.82 0.81 TORAI 0.83 0.95 0.93 - 0.77 0.92 0.89 Data Source

Method

(-) Sock Shop’s trace data is unavailable.

4.7

RQ4. Ablation Study

In this section, we conduct an ablation analysis to assess the contribution of each constituent component to TORAI’s overall performance. Table 9 presents the effectiveness comparison of TORAI and its three components: SeverityScorer, CausalRanker, and FineGrainer, across three datasets. Specifically, SeverityScorer ranks the root cause services based on the average value of the severity vector [𝜌𝑚 , 𝜌𝑙 , 𝜌𝑡𝑐 ]. CausalRanker uses RCD [25] to rank the root cause services from multi-source time series data, (Sec. 4.3). Meanwhile, FineGrainer performs hypothesis testing on multi-source time series data and ranks root causes based on the magnitude of 𝛾. The ablation results from Table 9 demonstrate that all constituent components have positive effects on TORAI’s overall performance. For instance, on the Train Ticket dataset, TORAI achieves T1, T3, and A5 accuracies of 0.77, 0.92, and 0.89, respectively, while SeverityScorer achieves 0.71, 0.87, and 0.85, and FineGrainer achieves 0.69, 0.82, and 0.81. SeverityScorer relies on the anomaly severity Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE130. Publication date: July 2026.

TORAI: Multi-source Root Cause Analysis for Blind Spots in Microservice Service Call Graph

FSE130:17

score to rank the root causes without considering the causal structure between services, hence its performance is lower than TORAI. Similarly, FineGrainer also relies purely on the hypothesis testing results of the time series individually without taking into account the causal relationship among them, resulting in poorer performance compared to our TORAI. Meanwhile, in the same scenario, CausalRanker achieves scores of 0.1, 0.64, and 0.54, respectively. It has relatively poor performance by itself because it solely focuses on causal learning without considering the anomaly severity score. 4.8

RQ5. How Does TORAI Perform in Real-World Scenarios?

In this section, we further evaluate TORAI on 10 real-world failures collected from a production system. We also demonstrate the ability of TORAI in diagnosing code-level failures in Section 4.10. 4.8.1 System and Data Description. The real-world failures are collected from a production microservice system of a major Internet service provider. The system consists of multiple components (e.g., load balancers, web/app servers, databases) categorized into five classes: OSB (Oracle Service Bus), service, DB, Docker, and OS. The system serves more than 50 million users. To monitor the system, engineers collect multi-source telemetry data. Due to confidentiality concerns, only metrics and traces are available, whereas logs are not collected. The dataset contains 10 failures with ground-truth labels for root cause components (e.g., services, databases). Each failure originates from a single component and propagates to others due to complex dependencies, causing their telemetry data to become abnormal. Regarding the collected telemetry data, metrics Table 10. Example of collected data. are recorded at multiple layers. For service instances (a) Metrics. (i.e., Docker containers), 10 metrics are collected (e.g., time docker1_cpu docker1_mem thread_total, fgct, session_used, cpu_used, mem_17336 0.216 0.352 used, etc.). For physical Linux host machines, 50 met17337 0.115 0.401 ric types are collected (e.g., Agent_ping, Buffers_17338 0.116 0.386 used, Zombie_Process, ss_total, etc.). For Oracle 17339 0.118 0.398 databases, 47 database-specific metrics are collected (b) Traces. (e.g., UndoTbs_Pct, Used_Tbs_Size, User_Commit, time id service callType elapsedTime tnsping_result_time). The dataset contains over 17336 cf8b.. osb_001 OSB 497 169,000 traces in total. The data was collected since 17337 60cf.. os_021 CSF 102 May 31, 2020, spanning 15 days. Each fault lasts for 17338 4a93.. docker_003 Remote.. 1310 5 minutes, and the root cause is confirmed by engi- 17338 fe23.. product.. ListProd.. 56 neers. There are five fault types: CPU exhaustion, network delay, packet loss, container network error, and database failures. Fault locations are in databases, hosts, or service instances (containers). Table 10 presents examples of metrics and trace data. Each column in the metric data represents a time series, and each row contains a value for a specific timestamp. Each trace has a unique ID, timestamp, corresponding service, call type, and elapsed time. 4.8.2 Experimental setup. In the collected dataset, logs are not available; we only have metrics and traces. We transform metrics and traces into time series as described in Sec. 3.1. To ensure fairness, we feed the processed time series into TORAI along with four state-of-the-art RCA methods, namely BARO, RCD, CIRCA, and PDiagnose. We use AC@1, AC@3, AC@5, and Avg@5 as evaluation metrics to assess the ability to identify the root cause service of the failures. 4.8.3 Results. Table 11 presents the experimental results, demonstrating that TORAI outperforms state-of-the-art baselines. Notably, TORAI achieves 100% in AC@3, meaning it can correctly rank Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE130. Publication date: July 2026.

FSE130:18

Luan Pham, Huong Ha, Xiuzhen Zhang, and Hongyu Zhang

Table 11. Real-world RCA Performance.

Method BARO [44] CIRCA [29] PDiagnose[24] RCD [25] TORAI

AC@1 0.70 0.20 0.60 0.10 0.60

AC@3 0.90 0.50 0.80 0.20 1.00

AC@5 0.90 0.70 0.80 0.20 1.00

Avg@5 0.82 0.46 0.74 0.16 0.88

the root cause within the top three recommendations with perfect accuracy. On average, TORAI attains an Avg@5 score of 0.88, outperforming BARO (0.82), PDiagnose (0.74), RCD (0.16), and CIRCA (0.46). RCD performs poorly because it processes all time series data indiscriminately, a limitation previously highlighted in [45]. Similarly, CIRCA constructs a causal graph using the PC algorithm on all time series data, leading to suboptimal performance. PDiagnose, which relies on traces to identify the root causes without incorporating metrics, achieves relatively strong results. Meanwhile, BARO employs hypothesis testing and delivers competitive performance; however, it does not account for causal relationships or severity symptoms. TORAI achieves the highest overall performance by effectively grouping abnormal services and conducting precise causal analysis, resulting in superior accuracy compared to the baselines. A replicable notebook is available in our replication package.

Accuracy

AC@1 1.0 0.8 0.6 0.4 0.2 0.0

AC@3

Avg@5

(a) Online Boutique

(b) Train-Ticket

0 10 20 30 40 50 60 70 80 90 100 Blind Spot Percentage (%)

0 10 20 30 40 50 60 70 80 90 100 Blind Spot Percentage (%)

Fig. 6. Sensitivity analysis of TORAI performance under varying blind spot percentages (i.e., proportion of services without trace data) on (a) Online Boutique and (b) Train Ticket systems. TORAI mostly maintains robust diagnostic accuracy (AC@1, AC@3, and Avg@5) across different blind spot levels.

4.9

How Robust Is TORAI Under Varying Blind Spots?

To evaluate TORAI’s sensitivity to varying levels of blind spots (i.e., trace unavailability), we conduct a sensitivity analysis by randomly removing traces from 0% to 100% of the instrumented services in 10% increments on the Online Boutique and Train Ticket systems1 . Figure 6 presents TORAI’s performance across these blind spot levels. We draw the following observations: (1) TORAI remains robust on the Online Boutique system. As shown in Figure 6(a), AC@1 ranges from 64.4% to 88.9% across all blind spot levels, with AC@3 remaining consistently above 82%. Notably, even at 100% blind spots (i.e., no trace data available), TORAI achieves 80% AC@1 and 95% AC@3, demonstrating that metrics and logs alone provide sufficient diagnostic signals for this system. 1 The Sock Shop system has no traces by default (i.e., 100% blind spots).

Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE130. Publication date: July 2026.

TORAI: Multi-source Root Cause Analysis for Blind Spots in Microservice Service Call Graph

FSE130:19

(2) TORAI shows graceful degradation on the larger Train Ticket system. Figure 6(b) reveals that the 64-service Train Ticket system exhibits more sensitivity to blind spots, with Avg@5 dropping from 89% (at 0%) to a minimum of 71.6% (at 70%). However, the degradation is gradual rather than catastrophic. Interestingly, performance recovers at 100% blind spots (74% AC@1, 84% AC@3). We found that, when traces are entirely unavailable, the method relies purely on metrics and logs without the potential noise introduced by incomplete or partial trace information. (3) AC@3 is more stable than AC@1 across both systems. While AC@1 fluctuates more noticeably, AC@3 remains relatively high (above 74% for Train Ticket and above 82% for Online Boutique) across all blind spot levels. This indicates that the true root cause is consistently ranked within the top three candidates, which is practical for engineers who can efficiently investigate a small set of services. These findings are further supported by TORAI’s strong performance on the Sock Shop system (Table 3), which has 100% blind spots by default and where TORAI achieves 0.84 AC@1 and 0.94 Avg@5. In summary, TORAI’s multi-source design enables reliable RCA even under the presence of blind spots, addressing a key practical concern for systems where full distributed tracing instrumentation is infeasible. 4.10

Diagnosing code-level faults

info : Grpc . AspNetCore . Server . CallHandler [7] Error status code ' FailedPrecondition '

Event Count

While we follow established practice to with detail 'Can ' t access cart storage . System . OverflowException : Value was either benchmark on resource and network-related too large or too small for an Int32 . at System . Number . ThrowOverflowException () faults, TORAI can diagnose other fault types at cartservice . cartstore . RedisCartStore . AddItemAsync ( String , String , String ) (e.g., code-level faults [18]) as long as the issues in / RedisCartStore . cs : line 54 ' raised . manifest symptoms in telemetry data. For example, if a code-level fault produces observable symptoms, such as exception logs or erroneous Fig. 7. Stack traces as Fine-grained root cause of the code-level faults indicating line 54 of file RedisCarttraces, our method can detect the root cause serStore.cs is the root cause. vices with these indicators to assist engineers in diagnosing the actual fault more efficiently. To demonstrate this capability, we 100 modified the cartservice in the Online 80 Boutique system to inject a code-level fault, namely Incorrect parameter values, 60 Normal Logs as described in [18]. Empirical studies Stack Traces 40 show that incorrect parameter values 20 Fault Time are among the five most common faults in real-world projects and that injected 0 0 200 400 600 800 1000 faults can realistically simulate actual Time (seconds) software faults [18]. After injecting this Fig. 8. The frequency of normal logs/stack traces of cartservice. fault, the cartservice became unstable, resulting in failures in all its callers. The front-end services reported error codes, and the resource usage of the cartservice spiked. Other services, such as recommendation and shipping, were also affected. The root cause was traced back to the cartservice (RedisCartStore.cs:54), with the fine-grained root cause indicator being the stack trace presented in Figure 7. Our FineGrainer precisely identifies the stack trace in the cartservice as a fine-grained root cause indicator, as its frequency during the failure period deviates significantly from the expected median, which is 0. The frequency of normal logs and stack traces for the cartservice is shown in Figure 8. This demonstration is included in our replication package. Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE130. Publication date: July 2026.

FSE130:20

5

Luan Pham, Huong Ha, Xiuzhen Zhang, and Hongyu Zhang

Threats to Validity

We now discuss threats to the validity of our study and the means we undertook to mitigate these threats. The internal threat concerns the implementation, where bugs may affect the reliability of the results. To address this, we reused the public code for the baselines and performed experiments to replicate their results, ensuring their correctness. To avoid the influence of randomness, we repeated the experiments five times and reported the average results together with the statistical analysis. The construct threat concerns the evaluation metrics. To address this, we used standard evaluation metrics extensively employed in the literature to evaluate the performance of RCA methods [25, 44, 45]. The external threat concerns the deployment of microservice applications and data collection strategies. To address this, we follow established practice to deploy these systems and collect data, as described in Section 4.1. These systems are widely recognized in academia for testing microservices-related methods [23, 25, 32, 56–58, 60, 61, 68]. Another potential threat concerns the assumptions underlying our method. We assume that root cause indicators exhibit significant distributional changes, which is supported by prior works [44, 51] and recent theoretical analysis [39]. We do not assume the root cause service always exhibits the highest raw anomaly score, instead, severity-based clustering serves as a coarse candidate selection mechanism, with final decisions made by CausalRanker and FineGrainer. The conclusion threat is tied to the fault types as microservices can experience various faults [36]. We acknowledge that different software applications and faults could have different properties and failure propagation mechanisms, which could impact the conclusions in this paper. However, we believe that these fault types are representative since they have been used in many previous studies [23, 25, 30, 32, 58, 60, 68]. Furthermore, our method is specifically designed for microservices systems; however, if a software system has multiple components interacting with each other, our method can be adapted to identify the root cause of failures in these systems. Expanding TORAI to work with other types of systems, such as distributed database systems, could be a potential future work. There may be other threats related to the underlying tools, our extracted data, that we have not considered here. To enable exploration of these potential threats and to facilitate replication and extension of our work, we make available our tools and data. 6

Conclusion

In this work, we propose TORAI, a novel unsupervised fine-grained RCA method that leverages multi-source telemetry data to accurately identify both coarse-grained root cause services and fine-grained root cause indicators of failures in microservice systems. The novelty of our proposed TORAI lies in the effective combination of unsupervised techniques, including clustering, causal analysis, and hypothesis testing, which enables the integration of multi-source telemetry data for RCA without requiring full trace coverage or labeled data, addressing limitations of existing RCA approaches. Extensive experiments on three benchmark systems and real-world failures demonstrate TORAI’s superiority in both effectiveness and efficiency compared to existing methods. 7

Data Availability

We have integrated TORAI into our open-source benchmark RCAEval [47], which can be accessed on GitHub at https://github.com/phamquiluan/rcaeval. Additionally, an immutable artifact for TORAI is available on Figshare [43], together with the experimental datasets [41]. Acknowledgments This research was supported by the Australian Research Council Discovery Project (DP220103044), Google Cloud Credit for Research, and RMIT Race Hub via the RMAS scheme. Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE130. Publication date: July 2026.

TORAI: Multi-source Root Cause Analysis for Blind Spots in Microservice Service Call Graph

FSE130:21

References [1] 2025. Container Advisor - an open-source tool to monitor containers. Retrieved Apr 14, 2026 from https://github.com/ google/cadvisor [2] 2025. Elasticsearch Log Monitoring - Centralized Log Management. Retrieved Apr 14, 2026 from https://www.elastic.co/ [3] 2025. Gaussian Mixture Model. Retrieved Apr 14, 2026 from https://scikit-learn.org/stable/modules/generated/sklearn. mixture.GaussianMixture.html [4] 2025. Google - Site Reliability Engineering. Retrieved Apr 14, 2026 from https://sre.google/sre-book/monitoringdistributed-systems/ [5] 2025. Grafana Loki - A horizontally scalable, highly available, multi-tenant log aggregation system. Retrieved Apr 14, 2026 from https://grafana.com/oss/loki/ [6] 2025. The Istio service mesh. Retrieved Apr 14, 2026 from https://istio.io/ [7] 2025. Jaeger: open source, distributed tracing platform. Retrieved Apr 14, 2026 from https://www.jaegertracing.io/ [8] 2025. A lightweight, ultra-fast tool for building observability pipelines. Retrieved Apr 14, 2026 from https://vector.dev/ [9] 2025. Online Boutique is a cloud-first microservices demo application. Retrieved Apr 14, 2026 from https://github. com/GoogleCloudPlatform/microservices-demo [10] 2025. An open-source monitoring and alerting toolkit. Retrieved Apr 14, 2026 from https://prometheus.io/ [11] 2025. Sock Shop - A Microservices Demo Application. Retrieved Apr 14, 2026 from https://github.com/microservicesdemo/microservices-demo [12] 2025. Stress test for Computer system. Retrieved Apr 14, 2026 from https://manpages.ubuntu.com/manpages/bionic/ man1/stress-ng.1.html [13] 2025. Traffic Control. Retrieved Apr 14, 2026 from https://man7.org/linux/man-pages/man8/tc.8.html [14] 2025. Train Ticket Benchmark System. Retrieved Apr 14, 2026 from https://github.com/FudanSELab/train-ticket [15] 2025. Uber Microservice Systems. Retrieved Apr 14, 2026 from https://www.uber.com/en-AU/blog/up-portablemicroservices-ready-for-the-cloud [16] Sachin Ashok, Vipul Harsh, Brighten Godfrey, Radhika Mittal, Srinivasan Parthasarathy, and Larisa Shwartz. 2024. TraceWeaver: Distributed Request Tracing for Microservices Without Application Modification. In Proceedings of the ACM SIGCOMM 2024 Conference. 828–842. [17] Algirdas Avizienis, J-C Laprie, Brian Randell, and Carl Landwehr. 2004. Basic concepts and taxonomy of dependable and secure computing. IEEE transactions on dependable and secure computing 1, 1 (2004), 11–33. [18] Domenico Cotroneo, Luigi De Simone, Pietro Liguori, Roberto Natella, and Nematollah Bidokhti. 2019. How bad can a bug get? an empirical analysis of software failures in the openstack cloud computing platform. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 200–211. [19] Luca Giamattei, Antonio Guerriero, Roberto Pietrantuono, Stefano Russo, Ivano Malavolta, Tanjina Islam, Madalina Dinga, Anne Koziolek, Snigdha Singh, Martin Armbruster, et al. 2024. Monitoring tools for DevOps and microservices: A systematic grey literature review. Journal of Systems and Software 208 (2024), 111906. [20] Shenghui Gu, Guoping Rong, Tian Ren, He Zhang, Haifeng Shen, Yongda Yu, Xian Li, Jian Ouyang, and Chunan Chen. 2023. TrinityRCL: Multi-Granular and Code-Level Root Cause Localization Using Multiple Types of Telemetry Data in Microservice Systems. IEEE Transactions on Software Engineering 49, 5 (2023), 3071–3088. [21] Pinjia He, Jieming Zhu, Zibin Zheng, and Michael R Lyu. 2017. Drain: An online log parsing approach with fixed depth tree. In 2017 IEEE international conference on web services (ICWS). IEEE, 33–40. [22] Shilin He, Pinjia He, Zhuangbin Chen, Tianyi Yang, Yuxin Su, and Michael R Lyu. 2021. A survey on automated log analysis for reliability engineering. ACM computing surveys (CSUR) 54, 6 (2021), 1–37. [23] Zilong He, Pengfei Chen, Yu Luo, Qiuyu Yan, Hongyang Chen, Guangba Yu, and Fangyuan Li. 2022. Graph based Incident Extraction and Diagnosis in Large-Scale Online Systems. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering (ASE’22). 1–13. [24] Chuanjia Hou, Tong Jia, Yifan Wu, Ying Li, and Jing Han. 2021. Diagnosing performance issues in microservices with heterogeneous data source. In 2021 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking (ISPA/BDCloud/SocialCom/SustainCom). IEEE, 493–500. [25] Azam Ikram, Sarthak Chakraborty, Subrata Mitra, Shiv Saini, Saurabh Bagchi, and Murat Kocaoglu. 2022. Root Cause Analysis of Failures in Microservices through Causal Discovery. In Advances in Neural Information Processing Systems (NeurIPS’22), Vol. 35. 31158–31170. [26] Amin Jaber, Murat Kocaoglu, Karthikeyan Shanmugam, and Elias Bareinboim. 2020. Causal Discovery from Soft Interventions with Unknown Targets: Characterization and Learning. In Advances in Neural Information Processing Systems (NeurIPS’20), Vol. 33. 9551–9561.

Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE130. Publication date: July 2026.

FSE130:22

Luan Pham, Huong Ha, Xiuzhen Zhang, and Hongyu Zhang

[27] Andrea Janes, Xiaozhou Li, and Valentina Lenarduzzi. 2023. Open tracing tools: Overview and critical comparison. Journal of Systems and Software (2023), 111793. [28] Cheryl Lee, Tianyi Yang, Zhuangbin Chen, Yuxin Su, and Michael R Lyu. 2023. Eadro: An End-to-End Troubleshooting Framework for Microservices on Multi-source Data. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 1750–1762. [29] Mingjie Li, Zeyan Li, Kanglin Yin, Xiaohui Nie, Wenchi Zhang, Kaixin Sui, and Dan Pei. 2022. Causal Inference-Based Root Cause Analysis for Online Service Systems with Intervention Recognition. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD’22). 3230–3240. [30] Zeyan Li, Junjie Chen, Rui Jiao, Nengwen Zhao, Zhijun Wang, Shuwei Zhang, Yanjun Wu, Long Jiang, Leiqin Yan, Zikai Wang, Zhekang Chen, Wenchi Zhang, Xiaohui Nie, Kaixin Sui, and Dan Pei. 2021. Practical Root Cause Localization for Microservice Systems via Trace Analysis. In 2021 IEEE/ACM 29th International Symposium on Quality of Service (IWQOS’21). 1–10. [31] Zeyan Li, Nengwen Zhao, Mingjie Li, Xianglin Lu, Lixin Wang, Dongdong Chang, Xiaohui Nie, Li Cao, Wenchi Zhang, Kaixin Sui, Yanhua Wang, Xu Du, Guoqing Duan, and Dan Pei. 2022. Actionable and Interpretable Fault Localization for Recurring Failures in Online Service Systems. In Proceedings of the 30th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE’22). [32] Jinjin Lin, Pengfei Chen, and Zibin Zheng. 2018. Microscope: Pinpoint Performance Issues with Causal Graphs in Micro-service Environments. In International Conference on Service-Oriented Computing. Springer, 3–20. [33] Chenghao Liu, Wenzhuo Yang, Himanshu Mittal, Manpreet Singh, Doyen Sahoo, and Steven CH Hoi. 2023. PyRCA: A Library for Metric-based Root Cause Analysis. arXiv preprint arXiv:2306.11417 (2023). [34] Fengrui Liu, Yang Wang, Zhenyu Li, Rui Ren, Hongtao Guan, Xian Yu, Xiaofan Chen, and Gaogang Xie. 2022. MicroCBR: Case-Based Reasoning on Spatio-temporal Fault Knowledge Graph for Microservices Troubleshooting. In International Conference on Case-Based Reasoning. Springer, 224–239. [35] Shutian Luo, Huanle Xu, Chengzhi Lu, Kejiang Ye, Guoyao Xu, Liping Zhang, Jian He, and Chengzhong Xu. 2022. An in-depth study of microservice call graph and runtime performance. IEEE Transactions on Parallel and Distributed Systems 33, 12 (2022), 3901–3914. [36] Leonardo Mariani, Cristina Monni, Mauro Pezzé, Oliviero Riganelli, and Rui Xin. 2018. Localizing faults in cloud systems. In 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST). IEEE, 262–273. [37] Yuan Meng, Shenglin Zhang, Yongqian Sun, Ruru Zhang, Zhilong Hu, Yiyin Zhang, Chenyang Jia, Zhaogang Wang, and Dan Pei. 2020. Localizing Failure Root Causes in a Microservice through Causality Inference. In IEEE/ACM 28th International Symposium on Quality of Service (IWQoS’20). 1–10. [38] Odigos. 2025. Solving the Pitfalls of Distributed Tracing in Real-World Microservices. https://odigos.io/blog/solvingpitfalls-of-distributed-tracing-in-real-world-microservices. Accessed on September 2, 2025. [39] William Roy Orchard, Nastaran Okati, Sergio Hernan Garrido Mejia, Patrick Blöbaum, and Dominik Janzing. 2025. Root Cause Analysis of Outliers with Missing Structural Knowledge. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=7Nxq4RQApu [40] Eva Patel and Dharmender Singh Kushwaha. 2020. Clustering cloud workloads: K-means vs gaussian mixture model. Procedia computer science 171 (2020), 158–167. [41] Luan Pham. 2026. Datasets for "TORAI: Multi-Source Root Cause Analysis for Blind Spots in Microservice Service Call Graph". (4 2026). doi:10.6084/m9.figshare.31925976.v1 [42] Luan Pham. 2026. Graph-Free Root Cause Analysis. arXiv preprint arXiv:2601.21359 (2026). [43] Luan Pham. 2026. Source code of "TORAI: Multi-Source Root Cause Analysis for Blind Spots in Microservice Service Call Graph". (4 2026). doi:10.6084/m9.figshare.31938495.v1 [44] Luan Pham, Huong Ha, and Hongyu Zhang. 2024. BARO: Robust Root Cause Analysis for Microservices via Multivariate Bayesian Online Change Point Detection. Proceedings of the ACM on Software Engineering 1, FSE (2024), 2214–2237. [45] Luan Pham, Huong Ha, and Hongyu Zhang. 2024. Root Cause Analysis for Microservice System based on Causal Inference: How Far Are We?. In The 39th IEEE/ACM International Conference on Automated Software Engineering (ASE 2024). [46] Luan Pham, Victor Nicolet, Joey Dodds, Hui Guan, and Daniel Kroening. 2026. EventADL: Open-Box Anomaly Detection and Localization Framework for Events in Cloud-Based Service Systems. Proceedings of the ACM on Software Engineering 3, FSE (2026). [47] Luan Pham, Hongyu Zhang, Huong Ha, Flora Salim, and Xiuzhen Zhang. 2025. RCAEval: a benchmark for root cause analysis of microservice systems with telemetry data. In Companion Proceedings of the ACM on Web Conference 2025. 777–780. [48] Raphael Rouf, Mohammadreza Rasolroveicy, Marin Litoiu, Seema Nagar, Prateeti Mohapatra, Pranjal Gupta, and Ian Watts. 2024. InstantOps: A Joint Approach to System Failure Prediction and Root Cause Identification in Microserivces Cloud-Native Applications. In Proceedings of the 15th ACM/SPEC International Conference on Performance Engineering.

Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE130. Publication date: July 2026.

TORAI: Multi-source Root Cause Analysis for Blind Spots in Microservice Service Call Graph

FSE130:23

119–129. [49] Jakob Runge, Peer Nowack, Marlene Kretschmer, Seth Flaxman, and Dino Sejdinovic. 2019. Detecting and quantifying causal associations in large nonlinear time series datasets. Science Advances 5, 11 (2019). [50] Gideon Schwarz. 1978. Estimating the Dimension of a Model. The Annals of Statistics 6, 2 (1978), 461 – 464. [51] Huasong Shan, Yuan Chen, Haifeng Liu, Yunpeng Zhang, Xiao Xiao, Xiaofeng He, Min Li, and Wei Ding. 2019. 𝜖Diagnosis: Unsupervised and Real-Time Diagnosis of Small-Window Long-Tail Latency in Large-Scale Microservice Platforms. In The World Wide Web Conference (WWW’19). 3215–3222. [52] Junxian Shen, Han Zhang, Yang Xiang, Xingang Shi, Xinrui Li, Yunxi Shen, Zijian Zhang, Yongxiang Wu, Xia Yin, Jilong Wang, et al. 2023. Network-centric distributed tracing with DeepFlow: Troubleshooting your microservices in zero code. In Proceedings of the ACM SIGCOMM 2023 Conference. 420–437. [53] Jacopo Soldani and Antonio Brogi. 2022. Anomaly Detection and Failure Root Cause Analysis in (Micro) Service-Based Cloud Applications: A Survey. Comput. Surveys 55, 3 (2022). [54] Peter Spirtes, Christopher Meek, and Thomas Richardson. 1995. Causal Inference in the Presence of Latent Variables and Selection Bias. In Proceedings of the Eleventh Conference on Uncertainty in Artificial Intelligence (UAI’95). 499–506. [55] Lingzhi Wang, Nengwen Zhao, Junjie Chen, Pinnong Li, Wenchi Zhang, and Kaixin Sui. 2020. Root-cause metric location for microservice systems via log anomaly detection. In 2020 IEEE international conference on web services (ICWS). IEEE, 142–150. [56] Qing Wang, Larisa Shwartz, Genady Ya. Grabarnik, Vijay Arya, and Karthikeyan Shanmugam. 2021. Detecting Causal Structure on Cloud Application Microservices Using Granger Causality Models. In IEEE 14th International Conference on Cloud Computing (CLOUD’21). 558–565. [57] Li Wu. 2022. Automatic performance diagnosis and recovery in cloud microservices. TU Berlin (Germany). [58] Li Wu, Johan Tordsson, Jasmin Bogatinovski, Erik Elmroth, and Odej Kao. 2021. Microdiag: Fine-grained performance diagnosis for microservice systems. In 2021 IEEE/ACM International Workshop on Cloud Intelligence. IEEE, 31–36. [59] Shuaiyu Xie, Jian Wang, Hanbin He, Zhihao Wang, Yuqi Zhao, Neng Zhang, and Bing Li. 2026. TVDiag: A Task-oriented and View-invariant Failure Diagnosis Framework for Microservice-based Systems with Multimodal Data. ACM Trans. Softw. Eng. Methodol. 35, 2, Article 40 (2026), 39 pages. doi:10.1145/3734868 [60] Ruyue Xin, Peng Chen, and Zhiming Zhao. 2023. CausalRCA: Causal inference based precise fine-grained root cause localization for microservice applications. Journal of Systems and Software 203 (2023), 111724. [61] Guangba Yu, Pengfei Chen, Hongyang Chen, Zijie Guan, Zicheng Huang, Linxiao Jing, Tianjun Weng, Xinmeng Sun, and Xiaoyun Li. 2021. Microrank: End-to-end latency issue localization with extended spectrum analysis in microservice environments. In Proceedings of the Web Conference (WWW’21). 3087–3098. [62] Guangba Yu, Pengfei Chen, Yufeng Li, Hongyang Chen, Xiaoyun Li, and Zibin Zheng. 2023. Nezha: Interpretable Fine-Grained Root Causes Analysis for Microservices on Multi-modal Observability Data. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 553–565. [63] Yue Yu, Jie Chen, Tian Gao, and Mo Yu. 2019. DAG-GNN: DAG Structure Learning with Graph Neural Networks. In Proceedings of the 36th International Conference on Machine Learning (ICML’19), Vol. 97. 7154–7163. [64] Lei Zhang, Zhiqiang Xie, Vaastav Anand, Ymir Vigfusson, and Jonathan Mace. 2023. The Benefit of Hindsight: Tracing {Edge-Cases} in Distributed Systems. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). 321–339. [65] Shenglin Zhang, Pengxiang Jin, Zihan Lin, Yongqian Sun, Bicheng Zhang, Sibo Xia, Zhengdan Li, Zhenyu Zhong, Minghua Ma, Wa Jin, et al. 2023. Robust failure diagnosis of microservice system through multimodal data. IEEE Transactions on Services Computing (2023). [66] Yingying Zhang, Zhengxiong Guan, Huajie Qian, Leili Xu, Hengbo Liu, Qingsong Wen, Liang Sun, Junwei Jiang, Lunting Fan, and Min Ke. 2021. CloudRCA: A root cause analysis framework for cloud computing platforms. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 4373–4382. [67] Lecheng Zheng, Zhengzhang Chen, Jingrui He, and Haifeng Chen. 2024. MULAN: Multi-modal Causal Structure Learning and Root Cause Analysis for Microservice Systems. In Proceedings of the ACM on Web Conference 2024. 4107–4116. [68] Xiang Zhou, Xin Peng, Tao Xie, Jun Sun, Chao Ji, Wenhai Li, and Dan Ding. 2018. Fault analysis and debugging of microservice systems: Industrial survey, benchmark system, and empirical study. IEEE Transactions on Software Engineering 47, 2 (2018), 243–260. [69] Zhouruixing Zhu, Cheryl Lee, Xiaoying Tang, and Pinjia He. 2024. HeMiRCA: Fine-Grained Root Cause Analysis for Microservices with Heterogeneous Data Sources. ACM Transactions on Software Engineering and Methodology (2024).

Received 2026-02-16; accepted 2026-03-24

Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE130. Publication date: July 2026.

Related documents

• Requirements After the First Edit: Mining Late Requirement Emergence and Rework in Real-World Coding-Agent Sessions#639748