Beyond Fault Localization: A Trajectory-Level Study of LLM Agents for Microservice Root Cause Analysis

Beyond Fault Localization: A Trajectory-Level Study of LLM Agents for Microservice Root Cause Analysis Qisheng Lu∗ , Aoyang Fang∗ , Junjielong Xu∗ , Jin’ao Shang† , Songhan Zhang∗ , Yifan Yang∗ , Xiaochuan Yan∗ , and Pinjia He∗

arXiv:2608.21310v1 [cs.SE] 21 Aug 2026

∗ The Chinese University of Hong Kong, Shenzhen, China

{qishenglu, aoyangfang, songhanzhang, yifanyang6, xiaochuanyan}@link.cuhk.edu.cn siyuexi@foxmail.com, hepinjia@cuhk.edu.cn † Xi’an Jiaotong University, Xi’an, China jinao s@stu.xjtu.edu.cn

Abstract—Existing evaluations of automated root cause analysis (RCA) for microservices uniformly assess diagnostic performance by endpoint correctness: whether a method localizes the responsible service. Although this criterion enables direct comparison, it provides no indication of the evidentiary basis for a diagnosis or the propagation route linking the fault source to the observed symptoms. Both are necessary for an on-call site reliability engineer (SRE) to assess whether an automated diagnosis warrants action. We therefore treat RCA as an observable diagnostic process. Our trajectory-level framework evaluates agent executions against manually curated service-level fault-propagation paths. Applied to a public microservice RCA benchmark, the framework supports an analysis of 3,500 diagnostic trajectories, characterizing where agents investigate and how they incorporate retrieved telemetry. Our results reveal a disconnect between answer correctness and diagnostic quality. An agent may correctly localize the fault source yet fail to reconstruct its propagation across the affected services; recovering this path is harder and more discriminative than identifying the affected services alone. Successful investigations tend to remain on the fault-impact surface, act on retrieved evidence, and exhibit a broader query repertoire as the search reaches deeper. Unsuccessful investigations instead depart from the relevant route, fail to use decisive observations, or stagnate in shallow, repetitive queries. Erroneous diagnoses reduce to three recurring evidence-handling failures: decisive evidence is omitted, retrieved evidence is misinterpreted, or unsupported inference substitutes for missing evidence. We operationalize this failure taxonomy as D IAG G UARD, a twostage defense-in-depth architecture: grounding systematically surveys the available observations before localization, and verification audits the diagnosis against them. In an independent validation setting with a different model, benchmark, and service topology, the prototype raises Acc@1 from 43.5% to 52.5%. These findings indicate that trajectory-level evaluation can expose limitations hidden by final-answer metrics and that recurring failure patterns provide an actionable basis for improving automated RCA. Index Terms—Root Cause Analysis, Microservices, LLM Agents, AIOps

Service-Level Fault Propagation Path

Root Cause Service

Agent Reasoning Trajectory

Affected Service

Observed Symptom

The RCA agent queries telemetry, revises hypotheses, synthesizes evidence, and reports the final answer.

Trajectory-Level Diagnosis Steps 4–7 Investigation

Rank services by error count → affected service ranks first

Steps 9–13 Hypothesis

Hypothesize that the top-ranked service is the root cause

Steps 16–30 Telemetry Query

Query telemetry to verify the hypothesis

Steps 37–40 Evidence Omission

Reach the root cause service but leave its metrics uninspected

Steps 45–47 Wrong Diagnosis

Recheck the symptom → report final answer: affected service

Outcome-Level Metrics Final Answer: Affected Service

Acc@k MRR MAP

…

Fig. 1: Outcome-level evaluation captures only an RCA diagnosis’s final answer and scores; the diagnostic trajectory explains the wrong diagnosis: the agent reaches the fault source but omits decisive evidence before reporting the final answer.

coupled services and trigger cascading failures. When such incidents occur, on-call site reliability engineers (SREs) must diagnose the fault source from distributed traces, logs, and metrics before remediation can begin [2]. Because remediation acts on a diagnosis, the SRE needs not only a root-cause label but also the evidence and reasoning that make it trustworthy. Automated root cause analysis (RCA) has produced a large body of methods [3]–[5]. Benchmark-driven studies usually turn RCA into a service-localization task: predict which service caused the incident. Top-k localization accuracy (Acc@k), mean reciprocal rank, and mean average precision make this task comparable across methods, but they collapse the diagnostic process into a final label [6]. That collapse matters because the same correct label can be supported by strong or weak evidence, and a wrong label is only useful for continuous I. I NTRODUCTION improvement if its failed reasoning can be inspected (Fig. 1). Microservice architectures underpin large-scale cloud appli- Across incidents, inspected reasoning failures can become cations [1], where local faults can propagate across loosely reusable failure modes and guide better automated RCA design. Tool-using LLM systems make this missing diagnostic Pinjia He and Aoyang Fang are co-corresponding authors.

1

process newly visible. General-purpose coding and deepresearch agents now plan over many steps, invoke tools, and synthesize evidence [7], [8]. LLM-based RCA methods follow the same pattern by querying telemetry and revising hypotheses during diagnosis [9], [11], [12], [21]. Because these agents act through observable steps, their runs leave trajectories: records of agent messages, telemetry queries, and tool-returned evidence. Such trajectories can expose how an agent investigates an incident, not only what answer it returns. However, raw trajectories cannot be evaluated directly as process-level evidence. Prior RCA work studies causal graphs and fault propagation for localization [3]–[5], and agent studies analyze tool traces [7], [8]. However, causal graphs in prior work capture statistically inferred service dependencies, not the incident-specific route a fault actually traveled; agent-trace analyses, in turn, lack a domain-level ground truth to evaluate against. To evaluate diagnostic trajectories in RCA, what is still missing is a reference for what the diagnostic process should have recovered. For our setting, that reference is a manually validated service-level fault propagation path: the route by which a fault moves from the faulty service to the observed symptom through service-dependency hops. RCABench [13], the microservice benchmark used for our main study, provides final root-cause labels but not these process-level annotations. Even with such paths, trajectories still need normalization because heterogeneous agents expose different trace formats and tool interfaces. Without both process-level ground truth and normalized steps, existing metrics cannot say whether an agent navigated relevant evidence, what operational purpose each step served, or why wrong diagnoses recurred. We address this gap with a trajectory-level characterization framework built around service-level fault propagation paths. These paths define the fault-impact surface: the services and edges through which a fault reaches the observed symptoms. To compare agent behavior against this surface, we normalize each run into agent messages, telemetry queries, and tool-returned evidence. The framework applies outcome-level metrics to all six frameworks and adds step-level trajectory analysis where the shared telemetry interface makes tool traces comparable. The resulting analysis first checks whether a run finds the root service and reconstructs the propagation path. The analysis then explains the route the agent took by mapping how the agent navigates the fault-impact surface, labeling SRE diagnostic intent and coding the failure modes behind wrong diagnoses. Together, these steps make RCA trajectories measurable, comparable, and useful for mining recurring diagnostic failures. With this framework in place, the study pursues two questions. The first is characterization: what separates a sound diagnostic process from a failing one? We pursue it at three progressively deeper levels (RQ1–RQ3), moving from the outcome and cost of localization, to how an agent navigates the fault-impact surface and what diagnostic intent drives each query, to why wrong diagnoses occur. We compare six frameworks under a shared backbone model and isolate the backbone-model effect by rerunning one framework on a second model. The trajectory-level analysis (RQ2–RQ3) focuses on

the four frameworks whose tool calls are SQL queries against a shared telemetry store, because only they expose comparable action traces. The second is intervention (RQ4–RQ5): can the characterized failures become a usable improvement? We distill them into D IAG G UARD, a defense-in-depth architecture that wraps the diagnostic agent in a grounding defense (the Grounder) and a verification defense (the Verifier). On a heldout setting, we test whether D IAG G UARD raises localization and isolate which defense drives the gain. Our study yields three core findings. 1) Localization difficulty is uneven, accuracy falls as the causal chain deepens, and reconstructing how a fault propagates between services is harder than identifying which services it reaches, even when the fault source is correctly localized. 2) Success and failure hinge on the diagnostic process itself: effective runs stay on the fault-impact surface and act on the evidence they retrieve, and the breadth of their intent repertoire sets how deep that search reaches; failing runs drift off the surface, leave decisive evidence unused, or circle in shallow loops. 3) Wrong diagnoses collapse into a small, evidence-handling failure taxonomy, and experience distilled from these modes remains effective on a held-out backbone model, dataset, and service topology, carrying the characterization into intervention. This paper makes the following contributions: • Annotated propagation-path dataset. We manually annotate service-level fault propagation paths for a microservice RCA benchmark, supplying process-level ground truth beyond final root-cause labels. • Trajectory-level characterization framework. We introduce a framework that normalizes heterogeneous RCA trajectories and makes them comparable against the annotated propagation paths. • Process-level empirical study. We analyze 3,500 reasoning trajectories across seven framework–model configurations (six frameworks under a shared model, one rerun on a second model), linking diagnostic behavior and failure modes to final RCA outcomes. • Failure-mode-derived intervention. We use the mined failure modes to design D IAG G UARD and validate it on a held-out backbone model, dataset, and service topology, showing that trajectory-level failure analysis can guide an intervention beyond the original characterization setting. We will release the annotated propagation-path dataset, trajectory-analysis pipeline, the failure-mode codebook, and the D IAG G UARD defense prompts to support reproducible process-level research on automated RCA. II. R ELATED W ORK Our work sits at the intersection of microservice RCA methods, tool-using LLM agents, and agent trajectory analysis. We review traditional and LLM-based RCA (Sections II-A and II-B), the general-purpose agents we repurpose for RCA (Section II-C), and trajectory-analysis techniques (Section II-D). Across these lines, general-purpose deep-research and coding agents have not been systematically compared with RCAspecific designs on end-to-end RCA, and diagnostic processes

2

are either not exposed or not evaluated against process-level ground truth; our study addresses both gaps.

For failure understanding, Bouzenia and Pradel [7] and Zhang et al. [51] analyze trajectories of software-engineering and multi-agent systems. Closest to ours, three concurrent efforts A. Traditional RCA in Microservice Systems target RCA agents: Riddell et al. [52] study post-detection RCA, Automated RCA research spans four families of methods: where externally extracted anomalies are analyzed by agents statistical correlation [14], [15], graph-based fault propaga- over a static system graph, Kim et al. [53] diagnose trajectories tion [16], [17], causal inference [10], [18]–[20], [22], and from a single O PEN RCA architecture across multiple models, learning-based frameworks over multimodal observability and STAR [54] repairs failing diagnostic stages post-hoc. We data [23]–[25]. These methods return a root-cause ranking instead evaluate the end-to-end RCA capability of multiple real rather than a reasoning process; LLM-based and agent-based agent frameworks and backbone models directly over telemetry methods have recently emerged as a more process-oriented against manually annotated propagation-path ground truth and direction [5]. turn the mined failure modes into a preventive intervention (Section VI). B. LLM-Based RCA for Microservices The application of large language models to microservice RCA has produced two distinct lines of work, following the workflow-versus-agent distinction [26]. Workflow approaches. These methods preprocess multimodal telemetry into a structured prompt for single-pass inference. They target classification and confidence estimation [9], [27], [28], explanation and remediation [29]–[33], or enrich the prompt with structured knowledge such as mined code or offline causal graphs [21], [34]. A single snapshot of system state, however, cannot capture the progressive, hypothesis-driven nature of expert fault diagnosis. Agentic approaches. These methods reason over multi-turn loops of diagnostic tool calls, telemetry queries, and hypothesis refinement. M ABC [11] and O PEN RCA [12] are representative iterative RCA agents, which we adopt as the two RCA-specific frameworks in our study (Section III-A); other designs add SOPderived tools, self-consistency, recursive cross-alert memory, or coordinated SRE-role agents [35]–[39], and a cross-cutting line fine-tunes the backbone model specifically for RCA [40], [41], unlike the off-the-shelf agents our study examines. Despite this progress, these methods are assessed almost exclusively at the outcome level, leaving the diagnostic process unexamined for lack of process-level ground truth.

III. S TUDY S ETUP This section establishes the study setup used throughout the paper: the evaluated agents, backbone models, and benchmarks. Fig. 2 gives the study design and the RQ1–RQ3 analysis pipeline; the validation (RQ4–RQ5) follows in Section VI-B. A. Evaluated Agent Frameworks We select six frameworks spanning two categories: generalpurpose investigative agents and RCA-specific agents. • T HINK D EPTH . AI [45]: open-source deep-research framework, top-ranked before April 2026 [8], with a self-balancing ReAct-style loop [55]. • AIQ [43]: open-source deep-research framework, also topranked, with a requery-and-refine ReAct-style loop. • TASK W EAVER [44]: Code-first plan-executor that decomposes a task into executable plans and tool-backed actions. • C LAUDE C ODE : commercial terminal-style coding agent. • O PEN RCA [12]: RCA-specific agent driving an LLM-based loop over telemetry to produce root-cause predictions • M ABC [11]: RCA-specific two-stage multi-agent design in which agents analyze incident symptoms, then aggregate conclusions via a blockchain-inspired consensus step. B. Backbone Models

C. Advanced Agents

Our study uses the following three backbone models (USD per 1M input/output tokens, as of 2026-02-15): • qwen3.5-plus-2026-02-15: $0.115/$0.688. • claude-sonnet-4.6: $3/$15. • doubao-seed-2.0-pro-2026-02-15: $0.44/$2.22.

General-purpose deep-research and coding agents now match or exceed domain-specific methods on knowledge-intensive tasks, raising the question of whether investigative architecture matters more than domain specialization. The DeepResearch Bench [8], [42] provides the first systematic evaluation of deepresearch agents (DRAs); on the open-source side, AIQ [43], TASK W EAVER [44], and T HINK D EPTH . AI [45] are representative frameworks, which we repurpose for RCA under the same harness (Section III-A). Commercial coding agents such as C LAUDE C ODE (Anthropic’s Claude Code) integrate with development environments to navigate codebases and iterate on solutions; we adapt one to RCA through the same harness, comparing it directly with RCA-specific agents.

C. Benchmarks We use two public benchmarks with different topologies and fault granularities. • RCABench [13]: a microservice RCA benchmark with hierarchical service-to-code-level fault labels; we evaluate on a 500-case sample drawn by simple random sampling without replacement from its 1,430 cases. • AIOps 2025 [56]: a microservices e-commerce RCA benchmark of 400 incidents over ten core services. On RCABench, the empirical study (RQ1–RQ3) runs a horizontal comparison of the six frameworks under the shared Qwen and a vertical comparison of Qwen against Sonnet under

D. Agent Trajectory Analysis Inspecting agent trajectories has attracted growing interest: tools visualize and debug agent behavior [46], [47], and a parallel line mines historical trajectories to optimize agents [48]–[50].

3

Phase 1: Preparation 3500 Trajectories

Observe

SQL Tools

Agent

metrics

Query

traces

logs

RCA Benchmark

Trajectory collection

Root cause propagation path Propagation-Path Annotation

Phase 2: Study Analysis

RQ 3: Failure-Mode Coding and Mining

RQ 2: Behavior & Intent Analysis

RQ 1: Outcome Metrics

Failure Family

Accuracy

Node & Edge F1

Depth

Cost

Navigation Evidence Intent Transition utilization Distribution

Intent N-gram

Protocol omissions

Semantic misreads

General reasoning

Fig. 2: Study overview. Phase 1 collects 3,500 agent reasoning trajectories on RCABench: each agent issues SQL over telemetry, and we annotate every case’s fault propagation path. Phase 2 characterizes them at three deepening levels: RQ1 outcome and cost (Section V-A), RQ2 behavior and intent (Section V-B), and RQ3 failure-mode families (Section V-C); and D IAG G UARD (RQ4–RQ5) appears in Fig. 7.

G = (V, E), whose edges need not be query-verified. Both ground the outcome metrics (Section IV-B).

T HINK D EPTH . AI to isolate the backbone-model effect, giving seven framework–model configurations (arms). The method evaluation (RQ4–RQ5) builds the baseline and D IAG G UARD on T HINK D EPTH . AI and validates them on AIOps 2025 with the different-vendor Seed 2.0 Pro (the held-out setting).

B. Metrics Definitions a) Trajectory sequence.: For each case, the agent’s reasoning is an ordered sequence of rounds T = [τ1 , . . . , τn ], each a triple τi = (ti , ai , ri ) of thought, action, and result. For the four SQL-call frameworks, each action generates SQL over a shared telemetry interface; the two RCA-specific frameworks (M ABC, O PEN RCA) retain their native data access. b) Reasoning budget.: We report each run’s per-case reasoning budget as three quantities: rds (effective rounds: rounds issuing at least one tool call), ktok (mean input+output tokens, in thousands), and $ (those tokens priced at the backbone model’s list price; Section III-B). c) Root-cause localization.: Each case has a ground-truth root-cause set Ri⋆ of one or two services. Acc@1 scores a case correct when the agent’s predicted root cause r̂i lies in Ri⋆ :

IV. C HARACTERIZATION M ETHODOLOGY This section details how we turn heterogeneous agent runs into path-grounded, comparable measurements: annotating ground-truth propagation paths and collecting trajectories (Section IV-A), defining outcome and difficulty metrics (Section IV-B), and analyzing trajectory behavior, intent, and failure modes (Section IV-C, Section IV-D). A. Propagation-Path Annotation and Trajectory Collection

a) Propagation-path annotation.: RCABench labels each case’s root cause but not the service-level path to the SLOalarming service, so we annotate this process-level ground truth ourselves. From the injection metadata (targeted service and N 1 X fault window), the pre-/post-injection traces, metrics, and logs, 1(r̂i ∈ Ri⋆ ) , (1) Acc@1 = and the static TrainTicket service-call graph [57], we reconstruct N i=1 ⋆ ⋆ ⋆ a service-level fault propagation graph G = (V , E ), the union of one or more propagation paths from the root cause where N is the number of incidents. For configurations run VI), we to the symptom. Its nodes are the fault-affected services; each multiple times (the D IAG G UARD validation, Section(j) also report pass@k, the fraction of incidents with r̂ ∈ Ri⋆ i directed edge is a propagation step, realized through a logical for at least one of the k runs: service call or physical resource sharing and grounded in the N  abnormal spans, metric shifts, or log signatures within the fault 1 X  (j) 1 ∃ j ∈ {1, . . . , k} : r̂i ∈ Ri⋆ . (2) pass@k = window. This per-case G⋆ is the ground truth for Node F1 and N i=1 Edge F1 (Section IV-B). b) Trajectory collection.: We collect 3,500 reasoning d) Node and edge F1.: Let G⋆ = (V ⋆ , E ⋆ ) be the groundtrajectories. Each run yields thought-action-result triples; a final truth service-level fault propagation graph and G = (V, E) the compression prompt then instructs the agent to reason over the agent’s predicted graph from its final result (Section IV-A). We collected evidence and report, in a uniform format, a predicted compute 2|V ∩ V ⋆ | root-cause service and a self-reported causal propagation graph Node F1 = , (3) |V | + |V ⋆ |

4

2|E ∩ E ⋆ | . |E| + |E ⋆ |

A fixed LLM classifier (claude-opus-4.8), following the LLM-as-judge paradigm [58]–[60], labels 92,501 SQL actions Node F1 measures service-set recovery; Edge F1, directed by reading the compacted trajectory while anchoring on the SQL operation and returned evidence rather than the agent’s propagation-link reconstruction. In Table II, rds, Node F1, and Edge F1 are further split by narrative. The six tag axes are derived deterministically from the SQL text; agreement and classifier accuracy are reported Acc@1 outcome (✓: correct; ×: incorrect; avg: all cases). e) Causal-chain depth.: We stratify cases by the causal- in Section IV-E. We summarize the intent view in two ways: the intent chain depth d⋆ , the shortest-path length from the root cause distribution, giving each experiment’s frequency over the to the SLO-alarming service along G⋆ . intents, and intent n-grams, capturing contiguous subsequences C. Behavior and Intent Analysis that surface recurring motifs and repeated-intent loops. The diagnostic goal is to localize a root cause, the set of services an agent queries and surfaces at each round indirectly D. Failure-Mode Coding and Mining We code each failed trajectory against the expected evidence reveals whether its behavior on the fault-impact surface (the services V ⋆ and edges E ⋆ of G⋆ ) is advancing toward that for its fault type, yielding three failure families that differ in cause, while each action’s SQL exposes the intent behind that how the agent relates to decisive evidence (Table III). 1) Coding failures against expected evidence: A correct behavior. We accordingly build two process views over the trajectories: a behavior view, tracking how each round navigates diagnosis must recover sufficient evidence for at least one the fault-impact surface and uses surfaced evidence, and an anomalous path from the observed symptom to a root cause on the fault-impact surface. Using the RQ2 intent and tag intent view, typing each SQL action by its diagnostic intent. vocabulary, we specify each fault’s expected evidence, the 1) Impact-surface navigation and evidence utilization: We typed queries that would confirm the relevant surface services define three per-round labels to characterize each trajectory: a and dependencies, and read a failed trajectory as a shortfall primary service that tracks where the investigation stands, a against it: evidence the agent omitted or misread, or reasoning navigation transition that captures its direction, and an evidencenot tied to that evidence. utilization label that records whether it acts on returned For each failed trajectory we compare expected evidence evidence. against the agent’s actual queries and reasoning, drawing on a) Per-round primary service.: Each round touches sevits compacted trajectory, the ground-truth causal context, and eral services, so to track this advance at round granularity we raw telemetry when needed. reduce it to a single primary service: foremost the round’s We code deviations into three families (Table III): protocol service nearest a root cause, capturing how far the round omission (OMIT), evidence reachable in telemetry but never advanced. Among equally near services, it is the newly visited queried; semantic misread (MIS), retrieved evidence misinone, which marks a new investigation direction, or else the terpreted or scoped to the wrong dependency; and general most frequently visited, where the round’s investigation has reasoning (GEN), a reasoning failure not tied to a single concentrated. Concretely, we draw candidates from the round’s evidence unit. own services (its action services, or its result services when Two authors jointly develop the codebook through open the round issues no query) and rank them lexicographically ⋆ coding of failed trajectories, letting the recurring failure codes by (i) distance from the nearest root cause along G , with emerge and refining them until the codebook stabilizes; an off-path services taking a large penalty; (ii) unvisited before LLM (claude-opus-4.8) assists evidence retrieval [51], visited; then (iii) higher cumulative visit count (both relative to [58], but every code is verified by hand against the cited SQL, earlier rounds). Key (i) sets the round’s status (on-path if any returned evidence, and the TrainTicket service-call graph [57]. candidate is on-path, i.e., the round’s own queries reach the Using the fixed codebook, the two authors independently code propagation path, and off-path otherwise), and keys (ii) and all 154 failed trajectories (50 Sonnet, 104 Qwen); agreement (iii) break the remaining ties in either case. is reported in Section IV-E. b) Navigation transitions.: For each round pair (i, i+1) we label a navigation transition from the on-path status E. Annotation Reliability and distance-to-root of their two primary services, capturing Three artifacts in our pipeline rest on manual judgment, and whether the investigation advances toward, stays level with, or we validate each by independent double-annotation, resolving moves away from a root cause (full label set in Fig. 3). c) Evidence utilization.: Independently, an evidence- disagreements by discussion to consensus. For the groundutilization label compares round i+1’s action services Ai+1 truth propagation graphs, one author annotated all 500 cases; against the on-path subset of round i’s result services Ri , a second author independently annotated a random sample of capturing whether the next round acts on the decisive evidence 100, matching the first author’s annotations on 93% of them. For the SRE intent labels, they judged a stratified random the previous one surfaced (Fig. 3). 2) Intent taxonomy and Sequence patterns: For the intent sample of 1,000 classifier labels, agreeing on 99% (Cohen’s view, every SQL action receives one intent label and a multi- κ = 0.83); against this consensus the fixed LLM classifier was valued set over six deterministic tag axes (Table I; Fig. 4). correct on 97%. For the failure-mode codes, they independently Edge F1 =

(4)

5

D

Legend Impacted Service Non-impacted Service

B

★

Root Cause Service

UI

SLO Service

utilized

cites a concrete fact from the result

partial

loose reference

ignored

disregards the result

no_data

result was empty

d2

OFF-PATH (Non-impacted Set)

UI

d3

4𝑅i → 𝐴𝑖+1 Evidence Utilization Types

Reasoning Trajectory Example

Ground-truth root-cause path

O2

O1

S2a

S2b

Depth

C

d3

d2

d1

UI

S2a

S2b

S2b

UI

S2a

★

S1

★

d0 r2

r1

r3

r4

• Step = down 1 level. (e.g.,UI→ 𝑆2𝑎) • Dive = down >=2 levels. (e.g.,S2a → ★)

Regress

r5 r6 round index

r7

r8

r9

r10

r11

Return

Lateral

Advance

A

S2a

d1

d0

O1

O2

S1

r0

10 Navigation transitions Types

O1

• Branch = a new same-level service. (e.g.,S2a → S2b) • Reprobe = an already-seen service / re-query. (e.g.,S2b → S2b)

Off-Path

• Ascend = up to a first-visited service. (e.g.,S2b → UI) • Backtrack = up to an already-seen service. (e.g.,S2a → 𝑈𝐼)

• Rejoin = onto a new service. (e.g.,O2 → S1) • Retrace = onto an already-seen service. (e.g.,O1 → S2a)

• Stray = Leave the on-path band into the off-path band. (e.g.,S1 → O1) • Wander = stay within the OFF-path band. (e.g.,O1 → 𝑂2)

Fig. 3: Trajectory label vocabularies (RQ2 behavior view). Each reasoning round is reduced to a primary service at some depth on the graph (Section IV-C1). Panel A: the 10 navigation-transition types, grouped into five families by how the investigation moves relative to a root cause. Panel B: the 4 evidence-utilization types, i.e., whether a round acts on the previous round’s on-path evidence. Panel C: node legend for panel D. Panel D: a worked example plotting each round’s primary-service depth (y) over round index (x); the grey band is the off-path region and the shaded box under each round marks its evidence-utilization type. These labels are defined in Section IV-C1 and quantified in Fig. 5a. TABLE I: The intent/tag axis: eleven SRE diagnostic intents and six SQL-derived tag axes. Axis

Intent

Tag

Family

Name

Definition

Symptom

symptom_scan log_inspect keyword_search

Compares service-level symptoms, including errors, latency, status codes, payload shape, and request volume. Inspects log records for errors, exceptions, stack traces, abnormal volume, and other textual symptoms. Searches telemetry text for service names, error patterns, fault keywords, and suspicious literals.

Topology

trace_follow callgraph network_flow

Follows trace IDs, spans, and parent-child links to localize propagation along service calls. Queries caller-callee relations, span kinds, and call ratios to recover dependency direction and structure. Inspects request flow, failures, connectivity, packet/drop signals, and service-to-service traffic.

Resource

resource_probe metric_dump

Tests resource hypotheses with metrics for named services, pods, nodes, JVMs, DBs, or runtimes. Retrieves broad metric values or snapshots to scan resource state before narrowing a hypothesis.

Liveness

lifecycle_probe

Checks lifecycle/emission signals such as restarts, silence, span export, pod state, and JVM state.

Inventory

metric_enum service_enum

Enumerates metric names, schemas, coverage, resource attributes, and measurement inventory. Enumerates services, pods, nodes, tables, topology entities, and telemetry inventory.

Derived

granularity window time emission resource_family symptom_signal

Target level over which evidence is grouped, e.g., global, node, pod, service, endpoint, or trace. Telemetry window compared by the query, e.g., abnormal, normal, or normal-abnormal contrast. Temporal shape requested by the query, e.g., snapshot, timeline, or onset analysis. Whether the query references telemetry emission, silence, exporter, queue, or liveness fields. Resource axis referenced by the query, e.g., CPU, memory, GC, pool, thread, or disk. Symptom signal referenced by the query, e.g., error, status code, latency, payload, or volume.

coded all 154 failed trajectories under the fixed codebook and assigned identical code sets on 144 (93.5%).

wrong diagnoses into recurring failure modes, which Section VI later turns into an intervention.

V. C HARACTERIZATION R ESULTS

A. RQ1: Effectiveness and Cost

This section characterizes RCA agents along three deepening levels: RQ1 establishes outcome and cost, RQ2 explains the outcome gaps through behavior and intent, and RQ3 distills

a) Framework and Model Comparison: Table II compares agents along two axes. Across frameworks, effectiveness is set by investigative architecture: the agents that run an open-

6

TABLE II: RQ1 effectiveness, reasoning budget, and depth-stratified Acc@1. Across all rows, Edge F1 trails Node F1 by a wide margin, and Acc@1 falls as the causal chain deepens, a decline the Sonnet row largely escapes. Metrics and the ✓/× outcome split defined in Section IV-B; Acc@1 and depth columns in %, F1 in [0, 1]. Node F1

Sonnet

Rounds (rds)

Budget / case

Acc@1 by depth d⋆

Acc@1

✓

×

avg

✓

×

avg

✓

×

avg

ktok

$

d⋆ =2 d⋆ =3 d⋆ =4 d⋆ =5

AIQ C LAUDE C ODE T HINK D EPTH . AI

42.6 46.6 65.2 77.6 79.6 79.2

0.49 0.57 0.78 0.81 0.81 0.83

0.35 0.48 0.55 0.63 0.58 0.59

0.41 0.52 0.70 0.77 0.76 0.78

0.15 0.36 0.58 0.61 0.65 0.66

0.05 0.16 0.26 0.35 0.28 0.33

0.09 0.25 0.47 0.55 0.57 0.59

– 6.1 13.2 50.0 51.3 43.1

– 6.7 14.5 55.1 62.7 50.7

– 6.4 13.7 51.1 53.6 44.7

163 364 672 1516 1512 1744

0.04 0.05 0.09 0.19 0.18 0.21

53.3 58.6 79.6 86.8 89.5 85.5

36.4 46.9 68.9 82.0 79.8 82.0

40.4 30.9 39.4 59.6 68.1 66.0

42.9 19.0 33.3 42.9 57.1 57.1

T HINK D EPTH . AI

90.0

0.82 0.56 0.79 0.67 0.37 0.64 30.5 38.2 31.2 1465

4.64

88.2

88.6

96.8

85.7

Backbone model Framework

Qwen

Edge F1

M ABC O PEN RCA TASK W EAVER

likewise (85.5 → 57.1%) while Sonnet holds flat (85.7–96.8%), widening the backbone-model gap on the hardest cases. Acc@1 and surface reconstruction are positively coupled across Table II, yet the backbone-model swap lifts localization without lifting the reconstruction ceiling. Weaker agents do intent = resource_probe not merely select the wrong final service: they reconstruct less window = abnormal · time = timeline · granularity = {pod, service} resource_family = {cpu, memory, pool, thread} of the fault-impact surface even on their correctly localized emission = true · symptom_signal = ∅ cases (✓ Node F1 rises from 0.49 for M ABC to 0.83 for Fig. 4: Intent/tag labeling of one SQL action. A real abnormal- T HINK D EPTH . AI, ✓ Edge F1 from 0.15 to 0.66). Even correct window metric query labeled with a single SRE intent localizations, however, rest on partial causal maps: ✓ Edge F1 (resource_probe) plus the six deterministic tag axes of never exceeds 0.67, and the model swap that raises Acc@1 Table I. Color-matched highlights link each SQL fragment to by 10.8 pp leaves this ceiling nearly untouched (Node F1 0.82 vs. 0.83, Edge F1 0.67 vs. 0.66). In every row, Edge F1 falls the tag axis it sets (Section IV-C2). well below Node F1 (M ABC: 0.41 vs. 0.09; C LAUDE C ODE: 0.76 vs. 0.57), and failure widens this imbalance: from correct ended, adaptive investigation, requerying telemetry as evidence to incorrect runs, Node F1 drops by 16–32% while Edge F1 emerges (T HINK D EPTH . AI 79.2%, AIQ 77.6%, C LAUDE C ODE drops by 43–67%, so failed runs still detect most anomalous 79.6%), hold the top band in a narrow range, while the more services yet no longer chain them into a propagation path. constrained designs trail in order: TASK W EAVER (65.2%) Identifying anomalous nodes requires only detecting aberrant follows a largely fixed plan, O PEN RCA (46.6%) commits telemetry; recovering directed edges requires inferring how the as soon as it can emit an answer, and M ABC (42.6%) does not fault propagates between services, making edge reconstruction iterate over evidence at all. Notably, none of these top agents the harder and more discriminative capability. was built for RCA: they are open-ended agents repurposed for Finding 2 it, yet prove highly effective on the task. Across backbone models, the Sonnet T HINK D EPTH . AI arm Accuracy degrades with causal-chain depth, a decline the reaches 90.0% Acc@1, a 10.8-pp gain over Qwen. The topstronger backbone model largely escapes. Reconstructing band frameworks span only 2.0 pp under the same model the fault-impact surface is harder than selecting the correct (77.6%–79.6%), while the model swap yields over five times final answer and remains partial even when localization that gap, suggesting the backbone model as the dominant lever. succeeds; within the reconstruction, recovering directed propagation edges proves more demanding than identifying Finding 1 anomalous nodes. Adaptive, minimally constrained frameworks outperform c) Reasoning Budget and Diagnostic Outcome: Budget more tightly orchestrated designs; among the top-band scales with investigative openness: the top-band agents sustain frameworks, the backbone model is the dominant lever. 45–54 effective rounds, over 1500 ktok, and roughly $0.2 per b) Causal-Chain Depth and Reconstruction Granularity: case, whereas the constrained agents stop far sooner (under Causal-chain depth serves as a proxy for diagnostic difficulty: 700 ktok, at most $0.09). Within T HINK D EPTH . AI, Sonnet a longer chain demands more investigative steps and more stops in fewer rounds than Qwen, locking onto decisive evidence to reach the root cause. Empirically, accuracy declines evidence quickly where Qwen enumerates more broadly, though accordingly. Every Qwen framework loses ground from depth 2 Sonnet’s higher per-token price still dominates per-case cost. Yet one pattern is shared across all agents: in every iterative to depth 5 (C LAUDE C ODE 89.5 → 57.1%, AIQ 86.8 → 42.9%); on the matched T HINK D EPTH . AI pair, Qwen falls arm, incorrect cases run more rounds than correct ones, by up SELECT time , metric, service_name, attr_k8s_pod_name , value FROM abnormal_metrics_sum WHERE service_name = ’ts-order-service’ AND metric IN ( ’jvm.memory.used’ , ’jvm.memory.limit’ , ’jvm.cpu.time’ , ’jvm.thread.count’ , ’db.client.connections.usage’ , ’queueSize’ ) ORDER BY time ASC, metric

7

to 11.4 rounds. The extra rounds buy no partial credit either: incorrect runs trail correct ones on both F1 metrics, so the added budget reflects unproductive search rather than evidence accumulation.

drills into them. Sonnet runs the same early symptom/log/trace survey but then pivots in mid-trajectory to exploring the metric query space (metric_enum), recovering dependency structure with callgraph, and drilling named resources and flows via resource_probe (used in 82.8% of its trajectories vs. 16.6% for Qwen) and network_flow. This maps onto per-fault accuracy: Qwen trails Sonnet most on categories requiring named-resource metrics (R ESOURCE 75% vs. 94%, C ODE (JVM) 66% vs. 93%), while N ETWORK nearly closes (82% vs. 89%) and HTTP is tied (88% vs. 87%). Qwen further perseverates in shallow loops that Sonnet avoids: intent 5-grams such as sustained trace-following (trace5 , 0.96 per trajectory) and log_inspect/symptom_scan oscillation recur far more in the Qwen arm than in Sonnet (≤ 0.12 per trajectory), the sequence-level signature of its narrow repertoire.

Finding 3 Reasoning budget varies across frameworks and backbone models but does not predict diagnostic quality: failed runs consume more rounds than successful ones, as unproductive search inflates cost without improving outcomes. B. RQ2: Agent Behavior Having established outcome and cost differences, we now examine the diagnostic process behind them. Because RQ2– RQ3 analyze SQL-call trajectories, they exclude M ABC and O PEN RCA, which expose no comparable traces. We center on T HINK D EPTH . AI’s Qwen and Sonnet arms, the only matched pair; the Qwen arm is representative because all Qwen SQL-call frameworks show similar behavior and intent patterns. a) Surface Navigation and Evidence Grounding: Correct runs ground each round in retrieved evidence; incorrect runs drift off-path or ignore decisive clues (Fig. 5a). Specifically, correct runs lean on Reprobe, re-querying on-path services at the same depth (higher than incorrect runs in both arms), whereas incorrect runs oscillate on and off the surface (higher Stray/Wander and Retrace). The two arms navigate in similar proportions. Evidence grounding, not navigation volume, is where the two backbone models diverge. Correct runs act on the evidence they retrieve (higher utilized), whereas incorrect runs more often ignore results or hit empty queries (higher ignored and no_data). Sonnet acts on the prior result (utilized or partial) on 54.3% of rounds versus Qwen’s 24.9%. In effect, Sonnet’s backtracks and advances are evidence-guided confirmations, while Qwen swings between shallow and deep nodes regardless of the returned evidence.

Finding 5 A richer diagnostic intent repertoire corresponds to broader, SRE-style search, while a narrow one keeps the agent in shallow loops over a limited slice of the evidence, producing fault-type-specific accuracy gaps that concentrate on the types requiring deeper evidence. C. RQ3: Failure Taxonomy The behavioral patterns above show how runs diverge; we now ask why wrong diagnoses occur. Wrong diagnoses reduce to three evidence-handling failure families that differ in how the agent relates to decisive evidence (Table III). a) Failure families and cross-model patterns: The generalreasoning failures in Table III are untied to any single signal: a model that cannot aggregate scattered evidence or drill deep enough to discard distractors locks onto an early anchor (GEN1); an overconfident one fills gaps by fabricating an outof-schema cause (GEN2); and one without the persistence to retry a failed or empty query abandons the evidence channel altogether (GEN3). The evidence-anchored families track the two backbone models’ reasoning posture. Qwen fails predominantly by omission, Sonnet by misreading. Qwen has less working knowledge of the RCA problem domain, so it stays redundantly confined to the most superficial, easily found signals (OMIT1, OMIT3, OMIT4) and rarely redirects its exploration from the evidence already gathered (GEN1). Sonnet has a stronger ability to hypothesize new investigation directions from the evidence, but its strong associative reach sometimes errs through hallucination (GEN2), and its synthesis often falls short of pinning down the crux among contradictory signals (MIS2, MIS3).

Finding 4 Successful RCA stay on the fault-impact surface and ground each round in retrieved evidence; incorrect RCA drift off the surface or ignore decisive clues. b) Intent Repertoire and Fault-Type Sensitivity: The two backbone models deploy qualitatively different diagnostic strategies, with Sonnet pivoting mid-trajectory to resourceand topology-oriented intents where Qwen persists in shallow trace and log loops (Fig. 5b). Over the full eleven-intent profile, the Sonnet arm is the least similar to every Qwen arm (cosine similarity 0.907–0.928), while the four Qwen frameworks cluster tightly (0.947–0.989). Both arms open with a shared symptom-scanning phase, after which their strategies diverge. Qwen stays trace- and log-bound: as the trajectory advances it relies increasingly on fuzzy keyword_search over log and trace text and on trace_follow span enumeration, and although it inventories metrics via metric_enum and metric_dump, it rarely

Finding 6 Agent failures reduce to three evidence-handling modes: decisive evidence is never gathered, is gathered but misread, or is overridden by ungrounded reasoning.

8

Correct

ThinkDepth.ai Sonnet (correct 450, incorrect 50)

Incorrect

70% 60% 50% 40% 30% 20% 10% 0%

p e e h e n k d y r Ste DiveprobBranc etrac RejoicktracAscen Stra ande W R R Ba

a d l ize rtia red at util pa igno no_d

Ri Ai + 1

ThinkDepth.ai Qwen (correct 396, incorrect 104)

Ri Ai + 1

70% 60% 50% 40% 30% 20% 10% 0%

p e e h e n k d y r Ste DiveprobBranc etrac RejoicktracAscen Stra ande W R R Ba

a d l ize rtia red at util pa igno no_d

(a) Navigation and evidence-utilization label shares, correct (blue) vs. incorrect (red) runs per arm. In the navigation block (left), the correct–incorrect contrast is consistent across both arms; in the evidence-utilization block (right, after the dashed line), the two base models diverge sharply. Labels defined in Fig. 3; bars macro-averaged per trajectory.

symptom_scan log_inspect

Move type (stream bands) metric_enum trace_follow

callgraph network_flow

ThinkDepth.ai Sonnet

1.0

Rel. freq.

1.0

keyword_search service_enum

0.5

0.0 0.0

resource_probe metric_dump

lifecycle_probe

ThinkDepth.ai Qwen

0.5

0.5

Norm. progress

0.0 0.0

1.0

0.5

Norm. progress

1.0

(b) SRE diagnostic-intent composition over normalized trajectory progress. Both arms share a symptom-dominated opening phase; band shapes diverge from mid-trajectory onward, with the Sonnet arm shifting weight toward intent categories largely absent in Qwen. Band thickness is each intent’s relative frequency (colors: the 11 intents of Table I); in-plot labels give overall shares.

Fig. 5: RQ2 behavior and intent analysis: what separates correct from incorrect runs (a) and what drives the backbone-model gap in diagnostic strategy (b).

Whether distilling these recurring modes into defenses can improve localization beyond the characterization setting remains to be tested.

OMIT5, OMIT6, GEN3; Table III), confirming that the modes recur on an unseen model and dataset. Under D IAG G UARD they are overcome, localizing the storage node as the true root cause.

VI. I NTERVENTION AND VALIDATION

A. D IAG G UARD Design

This section tests whether the mined failure modes can guide practical improvement beyond the characterization setting: we design D IAG G UARD from the RQ3 taxonomy (Section VI-A) and validate it on the held-out setting (Section VI-B). A held-out incident illustrates the design (Fig. 6): a disk-I/O fault on the TiKV storage node tidb-tikv-0 surfaces only as downstream tail latency, so localizing it requires descending past the service tier and chronic background noise to the storage node. The baseline T HINK D EPTH . AI agent fails through several already-catalogued modes at once (MIS2, OMIT1, OMIT3,

The taxonomy of Section V-C is not only a diagnosis of why agents fail: it is a design specification. Its three families expose a single structural deficit. A bare reasoning loop is at once under-grounded, committing to evidence it never gathered or to states it fabricated, and under-verified, never re-examining a reading or a conclusion before it commits. We read each mined mode as a requirement and close it with a dedicated, answeragnostic defense, yielding D IAG G UARD: a failure-taxonomy-

9

TABLE III: Three evidence-handling failure families. Each family addresses a different relationship between the agent and decisive evidence: never gathered (OMIT), gathered but misread (MIS), or overridden by ungrounded reasoning (GEN). Family

Mode

Definition

Consequence

Sonnet Qwen

Protocol omissions

OMIT1 Baseline gap OMIT2 Edge untested OMIT3 Resource gap OMIT4 Ranking gap OMIT5 Granularity gap OMIT6 Timing gap

No baseline-window comparison. No caller–callee edge test. No resource/lifecycle probe using concrete names. No peer ranking on the same signal. No pod, endpoint, edge, or metric-level drill-down. No timing check for the proposed cause.

Pre-existing signal misjudged as incident evidence. Propagation edge lacks span-level support. Decisive memory, restart, etc. signal remains unseen. Collateral spike is treated as uniquely causal. Failing pod, endpoint, edge, or metric remains unidentified. Causal order lacks temporal corroboration.

26.0% 54.0% 34.0% 2.0% 18.0% 4.0%

Semantic misreads

MIS1 Silence-as-health Silence, flatness, or missing data read as health. MIS2 Salience bias Loudest signal over-trusted. MIS3 Semantics misread Telemetry text read with wrong service semantics.

Suppressed callee fault is cleared as healthy. Noisy victim service is selected as the cause. Caller-side client fault attributed to downstream callee.

30.0% 44.2% 42.0% 32.7% 86.0% 61.5%

General reasoning

GEN1 Anchor lock Early anchor retained after conflicting evidence. GEN2 Schema-out claim Out-of-schema causal state fabricated. GEN3 Friction stop Evidence channel dropped after tool friction.

Contradictory evidence is discounted in the final answer. Unsupported out-of-schema state is returned as the cause. Resource evidence remains unavailable after tool friction.

28.0% 34.6% 82.0% 20.2% 0% 12.5%

51.9% 68.3% 63.5% 25.0% 16.3% 8.7%

Note. Percentages report within-model failure-pool prevalence (n=50 Sonnet, n=104 Qwen); subcategories may overlap and do not sum to 100%.

Disk-Write rates: From 15 up to 105 Mbps

Multi-Layer Telemetry

Service tier

Infrastructure tier Slow-Query count: From 0 up to 15

Time for Product Lookup: From ~10ms up to 14s

Diagnostic Discipline Soundness Check

Distinct Values Wider evidence

Right Root Cause Query

TiKV storage node Ground Truth: tidb-tikv-0

TiDB database

Observe

Verifier

Product-catalog service

Grounder

Protocol omissions Semantic misreads General reasoning

Agent

Frontend DIAGGUARD

Fig. 7: The D IAG G UARD architecture. A reasoning core is wrapped by two answer-agnostic defenses derived from Product-page 99th-percentile latency: Baseline the RQ3 failure taxonomy. The Grounder supplies grounding, From 0.25s up to 10s confronting the agent with the telemetry that exists (distinct values, wider evidence); the Verifier supplies verification, a Fig. 6: Motivating incident (Seed 2.0 Pro / AIOps 2025). diagnostic-discipline self-audit and a soundness check beAn injected disk-I/O fault on the TiKV storage node fore any commitment. Together they close the three failure tidb-tikv-0 surfaces only as downstream tail latency; families (Table III) that a bare reasoning loop leaves open red-highlighted values mark abnormal-vs-normal shifts. The (Section VI-A). baseline (red cross) anchors on the chronically noisy Ad service and never leaves the service tier, whereas D IAG G UARD (green check) descends through the infrastructure tier to the true root how much each component contributes to that improvement. cause tidb-tikv-0 (Section VI). a) RQ4: D IAG G UARD effectiveness: Table IV shows that D IAG G UARD raises Acc@1 from 43.5% to 52.5% (+9.0), grounded, defense-in-depth diagnostic architecture that wraps pass@3 from 56.9% to 67.1% (+10.2), and pass@5 from 62.3% to 73.0% (+10.7). The improvement holds across every a reasoning core in two complementary defenses (Fig. 7). The core is the Diagnostician, the T HINK D EPTH . AI agent fault type, with per-type gains from +1.0 (DNS) to +20.8 of Section III adapted to the AIOps 2025 telemetry interface; (I/O). Absolute accuracy is lower than on RCABench for two run alone, it is the baseline of Section VI-B. Around it sit two reasons: half of its fault types lie below the service tier and raise answer-agnostic defenses that adapt techniques studied in isola- no SLO, so the agent starts blind; and the target granularity tion for general agents: the Grounder (agentic grounding [61]) varies by fault: each AIOps 2025 fault must be localized at and the Verifier (inference-time verification [62]), shown in one prescribed layer (a service, a pod, or a node), rather than Fig. 7. Complementary, they span the three families between at the single service level we score on RCABench. Error rate: From 0.25% up to 0.69% ( )

them: evidence that is never gathered, gathered but misread, or overridden by ungrounded reasoning. Each check is mapped from a specific mined mode and frozen before the validation runs; the excerpt in Section VI-B, for instance, is mapped from the baseline-gap mode OMIT1, under which a chronic signal is misread as incident evidence.

Verifier self-audit (excerpt) Establish a baseline before relying on any signal: compare the abnormal window to the normal window for the same entity and the same metric, log-pattern, or volume. If an error, a latency, or a ‘‘first failure’’ already exists in the normal period, it is chronic background and cannot be the root-cause anchor.

B. Validation With D IAG G UARD designed, we validate it on the held-out setting along two axes: whether it improves localization, then

10

TABLE IV: D IAG G UARD versus the baseline on the held-out setting (Seed 2.0 Pro / AIOps 2025). Gains hold across all nine fault types (lower panel). Net.: network attack; Chg.: erroneous change; Cfg.: misconfiguration. Configuration

Acc@1

pass@3

pass@5

Baseline D IAG G UARD (ours)

43.5±0.6 52.5±1.1

56.9 67.1

62.3 73.0

reported differences are indicative; the validation (RQ4–RQ5) reports pass@k and mean±sd over five runs. VIII. C ONCLUSION We presented a process-level study of how LLM agents reach a root-cause diagnosis, not merely whether it is correct. Against manually annotated fault propagation paths, we characterized agent reasoning trajectories across backbone models along outcome, behavior, and intent. We find that final-answer accuracy masks how completely an agent reconstructs the faultimpact surface: recovering the propagation path is harder than naming the root-cause service, and even correctly localized runs leave the path incomplete. Effective diagnosis is an evidencegrounded, on-surface process whose intent breadth sets its search depth; failing runs drift off the surface or leave decisive evidence unused. Wrong diagnoses reduce to a small taxonomy of omitted or misread evidence and ungrounded reasoning, and distilling it into D IAG G UARD improves localization on a heldout backbone model, dataset, and service topology. Trajectorylevel analysis thus exposes agent capability boundaries that final-answer evaluation alone cannot reveal. Future work will turn trajectory labels into reinforcement-learning signals and target harder, more production-like incidents.

Per fault type (Acc@1) Net. Node Pod JVM Stress I/O Chg. DNS Cfg. Baseline 78.1 4.1 42.7 65.1 D IAG G UARD 88.2 19.5 45.0 67.3

52.4 62.4

2.1 32.4 43.8 75.6 22.9 39.0 44.8 84.4

TABLE V: Component ablation of D IAG G UARD. The ∆ column shows near-additive gains that do not track the budget columns (rds, ktok, $): the costliest arm gains least, and the full stack gains most while spending less than it. Acc@1 is mean±sd over five runs. Acc@1

pass@3 pass@5 Budget / incident ∆

%

Baseline (w/o both) 43.5±0.6 — w/o Grounder 47.6±1.6 +4.1 w/o Verifier 48.0±0.9 +4.5 D IAG G UARD (full) 52.5±1.1 +9.0

56.9 61.3 62.0 67.1

Configuration

%

%

rds

ktok

$

62.3 14.1 176.8 65.8 19.2 343.8 67.3 16.3 212.1 73.0 18.7 318.5

0.09 0.17 0.10 0.16

R EFERENCES

Finding 7 Diagnostic experience distilled from the mined failure modes remains effective on a held-out backbone model, dataset, and service topology. b) RQ5: Component contribution: Table V ablates the two defenses. Each helps on its own by a comparable margin: the Grounder alone (w/o Verifier) raises Acc@1 by +4.5 and the Verifier alone (w/o Grounder) by +4.1, with the full D IAG G UARD reaching +9.0 and the same ordering at pass@3 and pass@5. We thus read the Grounder’s breadth and the Verifier’s discipline as complementary and non-redundant, with each contributing a separable gain. VII. T HREATS TO VALIDITY External validity. Our characterization (RQ1–RQ3) rests on a single topology (RCABench’s TrainTicket), the fixed sample, and the chosen agents, and may not transfer to other topologies, stacks, or deployments; D IAG G UARD’s transfer to a held-out backbone model, dataset, and service topology partly mitigates this. Internal validity. Although our comparisons vary one factor at a time, residual implementation differences across frameworks and models may still confound them. Construct validity. Our process labels rest on manual annotation, LLM-assisted classification, and human review; fixed protocols, double coding, and released artifacts reduce but cannot eliminate annotation subjectivity and judge bias. Conclusion validity. Each characterization configuration (RQ1–RQ3) is run once without significance testing, so its

11

[1] X. Guo, X. Peng, H. Wang, W. Li, H. Jiang, D. Ding, T. Xie, and L. Su, “Graph-based trace analysis for microservice architecture understanding and problem diagnosis,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). New York, NY: ACM, 2020, pp. 1387–1397. [2] Google SRE Team, “Effective troubleshooting,” https://sre.google/ sre-book/effective-troubleshooting/, 2016, accessed: 2024-12-01. [3] N. Fu, G. Cheng, Y. Teng, G. Dai, S. Yu, and Z. Chen, “Intelligent root cause localization in microservice systems: A survey and new perspectives,” ACM Computing Surveys, vol. 57, no. 12, 2025. [4] S. Zhang, S. Xia, W. Fan, B. Shi, X. Xiong, Z. Zhong, M. Ma, Y. Sun, and D. Pei, “Failure diagnosis in microservice systems: A comprehensive survey and analysis,” ACM Transactions on Software Engineering and Methodology, 2024. [5] A. Fang, H. Yang, H. Dong, Q. Lu, J. Xu, and P. He, “A goal-driven survey on root cause analysis,” arXiv preprint arXiv:2510.19593, 2025. [6] Z. Li, J. Yu, Z. Huang, and Y. Huang, “Surveying root cause analysis techniques: A comprehensive review of aspects for multi-service applications,” IEEE Transactions on Services Computing, vol. 19, no. 1, pp. 808–825, 2026. [7] I. Bouzenia and M. Pradel, “Understanding software engineering agents: A study of thought-action-result trajectories,” arXiv preprint arXiv:2506.18824, 2025. [8] M. Du, B. Xu, C. Zhu, X. Wang, and Z. Mao, “Deepresearch bench: A comprehensive benchmark for deep research agents,” arXiv preprint arXiv:2506.11763, 2025. [9] Y. Chen, H. Xie, M. Ma, Y. Kang, X. Gao, L. Shi, Y. Cao, X. Gao, H. Fan, M. Wen et al., “Automatic root cause analysis via large language models for cloud incidents,” in Proceedings of the Nineteenth European Conference on Computer Systems, 2024, pp. 674–688. [10] Y. Gao, “An framework for fault localization in microservices,” Available at SSRN 5420754, 2025. [11] W. Zhang, H. Guo, J. Yang, Z. Tian, Y. Zhang, Y. Chaoran, Z. Li, T. Li, X. Shi, L. Zheng et al., “mabc: multi-agent blockchain-inspired collaboration for root cause analysis in micro-services architecture,” in Findings of the Association for Computational Linguistics: EMNLP 2024, 2024, pp. 4017–4033. [12] J. Xu, Q. Zhang, Z. Zhong, S. He, C. Zhang, Q. Lin, D. Pei, P. He, D. Zhang, and Q. Zhang, “Openrca: Can large language models locate the root cause of software failures?” in The thirteenth international conference on learning representations, 2025.

[13] A. Fang, S. Zhang, Y. Yang, H. Wu, J. Xu, X. Wang, R. Wang, M. Wang, Q. Lu, and P. He, “Rethinking the evaluation of microservice rca with a fault propagation-aware benchmark,” arXiv preprint arXiv:2510.04711, 2025. [14] X. Zhou, X. Peng, T. Xie, J. Sun, C. Ji, D. Liu, Q. Xiang, and C. He, “Latent error prediction and fault localization for microservice applications by learning from system trace logs,” in Proceedings of the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). ACM, 2019, pp. 683–694. [15] H. Shan, Y. Chen, H. Liu, Y. Zhang, X. Xiao, X. He, M. Li, and W. Ding, “ϵ-diagnosis: Unsupervised and real-time diagnosis of small-window long-tail latency in large-scale microservice platforms,” in The World Wide Web Conference, WWW 2019, San Francisco, CA, USA, May 13–17, 2019, L. Liu, R. W. White, A. Mantrach, F. Silvestri, J. J. McAuley, R. Baeza-Yates, and L. Zia, Eds. ACM, 2019, pp. 3215–3222. [Online]. Available: https://doi.org/10.1145/3308558.3313653 [16] L. Wu, J. Tordsson, E. Elmroth, and O. Kao, “MicroRCA: Root cause localization of performance issues in microservices,” in NOMS 2020 – 2020 IEEE/IFIP Network Operations and Management Symposium. IEEE, 2020. [17] G. Yu, P. Chen, H. Chen, Z. Guan, Z. Huang, L. Jing, T. Weng, X. Sun, and X. Li, “Microrank: End-to-end latency issue localization with extended spectrum analysis in microservice environments,” in WWW ’21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021, J. Leskovec, M. Grobelnik, M. Najork, J. Tang, and L. Zia, Eds. ACM / IW3C2, 2021, pp. 3087–3098. [Online]. Available: https://doi.org/10.1145/3442381.3449905 [18] S. Chakraborty, S. Garg, S. Agarwal, A. Chauhan, and S. K. Saini, “CausIL: Causal graph for instance level microservice data,” in Proceedings of the ACM Web Conference 2023 (WWW). ACM, 2023, pp. 2905–2915. [19] M. Li, Z. Li, K. Yin, X. Nie, W. Zhang, K. Sui, and D. Pei, “Causal inference-based root cause analysis for online service systems with intervention recognition,” in KDD ’22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, August 14–18, 2022, A. Zhang and H. Rangwala, Eds. ACM, 2022, pp. 3230–3240. [Online]. Available: https://doi.org/10.1145/3534678.3539041 [20] L. Pham, H. Ha, X. Zhang, and H. Zhang, “TORAI: Multi-source root cause analysis for blind spots in microservice service call graph,” in Proceedings of the ACM International Conference on the Foundations of Software Engineering (FSE), 2026. [21] Y. Li, Y. Wu, J. Liu, Z. Jiang, Z. Chen, G. Yu, and M. R. Lyu, “COCA: generative root cause analysis for distributed systems with code knowledge,” in 47th IEEE/ACM International Conference on Software Engineering, ICSE 2025, Ottawa, ON, Canada, April 26 - May 6, 2025. IEEE, 2025, pp. 1346–1358. [22] S. Zhang, A. Fang, Y. Yang, R. Cheng, X. Tang, and P. He, “Dynacausal: Dynamic causality-aware root cause analysis for distributed microservices,” arXiv preprint arXiv:2510.22613, 2025. [23] C. Lee, T. Yang, Z. Chen, Y. Su, and M. R. Lyu, “Eadro: An end-to-end troubleshooting framework for microservices on multi-source data,” in Proceedings of the 45th IEEE/ACM International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 1750–1762. [24] G. Yu, P. Chen, Y. Li, H. Chen, X. Li, and Z. Zheng, “Nezha: Interpretable fine-grained root cause analysis for microservices on multi-modal observability data,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), 2023, pp. 553–565. [25] L. Zheng, Z. Chen, J. He, and H. Chen, “MULAN: Multi-modal causal structure learning and root cause analysis for microservice systems,” in Proceedings of the ACM on Web Conference 2024 (WWW), 2024, pp. 4107–4116. [26] Anthropic, “Building effective agents,” https://www.anthropic.com/ research/building-effective-agents, 2024, accessed: 2026. [27] D. Zhang, X. Zhang, C. Bansal, P. Las-Casas, R. Fonseca, and S. Rajmohan, “Lm-pace: Confidence estimation by large language models for effective root causing of cloud incidents,” in Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, 2024, pp. 388–398. [28] Y. Han, Q. Du, Y. Huang, J. Wu, F. Tian, and C. He, “The potential of one-shot failure root cause analysis: Collaboration of the large language model and small classifier,” in Proceedings of the 39th IEEE/ACM

12

International Conference on Automated Software Engineering, 2024, pp. 931–943. [29] X. Zhang, S. Ghosh, C. Bansal, R. Wang, M. Ma, Y. Kang, and S. Rajmohan, “Automated root causing of cloud incidents using incontext learning with gpt-4,” in Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, 2024, pp. 266–277. [30] T. Ahmed, S. Ghosh, C. Bansal, T. Zimmermann, X. Zhang, and S. Rajmohan, “Recommending root-cause and mitigation steps for cloud incidents using large language models,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 1737–1749. [31] D. Goel, F. Husain, A. Singh, S. Ghosh, A. Parayil, C. Bansal, X. Zhang, and S. Rajmohan, “X-lifecycle learning for cloud incident management using llms,” in Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, 2024, pp. 417– 428. [32] A. Hrusto, P. Runeson, and M. C. Ohlsson, “Autonomous monitors for detecting failures early and reporting interpretable alerts in cloud operations,” in Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice, 2024, pp. 47– 57. [33] Y. Sun, T. Zheng, X. Wen, W. Kuang, H. Liu, S. Zhang, C. Shen, B. Wu, and D. Pei, “A multimodal intelligent change assessment framework for microservice systems based on large language models,” in Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering, 2025, pp. 378–388. [34] S. Liang, P. Chen, B. Tian, G. Tan, M. Xu, Y. Qu, Y. Zhao, Y. Shang, and C. Tan, “MetaRCA: A generalizable root cause analysis framework for cloud-native systems powered by meta causal knowledge,” in Proceedings of the ACM International Conference on the Foundations of Software Engineering (FSE), 2026. [35] C. Pei, Z. Wang, F. Liu, Z. Li, Y. Liu, X. He, R. Kang, T. Zhang, J. Chen, J. Li et al., “Flow-of-action: Sop enhanced llm-based multi-agent system for root cause analysis,” in Companion Proceedings of the ACM on Web Conference 2025, 2025, pp. 422–431. [36] D. Roy, X. Zhang, R. Bhave, C. Bansal, P. Las-Casas, R. Fonseca, and S. Rajmohan, “Exploring llm-based agents for root cause analysis,” in Companion proceedings of the 32nd ACM international conference on the foundations of software engineering, 2024, pp. 208–219. [37] Z. Wang, Z. Liu, Y. Zhang, A. Zhong, J. Wang, F. Yin, L. Fan, L. Wu, and Q. Wen, “RCAgent: Cloud root cause analysis by autonomous agents with tool-augmented large language models,” in Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, CIKM 2024, Boise, ID, USA, October 21–25, 2024, E. Serra and F. Spezzano, Eds. ACM, 2024, pp. 4966–4974. [38] L. Zhang, T. Jia, Y. Zhai, L. Pan, C. Duan, M. He, M. Jia, and Y. Li, “Agentic memory enhanced recursive reasoning for root cause localization in microservices,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP ’26). ACM, 2026. [39] Y. Chen, J. Pan, J. Clark, Y. Su, N. Zheutlin, B. Bhavya, R. R. Arora, Y. Deng, S. Jha, and T. Xu, “STRATUS: A multi-agent system for autonomous reliability engineering of modern clouds,” in Advances in Neural Information Processing Systems, 2025. [Online]. Available: https://neurips.cc/virtual/2025/poster/116834 [40] L. Zhang, Y. Zhai, T. Jia, C. Duan, S. Yu, J. Gao, B. Ding, Z. Wu, and Y. Li, “Thinkfl: Self-refining failure localization for microservice systems via reinforcement fine-tuning,” ACM Transactions on Software Engineering and Methodology, 2026, published online January 19, 2026 (Just Accepted). [41] Z. Xie, Z. Li, X. He, S. Zhang, L. Xu, Y. Yang, T. Zhang, J. Chen, R. Shi, and D. Pei, “Foundroot: Towards foundation model for root cause analysis via structured deep thinking,” in Proceedings of the IEEE/ACM 48th International Conference on Software Engineering, ser. ICSE ’26. ACM, 2026. [42] R. Li, M. Du, B. Xu, C. Zhu, X. Wang, and Z. Mao, “Deepresearch bench ii: Diagnosing deep research agents via rubrics from expert report,” arXiv preprint arXiv:2601.08536, 2026. [43] NVIDIA AI Blueprints, “AIQ research assistant,” https://github.com/ NVIDIA-AI-Blueprints/aiq/tree/drb1, 2025. [44] B. Qiao, L. Li, X. Zhang, S. He, Y. Kang, C. Zhang, F. Yang, H. Dong, J. Zhang, L. Wang et al., “Taskweaver: A code-first agent framework,” arXiv preprint arXiv:2311.17541, 2023.

[45] ThinkDepth.ai, “Deep Research,” https://github.com/thinkdepthai/Deep Research, 2025. [46] J. Lu, B. Pan, J. Chen, Y. Feng, J. Hu, Y. Peng, and W. Chen, “Agentlens: Visual analysis for agent behaviors in llm-based autonomous systems,” IEEE Transactions on Visualization and Computer Graphics, 2024. [47] W. Epperson, G. Bansal, V. C. Dibia, A. Fourney, J. Gerrits, E. Zhu, and S. Amershi, “Interactive debugging and steering of multi-agent ai systems,” in Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, 2025, pp. 1–15. [48] P. Gupta, S. Kirtania, A. Singha, S. Gulwani, A. Radhakrishna, G. Soares, and S. Shi, “Metareflection: Learning instructions for language agents using past reflections,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2024, pp. 8369–8385. [49] Y. Song, D. Yin, X. Yue, J. Huang, S. Li, and B. Y. Lin, “Trial and error: Exploration-based trajectory optimization of llm agents,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 7584–7600. [50] Z. Deng, Z. Dou, Y. Zhu, J.-R. Wen, R. Xiong, M. Wang, and W. Chen, “From novice to expert: Llm agent policy optimization via step-wise reinforcement learning,” arXiv preprint arXiv:2411.03817, 2024. [51] S. Zhang, M. Yin, J. Zhang, J. Liu, Z. Han, J. Zhang, B. Li, C. Wang, H. Wang, Y. Chen, and Q. Wu, “Which agent causes task failures and when? on automated failure attribution of LLM multi-agent systems,” in Proceedings of the 42nd International Conference on Machine Learning (ICML), ser. Proceedings of Machine Learning Research, vol. 267. PMLR, 2025, spotlight. [52] E. Riddell, J. Riddell, G. Sun, M. Antkiewicz, and K. Czarnecki, “Stalled, biased, and confused: Uncovering reasoning failures in LLMs for cloud-based root cause analysis,” in Proceedings of the 3rd ACM

13

International Conference on AI Foundation Models and Software Engineering (FORGE), 2026. [53] T. Kim, W. Park, H. Yun, and K. Lee, “Why do AI agents systematically fail at cloud root cause analysis?” 2026. [54] J. Wang, X. Liao, and W. Wu, “STAR: A stage-attributed triage and repair framework for RCA agents in microservices,” 2026. [55] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao, “ReAct: Synergizing reasoning and acting in language models,” in The Eleventh International Conference on Learning Representations, 2023. [56] AIOps Challenge Committee, “AIOps Challenge 2025: Live Benchmark for Automated Root Cause Analysis,” https://www.aiops.cn/gitlab/ aiops-live-benchmark/aiopschallenge2025, 2025. [57] FudanSELab, “Train Ticket: A Benchmark Microservice System,” https: //github.com/FudanSELab/train-ticket, 2026, accessed: 2026-02-26. [58] M. Zhuge, C. Zhao, D. R. Ashley, W. Wang, D. Khizbullin, Y. Xiong, Z. Liu, E. Chang, R. Krishnamoorthi, Y. Tian, Y. Shi, V. Chandra, and J. Schmidhuber, “Agent-as-a-judge: Evaluate agents with agents,” in Proceedings of the 42nd International Conference on Machine Learning (ICML), ser. Proceedings of Machine Learning Research, vol. 267. PMLR, 2025, pp. 80 569–80 611. [59] J. Gu, X. Jiang, Z. Shi, H. Tan, X. Zhai, C. Xu, W. Li, Y. Shen, S. Ma, H. Liu, Y. Wang, and J. Guo, “A survey on LLM-as-a-judge,” 2024. [60] H. Li, Q. Dong, J. Chen, H. Su, Y. Zhou, Q. Ai, Z. Ye, and Y. Liu, “LLMs-as-Judges: A comprehensive survey on LLM-based evaluation methods,” 2024. [61] M. Deng, A. Ramachandran, C. Xu, L. Hu, Z. Yao, A. Datta, and H. Zhang, “ReFoRCE: A text-to-SQL agent with self-refinement, format restriction, and column exploration,” 2025. [62] V. Venktesh, M. Rathee, and A. Anand, “Trust but verify! a survey on verification design for test-time scaling,” 2025.

Related documents

• Multi-Agent Systems for Root Cause Analysis in Microservices#155366
• eIRWR: Enhanced Iterative Random Walk with Restart for Scalable Root Cause Analysis in Microservices#441771
• Graphical Causal Reasoning for Root Cause Analysis in Cloud Networks#271744
• Towards In-Depth Root Cause Localization for Microservices with Multi-Agent Recursion-of-Thought#187371
• Does Graph Structure Earn Its Place in Microservice Root-Cause Analysis? A Controlled Study on RCAEval, and What the Benchmark Was Really Measuring#1063280