Free to Look, Free to Reason: Simple LLM Harnesses Outdiagnose Hand-Crafted Root Cause Analysis Agents

arXiv:2610.35767v1 [cs.AI] 26 Oct 2026

Free to Look, Free to Reason: Simple LLM Harnesses Outdiagnose Hand-Crafted Root Cause Analysis Agents Tingzhu Bi

Xun Zhang

Jinglin Li

Peking University Beijing, China bitingzhu@stu.pku.edu.cn

Peking University Beijing, China zhangxun@stu.pku.edu.cn

Beijing University of Posts and Telecommunications Beijing, China jlli@bupt.edu.cn

Ping Wang

Meng Ma

Peking University Beijing, China pwang@pku.edu.cn

Peking University Beijing, China mameng@pku.edu.cn

Abstract

benchmarks run one reference agent or none [6, 10]. Whether handcrafting still pays off as models and harnesses improve is therefore unknown. We compare the three lines on the same cases. RCAtlas pools eight public RCA datasets into one format, screens 542 cases for validity and samples 90 across 23 scenarios. On it we run three simple harnesses that prescribe none of the model’s calls (one call over an evidence package, a SQL agent over the raw telemetry, and Codex), five published LLM agents, and statistical and causal methods, with six models, three reasoning efforts and one scorer. We change a method only where it cannot run on our data or has an implementation error. Harnesses that leave the model free to choose what to look at and how to reason win (Figure 1). The simple harnesses reach PR@1 65–77% on three mid-tier models, ahead of the published agents (16–53%), which are mostly ahead of the non-LLM methods (6–39%). The trajectories show why: when the program decides what to read, it can skip the decisive evidence, such as a log line or a service that stops receiving calls. A stronger model mainly improves reliability, a multi-turn harness mainly improves aptitude, and more reasoning effort changes little. Among runs more accurate than the best nonLLM method, every Pareto-optimal run in cost or time uses a simple harness. Our contributions are: • RCAtlas, 90 validated cases from eight public datasets in one format, with a scoring rule for labels at different levels (Section 4); • a field-wide evaluation of simple harnesses, published LLM agents and non-LLM methods on the same cases, across six models and three reasoning efforts (Section 6.1); • analyses of why methods succeed or fail: a case study, the share of model calls each method prescribes, and the separate effects of model capability, reasoning effort and multiturn investigation on accuracy and reliability (Sections 6.2– 6.5); • cost- and time-aware recommendations from the Pareto frontier (Section 6.6).

Root cause analysis (RCA) of microservice failures has attracted many LLM agents whose investigation is hand-crafted, through standard operating procedures, fixed stages and role agents. Each is evaluated on its own data against few baselines, so it is unknown whether this design beats letting a capable model investigate on its own. We compare three simple harnesses that prescribe none of the model’s calls (one call over an evidence package, a SQL agent and the Codex coding agent) with five published LLM RCA agents and with statistical and causal methods on RCAtlas, 90 validated cases from eight public datasets, across six models and three reasoning efforts. The simple harnesses are the most accurate (PR@1 65–77% on three mid-tier models), ahead of the published agents (16–53%) and the non-LLM methods (6–39%), on every model on which both families were run. The published agents fix from a tenth to all of their model calls in the program, and a case study shows how these prescriptions skip the decisive evidence. A stronger model mainly improves reliability, a multi-turn harness mainly improves aptitude, and more reasoning effort changes accuracy by a few points at most while multiplying the time by up to seven. Among runs more accurate than the best non-LLM method, every Pareto-optimal run in cost or time uses a simple harness.

1

Introduction

Failures in the microservices behind web services cascade across components, and operators must find the root cause among hundreds of services, pods and nodes from metrics, traces, logs and events [3, 8, 46]. Three lines of work improve this task: RCA methods, from statistical and causal ranking [17, 18, 20, 26, 35] to LLM agents; the LLMs underneath them; and general-purpose agent harnesses. A harness is the program around a model that feeds it data and carries out its actions. The simplest harness only feeds data, in a single call; richer ones let the model query the data, or write and run code as Codex [23] does. Published LLM RCA agents hand-craft the investigation with standard operating procedures, fixed stages, expert or role subagents and votes, so the program decides much of what is read and when [21, 25, 37, 41, 44]. Each is evaluated against a few baselines on one to three systems, often collected by its own authors, and

2

Related Work

Root cause analysis methods. Classical RCA for microservices ranks components from telemetry by statistical deviation (NSigma, 1

Bi et al.

Sol xhigh · Codex

V4.1 Flash · Codex

0.9 Sol xhigh · SQL agent

0.9 V4.1 Flash · SQL agent

0.8

0.8

Sol xhigh · SQL agent

Sol xhigh · Codex

Model (colour) GPT-6 Luna GPT-6.1 Sol GPT-6 Astra Claude Haiku 5.5 DeepSeek V4.1 Flash MiMo-V2.6-Pro

V4.1 Flash · One-shot Sol · SQL agent

Luna xhigh · Codex Haiku · SQL agent

0.7

0.7

PR@1

PR@1

Luna · Codex

0.6 0.5

0.5

0.4

0.4

0.3

0.3

10 s

100 s

Method (shape) One-shot SQL agent Codex FoA RCAgent RCA-Agent DecompRCA

0.6

1000 s

Effort (fill)

$0.001

$0.01

End-to-end time per case (seconds, log scale)

$0.1

$1

medium xhigh max Pareto line best non-LLM (NSigma)

Cost per case (USD, log scale)

Figure 1: Accuracy versus time and cost per case for every run (first pass of each run). Colour = model, shape = method, fill depth = reasoning effort (Luna and Sol). Dashed: PR@1 of the best non-LLM method (NSigma, 38.9%), a usability threshold fixed in advance; dotted: Pareto line over all runs above it. ChatGPT-login runs priced at OpenRouter list price; time excludes network waits. BARO [26], 𝜖-Diagnosis [32]), causal discovery and inference (CIRCA [17], RCD [13], DyCause [24], FaultInsight [4]), propagation on dependency graphs (MicroRCA [39], MicroRank [42]), or multi-source fusion (TORAI [28]). LLM-based methods mostly fix the investigation in the program: standard operating procedures the agent must follow (FoA [25]), a controller that never sees data and an executor that writes code (RCA-Agent [41]), observation management with expert sub-agents (RCAgent [37]), voting among role agents (mABC [44]), one agent per trace span (RCLAgent-V2 [43]), tree search with self-reflection (LATS-RCA [22]), or program-side retrieval followed by one LLM reranking call (DecompRCA [21]). Failure studies trace recurrent errors to the shared agent architecture rather than the model [15], catalogue reasoning failures across agentic and non-agentic prompting [30], and contrast successful and failed trajectories [19].

priors reach high scores on some of them [5, 11]. RCAtlas pools eight of these sources, screens every case for validity, and evaluates simple harnesses, published LLM agents and non-LLM methods on the same cases with one scorer. Agents on complex tasks. Evidence from other domains questions elaborate agent pipelines. A fixed localise–repair–validate pipeline competed with software-engineering agents at a fraction of the cost [40]; splitting a task across conversational turns degrades LLMs mainly through lower reliability [16]; and input length alone can hurt even when retrieval is perfect [9]. More broadly, general methods that scale with computation tend to overtake hand-engineered knowledge [33]. We test this question directly on RCA.

3

Background and Problem

Systems and telemetry. The systems under test are microservice applications on Kubernetes. Their entities are services, the pods and containers that run them, the nodes (hosts) that run the pods, databases and middleware, and the operations a service exposes. Four kinds of telemetry are recorded: metrics, time series per entity such as CPU and memory use, request counts, error rates and latency (timestamp, entity, metric name, value); traces, one span per call between operations (trace and span identifiers, parent span, operation, duration, status), which reveal who calls whom and where errors and delays arise; logs, timestamped lines per pod with a level and a free-text message; and events, Kubernetes records such as restarts, evictions or scheduling failures. Each case also carries an inventory of entities and their relations (calls, deployment, placement).

General-purpose agent harnesses. Coding agents have grown into general harnesses such as Codex [23] and Claude Code [1]: the model decides what to do next and acts by writing and running code, with no task-specific procedure. For RCA, such a harness without any RCA design already outperforms purpose-built agents on OpenRCA and RCAEval [12]. OpsHarness [12] then adds a selfevolving layer that learns from labelled past incidents of the same system. We take the complementary view: we test simple harnesses without such a layer (Codex, a SQL agent and a single call) against prescribed pipelines and non-LLM methods on the same cases. Benchmarks and evaluation. Model-wide evaluations rank models on many tasks and sub-capabilities; the Artificial Analysis Intelligence Index [2] has recently added an RCA-style task suite, ITBench-AA [31], built from ITBench [14]. RCA-specific datasets include RCAEval [29], OpenRCA [10, 41], LEMMA-RCA [45], the AIOps challenge data and RCA-100 [6, 38], and a fault-propagationaware TrainTicket benchmark [11]. Each covers one to three systems, and each evaluates a narrow slice of methods: OpenRCA 2.0 runs one reference agent across models, AgenticOpsEval [6] releases data and a scoring protocol without method results, and new methods are often evaluated mainly on data collected by their own authors [26, 28]. Audits show that simple rules or telemetry-free

Task. A case gives an incident’s flagged period, a period known to precede it, the candidate inventory and the telemetry of both periods. The method returns a ranked list of root-cause entities; we score the first five. Labels name one root-cause entity per case (three cases have two), at the level its dataset annotates: service, pod or node (we count databases as services and hosts as nodes). Scoring. Datasets label at different levels, and an answer may be finer than the label, such as a pod of the faulty service. Before 2

Free to Look, Free to Reason

matching, we therefore map each answered entity upward: it is replaced by itself and those of its parents in the inventory (container, pod, service) that lie at the levels the dataset annotates. Duplicates are then dropped, the first five distinct entities form the answer list, and they are compared with the labels by name. For example, when a dataset labels services, an answer naming a pod of cart counts as cart, and a second pod of cart further down the list adds nothing. Methods that rank metric columns, as most non-LLM methods do, are mapped the same way through the entity that owns each column. Each answer can match at most one root cause. We report two metrics. Let case 𝑖 of 𝑁 have the set of root causes 𝑇𝑖 , with 𝑡𝑖 = |𝑇𝑖 |, and the mapped answer list 𝑎𝑖,1, . . . , 𝑎𝑖,𝑘𝑖 (𝑘𝑖 ≤ 5). For a root cause 𝑔 ∈ 𝑇𝑖 , let 𝑟𝑖 (𝑔) be the position of the first answer that matches 𝑔, and 𝑟𝑖 (𝑔) = ∞ if none does (in particular when the case has no answer). Then PR@1 =

RR =

𝑁  1 ∑︁  1 ∃ 𝑔 ∈ 𝑇𝑖 : 𝑟𝑖 (𝑔) = 1 , 𝑁 𝑖=1

𝑁 1 ∑︁ 1 ∑︁ 1
, 𝑁 𝑖=1 𝑡𝑖 𝑔∈𝑇 1 + max 0, 𝑟𝑖 (𝑔) − 𝑡𝑖

the fault is not clean, or if the fault starts late in the flagged window. The most frequent reasons are an unclean before-period (69 cases) and no evidence (56). Sampling. From the 348 eligible cases we draw the 90-case core set (RCAtlas-Core): a minimum per scenario, then cases that diversify fault class, root-cause level, data kinds and telemetry volume. The core set spans 23 scenarios and ten fault classes; the root cause is a service in 59 cases, a pod in 20 and a node in 11. Cases range from 37k to 5.6M telemetry rows, and the exact random-guess hit rate under our scoring rule ranges from 0.25% to 16.7% (median 3.45%). All experiments in this paper use the core set. Availability. We provide the core-set case list and our code at (https://ufile.io/5ww9iyfb). The file cases/core90_cases.csv gives, for each case, its identifier in the source dataset, the time window, the data kinds and the answer template, so that every case can be rebuilt from its public source; cases/core90_labels.json gives the root-cause labels and their levels (59 services, 20 pods and 11 nodes). The folder methods/ holds the three simple harnesses (the one-shot method, the SQL agent and the Codex run settings) and our reimplementations of FoA (with its SOP files), RCAgent, DecompRCA and SimpleRCA. Shared code in rcatlas/ covers path configuration, an OpenAI-compatible model client, answer parsing and the data tools shared by FoA and RCAgent; prompts/ holds the one-shot prompt templates and background notes, and answer_templates/ the answer templates of the 23 scenarios in the version used in the paper, without hints on the root-cause level. We plan to release the converted core-set data (about 11 GB) and an extended set of about 500 cases built with the same rules.

(1)

(2)

𝑖

with 1/∞ = 0. PR@1 is thus the share of cases whose first answer is a root cause. In RR the first 𝑡𝑖 positions all score one, so with two root causes A and B both “A, B” and “B, A” score 1. Both means run over all 𝑁 cases, so a case without an answer scores zero. We do not report PR@𝑘 for larger 𝑘, because methods return different numbers of answers: the simple harnesses one to five, chosen by the model; RCA-Agent and mABC always one; DecompRCA about 14; and non-LLM methods a full ranking. To show how hard each case is, we compute its exact random-guess hit rate: the probability that one entity drawn uniformly from the inventory is a hit under the same mapping. Its median over the core set is 3.45%, and Table 1 reports its inverse as the effective number of candidates.

4

5

Methods Under Test

We evaluate three families (Table 2). Every method receives the same information about a case, namely the flagged period, the candidate inventory and the telemetry, and returns a ranked list of root-cause entities, of which the first five are scored.

The RCAtlas Benchmark

RCAtlas pools eight public RCA datasets (ten suites, counting RCAEval RE1–3 separately) into one case format, screens every candidate case for validity, and samples the survivors across systems and fault types (Table 1; Appendix C.1 summarises the construction procedure).

Simple harnesses. One-shot sends the whole case to the model in a single call: an evidence package with the system inventory, the metric series that moved (as raw samples), per-minute trace statistics, log templates and events; the model answers with up to five candidates and no tools (Appendix C.2). The SQL agent is a single ReAct loop over the raw files with DuckDB, modelled on the reference agent of OpenRCA 2.0 [10]: tools to list files, read a schema, run SQL and submit, at most 100 calls. Codex runs the Codex CLI [23] in the case folder with a task statement and six rules (read only this folder; no network or installs; do not modify the data; do not try to identify the source dataset; always write a final report; one hour at most); the model writes its own shell and Python.

Sources and format. The sources span nine systems, from Online Boutique, Sock Shop and TrainTicket to a bank, a telecom system and a cloud platform, with faults injected by chaos tools, by challenge organisers, or in drills on a managed Kubernetes service. Each case is converted to the same tables: an entity inventory with call, deployment and placement relations, and the metrics, traces, logs and Kubernetes events of a window around the fault, all in UTC. The conversion removes traces of the injection tools (for example their pods and log lines) so that no case reveals its answer.

Published LLM agents. We run RCA-Agent [41] and mABC [44] from their released code, and reimplement FoA [25], RCAgent [37] and DecompRCA [21], which have no released code, from their papers; our DecompRCA reproduces the paper’s numbers on RCAEval RE1. The reimplemented agents share one set of data tools over the raw files. We follow one rule for fairness: fix only what prevents a method from running on our data or what is an implementation

Validity screening. The sources have released 2,359 cases; we screened 542 of them, and 348 pass eight exclusion rules. Among others, a case is excluded if its labels contradict each other, if it contains several faults, if the culprit is missing from the data, if neither a judge model nor an automatic check finds evidence of the fault in the telemetry, if data are incomplete, if the period before 3

Bi et al.

Table 1: The RCAtlas core set by source. Case pool: all cases the source has released; selected: cases in the core set. : number of selected cases that contain metrics, traces, logs and Kubernetes events. Levels: root causes at service / pod / node level (databases counted as services, hosts as nodes). Rows: metric, trace and log rows in a case’s window (median). Candidates: effective number of candidates under our scoring rule, the inverse of the exact random-guess hit rate (median). ∗ Fault records of the public release we obtained. Source

Systems

Scenarios

Case Pool / Selected

Levels

Rows

Cand.

AIOps 2022 [7] AIOps 2025 [6] ITBench-SRE [14] LEMMA-RCA [45] OpenRCA 1.0 [41] OpenRCA 2.0 [10] RCA-100 [6, 38] RCAEval RE1–3 [29]

Online Boutique Online Boutique, TiDB OTel Demo EKS platform, Book-Info Telecom, Bank, Market TrainTicket, Hotel, OTel Demo OTel Demo OB, Sock Shop, TrainTicket

1 1 1 2 4 4 1 9

241∗ / 6 400 / 9 35 / 5 10 / 5 335 / 14 500 / 12 103 / 12 735 / 27

6 9 5 5 14 12 12 27

6 9 5 – 14 12 12 12

6 9 5 4 10 12 12 18

– – 5 – – – 12 –

0/3/3 3/3/3 5/0/0 0/5/0 3/9/2 12/0/0 9/0/3 27/0/0

650k 251k 457k 2.1M 571k 97k 1.0M 681k

61 15 19 347 61 25 34 22

Total

8 sources, 10 suites

23

2,359 / 90

90

70

76

17

59/20/11

629k

29

Table 2: Methods under test. Source: original code, original code with interface fixes, or our reimplementation from the paper; the SQL agent follows the reference agent of [10]; Model calls decided by: who issues the method’s model calls (Section 6.3).

error, and add no capability. Examples are converting time zones for RCA-Agent, restoring modules missing from mABC’s release, and completing tool descriptions in FoA’s code agent; the appendix summarises the changes. We exclude RCLAgent-V2 [43] (about USD 0.9 and up to an hour per case), LATS-RCA [22] (its released search strips the agents’ tools), EviRCA [36] and MicroRCA-Agent [34] (each needs a hand-written adapter per system or is built around one benchmark’s data layout), and supervised methods, which need a labelled training split. Non-LLM methods. NSigma, BARO, CIRCA, RCD and TORAI come from RCAEval [29]; FaultInsight from the authors’ code; SimpleRCA [11] is reimplemented from its description. All receive the start of the flagged period; methods with randomness are averaged over three runs. A random ranking gives the floor. Models and settings. Setting 1 runs every LLM method on three mid-tier models: GPT-6 Luna, DeepSeek V4.1 Flash (1017) and Claude Haiku 5.5. Setting 2 runs the simple harnesses on six models, adding GPT-6.1 Sol, GPT-6 Astra (at xhigh) and MiMo-V2.6-Pro. Below we call the models Luna, Flash, Haiku, Sol, Astra and MiMo. Setting 3 raises the reasoning effort of Luna and Sol to xhigh and max. The simple harnesses on Luna, and the SQL agent and Codex on Sol, are run three times. Tables report the mean over passes; Figures 1 and 5 and Table 8 use the first pass. No generation parameter is sent except the effort in Setting 3; each case has one hour, and a case without an answer scores zero.

6

6.1

Method

Source

Model calls decided by

Simple harnesses One-shot SQL agent [10] Codex [23]

ours ours Codex CLI

one call the model the model

Published LLM agents FoA [25] reimplemented RCAgent [37] reimplemented RCA-Agent [41] original, fixed DecompRCA [21] reimplemented mABC [44] original, fixed

SOP flow + model model + experts controller + executor program, one call role agents + voting flow

Non-LLM methods NSigma [27] BARO [26] CIRCA [17] RCD [13] TORAI [28] FaultInsight [4] SimpleRCA [11]

– – – – – – –

RCAEval [29] RCAEval RCAEval RCAEval RCAEval original reimplemented

Overall effectiveness (RQ1)

Table 3 shows three layers. Averaged over the three mid-tier models, the simple harnesses reach PR@1 65–77%, the published LLM agents 16–53%, and the non-LLM methods 6–39%, against 5.5% for a random ranking. Codex is best overall, and even the weakest simple harness (one-shot) leads the best published agent (FoA) by 12 points. The lower two layers overlap: NSigma (38.9%) beats DecompRCA and mABC. Every LLM method is less accurate on pod- and node-level root causes than on service-level ones, and the published agents lose the most: from service to pod, Codex drops 9 points, FoA and RCAgent 31–38. NSigma is the only method more accurate on pods than on services (55.0% vs. 33.9%), since resource faults show on the culprit’s own metrics. Appendix C analyses the hardest groups. More kinds of data do not make a case easier.

Experiments

We answer five research questions. RQ1 (effectiveness): how do simple harnesses, published LLM agents and non-LLM methods compare on the same cases? RQ2 (mechanism): why do the simple harnesses win, and how much of each method’s investigation is prescribed by its program? RQ3 (capability and effort): how do a stronger model and more reasoning effort change how agents investigate and how accurate they are? RQ4 (aptitude and stability): how much accuracy is reproducible across passes, and how much appears only in some? RQ5 (cost-effectiveness): which model and method combinations are Pareto-optimal in accuracy against cost and time? 4

Free to Look, Free to Reason normal

errors

alert

root cause

database

calls to it are refused. The root cause raises no errors of its own: its request count collapses, while cart’s error rate rises and cart logs “Connection refused (inventory:9090)”. All runs below use Luna.

ad recommendation

redis (cache)

refused

frontend-proxy

frontend

cart

inventory

errors 7×

alert: 51% errors

errors 1.7% ! 32%

requests 6k ! 55 / 30 s

checkout

+6 services

product-catalog

One-shot. The program decides what the model reads (the evidence-package rules), and the model does the reasoning. The package holds a log template from the cart pod (E27.2) whose refusal line grows from 43 lines before the fault to 609 during it. All three passes cite this item and rank inventory first (Figure 3).

Figure 2: Case rca100-main-0104. Orange: calls on the failing path; dashed red: the refused call.

SQL agent. The SQL agent starts from the symptom and follows “who cannot reach whom” to the root cause (Figure 4, left). Its 31 queries reduce to six steps: it reads which tables exist (steps 1–7); counts failing calls by service and finds cart failing (step 12); searches the logs for error keywords and finds 1,291 lines in which cart’s connections to inventory are refused (step 14); queries inventory’s requests per minute and confirms that they fall from about 1,450 to 109 in the first 30 seconds of the fault and then to none; checks inventory’s pod logs and its place in the topology (steps 18–30, including an SQL error at step 17 that it fixes itself); and submits inventory (step 31).

Evidence package (1 of 136 items, abridged) E27.2

log templates, pod cart-…-q9w2g before during [ERROR] inventory call failed: Connection refused (inventory:9090) 43 609 [INFO] got stock of product … 3205 0 (log text translated)

Answer (one call)

"cart requests to inventory:9090 were refused; cart then returned an Internal error, which surfaced as frontend cart errors." [E27.2, …] ranked_candidates: ["inventory"] ✓

Figure 3: One-shot (Luna, pass 1): the evidence item the answer cites, and the answer. The 17 cases with all four kinds, which come from RCA-100 and ITBench-SRE, are the hardest for the simple harnesses (52–71%), and they form the only column in which a published agent matches the best simple harness: FoA reaches 70.6%, as Codex does. Conversely, the ten cases with metrics only are the easiest for the SQL agent (95.6%). The ordering does not depend on the model (Tables 4 and 5). On each of Luna, Haiku and Flash, every simple harness is ahead of every published agent (one-shot’s 5.9-point lead on Luna is not significant), and Codex is best or tied on every model it ran on. Across models, the simple harnesses fall into two tiers at the jump of the AA index from 34.5 to 45.1: 60–71% on Luna and Haiku, 74–87% on Flash, Sol and Astra. Leaving out MiMo (see below), 12 of the 13 cross-tier model pairs differ significantly, but only one of the nine within-tier pairs (Appendix B). MiMo is the exception: its AA index (46.3) places it in the upper tier, but its accuracy (58–74%) matches the lower one, partly because it finished only 69–86 of the 90 cases.

6.2

Codex. Codex casts a wide net first and then narrows down, in seven commands. After listing and loading the files, its third command runs comparison code it wrote itself over every entity and metric before and during the fault, together with events, spans and error logs; cart’s error rate (0.017 to 0.325) ranks first, and the refusal line is already in the output. It then pulls the events and the metrics and logs of inventory, cart and frontend side by side (commands 4–6), reads the logs of the two minutes around the onset (command 7), which show inventory serving until 01:59:22 and cart refused from 01:59:23, and writes its report naming inventory. FoA.. The SOP chosen by the program decides what is read. FoA fills in the fault description itself as “microservice incident”, which matches a generic SOP whose fixed steps never read logs; its judge declares the root cause found, cart, in the first round, after four calls of the controller (11 model calls in all). In the original paper the alert carries the fault type (“Network Issue” in its Figure 4), from which the SOP is chosen; we give every method only the anomalous period, because the fault type would reveal the answer. Four of our 12 SOPs, written after the paper’s examples, read logs, but none matched this case (the SOP files are in the released code; Appendix A).

Why simple harnesses win: a case study (RQ2)

The difference between the families is who decides what to look at and how to compute on it, which is what the prescribed-call ratio (Section 6.3) measures. The SQL agent writes its own queries and Codex writes its own analysis code, so the model chooses both what to read and how to compare it, and none of their calls is prescribed. The published agents fix part of this in the program. FoA can only choose among the steps of its built-in procedures, so when the matched procedure has no log step, it never sees the logs. RCA-Agent’s controller never touches the data and learns about the case only from its executor’s summaries, and mABC starts from an alert its program computes and walks the call chain from there. In the terms of our title, the SQL agent and Codex are free to look and free to reason; one-shot is free to reason over evidence that rules select. We follow one case through the methods, rca100-main-0104 from RCA-100 (Figure 2). The inventory Service selects no pods, so cart’s

mABC.. The framework walks down the call chain from the alert: the frontend-proxy ingress at 02:03, frontend’s POST, and its four callees. In that minute only cart’s AddItem still receives a call (one, failing); the other three receive none. Inventory, three hops down, is never compared at the interface level; it served 109 calls with no errors at 01:59 and none afterwards. mABC stops at cart. A method that walks the call chain and inspects only the calls a service receives cannot see a service that stops receiving them. Non-LLM methods. NSigma and BARO score each metric by its largest standardised deviation, keeping the sign, so only metrics that rise score high. Inventory’s request count falls sharply; its z-score is large and negative, and the metric ranks second to last among the 373 columns. 5

Bi et al.

Table 3: Main results on the 90 core cases (%). LLM methods: mean over Luna, Flash and Haiku. Data kinds: metrics, traces, logs, events. Red tint: drop below the column best (white = best, 50 points below , 100 points below ). Rel. to best: PR@1 as a share of the column best. Random: expected score of a random ordering of the inventory. All (90)

PR@1 by data kinds

Method

PR@1

RR

(10)

(4)

Codex† SQL agent One-shot FoA RCAgent RCA-Agent DecompRCA mABC§ NSigma TORAI‡ SimpleRCA BARO CIRCA FaultInsight‡ RCD‡ Random

77.2 72.8 65.3 53.3 49.3 43.3 28.5 15.6 38.9 33.3 33.3 22.2 17.8 18.9 5.9 5.5

80.6 74.9 69.8 58.7 53.7 41.9 34.4 16.1 47.5 46.8 45.1 31.7 29.2 29.7 10.5 12.0

85.0 95.6 76.7 50.0 56.7 33.3 53.3 5.0 80.0 80.0 20.0 40.0 40.0 33.3 0.0 7.1

91.7 80.6 83.3 41.7 41.7 16.7 0.0 0.0 25.0 25.0 50.0 0.0 0.0 25.0 0.0 2.8

(10) 86.7 93.3 70.0 53.3 36.7 33.3 16.7 5.0 50.0 40.0 40.0 10.0 10.0 16.7 0.0 6.6

PR@1 by level

(49)

(17)

74.8 67.8 65.1 49.0 51.0 49.7 30.6 15.3 36.7 32.7 34.7 24.5 16.3 16.3 6.8 5.7

70.6 60.1 52.3 70.6 49.0 43.1 21.6 32.4 17.6 5.9 29.4 17.6 17.6 17.6 11.8 3.9

PR@1

service (59)

pod (20)

node (11)

rel. to best

81.4 79.7 72.5 66.7 59.3 48.6 31.6 21.2 33.9 35.6 39.0 25.4 16.9 24.3 5.1 6.3

72.5 63.9 52.8 28.3 28.3 30.0 28.3 7.5 55.0 40.0 15.0 20.0 10.0 5.0 3.3 3.6

63.6 52.5 49.5 27.3 33.3 39.4 12.1 0.0 36.4 9.1 36.4 9.1 36.4 15.2 15.2 4.8

100.0 94.3 84.6 69.1 63.8 56.1 36.9 20.1 50.4 43.2 43.2 28.8 23.0 24.5 7.7 7.1

† Haiku not run. § DeepSeek not run. ‡ mean over three passes.

Table 4: The simple harnesses across six models (PR@1 / RR, %), ordered by the AA Intelligence Index [2]. Luna and Sol (SQL agent, Codex): mean of three passes. MiMo has no effort setting and finished 69–86 of 90 cases. 6

5.5

4.1

2.6

6.1

Luna

Haiku

Flash

Pro

Sol

29.9 medium

34.5 medium

45.1 default

SQL agent 68.5 / 69.5 Codex 68.9 / 73.1 One-shot 60.4 / 64.4

71.1 / 74.5 – 60.0 / 66.0

78.9 / 80.6 64.4 / 66.1 84.1 / 82.8 85.6 / 88.1 74.4 / 77.6 87.0 / 88.2 75.6 / 79.1 57.8 / 63.6 74.4 / 77.7

Method AA index Effort

6.3

46.3 –

6

47.8 medium

How much of the investigation is prescribed (RQ2)

Astra

52.4 xhigh – 86.7 / 88.3 82.2 / 84.3

Table 5: The published LLM agents on the three mid-tier models (PR@1 / RR, %; header as in Table 4). mABC: partial runs (47 and 85 of 90 cases answered). Method AA index Effort FoA RCAgent RCA-Agent DecompRCA mABC

6

5.5

Luna

Haiku

4.1

Flash

29.9 medium

34.5 medium

45.1 default

54.4 / 57.1 54.4 / 57.0 32.2 / 30.6 24.4 / 30.4 11.1 / 11.7

46.7 / 56.7 43.3 / 48.4 34.4 / 33.3 28.9 / 35.9 20.0 / 20.6

58.9 / 62.2 50.0 / 55.6 63.3 / 61.7 32.2 / 37.0 –

and summary calls, RCA-Agent’s summary after each successful execution, forced final reports, and DecompRCA’s reranking of 15 candidates selected by its program.

To measure what the case study shows across all cases, we compute each method’s prescribed-call ratio: the share of its model calls that its program issues at fixed points rather than the model choosing to make them, over the trajectories of all 90 cases on the three mid-tier models (Table 6). From the run records and each method’s code, we sort every model call into one of three kinds:

For a method run on model 𝑚, let 𝐶𝑚 be the cases with a trajectory, and let case 𝑖 have 𝑐𝑖 model calls, of which 𝑓𝑖 are prescribed. The ratio is averaged over cases and then over the three models M: 1 ∑︁ 1 ∑︁ 𝑓𝑖 . (3) 𝜌= |M| |𝐶𝑚 | 𝑖 ∈𝐶 𝑐𝑖 𝑚∈ M

𝑚

The three simple harnesses prescribe no call. Every published agent does: at least 0.10 for mABC, 0.29 for RCA-Agent, 0.35 for RCAgent, 0.51 for FoA, and 1.00 for DecompRCA, whose single call is placed by its program. Within the published agents the ratio does not order accuracy (RCA-Agent prescribes fewer calls than FoA yet is less accurate); the clear divide is between harnesses that prescribe nothing and agents that prescribe something. Three points qualify these numbers. First, the one call of one-shot counts as a main call, because the model receives the whole task, whereas DecompRCA’s single call counts as prescribed, because

• main: the next turn of the model in charge of the task, such as every call of Codex and the SQL agent, the main agent of FoA, and the controllers of RCAgent and RCA-Agent; • delegated: a call the main model chooses to hand to another role, such as FoA’s code agent or RCA-Agent’s executor writing code; • prescribed: a call the program makes whatever the model decides, such as FoA’s action agent at every step and its judge after every SOP run, RCAgent’s chunked analysis 6

Free to Look, Free to Reason

SQL agent: the model writes each query read the schema of six tables

1-7

12 which spans fail? WHERE status='error' GROUP BY svc, op frontend AddItem 637 cart GetInventory 562 14 what do the logs say? WHERE msg LIKE '%fail%' OR … cart: Connection refused (inventory:9090) 1,291 18-30

list and load every file

3

compare everything at once

inventory, cart, frontend side by side

judge: root cause found

4

✓

abnormal metrics per service

1 call

100% errors

"the only checked downstream candidate with a matching error"

✗

4 steps · 11 model calls (4 by the controller) · 51 s

31 queries · 22 model calls · 107 s

4-12

conclude

report

["deploy:cart", …]

alert: frontend-proxy ingress

19-25 callees at 02:03 cart AddItem 1 call 100% err cart GetCart no calls checkout no calls product-catalog no calls inventory (3 hops down): not compared (109 calls at 01:59, 0% err; then none)

"failures are localized to svc:cart … inventory … healthy"

report

1-3 02:03

13-18 walk down from the alert ingress → frontend-POST → 4 callees

2-3 run the SOP's fixed steps a. abnormal metrics 81 / 448 b. entry requests c. traces of those services (no step reads logs)

7 commands · 8 model calls · 57 s

✓

mABC: follow the alert down the call chain

pick an SOP from the alert

match_sop("microservice incident") → "Microservice Issue" 0.915 not matched (these read logs): Memory · Network Loss · Code/Config · DB/Cache

for (e, k), x in m.groupby([ent, kpi]): d = |med(during) - med(before)| + events, spans by service, error logs #1 cart error_rate 0.017 → 0.325 cart: Connection refused (inventory:9090) 01:59:23

["deploy:inventory", …]

submit

["deploy:inventory", …]

1

7 zoom in on 01:58:30–02:00:30 inventory serves until 01:59:22 cart refused from 01:59:23

GROUP BY time_bucket('30 s'), svc inventory spans 1,450/min → 109 in first 30 s → none (17: syntax error, rewritten in 18) 31

1-2

4-6

check the callee, inventory

FoA: a fixed SOP decides what to read

Codex: the model writes analysis code

report ["svc:cart"]

✗

26 actions · 32 model calls · 631 s

Figure 4: Four runs on case rca100-main-0104 (Luna). Grey: what the model wrote; black: what came back; red dot: the turning point. Times as recorded, including network waits. the model only reranks a list the program has already narrowed. One-shot’s evidence package is still selected by rules (it keeps only metric series that moved), which is why the case study says the program decides what one-shot reads. Second, mABC’s records do not say which role made each call, so its ratio is a lower bound. Third, simply counting calls does not separate the families: the SQL agent makes 27 calls per case, the same order as the multi-step published agents (35–50), while one-shot and DecompRCA both make one.

covers every effort level (Figure 8). In all three contrasts the runs query the root cause in two to five more cases out of a hundred (RC visited); the difference is what the model makes of it. Sol turns what it sees into a correct first answer, while a larger budget mostly adds queries that the model then does not use to change its answer. A larger budget buys exploration; a stronger model buys direction.

6.5

The comparisons so far use one pass per run, but an LLM’s answers vary between passes. Following Laban et al. [16], we run Luna and Sol three times at medium effort and measure aptitude and unreliability. Laban et al. [16] define them through the 90th and 10th percentiles of a score over many samples; with three passes 𝑝 we use the best and the worst pass. Let 𝑠𝑖 ∈ {0, 1} be the PR@1 of case 𝑖 in pass 𝑝 ∈ {1, 2, 3}. Then

Table 6: Prescribed-call ratio (Equation 3) on all 90 cases and the three mid-tier models. Calls, actions: per case; per-case range: lowest and highest value over all cases and models; fixed stages: rules or stages read from the code. mABC: lower bound. Method

6.4

Calls Actions Prescribed/ case / case call ratio

Per-case range

Fixed stages

Codex SQL agent One-shot

8.5 26.8 1.0

7.5 41.7 0.0

0.00 0.00–0.00 0.00 0.00–0.08 0.00 –

0 1 0

FoA RCAgent RCA-Agent mABC DecompRCA

36.4 35.1 50.3 41.2 1.0

22.3 21.5 32.3 35.9 2.0

0.51 0.35 0.29 ≥0.10 1.00

8 4 4 3 2

0.46–0.55 0.05–0.81 0.06–0.33 – 1.00–1.00

Aptitude and stability (RQ4)

𝑁

𝐴=

1 ∑︁ 𝑝 max 𝑠 , 𝑁 𝑖=1 𝑝 𝑖

𝑁

𝑈 =

 1 ∑︁  𝑝 𝑝 max 𝑠𝑖 − min 𝑠𝑖 , 𝑝 𝑝 𝑁 𝑖=1

(4)

so 𝐴 is the share of cases solved in at least one pass, 𝑈 the share solved in some passes but not all, and 𝐴 − 𝑈 the share solved in all three. Figure 6 shows both. A stronger model mainly buys reliability: from Luna to Sol, A rises by only 2–7 points, but U falls from 32–36 to 7–8. On Luna, a single pass of Codex ranges from 62% to 76%. A multi-turn agent mainly buys aptitude: on Luna, the SQL agent and Codex solve 10–11 points more cases at least once than one-shot, with similar U (30–36). One-shot never solves 26% of the cases in any pass, against 14–16% for the two agents, so it misses cases outright rather than by chance. This is the opposite of the finding of Laban et al. [16], where splitting a task over turns mainly costs reliability. Single passes on a weaker model are therefore noisy estimates. On Luna the three passes of Codex give 62%, 69% and 76%, and those of the SQL agent 66% to 71%, so a comparison between methods based on one pass of a weak model can flip. On Sol the passes differ by at most 4.5 points (Codex 86.7–87.8%, SQL agent 81.1–85.6%).

Model capability and reasoning effort (RQ3)

Because the simple harnesses leave the investigation to the model, their accuracy rests on the model: it rises with the model’s tier (Section 6.1). We now ask how a stronger model and more reasoning effort change the investigation itself. Figure 5 compares three pairs of runs. A stronger model investigates less and finds more (contrast A): from Luna to Sol, both agents make fewer calls and query fewer entities (Codex 16% → 12%, SQL agent 21% → 12%), yet reach the root cause in more cases and gain 14–18 points of PR@1. More reasoning effort does the opposite (contrasts B and C, Luna medium to max and Sol medium to xhigh): agents make more calls and query more entities, but gain only 2–4 points of PR@1 (first pass of the SQL agent and Codex), at 1.5 to 4 times the wall time. Appendix B

6.6

Cost-effectiveness (RQ5)

In operations, accuracy is not the only concern: an incident needs an answer quickly and at a sustainable cost. Figure 1 plots PR@1 against time and against cost per case. We count a run as usable only 7

Bi et al.

PR@1 (curve height)

Codex

A B C

1 Sol medium 87 0 1

Luna max 72

Luna medium 69

0 1 Sol medium 87 0

SQL agent Sol medium 86

Luna medium 69

Luna medium 71

Sol xhigh 89

Sol medium 86

more actions, later hit !

à fewer actions, earlier hit

à fewer actions, earlier hit

Luna medium 71

Contrast Method

Calls

Actions

Entities coverage

RC visited

ΔPR@1

A

Codex SQL

8.5→7.5 7.5→6.1 16%→12% 89%→94% 19.7→8.9 31.4→24.9 21%→12% 91%→93%

+17.8 +14.4

B

Codex SQL

8.5→13.3 7.5→12.2 16%→25% 89%→94% 19.7→24.3 31.4→53.9 21%→27% 91%→96%

+3.3 +4.4

C

Codex SQL

7.5→9.2 6.1→8.5 12%→17% 94%→99% 8.9→10.3 24.9→33.8 12%→18% 93%→96%

+2.2 +2.2

Luna max 76

Sol xhigh 88

more actions, later hit !

Figure 5: Stronger model versus more reasoning effort (90 cases, first pass). Contrasts: A = Luna → Sol (medium); B = Luna medium → max; C = Sol medium → xhigh. Left: cases placed by the geometric mean of tool actions and the step at which the root cause is first queried (log scale; left = fewer, earlier); curve height = the run’s PR@1; dotted line = median; arrow = shift from blue to orange. Right: per-case means; Entities coverage = share of inventory entities queried; RC visited = share of cases whose queries ever name the root cause.

100% A= 84

A= 91

Model: Luna ! Sol (SQL agent, medium)

100%

100% A= 86

U=7

A= 88

U=30

U=36

70%

U=36

70%

A= 84

A= 74

80 U=32

PR@1

A= 86 U=8

84

70%

methods may still differ from their originals; the appendix summarises the changes. Fifth, the case study follows a single case. The prescribed-call ratio extends its mechanism to all 90 cases, but it measures how much of an investigation is fixed, not that this fixing causes the lower accuracy; an ablation that lifts the prescriptions from one agent would test that directly.

Method (Luna, medium)

U=32

Model: Luna ! Sol (Codex, medium)

52 40%

Luna

50 Sol

40%

Luna

50 Sol

40%

8

52

44 Oneshot

A = aptitude (solved in at least one of three passes) U = unreliability (solved only sometimes)

Figure 6: Aptitude (A) and unreliability (U) over three passes (PR@1, medium effort). Whiskers: A−U to A; box: range of the three passes; line: their mean.

if it is more accurate than the best non-LLM method (NSigma, PR@1 38.9%), a threshold fixed before drawing the frontier, and draw the Pareto lines through all usable runs. Seven runs of published agents qualify (FoA, RCAgent and RCA-Agent, 43–63%), yet every run on either line uses a simple harness: each published agent is beaten in accuracy by a simple harness that is no more costly, and by one that is no slower. The DecompRCA runs, the cheapest and fastest of all, fall below the threshold (24–29%). The frontier gives three practical choices. When cost matters most, Luna with Codex reaches 68.9% for under one cent per case. When accuracy per dollar matters, Flash with Codex reaches 85.6% for about five cents. When the answer is needed quickly, Sol with the SQL agent answers in 45 s at 85.6%. The most accurate run, Sol at xhigh with Codex (88.9%), costs about 26 cents and takes 277 s per case, so the last few points of accuracy cost roughly five times as much as the cheapest strong setup.

7

Conclusion

For microservice root cause analysis, a capable model that is given the data and left to decide what to look at and how to reason outdiagnoses agents whose investigation is hand-crafted, and both mostly outperform statistical and causal methods. A stronger model improves accuracy and reliability; more reasoning effort adds little. Practitioners can start from a simple harness and pick the model and harness on the cost–accuracy frontier. Designers of RCA agents should compare against such harnesses on shared, validated cases, which RCAtlas provides. Our results do not show that domain knowledge is useless. They show that fixing it into the control flow hurts when a procedure does not fit the case, as in our case study, where the procedure that FoA matched never read the logs. Offering procedures to the model as optional guidance, rather than prescribing them, is a natural next step.

SQL Codex agent

References [1] Anthropic. 2026. Claude Code. https://github.com/anthropics/claude-code Software; accessed 2026-10-26. [2] Artificial Analysis. 2026. Artificial Analysis Intelligence Index. https: //artificialanalysis.ai/methodology/intelligence-benchmarking Version 4.3.2; accessed 2026-10-26. [3] Betsy Beyer, Chris Jones, Jennifer Petoff, and Niall Richard Murphy. 2016. Site reliability engineering: How Google runs production systems. O’Reilly Media, Inc. [4] Tingzhu Bi, Zhang Yang, Yicheng Pan, Yu Zhang, Meng Ma, Xinrui Jiang, Linlin Han, Feng Wang, Xian Liu, and Ping Wang. 2024. FaultInsight: Interpreting Hyperscale Data Center Host Faults. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery, 141–152. doi:10.1145/3637528.3672051 [5] Imad Buljić. 2026. Does Graph Structure Earn Its Place in Microservice RootCause Analysis? A Controlled Study on RCAEval, and What the Benchmark Was Really Measuring. arXiv:2609.27069 https://arxiv.org/abs/2609.27069 [6] Yuanhong Cai, Xiaohui Nie, Kanglin Yin, Changhua Pei, Yongqian Sun, Shenglin Zhang, Haibin Liu, Guiyang Liu, Xidao Wen, Fang Situ, and Dan Pei. 2026. A Multi-Dataset Benchmark for Evaluating LLM Agents in Microservice Failure Diagnosis. arXiv:2606.29193 https://arxiv.org/abs/2606.29193 [7] China Computer Federation, Tsinghua University, and China Construction Bank. 2022. CCF International AIOps Challenge 2022. https://www.aiops. cn/aiopschallenge/ Competition and dataset; accessed 2026-10-26.

Limitations

Our study has five limitations. First, Codex is the only generalpurpose coding harness we test; others, such as Claude Code, may behave differently. Second, all models are from 2026, so we cannot say whether the ordering holds for the weaker models for which many prescribed workflows were designed. Third, budget kept us from current frontier models such as Fable, Opus or Argon. Fourth, despite careful conversion and reimplementation, our datasets and 8

Free to Look, Free to Reason

[25] Changhua Pei, Zexin Wang, Fengrui Liu, Zeyan Li, Yang Liu, Xiao He, Rong Kang, Tieying Zhang, Jianjun Chen, Jianhui Li, Gaogang Xie, and Dan Pei. 2025. Flow-ofAction: SOP Enhanced LLM-Based Multi-Agent System for Root Cause Analysis. In Companion Proceedings of the ACM on Web Conference 2025. Association for Computing Machinery, 422–431. doi:10.1145/3701716.3715225 [26] Luan Pham, Huong Ha, and Hongyu Zhang. 2024. BARO: Robust Root Cause Analysis for Microservices via Multivariate Bayesian Online Change Point Detection. Proceedings of the ACM on Software Engineering 1, FSE (2024), 2214–2237. doi:10.1145/3660805 [27] Luan Pham, Huong Ha, and Hongyu Zhang. 2024. Root Cause Analysis for Microservice System based on Causal Inference: How Far Are We?. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. Association for Computing Machinery, 706–715. doi:10.1145/3691620.3695065 [28] Luan Pham, Huong Ha, Xiuzhen Zhang, and Hongyu Zhang. 2026. TORAI: Multi-source Root Cause Analysis for Blind Spots in Microservice Service Call Graph. Proceedings of the ACM on Software Engineering 3, FSE (2026), 2928–2951. doi:10.1145/3808137 [29] Luan Pham, Hongyu Zhang, Huong Ha, Flora Salim, and Xiuzhen Zhang. 2025. RCAEval: A Benchmark for Root Cause Analysis of Microservice Systems with Telemetry Data. In Companion Proceedings of the ACM on Web Conference 2025. Association for Computing Machinery, 777–780. doi:10.1145/3701716.3715290 [30] Evelien Riddell, James Riddell, Gengyi Sun, Michał Antkiewicz, and Krzysztof Czarnecki. 2026. Stalled, Biased, and Confused: Uncovering Reasoning Failures in LLMs for Cloud-Based Root Cause Analysis. In Proceedings of the 2026 IEEE/ACM Third International Conference on AI Foundation Models and Software Engineering. Association for Computing Machinery, 172–183. doi:10.1145/3793655.3793732 [31] Ayhan Sebin, Saurabh Jha, and Rohan Arora. 2026. ITBench-AA: Frontier Models Score Below 50% on the First Benchmark for Agentic Enterprise IT Tasks — by Artificial Analysis and IBM. https://huggingface.co/blog/ibm-research/itbenchaa Hugging Face community article; accessed 2026-10-26. [32] Huasong Shan, Yuan Chen, Haifeng Liu, Yunpeng Zhang, Xiao Xiao, Xiaofeng He, Min Li, and Wei Ding. 2019. 𝜖 -Diagnosis: Unsupervised and Real-time Diagnosis of Small-window Long-tail Latency in Large-scale Microservice Platforms. In The World Wide Web Conference. Association for Computing Machinery, 3215–3222. doi:10.1145/3308558.3313653 [33] Richard S. Sutton. 2019. The Bitter Lesson. http://www.incompleteideas.net/ IncIdeas/BitterLesson.html Online essay; 13 March 2019; accessed 2026-10-26. [34] Pan Tang, Shixiang Tang, Huanqi Pu, Zhiqing Miao, and Zhixing Wang. 2025. MicroRCA-Agent: Microservice Root Cause Analysis Method Based on Large Language Model Agents. arXiv:2509.15635 https://arxiv.org/abs/2509.15635 [35] Ping Wang, Jingmin Xu, Meng Ma, Weilan Lin, Disheng Pan, Yuan Wang, and Pengfei Chen. 2018. CloudRanger: Root Cause Identification for Cloud Native Systems. In 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGRID 2018, Washington, DC, USA, May 1-4, 2018. IEEE Computer Society, 492–502. [36] Yuhao Wang, Zhen Qin, Xingliang Wang, Guochang Li, Weize Li, and Shuiguang Deng. 2026. EviRCA: Decoupling Evidence Extraction from Reasoning for Microservice Root-Cause Analysis. arXiv:2609.19825 https://arxiv.org/abs/2609. 19825 [37] Zefan Wang, Zichuan Liu, Yingying Zhang, Aoxiao Zhong, Jihong Wang, Fengbin Yin, Lunting Fan, Lingfei Wu, and Qingsong Wen. 2024. RCAgent: Cloud Root Cause Analysis by Autonomous Agents with Tool-Augmented Large Language Models. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management. Association for Computing Machinery, 4966–4974. doi:10.1145/3627673.3680016 [38] Xidao Wen, Haibin Liu, Guiyang Liu, Cheng Zhang, Fang Situ, and Qi Zhou. 2026. RCA-100: A Chain-Reasoning Benchmark for Root Cause Analysis on CloudNative Microservices. https://www.aiops.cn/gitlab/aiops-live-benchmark/ agenticopseval/-/tree/main/RCA100 Dataset; 103 chaos-drill incidents; CC BYNC-SA 4.0; accessed 2026-10-26. [39] Li Wu, Johan Tordsson, Erik Elmroth, and Odej Kao. 2020. MicroRCA: Root Cause Localization of Performance Issues in Microservices. In NOMS 2020 - 2020 IEEE/IFIP Network Operations and Management Symposium. IEEE, 1–9. doi:10. 1109/noms47738.2020.9110353 [40] Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2025. Demystifying LLM-Based Software Engineering Agents. Proceedings of the ACM on Software Engineering 2, FSE (2025), 801–824. doi:10.1145/3715754 [41] Junjielong Xu, Qinan Zhang, Zhiqing Zhong, Shilin He, Chaoyun Zhang, Qingwei Lin, Dan Pei, Pinjia He, Dongmei Zhang, and Qi Zhang. 2025. OpenRCA: Can Large Language Models Locate the Root Cause of Software Failures?. In International Conference on Learning Representations. 84562–84590. https://proceedings.iclr.cc/paper_files/paper/2025/hash/ d29b8d53678015079e1d245c023e49d2-Abstract-Conference.html [42] Guangba Yu, Pengfei Chen, Hongyang Chen, Zijie Guan, Zicheng Huang, Linxiao Jing, Tianjun Weng, Xinmeng Sun, and Xiaoyun Li. 2021. MicroRank: End-to-End Latency Issue Localization with Extended Spectrum Analysis in Microservice Environments. In Proceedings of the Web Conference 2021. Association for Computing Machinery, 3087–3098. doi:10.1145/3442381.3449905

[8] Yingnong Dang, Qingwei Lin, and Peng Huang. 2019. AIOps: real-world challenges and research innovations. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). IEEE, 4–5. [9] Yufeng Du, Minyang Tian, Srikanth Ronanki, Subendhu Rongali, Sravan Babu Bodapati, Aram Galstyan, Azton Wells, Roy Schwartz, Eliu A Huerta, and Hao Peng. 2025. Context Length Alone Hurts LLM Performance Despite Perfect Retrieval. In Findings of the Association for Computational Linguistics: EMNLP 2025. Association for Computational Linguistics, 23281–23298. doi:10.18653/v1/ 2025.findings-emnlp.1264 [10] Aoyang Fang, Yifan Yang, Jin’ao Shang, Qisheng Lu, Junjielung Xu, Rui Wang, Songhan Zhang, Yuzhong Zhang, Boxi Yu, and Pinjia He. 2026. OpenRCA 2.0: From Outcome Labels to Causal Process Supervision. arXiv:2606.27154 https://arxiv.org/abs/2606.27154 [11] Aoyang Fang, Songhan Zhang, Yifan Yang, Haotong Wu, Junjielong Xu, Xuyang Wang, Rui Wang, Manyi Wang, Qisheng Lu, and Pinjia He. 2026. Rethinking the Evaluation of Microservice RCA with a Fault Propagation-Aware Benchmark. Proceedings of the ACM on Software Engineering 3, FSE (2026), 1605–1627. doi:10. 1145/3797100 [12] Haiyu Huang, Jiewei Lyu, Zhihan Jiang, Jinyang Liu, Xiao He, Tieying Zhang, Wu Xiang, and Michael R. Lyu. 2026. From General Agents to RCA Experts: A Self-Evolving Harness for Root Cause Analysis. arXiv:2608.25661 https: //arxiv.org/abs/2608.25661 [13] Azam Ikram, Sarthak Chakraborty, Subrata Mitra, Shiv Kumar Saini, Saurabh Bagchi, and Murat Kocaoglu. 2022. Root Cause Analysis of Failures in Microservices through Causal Discovery. In Advances in Neural Information Processing Systems, Vol. 35. Curran Associates, Inc., 31158–31170. doi:10.52202/068431-2259 [14] Saurabh Jha, Rohan R. Arora, Yuji Watanabe, Takumi Yanagawa, Yinfang Chen, Jackson Clark, Bhavya Bhavya, Mudit Verma, Harshit Kumar, Hirokuni Kitahara, Noah Zheutlin, Saki Takano, Divya Pathak, Felix George, Xinbo Wu, Bekir O Turkkan, Gerard Vanloo, Michael Nidd, Ting Dai, Oishik Chatterjee, Pranjal Gupta, Suranjana Samanta, Pooja Aggarwal, Rong Lee, Jae-Wook Ahn, Debanjana Kar, Amit Paradkar, Yu Deng, Pratibha Moogi, Prateeti Mohapatra, Naoki Abe, Chandrasekhar Narayanaswami, Tianyin Xu, Lav R. Varshney, Ruchi Mahindru, Anca Sailer, Laura Shwartz, Daby Sow, Nicholas C. M. Fuller, and Ruchir Puri. 2025. ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks. In Proceedings of the 42nd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 267). PMLR, 27134–27197. https: //proceedings.mlr.press/v267/jha25a.html [15] Taeyoon Kim, Woohyeok Park, Hoyeong Yun, and Kyungyong Lee. 2026. Why Do AI Agents Systematically Fail at Cloud Root Cause Analysis?. In Cloud Intelligence/AIOps Workshop. https://cloudintelligenceworkshop.org/papers/ aiops26-Kim.pdf [16] Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville. 2026. LLMs Get Lost In Multi-Turn Conversation. In The Fourteenth International Conference on Learning Representations. https://openreview.net/forum?id=VKGTGGcwl6 [17] Mingjie Li, Zeyan Li, Kanglin Yin, Xiaohui Nie, Wenchi Zhang, Kaixin Sui, and Dan Pei. 2022. Causal Inference-Based Root Cause Analysis for Online Service Systems with Intervention Recognition. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery, 3230–3240. doi:10.1145/3534678.3539041 [18] JinJin Lin, Pengfei Chen, and Zibin Zheng. 2018. Microscope: Pinpoint performance issues with causal graphs in micro-service environments. In International Conference on Service-Oriented Computing. Springer, 3–20. [19] Qisheng Lu, Aoyang Fang, Junjielong Xu, Jin’ao Shang, Songhan Zhang, Yifan Yang, Xiaochuan Yan, and Pinjia He. 2026. Beyond Fault Localization: A Trajectory-Level Study of LLM Agents for Microservice Root Cause Analysis. arXiv:2608.21310 https://arxiv.org/abs/2608.21310 [20] Meng Ma, Jingmin Xu, Yuan Wang, Pengfei Chen, Zonghua Zhang, and Ping Wang. 2020. AutoMAP: Diagnose Your Microservice-based Web Applications Automatically. In WWW ’20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020. ACM / IW3C2, 246–258. [21] Hada Melino Muhammad, Luan Pham, Laure Barrière, Sachin Shetty, Leonardo Pulga, and Flora D. Salim. 2026. Where Root Cause Analysis Fails: A RetrievalReranking Decomposition. arXiv:2609.36686 https://arxiv.org/abs/2609.36686 Accepted at NeurIPS 2026, Evaluations and Datasets Track; to appear. [22] Alexander Naakka, Yuqing Wang, and Mika V. Mäntylä. 2027. LATS-RCA: Language Agent Tree Search for Root Cause Analysis in Microservices. In Software Engineering and Advanced Applications (Lecture Notes in Computer Science, Vol. 16863). Springer Nature Switzerland, 196–212. doi:10.1007/978-3-032-365903_14 SEAA 2026; first published online 27 August 2026. [23] OpenAI. 2026. Codex CLI. https://github.com/openai/codex Software; accessed 2026-10-26. [24] Yicheng Pan, Meng Ma, Xinrui Jiang, and Ping Wang. 2021. Faster, deeper, easier: crowdsourcing diagnosis of microservice kernel failure from user space. In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis. Association for Computing Machinery, 646–657. doi:10. 1145/3460319.3464805 9

Bi et al. SQL agent

[43] Lingzhe Zhang, Tong Jia, Kangjin Wang, Chiming Duan, Minghua He, Rongqian Wang, Xi Peng, Meiling Wang, Gong Zhang, Renhai Chen, and Ying Li. 2026. Toward In-Depth Root Cause Localization for Microservices With Multi-Agent Recursion-of-Thought. IEEE Transactions on Dependable and Secure Computing 23, 5 (2026), 10702–10719. doi:10.1109/tdsc.2026.3706572 [44] Wei Zhang, Hongcheng Guo, Jian Yang, Zhoujin Tian, Yi Zhang, Chaoran Yan, Zhoujun Li, Tongliang Li, Xu Shi, Liangfan Zheng, and Bo Zhang. 2024. mABC: Multi-Agent Blockchain-inspired Collaboration for Root Cause Analysis in MicroServices Architecture. In Findings of the Association for Computational Linguistics: EMNLP 2024. Association for Computational Linguistics, 4017–4033. doi:10.18653/ v1/2024.findings-emnlp.232 [45] Lecheng Zheng, Zhengzhang Chen, Dongjie Wang, Chengyuan Deng, Reon Matsuoka, and Haifeng Chen. 2026. LEMMA-RCA: A Large Multi-modal Multi-domain Dataset for Root Cause Analysis. In Proceedings of the 35th ACM International Conference on Information and Knowledge Management. doi:10.1145/3799682.3840176 To appear. [46] Xiang Zhou, Xin Peng, Tao Xie, Jun Sun, Chao Ji, Wenhai Li, and Dan Ding. 2021. Fault Analysis and Debugging of Microservice Systems: Industrial Survey, Benchmark System, and Empirical Study. IEEE Trans. Software Eng. 47, 2 (2021), 243–260.

A: Luna ! Sol first hit (step)

32

10

Sol: PR@1 86, 25 actions, never 7%

Luna: PR@1 69, 7 actions, never 11% Sol: PR@1 87, 6 actions, never 6%

6 18 3 10

B: Luna med. ! max first hit (step)

16 63

25

40

4

Luna: PR@1 71, 30 actions, never 9%

6

10

Luna: PR@1 69, 7 actions, never 11%

Luna max: PR@1 76, 52 actions, never 4%

10

Luna max: PR@1 72, 11 actions, never 6%

32 6 16 3 8 16 40

first hit (step)

C: Sol med. ! xhigh

A

Codex

Luna: PR@1 71, 30 actions, never 9%

Per-method walkthroughs

We explain why the methods not covered by the case study (Section 6.2) succeed or fail; PR@1 is given as Luna / Flash / Haiku.

63

126

6

10

18

Sol: PR@1 87, 6 actions, never 6%

Sol xhigh: PR@1 88, 33 actions, never 4%

6

Sol xhigh: PR@1 89, 8 actions, never 1%

25 4 16 3 10 16

Published LLM agents. FoA (54.4 / 58.9 / 46.7) runs one of twelve SOPs written before any run after the examples in [25]. In FoA the alert names the fault type, which selects the SOP; our setting withholds it, since it would reveal the answer, so FoA describes the incident itself. Luna then matches the generic Microservice Issue SOP (abnormal metrics → entry requests → traces) in 69 of the 90 cases; Flash, which more often guesses a fault type, does so in 33. Only four SOPs contain a log step (Memory, Network Loss or Partition, Code or Configuration Error, Database or Cache). FoA’s accuracy varies widely by fault group: on Luna 93% on applicationcode faults but 41% on resource saturation. RCAgent (54.4 / 50.0 / 43.3) prescribes 35% of its calls and, with FoA, comes closest to the simple harnesses on Luna; Luna leaves 7 cases and Flash 19 cases unanswered within its 30 steps. RCA-Agent (32.2 / 63.3 / 34.4): its released executor discards outputs above 16,384 tokens after two tries, which costs Luna, which prints whole tables, 16% of its executor steps (Haiku 3%, yet Haiku scores as low). DecompRCA (24.4 / 32.2 / 28.9): the program’s at most 15 candidates contain the root cause in only 57.8% of the cases, which bounds the model’s rerank. mABC (11.1 / – / 20.0) walks down the call chain from the alert, so a service that receives no requests is never reached. Luna also declares the cause undetermined in 42 of 89 cases, concentrated in sources with few or no traces (LEMMA 5 of 5, RCAEval RE1 8 of 9); Haiku answers 85 of 90.

32

Sol: PR@1 86, 25 actions, never 7%

25 40 tool actions per case

63

2

. fewer actions, earlier hit 4

6 10 tool actions per case

Figure 7: The cases behind Figure 5 (first pass). x = tool actions per case, y = step of the first query that names the root cause (both log); filled = root cause ranked first, hollow = missed; contour = densest 65% of a run’s hits; X = median; arrow = first run (blue) to second (orange). Cases that never query the root cause are not drawn.

B

Additional results and significance

Significance. A paired bootstrap over the 90 cases (10,000 resamples, 95%) confirms the main gaps: the SQL agent and Codex beat the best published agent on every model they ran on, by 14 to 24 points (one-shot on Flash and Haiku only), and, with MiMo left out of the tiers as in Section 6.1, 12 of 13 cross-tier model pairs differ against 1 of 9 within-tier pairs (with MiMo grouped with the lower tier by its accuracy: 20 of 21 and 1 of 14). The best published agent beats NSigma on Luna and Flash, but mABC falls significantly below it, so the family order holds for the strongest members. Investigation orientation (Figure 7). Figure 5 summarises each run along one diagonal; Figure 7 shows the cases behind it. From Luna to Sol (A) both clouds move down and left: the SQL agent needs 25 instead of 30 actions and first queries the root cause at step 14 instead of 17, Codex at step 3 instead of 5. More effort (B, C) moves the clouds right without moving them down: Luna at max uses 52 instead of 30 actions with the SQL agent and first hits the root cause later (step 22), and Sol at xhigh hits it no earlier than at medium.

Non-LLM methods. The gap lies in where the evidence is. On resource saturation (34 cases) NSigma reaches 62%, close to oneshot on Luna (65%); on errors, outages, configuration faults and network breaks (33 cases), whose evidence is in logs, traces and events, NSigma and BARO reach 15% and 6% against 67% for the SQL agent and Codex on Luna. Three causes add to this. Many sources have thousands of series with about 15 pre-fault samples, half of them constant, so normal jumps outrank the root cause. NSigma and BARO score the largest signed deviation, so a drop ranks last (inventory’s request drop in the case study: 372nd of 373). Causal-graph methods cannot learn dependencies from tens of samples: RCD misses the root cause entirely in 71 of 89 cases.

By source (Table 7). The simple harnesses lead on seven of eight sources. On RCA-100 FoA leads (72.2), although Luna’s FoA runs used the generic SOP in 10 of its 12 cases. Non-LLM methods come closest on RCAEval, OpenRCA 1.0 and LEMMA, where NSigma reaches 57–60% and on OpenRCA 1.0 even exceeds one-shot (57.1 vs. 56.3); AIOps 2022 is the hardest source for every method (best 52.8). 10

Free to Look, Free to Reason

0.95 0.90

mean RR

pre-filter (5), an unclean before-period or nearby fault (69), and less than a third of the window remaining after the fault (37); rules overlap. Of 542 screened cases, 348 remain. RCAEval supplies three cases per sub-suite (27); other scenarios supply at least three where eligible. Remaining places diversify scenario, fault class, root-cause level, data kinds, volume and clue coverage, with fixed tie-breaking and seed, before method runs.

Model (colour) GPT-6 Luna

0.85

GPT-6.1 Sol

Method (shape)

0.80

One-shot SQL agent

0.75

Codex

0.70

Effort (fill) medium

0.65

xhigh max

0.60 0.55 50

100

200

300

500

seconds per case (mean, log scale)

Figure 8: RR versus wall time per case (without network waits) as reasoning effort rises from medium to xhigh and max. Colour = model, shape = method, fill depth = effort. Medium points of Luna and of Sol’s SQL agent and Codex: mean of three passes; others one pass. Sol’s SQL agent was not run at max.

C.2

Cost (Table 8). The table reports the first pass of each run, whereas Tables 3–5 average three passes for Luna and Sol, so the two can differ slightly. One-shot sends 235k–293k prompt tokens in one call. The agents’ totals range from 90k (Sol with the SQL agent) to over two million (Flash and MiMo), but 76–95% of them are served from the provider’s cache, so the SQL agent on Luna costs USD 0.011 per case against 0.052 for one-shot. Weaker models take more steps: Flash and MiMo make 21–47 calls per case with the agents, Luna and Sol 7.5–20. Reasoning effort (Figure 8). Over every effort level and harness in Figure 8, including one-shot at max (which Section 6.4 does not cover), raising effort above medium multiplies wall time by 1.5 to 7 and changes RR by −2 to +7 points, while replacing Luna with Sol at medium raises it by 13 to 15 points: a stronger model is worth more than a larger thinking budget.

C

Benchmark and method details

C.3

Benchmark. Cases are converted to one format (entity and relation tables, metrics, traces, logs, events; UTC), with traces of the injection tooling removed. Of 542 screened cases, 348 pass the exclusion rules; the most frequent reasons are an unclean pre-fault period (69), no evidence (56), incomplete data (38) and a fault late in the window (37). LEMMA has no traces and hundreds of pods per scenario. RCAEval injects faults into only five services per system, so guessing among the injected services hits 20%; against the full inventory our random-guess rate is far lower (Section 4).

C.1

One-shot evidence package

The prompt includes instructions, the question, recorded and flagged periods, all selected evidence and the scenario’s answer template, with neutral item titles. S1 gives the incident record, inventory, topology, placement and coverage. S2 supplies moved metrics as raw samples: using the before-period median and robust spread (MAD ×1.4826), a series needs a change score max(median shift, 0.6 × largest departure) ≥ 4 spreads, plus a median shift of at least 10% or a departure of at least 20% of the baseline level. Identical series are collapsed; unlabelled duplicaterow series are omitted. At most 120 series are selected in turns across moving entities and rendered on the source time grid, with missing samples marked. S3 gives per-minute entry requests, errors, duration quantiles and caller–callee statistics, including missing downstream spans. S4 groups logs into masked templates: the 30 most-changed entities receive separate items, with up to eight templates and twelve raw lines per entity, prioritising errors, warnings and new templates. S5 groups Kubernetes events by object and reason (at most 60 groups). Constant series are counted only; series beyond the cap, full raw traces and logs, injection traces and answer material are omitted. The case study contains 136 items (about 75k tokens); no one-shot prompt was truncated.

Method settings and hard groups

Methods. LLM methods run without generation parameters (except effort in Setting 3), one hour per case, first five answers scored. Interface fixes: shared raw-data tools for FoA, RCAgent and mABC; complete tool descriptions for FoA’s code agent; UTC+8 times for RCA-Agent; modules missing from mABC’s release, a loop limit, a cut at “Observation” and a format hint; resampling for TORAI; missing functions for FaultInsight. Setting differences: no temperatures; RCAgent’s expert also reads metrics and traces (at most 30 steps); mABC’s alert on the entry service’s requests. The SQL agent differs from the reference agent of [10] in temperature, an optional think tool, its answer format, and the topology files that every case folder carries.

Case construction

Cases share entity and relation tables and UTC telemetry; labels and answer material are kept separate. Duplicate metric observations are collapsed, source attributes distinguish otherwise duplicate rows, and injection-tool entities, spans, logs and events are removed. Source fault windows are retained where available; RCA-100 adds 20 minutes before the alert, and sources giving only an injection time use an initial before-period and flag the remainder without exposing the onset. Automatic checks cover label consistency, culprit coverage, fault timing, completeness, shortcuts and observable evidence. Claude Sonnet 5.5 reviews labels, check results, row counts and the twelve most-changed series; ITBench-SRE uses automatic checks only. Exclusions are contradictory labels (8), multiple faults outside the multi-fault scenario (20), a missing culprit or its data (14), no evidence (56), incomplete data (38), failure of the multi-fault

Hardest groups. On node-level faults the agents mention the true node in 85–90% of runs but mostly blame a pod or service on it. Three AIOps 2022 cases fail for reasons in the data: no clear metric on the faulty node, a fault that leaves almost no trace, and an unlabelled burst on another node. No one-shot prompt was truncated; cases with all four data kinds have a median of 115k tokens.

11

Bi et al.

Table 7: Results by source dataset (%), averaged as in Table 3. Red tint: drop below the column best (white = best). Rel. to best: PR@1 as a share of the column best. All (90)

PR@1 by source dataset

Method

PR@1

RR

RCAEval (27)

OpenRCA 1.0 (14)

Codex† SQL agent One-shot FoA RCAgent RCA-Agent DecompRCA mABC§ NSigma TORAI‡ SimpleRCA BARO CIRCA FaultInsight‡ RCD‡ Random

77.2 72.8 65.3 53.3 49.3 43.3 28.5 15.6 38.9 33.3 33.3 22.2 17.8 18.9 5.9 5.5

80.6 74.9 69.8 58.7 53.7 41.9 34.4 16.1 47.5 46.8 45.1 31.7 29.2 29.7 10.5 12.0

90.1 94.2 79.8 66.7 59.3 39.5 55.6 11.1 59.3 63.0 55.6 44.4 29.6 28.4 0.0 7.1

65.5 61.1 56.3 28.6 33.3 21.4 23.8 7.1 57.1 42.9 28.6 14.3 14.3 9.5 2.4 3.4

OpenRCA 2.0 (12) 72.2 74.1 72.2 66.7 63.9 63.9 8.3 25.0 25.0 25.0 16.7 8.3 0.0 19.4 5.6 8.2

PR@1

RCA-100 (12)

AIOps 2025 (9)

AIOps 2022 (6)

ITBench (5)

LEMMA (5)

rel. to best

66.7 50.0 50.0 72.2 52.8 38.9 30.6 37.5 25.0 8.3 41.7 25.0 16.7 22.2 8.3 3.0

81.5 65.4 66.7 37.0 51.9 63.0 22.2 11.1 11.1 11.1 44.4 22.2 33.3 22.2 25.9 8.3

52.8 29.6 42.6 5.6 22.2 38.9 0.0 8.3 16.7 16.7 0.0 0.0 0.0 5.6 0.0 1.9

80.0 84.4 57.8 66.7 40.0 53.3 0.0 20.0 0.0 0.0 0.0 0.0 20.0 6.7 20.0 5.8

96.7 95.6 64.4 46.7 33.3 46.7 13.3 0.0 60.0 20.0 0.0 0.0 0.0 6.7 0.0 1.1

100.0 94.3 84.6 69.1 63.8 56.1 36.9 20.1 50.4 43.2 43.2 28.8 23.0 24.5 7.7 7.1

† Haiku not run. § DeepSeek not run. ‡ mean over three passes.

Table 8: The five cost metrics of the three simple harnesses across models (setting 2, default effort; Astra at xhigh; first pass of each run): wall time per case, model calls per case, prompt and completion tokens per case, share of prompt tokens served from the provider’s cache, and billed USD per case. Wall time excludes waits caused by the network (failed connections, read timeouts, reconnects). Calls = requests that reached the model (retries after connection failures during the 10-25 network outage are not counted). ∗ = not billed per call (ChatGPT login); priced call by call at the OpenRouter list price, with the long-prompt tier (about twice the price) for calls above 272k input tokens, which is why the one-shot rows through Codex exceed the price of their mean token counts. MiMo: unfinished cases are not in the per-case means. Method One-shot

SQL agent

Codex

Model 6-Luna 6.1-Sol 5.5-Haiku 4.1-Flash 2.6-Pro 6-Astra (xhigh) 6-Luna 6.1-Sol 5.5-Haiku 4.1-Flash 2.6-Pro 6-Luna 6.1-Sol 4.1-Flash 2.6-Pro 6-Astra (xhigh)

Time s / case

Calls / case

Prompt tokens

Completion tokens

Cached share

Cost USD / case

46 64 55 381 1,037 316 134 45 51 171 997 69 105 574 609 216

1.0 1.0 1.0 1.0 1.0 1.0 19.7 8.9 13.8 47.4 46.3 8.5 7.5 34.0 21.0 8.9

235,644 241,783 269,460 239,332 293,483 241,824 381,235 90,160 290,719 2,098,190 2,628,405 275,018 289,443 2,552,729 1,336,211 428,257

3,949 1,392 10,731 29,056 57,216 7,640 6,268 2,260 7,814 24,249 38,658 4,295 3,685 80,595 21,362 7,803

0% 21% 0% 1% 14% 23% 90% 76% 88% 95% 93% 84% 85% 90% 90% 87%

0.0522 0.639∗ 0.151 0.0334 0.160 3.56∗ 0.0112 0.0827 0.0107 0.0511 0.116 0.00900∗ 0.149∗ 0.0549∗ 0.0790∗ 1.32∗

12

Related documents

• Free to Look, Free to Reason: Simple LLM Harnesses Outdiagnose Hand-Crafted Root Cause Analysis Agents#1459290
• Beyond Fault Localization: A Trajectory-Level Study of LLM Agents for Microservice Root Cause Analysis#1298102
• Beyond Fault Localization: A Trajectory-Level Study of LLM Agents for Microservice Root Cause Analysis#480119
• What a Pooled Leaderboard Does Not Say: Auditing the Scope and Provenance of Method Comparisons in Microservice RCA Benchmarks#1343626
• How Far Can Root Cause Analysis Go on Real-World Telemetry Data?#1423923