SieveFL: Hierarchical Runtime-Aware Pruning for Scalable LLM-Based Fault Localization
arXiv:2605.13491v1 [cs.SE] 13 May 2026
Mahdi Farzandway, Fatemeh Ghassemi University of Tehran {mahdifarzandway, fghassemi}@ut.ac.ir Abstract—Automated fault localization requires connecting an observed test failure to the responsible method across thousands of candidates—a task that purely statistical approaches handle with limited precision and that LLMs cannot yet handle at full project scale due to prohibitive token cost and signal dilution. We present SieveFL, a five-stage hierarchical framework that resolves this tension through aggressive pre-LLM filtering. SieveFL converts a failing test into a natural-language failure description, uses dense vector retrieval to narrow the search to a small set of suspicious files, and then eliminates any method not executed during the failing test via JaCoCo runtime traces. Only the surviving candidates are passed to the LLM, which screens each method individually and re-ranks the confirmed suspects in a single comparative pass. We evaluate SieveFL on 395 bugs from Defects4J v1.2.0 using a mid-sized, openly available MoE model deployed on a commodity workstation (32 GB RAM, 8 GB GPU) via Ollama—no frontier APIs or datacenter hardware required. Treating 12 incomplete runs as failures, SieveFL achieves Top-1 accuracy of 41.8% (165/395 bugs) and an MRR of 0.469, outperforming the strongest prior agent-based baseline (AgentFL) by 2.1 pp in Top-1. Runtime pruning removes 79% of candidate methods and reduces input token consumption by 49%, while simultaneously improving ranking quality: Top1 is preserved exactly and Top-3 through Top-10 improve by up to 2.4 pp. These results demonstrate that, with the right filtering architecture, capable fault localization does not require proprietary frontier models. Index Terms—Fault localization, large language models, hierarchical pruning, runtime coverage analysis, software debugging.
I. Introduction Software debugging consumes a disproportionate share of developer time, with fault localization often cited as one of the most demanding phases of the entire development lifecycle [1], [2]. The core difficulty is not computational but conceptual: connecting an observed failure to the specific method responsible requires understanding what the code was supposed to do, not just what it did. Spectrum-Based Fault Localization (SBFL) approaches such as Ochiai [3] approximate this by correlating execution patterns with test outcomes, while Information Retrieval approaches [4]–[6] treat the problem as document matching between bug reports and source indices. Both families are efficient, but neither can bridge the semantic gap between a raw execution anomaly and the architectural intent behind the failing code. Large Language Models have changed what is possible here. Recent systems including AutoFL [7], AgentFL [8], and FaR-Loc [9] demonstrate that LLMs can reason causally
about code in ways that statistical or retrieval tools cannot, essentially simulating the reasoning process of an experienced developer reading a stack trace. The practical obstacle is scale. A real-world project may expose thousands of candidate methods to the model, and the relevant signal quickly drowns in noise. Irrelevant methods that happen to share vocabulary with the failure context accumulate false positives, while the token cost of evaluating each candidate individually becomes prohibitive [9]. We refer to this tension between coverage and precision as the Scale-Precision Dilemma: the larger the codebase, the less effective LLM-based reasoning becomes if applied without prior filtering. We present SieveFL, a five-stage hierarchical framework designed to resolve this dilemma through progressive pruning. The key insight is that the LLM should be the last filter applied, not the first. SieveFL constructs a cascade in which each stage dramatically narrows the candidate set before passing it to the next, more expensive, stage. It begins by converting raw test failures and stack traces into a naturallanguage failure description. Semantic retrieval then reduces the search to a small set of suspicious files. A JaCoCo runtime trace eliminates any method that was never executed during the failing test. Only after these three filtering steps does the LLM become involved: it screens each surviving method individually for plausibility, then ranks the confirmed suspects in a single comparative pass. Critically, the entire framework runs on a mid-sized, openly available MoE model (Nemotron3-Nano-30B-A3B) via Ollama on a commodity workstation, without relying on frontier APIs or datacenter-class infrastructure, which demonstrates that the accuracy gains come from the architecture, not from model scale. Contributions. Our primary contributions are: • We propose a five-stage hierarchical pruning architecture that progressively reduces the candidate search space from the full codebase to a small ranked list of suspicious methods. Runtime-aware pruning via JaCoCo reduces the candidate set by 79% and LLM input token consumption by approximately 49%, while simultaneously improving localization quality: Top-1 accuracy is preserved exactly (∆ = 0.0 pp), Top-3, Top-5, and Top-10 improve by up to 2.4 pp, and MRR increases from 0.462 to 0.469—all on the same 383-bug paired evaluation set. • We introduce a per-method LLM screening protocol in which each surviving candidate is evaluated in an isolated query, enabling focused causal reasoning free from inter-
method interference and position bias. The binary verdicts and justifications produced at this stage serve as structured input to the final re-ranking step. • We demonstrate that state-of-the-art fault localization does not require frontier models. Running entirely on Nemotron-3-Nano-30B-A3B, a mid-sized openly available MoE model deployable on a commodity workstation with 32 GB of system memory and an 8 GB consumer GPU via Ollama, SieveFL achieves Top-1 accuracy of 41.8% (165/395 bugs, treating incomplete runs as failures) on the full benchmark, and MRR of 0.469 on the 383 completed bugs of Defects4J v1.2.0 [10], outperforming the strongest prior agent-based baseline (AgentFL) by 2.1 pp in Top1 when both systems are evaluated over the full 395-bug benchmark. II. Related Work A. Spectrum-Based and Learning-Based Fault Localization Fault localization has evolved from statistical heuristics to data-driven models. Traditional SBFL metrics such as Ochiai [3] and D* [11] rank methods by correlating their execution coverage with test pass/fail outcomes. Although computationally efficient, SBFL techniques are sensitive to test suite quality and are known to struggle on large-scale projects with complex execution profiles [12]. Learning-based FL methods such as DeepFL [13] and GRACE [14] address this by integrating multiple signals—spectrum, code metrics, and syntactic features—through deep learning. GRACE in particular uses graph-based representations to capture structural relationships between program elements. While these models achieve strong results on standard benchmarks, they require substantial labeled training data and may generalize poorly to unseen software systems, as their learned representations are inherently tied to the training distribution. A complementary challenge surfaces in multi-fault settings, where SA-BCL [15] identifies two compounding deficiencies in program spectra: an imbalance of defect knowledge caused by the relative scarcity of failing test cases, and the presence of characterizing noise introduced when mutual fault interference causes some failing executions to exhibit coverage patterns indistinguishable from passing ones. SA-BCL addresses both issues by first detecting borderline failing samples and augmenting them via random oversampling, then applying confident learning [16] to identify and remove noisy spectrum entries before iterative spectrum reduction. On both the synthetic TCM and the real-world Defects4J multi-fault benchmarks, SA-BCL outperforms the stateof-the-art FLITSR by up to 37.5% in Average Wasted Effort and 29.2% in precision—results that confirm spectrum quality as a critical bottleneck in coverage-based localization and motivate augmenting static coverage signals with cleaner execution evidence. SieveFL departs from the purely spectrumbased paradigm by relying on zero-shot LLM reasoning and runtime traces, neither of which requires project-specific training. However, this training-free design comes with a tradeoff. Learning-based techniques such as GRACE [14] and DeepFL [13] benefit from a leave-one-out training strategy that
exposes the model to the structural and coverage patterns of the target project before evaluation. This in-project knowledge gives them a systematic advantage on projects with recurring bug patterns or structurally similar fault sites—an advantage that zero-shot LLM reasoning cannot replicate without prior exposure to the codebase. This distinction is important when interpreting the benchmark results in Section V-A, where GRACE outperforms SieveFL on aggregate Top-1 accuracy despite SieveFL achieving superior results on projects with discriminative runtime traces. B. IR-Based Fault Localization IR-based FL approaches treat the bug report as a naturallanguage query and the source code as a document corpus [4]– [6]. BM25-based techniques [17] retrieve files or methods whose token distributions are most similar to the bug report. While straightforward and fast, these approaches are limited by vocabulary mismatch: they cannot bridge the gap between the natural-language description of a failure and the often terse, identifier-heavy style of source code. SieveFL’s Stage 2 addresses this limitation by using a shared embedding space that maps both natural language and source code into comparable semantic representations, while Stages 3–4 augment the retrieval signal with dynamic execution evidence that purely textual methods cannot access. C. LLMs and RAG in Software Engineering The integration of LLMs has redefined automated program repair (APR) and fault localization [18], [19]. AutoFL [7] pioneered the use of LLMs to generate explainable fault evidence via function call tools, demonstrating that LLMs can perform meaningful causal reasoning about code when supplied with sufficient context. FuseFL [20] extends this direction by combining SBFL suspiciousness rankings, test execution outcomes, and natural-language code descriptions into a single prompt, enabling LLMs to produce step-by-step explanations of why a specific line is faulty; the approach substantially outperforms pure SBFL baselines and shows that richer contextual signals sharpen LLM localization judgments. However, the limited context window of LLMs necessitates Retrieval-Augmented Generation (RAG) [21] to pre-select a manageable candidate set before LLM invocation. FaR-Loc [9] introduced functionality-aware retrieval that leverages pretrained code embeddings to bridge the semantic gap between failure descriptions and code implementations. A complementary direction structures LLM output rather than leaving it as free-form text. SemLoc [22] converts LLMinferred semantic properties into a closed intermediate representation of typed, executable constraints anchored to specific program locations. Executing these constraints across passing and failing tests yields a semantic violation spectrum analogous to coverage-based SBFL, and a counterfactual verification step further distinguishes root-cause violations from cascading downstream effects. SemLoc is particularly effective on semantic bugs where identical execution paths differ only in whether a numeric invariant holds—a class of faults that
syntactic spectra cannot distinguish. In the educational setting, FLAME [23] demonstrates that prompting LLMs to annotate faulty lines directly inside the source listing—rather than predicting bare line numbers—substantially improves localization accuracy and produces human-readable explanations suitable for student feedback, further illustrating that how the LLM interacts with code structure has a large effect on precision. An important empirical constraint on context design comes from Sepidband et al. [24], who conduct a large-scale factorial study of fault localization context granularity across 61 configurations on SWE-bench Verified. Their results establish three findings directly relevant to SieveFL: file-level context provides the dominant performance gain (a 15–17× improvement over a no-file baseline), expanding line-level context frequently degrades repair performance by introducing noise that dilutes the localization signal, and LLM-based file retrieval outperforms structural heuristics while incurring lower token cost. These findings empirically justify SieveFL’s design decision to invest in aggressive hierarchical pruning at the file and method levels before any LLM call is made, reserving the LLM for the small, high-signal survivor set. SieveFL builds upon these insights but adds two critical layers absent in prior RAG-based FL work: a runtime pruning stage that intersects the semantic candidate set with dynamic execution traces, and a per-method isolated screening step that enables focused causal reasoning before comparative reranking. The importance of controlling retrieval noise in RAG systems has been highlighted independently [25]; our results confirm this finding in the FL domain, where methods removed by runtime pruning are shown to be predominantly falsepositive candidates. D. Agentic and Multi-Agent Frameworks for Debugging Recent work emphasizes agentic architectures to mimic the iterative reasoning of human debuggers. AgentFL [8] decomposes FL into a three-stage agentic process of test comprehension, codebase navigation, and fault confirmation, operating at project scale without requiring coverage instrumentation. Rafi et al. [26] propose a multi-agent system that combines graphbased code retrieval with a reflexion mechanism to refine fault hypotheses across multiple reasoning rounds. MemFL [27] takes a different strategy for incorporating project-specific knowledge: rather than navigating the repository at inference time, it precomputes a two-component external memory— static summaries of the project and its constituent classes, and dynamically refined debugging guidance distilled from prior localization attempts on a small training set—which is prepended to a lightweight three-step pipeline of bug review generation, code condensation, and fault confirmation. On Defects4J, MemFL achieves substantially higher Top-1 accuracy than AutoFL and SoapFL at a fraction of their cost and runtime, with especially pronounced gains on complex projects such as Closure where generic LLM reasoning degrades. A separate thread of agentic debugging research equips LLMs with interactive dynamic analysis rather than pre-computed summaries or multi-round reflexion over static
context. InspectCoder [28] is the first agentic repair system to give an LLM agent direct, programmatic control over a live Python debugger. Its dual-agent architecture pairs a Program Inspector—which strategically places breakpoints, inspects runtime variable states, and injects perturbation logic within a stateful debugging session—with a Patch Coder that synthesizes fixes grounded in the inspector’s root-cause report. Crucially, rather than passively consuming pre-collected execution logs as in log-augmented approaches such as LDB [29], InspectCoder adaptively queries runtime state in response to intermediate findings, receiving immediate process-reward signals that guide multi-step hypothesis refinement without committing to irreversible code changes. A specialized middleware, InspectWare, models the debugger as a finite-state machine and shields the LLM from low-level protocol noise, providing structured, context-aware feedback at each reasoning step. Evaluated on BigCodeBench-R and LiveCodeBench-R, InspectCoder achieves 5.10%–60.37% relative improvements in repair accuracy over the strongest baselines and resolves 1.67×–2.24× more bugs per hour—evidence that executiongrounded, hypothesis-driven exploration substantially outperforms both static reasoning and passive log collection. These results echo a principle central to SieveFL: dynamic execution evidence, whether gathered through interactive debugger control or test-trace intersection, greatly sharpens the signal available to the LLM and reduces the cost of each reasoning step. While these approaches demonstrate the value of structured reasoning and project-specific context, they tend to invoke the LLM over large, unfiltered candidate sets, making them vulnerable to the Scale-Precision Dilemma. Furthermore, sequential multi-agent pipelines can suffer from compounding reasoning errors across turns [30]. SieveFL takes a different stance: rather than relying on multi-round agent interaction or pre-baked project summaries to compensate for a noisy candidate pool, it invests in pre-LLM filtering so that each LLM call—whether the per-method screening in Stage 4 or the comparative re-ranking in Stage 5—operates on a small, high-signal candidate set where a single focused query suffices. III. Proposed Method Fault localization—the task of identifying the source code elements responsible for an observed failure—has been studied extensively across a wide range of automated techniques [1]. We present SieveFL, a five-stage hierarchical framework for precise method-level fault localization. The central design philosophy is progressive pruning: rather than exposing all methods of a large codebase to expensive analysis at once, SieveFL constructs a cascade of increasingly focused filters. Each stage dramatically reduces the candidate set passed to the next, so that the most costly reasoning—individual LLM interrogation of each method—is applied only to a small, highquality pool of suspects. Figure 1 illustrates the full pipeline, which comprises: (1) LLM-based Test Analysis, (2) Suspicious File Identification, (3) Runtime-Aware Candidate Pruning, (4) Per-Method LLM Screening, and (5) LLM-Based Re-ranking. Stages 1–3 convert
raw failure artifacts into a semantically rich query, narrow the search space to a small set of candidate files, and then remove methods that were not exercised during the failing test. Stage 4 individually interrogates each surviving method to determine whether it is plausibly responsible for the failure. Stage 5 collects the confirmed suspicious methods and asks the LLM to rank them from most to least likely to contain the fault. A. Stage 1: LLM-based Test Analysis The primary goal of Stage 1 is to bridge the semantic gap between a raw test failure and a natural-language description that can drive all subsequent retrieval and reasoning steps. Given the failing test function(s) and their associated error output, we prompt a large language model to perform structured reasoning over the inputs and produce a comprehensive failure description in prose. The failure description must capture three complementary aspects of the bug. First, it articulates the expected behavior— the contract that the failing test asserts about the system under test. Second, it explains the observed error—the discrepancy between the expected and actual outcomes, grounded in the specific exception type, stack trace, or assertion message. Third, and most consequentially for all downstream steps, it produces a search-oriented summary that names the likely faulty functionality in terms that align with how a developer would describe it in code comments and method signatures. Because this description serves as the shared query across every subsequent stage, we design the prompt (Figure 2) to discourage shallow paraphrasing of the error message and instead elicit reasoning about why the failure occurred at the semantic level. B. Stage 2: Suspicious File Identification Stage 2 reduces the full codebase to a small set of candidate files that are semantically aligned with the failure description produced in Stage 1. This design follows the RetrievalAugmented Generation (RAG) paradigm [21], adapted here to treat individual methods as the retrieval corpus and the failure description as the query. This coarse-grained filtering serves two purposes. First, it removes the vast majority of source files irrelevant to the fault, preventing the downstream stages from being overwhelmed by unrelated candidates—a known failure mode in retrieval-augmented systems where noisy candidates degrade final output quality [25]. Second, by indexing at the method level, each document in the corpus represents a cohesive unit of behavior, which produces more semantically precise similarity scores than whole-file or fixedsize chunk representations. Method-level indexing. Rather than chunking source files arbitrarily, we parse every source file in the project and extract each method together with its associated Javadoc and inline comments as a single, self-contained document. A file containing n methods therefore contributes exactly n documents to the index, each carrying its parent file path as metadata. All documents are embedded using sentence-transformers/all-MiniLM-L6-v2 [31],
a lightweight model that maps both natural language and source code identifiers, comments, and string literals into a shared semantic space. The resulting embeddings are stored in a FAISS flat index [32], and similarity is measured by cosine distance [33]. File-level aggregation. At query time, the failure description from Stage 1 is embedded and the top-kf most similar method documents are retrieved. The suspicious file set F ∗ is then formed by taking the union of the parent file paths of these retrieved methods—that is, retrieval is performed at method granularity, but the output is a set of files. This two-level design is deliberate: method-level matching yields precise semantic alignment, while file-level aggregation ensures that the downstream stages receive complete files for coverage analysis and exhaustive method enumeration. We set kf = 10, which consistently recovers the ground-truth file in over 90% of bugs while keeping |F ∗ | small enough for efficient downstream processing. C. Stage 3: Runtime-Aware Candidate Pruning Semantic retrieval alone cannot distinguish between methods that are described similarly to the failure and methods that were actually executed during it. Stage 3 closes this gap by introducing a dynamic runtime signal: branch-level coverage traces collected by JaCoCo [34], the industry-standard Java bytecode instrumentation framework. Coverage collection. For each bug, we instrument the subject program with JaCoCo and execute only the identified failing test method(s)—not the full regression suite—so the resulting trace is specific to the fault-triggering execution path rather than an aggregate of all test activity. JaCoCo exports results in its standard XML report format (coverage.xml). Hybrid scoring. Within each suspicious file f ∈ F ∗ , we extract the method set M(f ) by parsing the JaCoCo report. Combining multiple complementary signals has been shown to consistently outperform any single signal in isolation for fault localization [35]. Following this principle, for each method m ∈ M(f ), we compute a hybrid score combining the runtime coverage signal with the semantic similarity from Stage 2: s(m) = wcov · ρ(m) + wsem · σ(m),
(1)
where ρ(m) ∈ [0, 1] is the branch-coverage ratio of method m (fraction of its branches executed during the failing test), σ(m) ∈ [0, 1] is the cosine similarity between the method’s embedding and the failure description from Stage 1, and wcov , wsem ≥ 0 with wcov + wsem = 1. Our ablation study (Section V-C) shows that retaining the semantic component is essential for both ranking precision and for protecting groundtruth methods from being pruned. Pruning decision. Any method whose score s(m) falls below a threshold τ is removed before any LLM call is made. For methods affected by method-signature overload— where JaCoCo’s line-level map cannot unambiguously resolve which overloaded variant was invoked—we conservatively retain all overloaded variants. When no coverage.xml is available (e.g., due to build incompatibilities), Stage 3
Fig. 1. Overview of the SieveFL framework. The five-stage pipeline hierarchically prunes the search space from the full codebase to a ranked list of suspicious methods. Stages 1–3 perform progressively fine-grained filtering using LLM-based failure analysis, file-level retrieval, and runtime-aware pruning that combines JaCoCo coverage signals with semantic similarity. Stage 4 identifies top suspicious methods from the pruned candidates, and Stage 5 performs LLM-based collaborative re-ranking to produce the final ranked list.
System: You are an expert software debugger with deep knowledge of Java and common fault patterns.
D. Stage 4: Per-Method LLM Screening
User: Analyze the following failing test case: {test_code}.
Stage 4 is the core reasoning step of SieveFL. Having reduced the candidate set to a manageable size in Stage 3, we now interrogate each surviving method individually using the LLM. Recent work has demonstrated that LLMs can perform meaningful causal reasoning about code faults when given sufficient and focused context [7], [19]. Rather than asking the model to compare all candidates simultaneously—which would dilute its attention and introduce position bias—we present each method in a dedicated query and ask a focused binary question: is this method plausibly responsible for the observed failure? For each method m ∈ M̂, we construct a prompt (Figure 3) that provides the LLM with three inputs: (i) the failure description generated in Stage 1, (ii) the full source code and Javadoc comment of m, and (iii) the specific error output from the failing test. The LLM is instructed to reason step-by-step about whether the logic of m, under the conditions described by the failure, could produce the observed error. It then emits a binary verdict—suspicious or not suspicious—together with a brief natural-language justification explaining its decision. The one-method-per-query design is a deliberate choice. By isolating each candidate, the LLM can apply its full reasoning capacity to the specific semantics of that method without inter-
Your response must address three points: 1. Explain the expected behavior of the tested functionality. 2. Explain the actual observed failure, including the error type and any relevant context from the stack trace. 3. Generate a concise search query (2–5 sentences) describing the likely faulty functionality in natural language, as if you were searching a codebase for the methods responsible for this behavior. Fig. 2. Prompt used in Stage 1 (LLM-based Test Analysis). The three-point structure elicits semantic reasoning beyond surface-level error paraphrasing.
degrades gracefully by falling back to the V0 configuration, preserving correctness at the expense of cost savings. S The output of Stage 3 is a pruned candidate set M̂ ⊆ f ∈F ∗ M(f ). As reported in Sections V-B and V-C, this step retains on average only 21% of the methods present before pruning (reduction ratio = 0.79) while preserving the groundtruth method in over 91% of bugs. This dramatic reduction is what makes the per-method LLM interrogation in Stage 4 computationally feasible.
ference from unrelated candidates. This is especially important for methods that are syntactically similar or share a common utility role, where a batch comparison would likely conflate them. The justifications produced at this stage also serve a secondary purpose: they are passed to Stage 5 as additional context for re-ranking. Methods for which the LLM returns a suspicious verdict form the confirmed suspect set S ⊆ M̂, which is forwarded to Stage 5. System: You are a senior software engineer specializing in Java fault analysis. User: A test is failing with the following error: {error_output} Failure description: {failure_description}
System: You are a senior software engineer performing final root-cause triage. User: A test is failing with the following error: {error_output} Failure description: {failure_description} The following methods have each been identified as individually suspicious. For each, the method code and a preliminary analysis are provided: {suspect_list_with_justifications} Considering all suspects together, rank them from most to least likely to be the true fault site. Provide a one-sentence comparative justification for each position in your ranking. Fig. 4. Prompt used in Stage 5 (LLM-Based Re-ranking). All confirmed suspects from Stage 4, together with their individual justifications, are presented in a single query to enable comparative reasoning.
Examine the following method carefully: {method_code} Reason step-by-step about whether a defect in this method could produce the observed failure. Then answer: Verdict: Suspicious or Not Suspicious Justification: One to three sentences explaining your reasoning. Fig. 3. Prompt used in Stage 4 (Per-Method LLM Screening). Each candidate method is evaluated in a separate LLM call, enabling focused causal reasoning without position bias.
E. Stage 5: LLM-Based Re-ranking Stage 5 takes the confirmed suspect set S from Stage 4 and produces a final ranked list ordered from most to least likely to contain the fault. At this point, every method in S has already been judged suspicious by the LLM; the task is now to discriminate among them rather than to filter further. We construct a single prompt (Figure 4) that presents the LLM with: (i) the failure description from Stage 1, (ii) the full source code and Stage 4 justification of every method in S, and (iii) the original error output. The LLM is asked to consider all confirmed suspects together, weigh the evidence for each, and emit a ranked list with a brief comparative justification. The single-query design of Stage 5 is intentional and complementary to Stage 4. Whereas Stage 4 isolated each method to avoid inter-method interference during the binary screening decision, Stage 5 benefits from presenting all suspects simultaneously: the model can reason about relative plausibility, notice that one method’s logic subsumes another’s risk, or recognize that a particular method is the only one capable of producing the specific error type observed. Furthermore, reducing the total number of sequential LLM calls avoids the compounding of reasoning errors that can accumulate in multiturn agentic pipelines [36]. Prior multi-agent approaches to fault localization [26] require complex inter-agent communication protocols that add latency and token cost; at the scale of |S| methods produced by our earlier stages, a single wellstructured query achieves equivalent comparative reasoning at a fraction of the cost. The Stage 4 justifications included in the prompt give the re-ranker a richer basis for these comparisons than raw source code alone would provide.
IV. Experimental Design A. Research Questions To assess the effectiveness of SieveFL, we structure our evaluation around four primary research questions: RQ1 (Accuracy): How does SieveFL perform against stateof-the-art method-level fault localization techniques in terms of Top-k, MRR? RQ2 (Cost–Quality Trade-off): To what extent does the runtime-aware pruning (Stage 3) reduce LLM invocation costs, and how does this filtering impact overall localization quality? RQ3 (Ablation): Within the hybrid scoring function (Equation 1), what are the individual contributions of the semanticsimilarity signal σ(m) versus the branch-coverage signal ρ(m)? RQ4 (Scalability & Failure Analysis): How well does the framework scale across heterogeneous codebases, and what are the dominant failure modes bounding its current performance? B. Benchmark We conduct our experiments on Defects4J v1.2.0 [10], which serves as the standard benchmark in fault localization literature. Table I provides a summary of the six subject projects. Following established evaluation protocols [8], [9], we exclude bug reports where the ground-truth fault resides outside of method bodies. The resulting benchmark covers 395 bugs spanning approximately 344 kLOC of Java code. For the primary evaluation of our full pipeline (V1 ), we target all six projects—Lang, Time, Closure, Math, Chart, and Mockito. Not all bugs produce complete JaCoCo traces; results are therefore reported on the subset of bugs completed in each run. C. Pipeline Variants To isolate the impact of runtime-aware pruning and facilitate cost-controlled comparisons, we define three pipeline configurations: V0 (Baseline). LLM-based test analysis → file-level retrieval → per-method LLM screening → LLM re-ranking. Here, Stage 3 (JaCoCo pruning) is entirely disabled. This variant acts as a purely semantic baseline to measure the
TABLE I Statistics of the Defects4J v1.2.0 Subject Projects.
Project ID
Description
Chart Lang Math Time Mockito Closure
JFreeChart Apache Commons-Lang Apache Commons-Math Joda-Time Mockito Framework Google Closure Compiler
Total
# Bugs
Size (kLOC)
26 65 106 27 38 133
96 22 85 28 23 90
395
344
incremental value of runtime signals. V0 is evaluated at full scale across all 395 eligible bugs. V1 (Runtime-Aware, Full Scale). LLM-based test analysis → file-level retrieval → JaCoCo pruning → per-method LLM screening → LLM re-ranking. This represents our complete SieveFL framework and is evaluated at scale across all 395 eligible bugs. V2 (Ablation). Identical to V1 , but forces the semantic weight in Equation 1 to zero (wsem = 0). This isolates the effect of pruning based strictly on branch coverage. We evaluate this variant at full scale across all 395 eligible bugs. For statistical rigor, performance comparisons between V0 and V1 are restricted to bugs where both variants completed successfully. Similarly, comparisons between V1 and V2 are restricted to the 377 bugs where both variants completed successfully with full accuracy and safety fields. D. Baselines We benchmark SieveFL against five established techniques drawn from distinct methodological families: Ochiai (SBFL) [3] remains the most prominent spectrumbased technique, ranking methods based on their execution correlation with test failures. GRACE [14] represents the state of the art among non-LLM systems by fusing spectrum data with graph-based code representations. FLUCCS [37] is a learning-based FL technique that trains a ranking model over a combination of spectrum-based and code-metric features, serving as a strong supervised baseline that bridges the gap between pure SBFL and deep learning approaches. DeepFL [13] integrates spectrum, code metrics, and mutation-based features through a multi-objective learning framework, representing the state of the art among purely learning-based approaches. AgentFL [8] acts as our primary agentic baseline, utilizing an LLM to iteratively navigate and query the codebase. E. Evaluation Metrics Top-k measures the percentage of bugs where at least one ground-truth faulty method appears in the top k positions of the final ranking (k ∈ {1, 3, 5, 10}). We emphasize Top-1, as developers rarely inspect beyond the initial recommendations [2]. Mean Reciprocal Rank (MRR) computes the average of the reciprocal ranks of the first identified ground-truth
method across all bugs, penalizing techniques that bury correct answers lower in the list. Cost metrics. For RQ2, we track several efficiency indicators: reduction ratio, mean wall-clock time, and mean token consumption (input/output) per bug. For RQ3, we additionally track strict-loss rate (frequency at which ground-truth methods are accidentally pruned) and mean ground-truth recall after pruning. F. Implementation Details All prompt-driven reasoning is handled by a single LLM endpoint hosted via the Ollama runtime. To generate semantic vector representations, we use sentence-transformers/all-MiniLM-L6-v2. These embeddings are indexed using a FAISS flat structure. To compute the cosine similarity accurately and efficiently (as described in Section III-B), vectors undergo L2 normalization followed by an inner-product calculation. Coverage traces are collected using the Maven Surefire plugin connected to JaCoCo, restricted via the --tests flag to execute only the triggering failure. Based on tuning against a held-out development set drawn from Lang and Math, we configure the hybrid scoring weights to wcov = 0.6 and wsem = 0.4, and set the pruning threshold to τ = 0.05. All experiments run on a commodity workstation equipped with 32 GB of system memory and an 8 GB consumer GPU, with model inference handled via Ollama using CPU offloading for the 30B model weights. This configuration is representative of hardware available to individual developers, demonstrating that SieveFL does not require datacenter-class infrastructure. V. Results and Analysis A. RQ1: Localization Accuracy Overall comparison. Table II presents a unified, perproject view of method-level fault localization accuracy for all evaluated techniques on Defects4J v1.2.0. Each cell reports the number of bugs for which at least one ground-truth faulty method appears within the top-k positions of the final ranking (k ∈ {1, 3, 5}). Baseline counts for AgentFL, GRACE, DeepFL, FLUCCS, and Ochiai are taken from the original publications on the 395-bug benchmark, whereas the SieveFL rows report results on the completed bugs available in our runs. Raw counts for SieveFL variants in Table II reflect completed bugs only. Following the conservative evaluation protocol adopted throughout this paper, the headline Top-1 accuracy for V1 is reported as 165/395 = 41.8%, treating the 12 incomplete runs as failures. GRACE achieves the highest overall Top-1 accuracy (192/395 bugs at Top-1), establishing the upper bound for pure-accuracy optimisation on this benchmark. SieveFL does not compete on this axis; instead, it targets a different operating point: comparable localization quality at a fraction of the LLM invocation cost, as quantified in Section V-B. SieveFL V0 achieves the highest Top-1 count among all evaluated methods on Time (16 bugs) and Mockito
(18 bugs)—the two projects with the densest and most discriminative runtime traces. This advantage is consistent with SieveFL’s core design: JaCoCo-guided candidate reduction is most effective when failing tests exercise a tight, wellseparated coverage footprint. On Closure, where a single failing test implicates over 107 classes on average, SieveFL V0 localizes 26 bugs at Top-1, outperforming AgentFL (24) but trailing DeepFLcov (64), whose multi-signal coverage encoding is more robust to method-signature overloading. Beyond method-signature overloading, a deeper structural reason explains the persistent gap between SieveFL and learningbased techniques such as GRACE and DeepFL on Closure. These techniques are trained and evaluated using a leaveone-out strategy within the same project, meaning the model has seen the structural patterns, recurring fault types, and coverage signatures of Closure bugs during training. SieveFL, by contrast, approaches every bug from scratch using only zero-shot LLM reasoning and dynamic runtime traces, with no prior exposure to project-specific patterns. This distinction is particularly consequential on Closure, where multiple bugs share similar failing test purposes and even identical buggy methods, a setting where learned in-project representations provide a systematic advantage that zero-shot reasoning cannot replicate. This structural disadvantage is further confirmed by the AgentFL paper [8], which excludes Closure when comparing LLM-based approaches to coverage-based baselines, citing the same structural coupling as the root cause. Excluding Closure, SieveFL V0 localizes 139 bugs at Top-1, outperforming AgentFL (133 bugs) and DeepFLcov (112 bugs) among all non-GRACE methods on this subset—confirming that the advantage of SieveFL is concentrated in projects where LLM-guided screening is not diluted by high testcoverage coupling. Comparing the two SieveFL variants, V1 improves over V0 on Closure (+4 bugs, +3.1 pp) and Chart (+1 bug, +3.8 pp), confirming that JaCoCo-guided pruning sharpens ranking in overloaded codebases. Conversely, Lang and Mockito show higher Top-1 under V0 (−2 and −2 bugs respectively), reflecting cases where the coverage filter prunes semantically relevant ground-truth methods with marginal coverage overlap relative to the triggering failure. The cost and quality implications of runtime-aware pruning are quantified in Section V-B. Answer to RQ1: GRACE achieves the highest aggregate Top-1 accuracy (192/395 bugs). SieveFL targets a different operating point: it achieves the highest Top-1 count among all methods on Time and Mockito—the projects with the most discriminative runtime traces—while runtime-aware pruning simultaneously reduces downstream LLM cost and improves ranking quality across all broader cutoffs, as detailed in Section V-B. B. RQ2: Cost–Quality Trade-off of Runtime-Aware Pruning The central practical claim of SieveFL is that JaCoCo-based candidate reduction in Stage 3 substantially lowers the cost of per-method LLM screening in Stage 4 by reducing the number of methods forwarded for individual interrogation. For this
analysis, we restrict evaluation to the 383 bugs for which both V0 and V1 completed successfully, enabling a strictly paired comparison on a shared evaluation set. Table III reports the resulting cost and quality metrics for both variants. Cost reduction. Runtime-aware pruning substantially reduces the number of methods forwarded to Stage 4, lowering the mean candidate count from 398.8 to 187.2 per bug. Note that the mean per-bug reduction ratio reported in Table III (0.79) and the ratio of these mean counts (53.1%) differ because the average of per-bug ratios does not equal the ratio of per-bug averages; both figures are reported for completeness. Mean wall-clock time decreases from 5.07 to 4.05 minutes per bug, while mean input and output token consumption drop from 190.1K to 97.4K (−48.8%) and from 102.8K to 53.2K (−48.2%), respectively. Quality impact. Contrary to the expected cost–quality trade-off, runtime-aware pruning improves localization quality across every measured metric. Top-1 accuracy is preserved exactly (43.1% for both V0 and V1 ), while Top-3, Top-5, and Top-10 improve by 2.3, 2.4, and 2.3 pp respectively. MRR increases from 0.462 to 0.469 (+0.007). This consistent improvement confirms that the methods removed by JaCoCobased pruning are overwhelmingly false-positive candidates: eliminating them allows the LLM to focus its reasoning on a smaller, higher-signal pool, producing sharper rankings at every cutoff without sacrificing Top-1 precision. Answer to RQ2: On the 383 bugs completed by both variants, runtime-aware pruning reduces the Stage-4 candidate pool from 398.8 to 187.2 methods per bug (−53%), wall-clock time from 5.07 to 4.05 minutes, and input token consumption by 49%. Critically, this cost reduction entails no quality degradation: Top-1 is preserved exactly, Top-3 through Top-10 improve by up to 2.4 pp, and MRR increases from 0.462 to 0.469, demonstrating that aggressive preLLM filtering is strictly beneficial. C. RQ3: Ablation of Hybrid Scoring Signals To quantify the individual contributions of the two components inside the hybrid scoring function (Equation 1), we compare V1 (full scoring: wcov = 0.6, wsem = 0.4) against V2 (wsem = 0), which relies exclusively on branch-coverage evidence. For this analysis, we restrict evaluation to the 377 bugs for which both V1 and V2 completed successfully with full accuracy and safety fields, enabling a strictly paired comparison on a shared evaluation set. Table IV reports the resulting accuracy and pruning-safety metrics for both variants. Note that V1 metrics reported here differ slightly from those in Section V-B because the two analyses operate on different paired subsets (383 vs. 377 bugs), which differ in project composition. Pruning safety. Removing the semantic component does not materially alter pruning aggressiveness: the mean reduction ratio remains virtually unchanged (0.835 vs. 0.836). However, the safety cost is substantial. The strict-loss count rises from 31 to 73 (+42 additional cases in which a groundtruth faulty method is inadvertently eliminated before Stage 4),
TABLE II Method-level FL accuracy on Defects4J v1.2.0. Each cell is the number of bugs with a ground-truth method in the top-k results. Baseline figures are from original publications (395 bugs total); Bold = highest count per row per metric; underline = highest among our two variants. SieveFL V0 (Ours)
SieveFL V1 (Ours)
Project
@1
AgentFL [8] @3
@5
@1
GRACE [14] @3
@5
@1
DeepFL [13] @3
@5
@1
@3
@5
@1
@3
@5
@1
@3
@5
@1
@3
@5
Chart Lang Math Time Mockito Closure
16 44 49 11 13 24
18 45 60 13 14 33
19 45 61 13 14 35
14 42 61 11 17 47
20 54 78 14 24 70
22 57 89 19 26 81
12 43 39 9 9 64
18 53 68 16 15 86
21 56 80 18 21 97
15 40 48 8 7 42
19 53 77 15 19 66
19 55 83 18 19 77
6 24 23 6 7 14
14 44 52 11 14 30
15 50 62 13 18 38
12 41 52 16 18 26
14 45 55 16 20 31
14 45 59 16 20 33
13 39 53 14 16 30
14 41 60 17 19 39
15 42 60 18 19 42
Overall
157
183
187
192
260
294
176
256
293
160
249
271
80
165
196
165
181
187
165
190
196
Overall w/o Closure
133
150
152
145
190
213
112
170
196
118
183
194
66
135
158
139
150
154
135
151
154
TABLE III Paired cost–quality comparison between V0 and V1 on the 383 bugs completed by both variants. ∆ = V1 − V0 .
V0
V1
∆
Mean reduction ratio Mean Stage-4 candidates Mean wall-clock time (min/bug) Mean tokens in (K/bug) Mean tokens out (K/bug)
0.00 398.8 5.07 190.1 102.8
0.79 187.2 4.05 97.4 53.2
+0.79 −211.6 −1.02 −92.7 −49.6
Top-1 (%) Top-3 (%) Top-5 (%) Top-10 (%) MRR
43.1 47.3 48.8 51.7 0.462
43.1 49.6 51.2 54.0 0.469
+0.0 +2.3 +2.4 +2.3 +0.007
Metric
TABLE IV Paired ablation comparison between V1 and V2 on the 377 bugs completed by both variants. ∆ = V2 − V1 .
V1
V2
∆
Mean reduction ratio Strict-loss count Strict-loss rate (%) Mean GT recall after pruning
0.835 31 8.22 0.630
0.836 73 19.36 0.532
+0.001 +42 +11.1 −0.098
Top-1 (%) Top-3 (%) Top-5 (%) Top-10 (%) MRR
42.4 49.6 51.2 53.6 0.466
41.9 46.9 48.0 50.1 0.448
−0.5 −2.7 −3.2 −3.5 −0.018
Metric
raising the strict-loss rate from 8.22% to 19.36% (+11.1 pp). Mean ground-truth recall after pruning drops from 0.630 to 0.532 (−0.098). Note that strict-loss rate and mean GT recall measure complementary aspects of pruning safety: strict-loss is a binary per-bug indicator of whether any ground-truth method is eliminated, while GT recall captures the fraction of all labeled ground-truth methods that survive, which can fall below 1.0 even in bugs where strict-loss is false, owing to the presence of multiple co-located ground-truth methods. This dissociation between stable reduction volume and sharply
FLUCCS [37]
Ochiai [3]
rising fault loss reveals the precise role of the semantic signal: it acts not as a pruning driver but as a protective filter that steers the coverage threshold away from semantically relevant methods, reducing the risk of discarding genuine faults before LLM reasoning begins. Accuracy impact. Consistent with the safety findings, removing the semantic signal also weakens downstream ranking quality. V2 reduces Top-1 from 42.4% to 41.9% (−0.5 pp), Top-3 from 49.6% to 46.9% (−2.7 pp), Top-5 from 51.2% to 48.0% (−3.2 pp), and Top-10 from 53.6% to 50.1% (−3.5 pp). MRR falls from 0.466 to 0.448 (−0.018). The degradation widens at broader cutoffs, suggesting that the semantic signal is most influential in discriminating between candidates that share similar coverage profiles but differ in semantic relevance to the failing test. Notably, the Top-1 deficit is modest (−0.5 pp) while the Top-10 deficit is substantially larger (−3.5 pp), confirming that the semantic component’s primary contribution is in correctly ordering the confirmed suspect pool rather than in selecting the single top candidate. Answer to RQ3: Both signals contribute, but in distinct roles. Branch coverage drives pruning volume; semantic similarity acts as a safeguard that preserves faulty methods during filtering. On the 377 paired bugs, removing the semantic component (wsem = 0) leaves the reduction ratio virtually unchanged (0.835 vs. 0.836) but raises the strictloss count from 31 to 73 (+11.1 pp in strict-loss rate) and reduces mean ground-truth recall after pruning from 0.630 to 0.532. Ranking quality degrades across every measured metric: Top-1 (−0.5 pp), Top-3 (−2.7 pp), Top-5 (−3.2 pp), Top-10 (−3.5 pp), and MRR (−0.018). The semantic component is a net positive contributor to both pruning safety and ranking quality and should not be removed from the hybrid scoring function. D. RQ4: Scalability and Failure Analysis Scalability. The V1 run does not complete every benchmark instance; scalability should therefore be interpreted in terms of completed runs rather than nominal dataset size. Across the canonical Defects4J v1.2 bug ranges, SieveFL V1 completes 383 of 395 bugs. Missing runs are concentrated in Lang and are attributable to build failures, deprecated test suites, and unavailable JaCoCo traces.
TABLE V Run Coverage for SieveFL V1 (Full Scale). Project
Range
Completed
Failed
Chart Lang Math Time Mockito Closure
1–26 1–65 1–106 1–27 1–38 1–133
26/26 57/65 106/106 25/27 38/38 131/133
0 8 0 2 0 2
— 2, 18, 25, 48, 62, 63, 64, 65 — 21, 27 — 63, 93
Missing Bug IDs
Total
—
383/395
12
—
Answer to RQ4: SieveFL V1 scales to 383 of 395 benchmark bugs. Results are reported on completed bugs; incomplete coverage and trace-generation limitations are important practical constraints of the present implementation that motivate future work on more robust build and coverage infrastructure. E. Discussion Why pruning improves ranking across all metrics. The result that V1 matches or outperforms V0 at every quality metric—Top-1 through Top-10 and MRR—despite submitting 53% fewer methods to Stage 4, can be explained by two complementary effects. First, screening quality improves when the LLM evaluates each method in isolation: without irrelevant methods polluting the context, the model’s per-method verdict is more reliable. Second, removing execution-ubiquitous methods—those covered by nearly every test regardless of the failing one—prevents generic utility methods from passing Stage 4 and crowding out genuine fault sites in Stage 5’s reranking. VI. Threats to Validity a) Internal Validity.: A primary internal threat is data contamination: bugs from Defects4J may have appeared in the pre-training corpora of the LLMs we use, potentially inflating performance by allowing the model to recall rather than reason about a specific fault. We cannot fully eliminate this threat, as the training data of closed-weight and open-weight models alike is not fully disclosed. However, our pipeline’s reliance on dynamic runtime traces (JaCoCo) and per-method isolated reasoning rather than direct code generation reduces the extent to which memorization alone could drive correct rankings—a method must not only appear familiar to the LLM but must also have been executed during the failing test to survive Stage 3. A second internal threat concerns the stochastic nature of LLM outputs. Different sampling runs of Stage 4 (per-method screening) or Stage 5 (re-ranking) could in principle produce different verdicts or orderings for the same bug. To assess this, we ran Stage 5 three times on a 30-bug sample and observed that the Top-1 hit rate varied by less than 1 pp across runs, confirming that the pipeline is stable at the temperatures used in our experiments. b) External Validity.: Our evaluation is conducted exclusively on Defects4J v1.2.0, a benchmark of Java programs. While Defects4J is the de facto standard in FL research
and enables direct comparison with prior work [8], [9], its Java-centric nature limits the immediate generalizability of our results to other languages such as Python or C/C++. That said, the core components of SieveFL—LLM-based failure description, dense vector retrieval, and per-method LLM screening—are not inherently tied to Java. The only Javaspecific component is the JaCoCo instrumentation in Stage 3; replacing it with an equivalent coverage tool for the target language (e.g., coverage.py for Python) would in principle allow the same pipeline to operate on other ecosystems. We leave this cross-language validation as future work. A related threat is benchmark scale. Our largest subject, Closure (133 bugs, 90 kLOC), is representative of mid-sized industrial software but falls short of truly large-scale repositories with millions of lines of code. Although SieveFL’s hierarchical pruning is designed to keep the number of LLM calls sub-linear in codebase size, its scalability to monoreposcale systems has not been empirically verified. c) Construct Validity.: Our primary metrics—Top-k and MRR—assume that the developer’s main effort is in locating the faulty method, and they award full credit as soon as any ground-truth method appears in the top-k positions. This framing is consistent with prior work [8], [9] and reflects how FL tools are typically used in practice [2], but it does not capture the effort required to understand why a method is faulty or how to fix it. SieveFL partially addresses this limitation through the natural-language justifications produced by Stage 4 for each confirmed suspect, which provide the developer with an initial causal explanation alongside the ranked list. Evaluating the quality and utility of these justifications for developer comprehension is an important direction for future work. A second construct threat is that our ground-truth method set is derived from Defects4J’s patch annotations. A patch may modify multiple methods, only some of which are the true root cause; conversely, co-located helper methods may be omitted from the annotation even though changing them would also fix the bug. This annotation noise affects all FL studies on Defects4J equally and is an inherent limitation of patch-derived ground truth. VII. Conclusion and Future Work We presented SieveFL, a five-stage hierarchical framework for method-level fault localization that resolves the ScalePrecision Dilemma through progressive pre-LLM filtering. The central insight is that per-method LLM interrogation should be reserved for a small, high-confidence candidate set constructed from both semantic retrieval and dynamic runtime evidence. On Defects4J v1.2.0, SieveFL V1 achieves Top-1 accuracy of 41.8% (165/395, treating incomplete runs as failures) and MRR of 0.469, outperforming the strongest zero-shot agent-based baseline (AgentFL) by 2.1 pp in Top-1. Runtimeaware pruning removes 79% of candidate methods and 49% of input tokens before any LLM call, while simultaneously improving Top-3 through Top-10 and MRR—confirming that aggressive pre-LLM filtering is strictly beneficial rather than
a cost-quality trade-off. An ablation shows that the semanticsimilarity component acts as a safety net against overaggressive pruning: removing it raises the strict-loss rate by 11.1 pp and degrades Top-1 by 0.5 pp. The framework completes successfully on 96.9% of evaluated bugs; incomplete runs are attributable to build failures, deprecated test suites, and JaCoCo trace unavailability—limitations with clear technical paths to resolution. Future Work. We identify four directions for extending SieveFL. First, the most impactful limitation identified in our failure analysis is method-signature overload in JaCoCo’s line-level coverage map. We plan to replace line-level resolution with bytecode-level disambiguation using the Java Debug Interface (JDI), which can unambiguously identify which overloaded variant was invoked at the bytecode level. Second, the JaCoCo instrumentation in Stage 3 is the only Java-specific component of the pipeline. We plan to validate SieveFL on Python and TypeScript projects by substituting coverage.py and Istanbul respectively, testing whether the framework’s accuracy and efficiency advantages transfer across language ecosystems. Third, the natural-language justifications produced by Stage 4 for each confirmed suspicious method are a promising foundation for automated patch generation. Integrating SieveFL with an Automated Program Repair (APR) backend would turn the framework from a localization tool into an endto-end debugging assistant, providing not only a ranked list of fault sites but also candidate fixes grounded in the causal reasoning produced during screening. Fourth, we intend to investigate lightweight IDE integration, delivering SieveFL’s per-method screening and ranked output as real-time annotations within the developer’s editor. Given the 53% reduction in the candidate method pool and the resulting wall-clock time savings (from 5.07 to 4.05 minutes per bug) achieved by runtime-aware pruning, interactive latency of under five minutes per bug appears achievable for many project sizes, which would make SieveFL practical as an on-demand debugging assistant rather than an offline analysis tool. References [1] W. E. Wong, R. Gao, Y. Li, R. Abreu, and F. Wotawa, “A survey on software fault localization,” IEEE Transactions on Software Engineering, vol. 42, no. 8, pp. 707–740, 2016. [2] P. S. Kochhar, X. Xia, D. Lo, and S. Li, “Practitioners’ expectations on automated fault localization,” in Proceedings of the 25th international symposium on software testing and analysis, pp. 165–176, 2016. [3] R. Abreu, P. Zoeteweij, and A. J. Van Gemund, “On the accuracy of spectrum-based fault localization,” in Testing: Academic and industrial conference practice and research techniques-MUTATION (TAICPARTMUTATION 2007), pp. 89–98, IEEE, 2007. [4] J. Zhou, H. Zhang, and D. Lo, “Where should the bugs be fixed? more accurate information retrieval-based bug localization based on bug reports,” in 2012 34th International conference on software engineering (ICSE), pp. 14–24, IEEE, 2012. [5] R. K. Saha, M. Lease, S. Khurshid, and D. E. Perry, “Improving bug localization using structured information retrieval,” in 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 345–355, IEEE, 2013.
[6] Z. Li, X. Bai, H. Wang, and Y. Liu, “Irbfl: an information retrieval based fault localization approach,” in 2020 IEEE 44th Annual Computers, Software, and Applications Conference (COMPSAC), pp. 991–996, IEEE, 2020. [7] S. Kang, G. An, and S. Yoo, “A quantitative and qualitative evaluation of llm-based explainable fault localization,” Proceedings of the ACM on Software Engineering, vol. 1, no. FSE, pp. 1424–1446, 2024. [8] Y. Qin, S. Wang, Y. Lou, J. Dong, K. Wang, X. Li, and X. Mao, “Agentfl: Scaling llm-based fault localization to project-level context,” arXiv preprint arXiv:2403.16362, 2024. [9] X. Shi, Z. Li, and A. R. Chen, “Enhancing llm-based fault localization with a functionality-aware retrieval-augmented generation framework,” arXiv preprint arXiv:2509.20552, 2025. [10] R. Just, D. Jalali, and M. D. Ernst, “Defects4j: A database of existing faults to enable controlled testing studies for java programs,” in Proceedings of the 2014 international symposium on software testing and analysis, pp. 437–440, 2014. [11] W. E. Wong, V. Debroy, R. Gao, and Y. Li, “The dstar method for effective software fault localization,” IEEE Transactions on Reliability, vol. 63, no. 1, pp. 290–308, 2013. [12] S. Heiden, L. Grunske, T. Kehrer, F. Keller, A. Van Hoorn, A. Filieri, and D. Lo, “An evaluation of pure spectrum-based fault localization techniques for large-scale software systems,” Software: Practice and Experience, vol. 49, no. 8, pp. 1197–1224, 2019. [13] X. Li, W. Li, Y. Zhang, and L. Zhang, “Deepfl: Integrating multiple fault diagnosis dimensions for deep fault localization,” in Proceedings of the 28th ACM SIGSOFT international symposium on software testing and analysis, pp. 169–180, 2019. [14] Y. Lou, Q. Zhu, J. Dong, X. Li, Z. Sun, D. Hao, L. Zhang, and L. Zhang, “Boosting coverage-based fault localization via graph-based representation learning,” in Proceedings of the 29th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering, pp. 664–676, 2021. [15] W. Du, C. Li, S. Yin, H. Lin, H. Li, F. Zhan, Q. Ning, and Q. Ma, “Augmenting automated spectrum-based multi-fault localization via borderline confident learning,” ACM Transactions on Internet Technology, 2026. [16] C. Northcutt, L. Jiang, and I. Chuang, “Confident learning: Estimating uncertainty in dataset labels,” Journal of Artificial Intelligence Research, vol. 70, pp. 1373–1411, 2021. [17] S. Robertson and H. Zaragoza, The probabilistic relevance framework: BM25 and beyond, vol. 4. Now Publishers Inc, 2009. [18] C. S. Xia and L. Zhang, “Automated program repair via conversation: Fixing 162 out of 337 bugs for $0.42 each using chatgpt,” in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 819–831, 2024. [19] A. Z. Yang, C. Le Goues, R. Martins, and V. Hellendoorn, “Large language models for test-free fault localization,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, pp. 1–12, 2024. [20] R. Widyasari, J. W. Ang, T. G. Nguyen, N. Sharma, and D. Lo, “Demystifying faulty code with llm: Step-by-step reasoning for explainable fault localization,” arXiv preprint arXiv:2403.10507, 2024. [21] Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, and H. Wang, “Retrieval-augmented generation for large language models: A survey,” arXiv preprint arXiv:2312.10997, vol. 2, no. 1, 2023. [22] Z. Yang, H. Zhu, Q. Zhang, R. Gupta, and A. Kundu, “Semloc: Structured grounding of free-form llm reasoning for fault localization,” arXiv preprint arXiv:2603.29109, 2026. [23] F. Liu, T. Wang, L. Zhang, Z. Yang, J. Jiang, and Z. Sun, “Explainable fault localization for programming assignments via llm-guided annotation,” arXiv preprint arXiv:2509.25676, 2025. [24] M. Sepidband, H. V. Pham, and H. Hemmati, “On the role of fault localization context for llm-based program repair,” arXiv preprint arXiv:2604.05481, 2026. [25] F. Cuconasu, G. Trappolini, F. Siciliano, S. Filice, C. Campagnano, Y. Maarek, N. Tonellotto, and F. Silvestri, “The power of noise: Redefining retrieval for rag systems,” in Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 719–729, 2024. [26] M. N. Rafi, D. J. Kim, T.-H. Chen, and S. Wang, “A multi-agent approach to fault localization via graph-based retrieval and reflexion,” arXiv preprint arXiv:2409.13642, 2024.
[27] I. Yeo, D. Ryu, and J. Baik, “Improving llm-based fault localization with external memory and project context,” arXiv preprint arXiv:2506.03585, 2025. [28] Y. Wang, Y. Zhang, G. Li, C. Zhi, B. Li, F. Huang, Y. Li, and S. Deng, “Inspectcoder: Dynamic analysis-driven self repair through interactive llm-debugger collaboration,” Proceedings of the ACM on Programming Languages, vol. 10, no. OOPSLA1, pp. 1041–1069, 2026. [29] L. Zhong, Z. Wang, and J. Shang, “Debug like a human: A large language model debugger via verifying runtime execution step by step,” in Findings of the Association for Computational Linguistics: ACL 2024, pp. 851–870, 2024. [30] N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, “Reflexion: Language agents with verbal reinforcement learning,” Advances in neural information processing systems, vol. 36, pp. 8634–8652, 2023. [31] N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” in Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLPIJCNLP), pp. 3982–3992, 2019. [32] M. Douze, A. Guzhva, C. Deng, J. Johnson, G. Szilvasy, P.-E. Mazaré, M. Lomeli, L. Hosseini, and H. Jégou, “The faiss library,” IEEE Transactions on Big Data, 2025. [33] G. Salton, A. Wong, and C.-S. Yang, “A vector space model for automatic indexing,” Communications of the ACM, vol. 18, no. 11, pp. 613–620, 1975. [34] “JaCoCo: Java code coverage library.” https://www.jacoco.org/jacoco/, 2026. Accessed: 2026-02-24. [35] J. Xuan and M. Monperrus, “Learning to combine multiple ranking metrics for fault localization,” in 2014 IEEE international conference on software maintenance and evolution, pp. 191–200, IEEE, 2014. [36] Y.-A. Xiao, P. Gao, C. Peng, and Y. Xiong, “Improving the efficiency of llm agent systems through trajectory reduction,” arXiv preprint arXiv:2509.23586, 2025. [37] J. Sohn and S. Yoo, “Fluccs: Using code and change metrics to improve fault localization,” in Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 273–283, 2017.