Efficient Black-Box Fault Localization for System-Level Test Code Using Large Language Models

[2506.19045] Efficient Black-Box Fault Localization for System-Level Test Code Using Large Language Models Skip to main content Search Submit Donate Log in Search arXiv Press Enter to search · Advanced search Computer Science > Software Engineering arXiv:2506.19045 (cs) [Submitted on 23 Jun 2025 ( v1 ), last revised 21 Jul 2026 (this version, v6)] Title: Efficient Black-Box Fault Localization for System-Level Test Code Using Large Language Models Authors: Ahmadreza Saboor Yaraghi , Golnaz Gharachorlu , Sakina Fatima , Lionel C. Briand , Ruiyuan Wan , Ruifeng Gao View a PDF of the paper titled Efficient Black-Box Fault Localization for System-Level Test Code Using Large Language Models, by Ahmadreza Saboor Yaraghi and Golnaz Gharachorlu and Sakina Fatima and Lionel C. Briand and Ruiyuan Wan and Ruifeng Gao View PDF HTML (experimental) Abstract: Fault localization (FL) is a critical step in debugging, which typically relies on repeated executions to pinpoint faulty code regions. However, repeated executions can be impractical in the presence of non-deterministic failures or high execution costs. While recent efforts have leveraged Large Language Models (LLMs) to aid execution-free FL, these have primarily focused on identifying faults in the system-under-test (SUT) rather than in the often complex system-level test code. However, the latter is also important, as in practice, many failures are triggered by faulty test code. To overcome these challenges, we introduce a fully static, LLM-driven approach for system-level test code fault localization (TCFL) that does not require executing the test case. Our method uses a single failure execution log to estimate the test's execution trace through three novel algorithms that identify only code statements likely involved in the failure. This pruned trace, combined with the error message, is used to prompt the LLM to rank potential faulty locations. Our black-box, system-level approach requires no access to the SUT source code and is applicable to complex test scripts that assess full system behavior. We evaluate our technique at the function, block, and line levels using an industrial dataset of faulty Python test cases that were not used in pre-training LLMs. Results show that our best-estimated traces closely match the actual traces, with an F1 score of around 90%. Additionally, pruning the complex system-level test code reduces the LLM's inference time by up to 34% without any loss in FL performance. Our method achieves equal or higher FL accuracy, requiring over 85% less average inference time per test case and 93% fewer tokens than the latest LLM-guided FL method. Comments: Accepted for publication at IEEE Transactions on Software Engineering (TSE) 2026, 33 pages. Link to Github repository: this https URL Subjects: Software Engineering (cs.SE) Cite as: arXiv:2506.19045 [cs.SE] (or arXiv:2506.19045v6 [cs.SE] for this version) https://doi.org/10.48550/arXiv.2506.19045 Focus to learn more arXiv-issued DOI via DataCite Related DOI : https://doi.org/10.1109/TSE.2026.3707426 Focus to learn more DOI(s) linking to related resources Submission history From: Golnaz Gharachorlu [ view email ] [v1] Mon, 23 Jun 2025 19:04:51 UTC (677 KB) [v2] Tue, 9 Dec 2025 19:46:05 UTC (706 KB) [v3] Fri, 10 Apr 2026 20:55:05 UTC (210 KB) [v4] Tue, 12 May 2026 14:13:32 UTC (210 KB) [v5] Sat, 27 Jun 2026 20:06:02 UTC (210 KB) [v6] Tue, 21 Jul 2026 15:45:12 UTC (210 KB) Full-text links: Access Paper: View a PDF of the paper titled Efficient Black-Box Fault Localization for System-Level Test Code Using Large Language Models, by Ahmadreza Saboor Yaraghi and Golnaz Gharachorlu and Sakina Fatima and Lionel C. Briand and Ruiyuan Wan and Ruifeng Gao View PDF HTML (experimental) TeX Source view license Current browse context: cs.SE < prev | next > new | recent | 2025-06 Change to browse by: cs References & Citations NASA ADS Google Scholar Semantic Scholar export BibTeX citation Loading... BibTeX formatted citation × loading... Data provided by: Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer ( What is the Explorer? ) Connected Papers Toggle Connected Papers ( What is Connected Papers? ) Litmaps Toggle Litmaps ( What is Litmaps? ) scite.ai Toggle scite Smart Citations ( What are Smart Citations? ) Code, Data, Media Code, Data and Media Associated with this Article alphaXiv Toggle alphaXiv ( What is alphaXiv? ) Links to Code Toggle CatalyzeX Code Finder for Papers ( What is CatalyzeX? ) DagsHub Toggle DagsHub ( What is DagsHub? ) GotitPub Toggle Gotit.pub ( What is GotitPub? ) Huggingface Toggle Hugging Face ( What is Huggingface? ) ScienceCast Toggle ScienceCast ( What is ScienceCast? ) Demos Demos Replicate Toggle Replicate ( What is Replicate? ) Spaces Toggle Hugging Face Spaces ( What is Spaces? ) Spaces Toggle TXYZ.AI ( What is TXYZ.AI? ) Related Papers Recommenders and Search Tools Link to Influence Flower Influence Flower ( What are Influence Flowers? ) Core recommender toggle CORE Recommender ( What is CORE? ) Author Venue Institution Topic About arXivLabs arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs . Which authors of this paper are endorsers? | Disable MathJax ( What is MathJax? ) We gratefully acknowledge support from our major funders , member institutions , , and all contributors. About · Help · Contact · Subscribe · Copyright · Privacy · Accessibility · Operational Status (opens in new tab) Major funding support from

Related documents

• Supporting Industrial Test-Failure Analysis with LLM-Based Systems: An Experience Report#1036232
• Supporting Industrial Test-Failure Analysis with LLM-Based Systems: An Experience Report#1006923
• How do Execution Features Improve Statistical Fault Localization? An Empirical Study#321896
• How Developers Experience Debugging Unfamiliar Codebases with Code Tours Generated and Evaluated by Local LLMs#411102
• LLVM-Bench: Benchmarking and Advancing Large Language Models for LLVM Compiler Issue Resolution#329156