LLM-Based Automated Diagnosis Of Integration Test Failures At Google Celal Ziftci
Ray Liu
[email protected] Google New York, NY, USA
[email protected] Google New York, NY, USA
arXiv:2604.12108v1 [cs.SE] 13 Apr 2026
Spencer Greene
Livio Dalloro
[email protected] Google New York, NY, USA
[email protected] Google New York, NY, USA
Abstract
Preprint Notice
Integration testing is critical for the quality and reliability of complex software systems. However, diagnosing their failures presents significant challenges due to the massive volume, unstructured nature, and heterogeneity of logs they generate. These result in a high cognitive load, low signal-to-noise ratio, and make diagnosis difficult and time-consuming. Developers complain about these difficulties consistently and report spending substantially more time diagnosing integration test failures compared to unit test failures. To address these shortcomings, we introduce Auto-Diagnose, a novel diagnosis tool that leverages Large Language Models (LLMs) to help developers efficiently determine the root cause of integration test failures. Auto-Diagnose analyzes failure logs, produces concise summaries with the most relevant log lines, and is integrated into Critique, Google’s internal code review system, providing contextual and in-time assistance. Based on our case studies, Auto-Diagnose is highly effective. A manual evaluation conducted on 71 real-world failures demonstrated 90.14% accuracy in diagnosing the root cause. Following its Google-wide deployment, Auto-Diagnose was used across 52, 635 distinct failing tests. User feedback indicated that the tool was deemed "Not helpful" in only 5.8% of cases, and it was ranked #14 (in the top 3.78%) in helpfulness among 370 tools that post findings in Critique. Finally, user interviews confirmed the perceived usefulness of Auto-Diagnose and positive reception of integrating automatic diagnostic assistance into existing workflows. We conclude that LLMs are highly successful in diagnosing integration test failures due to their capacity to process and summarize complex textual data. Integrating such AI-powered tooling automatically into developers’ daily workflows is perceived positively, with the tool’s accuracy remaining a critical factor in shaping developer perception and adoption.
This is a preprint of a paper accepted at the IEEE/ACM 48th International Conference on Software Engineering (ICSE) 2026.
CCS Concepts • Software and its engineering → Software testing and debugging; Empirical software validation; Maintaining software; • Computing methodologies → Information extraction.
Keywords Software, Testing, Debugging, Diagnosis, Productivity, LLM
1
Introduction
Software testing is a crucial phase in the software development lifecycle, ensuring the quality and reliability of complex systems. Unit testing and integration testing are two of the most commonly employed types of software testing [38]. Unit testing focuses on individual components or modules of a software application. The primary goal of unit tests is to verify that each unit of the software performs as expected in isolation. Developers typically write these tests during the coding phase, and they are often automated. They help in early detection of bugs, facilitate easier debugging, and provide documentation for the individual components. Integration testing verifies the interaction, communication, and data exchange between separate software components that are typically already unit-tested [14, 38]. In modern, complex, distributed software systems, integration testing is a critical part of quality assurance. It aims to expose defects in the interfaces and interactions between integrated components, rather than within the components themselves. This type of testing is essential for ensuring that different parts of a system work together seamlessly to achieve the desired functionalities. When these tests fail, the primary artifacts available for postmortem analysis and debugging are the semi-structured text logs [11, 12] generated by the test runtime itself and the system under test (SUT). However, as software systems have scaled, the manual analysis of these logs has become infeasible. Developers face significant challenges, including massive log volumes [12], unstructured and heterogeneous text formats [60], and a low signal-to-noise ratio where crucial errors are often obscured by routine operational data [64]. This complexity results in a high cognitive load required to interpret the failure context across numerous log files. The difficulty of this process is empirically documented within Google: diagnosing integration test failures was identified as one of the top five most frequent complaints in a company-wide survey [5] of 6, 059 developers. Furthermore, developers consistently report spending substantially more time diagnosing integration test failures, often more than an hour and sometimes exceeding a day, compared to unit test failures, highlighting the inherent difficulty
Celal Ziftci, Ray Liu, Spencer Greene, and Livio Dalloro
of this task. The scale of the challenge is reflected in the properties of failing tests analyzed, which had a median of 16 log files and 2, 801 log lines. While many automated program diagnosis and repair techniques exist, from statistical debugging to spectrum-based analysis, recent work has increasingly explored the capabilities of Large Language Models (LLMs). However, this research, along with its associated benchmarks, has concentrated primarily on small-scale or unit-testlevel debugging and repair. The unique challenges of integration testing, involving multiple interacting components, complex environment setups, and massive, distributed log outputs, remain largely under-explored. To address the shortcomings and inefficiency of manual log investigation, this paper proposes Auto-Diagnose, a novel failure diagnosis tool leveraging LLMs. Given the proficiency of LLMs in text processing and summarization tasks, Auto-Diagnose is designed to help developers efficiently determine the root cause of complex integration test failures. This functionality is integrated into Google’s internal code review system Critique [45], providing contextual and in-time assistance to streamline the debugging process. The effectiveness of Auto-Diagnose was evaluated through quantitative and qualitative methods. A manual evaluation on 71 realworld integration test failures demonstrated a high accuracy rate of 90.14% in detecting and summarizing the root cause. Following its deployment, Auto-Diagnose was used across 52, 635 distinct tests. User feedback gathered from code changes indicated that the tool was deemed "Not helpful" in only 5.8% of cases. User interviews confirmed the perceived usefulness and positive reception of integrating automatic diagnostic assistance into existing workflows. We conclude that LLMs are highly successful in diagnosing integration test failures due to their capacity to process and summarize complex textual data, and developers prefer using such tooling when integrated into their workflows.
2
Diagnosing Integration Test Failures
In this section, we discuss our motivation to focus specifically on integration tests, showcase a specific type of integration test at Google with its various characteristics on a motivating example, explain how code reviews are conducted and integration test failures are reported during the code review process, showcase how developers diagnose test failures traditionally, and finally discuss the shortcomings of manual diagnosis.
2.1
Why Focus On Integration Tests?
Our work focuses on diagnosing integration tests rather than unit tests, informed by several internal surveys conducted at Google. First, in a Google-wide survey of 6, 059 developers about their use of and experience on various internal developer tools, named EngSat [5], diagnosing integration test failures was identified as one of the top five most frequent complaints. To understand the differences between unit and integration test failures, we conducted a follow-up survey to ask 116 developers to self-report usage and pain points on diagnosing unit and integration tests, named Survey-2. As shown in Figure 1a, developers reported encountering unit test failures more often than integration
(a) Frequency of encountering unit and integration test failures
(b) Duration of failure diagnosis of unit and integration test failures
Figure 1: Differences in the encounter and diagnosis of unit and integration test failures as reported by 116 survey respondents. test failures daily and weekly, while integration test failures are reported to occur more frequently monthly. Unit tests are typically executed earlier and more often in the software development lifecycle, while integration tests are executed later, as they are typically more expensive and slower. As a result, the survey responses support our expectations of the frequency of unit and integration test failures. Additionally, as shown in Figure 1b, developers reported spending significantly more time diagnosing integration test failures, often more than an hour and sometimes longer than a day, pointing to its difficulty and supporting the complaints identified in EngSat. We discuss potential reasons for this in Section 2.4.
2.2
Hermetic Functional Integration Tests
At Google, there are different kinds of integration tests employed by teams, including functional, performance, reliability and security. The tests under consideration for this work are the hermetic functional tests, those that are brought up entirely within isolated environments without relying on external services or shared infrastructure, and that exercise business logic, as opposed to qualitative aspects of the system such as performance, security and reliability.
LLM-Based Automated Diagnosis Of Integration Test Failures At Google
Figure 3: Integration test failure information surfaced in Critique [46], Google’s web-based internal code review system. System Under Test (SUT)
Figure 2: Different kinds of integration tests employed by teams at Google, based on a survey of 239 respondents. Note that a single team can have several types of these integration tests, therefore, the sum exceeds 100%. We focus on hermetic tests, as they provide a consistent and reproducible testing environment, simplifying debugging by eliminating external dependencies as potential sources of failure. We focus on functional tests, as they are the most frequently used integration tests across Google based on a survey of 239 respondents, as shown in Figure 2. Note that a single team can have several types of these integration tests, therefore, the sum exceeds 100%.
2.3
Integration Test Failure Findings During Code Review
Google’s internal code review system, Critique [46], is a web-based interface that facilitates the code review process, enabling discussions between authors and reviewers, and ensuring adherence to code quality standards and policies. Beyond human reviews, Critique also integrates automated analysis results crucial for maintaining code health. A key mechanism for surfacing automated analysis feedback within Critique is the findings feature [47] which allows various tools and services to post structured annotations as comments, directly within the context of a pending change. These findings may range from style linting errors, to static analysis warnings and the results of continuous integration systems and test executions [30]. Relevant to test-driven development, as shown in Figure 3, the functional hermetic integration tests discussed earlier are typically executed automatically when a code change is sent to a reviewer, and failures are surfaced as actionable findings within Critique, preventing submission until resolved. This tight integration ensures that test results are highly visible and directly linked to the code changes under review, reinforcing the practice of submitting welltested code.
2.4
Diagnosing Integration Test Failures Is Hard
Functional integration tests, as shown in Figure 4, typically comprise of the test driver and the system under test (SUT). The SUT typically consists of a collection of servers, called components, that
server-x.info
server-x
server-a Test driver
my-server.info
my-server
server-a.info
test-driver.info test-driver.error
my-server.error
their-server infra-server
their-server.info
infra-server.info
Figure 4: Test driver and the system under test (SUT) that it controls to exercise business logic in the integration test. Each component produces its own log files, typically split by log level, and named dynamically after the component.
communicate with each other. The test driver is responsible for configuring the SUT and bringing up the components needed by the test, and is itself a separate component written in languages such as C++, Java or Python. During the execution of an integration test, logs [60] are generated from several sources: Test driver logs: The test driver produces its own logs to record its actions, configurations, and any issues it encounters while setting up the components or interacting with the SUT. SUT component logs: Each individual component within the SUT generates its own logs output to distinct files, detailing its internal operations, communication with other components, and any errors or events that occur during its execution. Each component produces log files dynamically named after the component name and can produce several log files split by log level.
Celal Ziftci, Ray Liu, Spencer Greene, and Livio Dalloro
These logs are crucial to understand the behavior of the distributed system. Both the test driver logs and the SUT component logs are collected and made available to developers for diagnosis when a test fails. The default logs shown to the developer as they investigate the failure are the test driver logs, as these are the highest level logs about the execution of the test. When a developer receives a finding for a failing integration test on their code change during code review, shown in Figure 3, the diagnostic process typically begins with an examination of the test driver’s logs that provide a high-level summary of the test execution. However, there are several shortcomings of test driver logs, as they lack the necessary detail to pinpoint the root cause of the failure. First, most of the time, a failure manifests as a generic error, e.g. a timeout, in the test driver logs, indicating that a component of the SUT failed to become operational within the expected time frame, or that a specific test assertion caused a failure. In these cases, as shown in Figure 4, developers need to investigate the list of created log files one by one to identify which one contains the root cause. Second, the sheer volume of logs, originating from both the test driver and numerous SUT components, presents a significant challenge [12]. Developers must manually sift through a multitude of log files, each with its own formatting and conventions, to find the relevant error messages. Third, many logs contain warnings and errors that are not related to the current failure, or the execution of the test at all, as they happen to be logged at warning or error levels even though they are recoverable. This creates a low signal-to-noise ratio that can mislead the developers during investigation [64]. The cognitive load required to correlate events across different logs and distinguish between benign errors and the actual cause of the failure is substantial. This complexity often overwhelms developers, leading them to seek alternative ways for diagnosis of the failure. Two of the most common ways are seeking assistance from colleagues with more experience in the specific system or test, and consulting the infrastructure team that builds and maintains the testing framework itself. This reliance on manual expertise results in inefficiency and scalability issues inherent in diagnosing integration test failures, highlighting the need for more efficient tools to investigate such failures.
3
Auto-Diagnose: Automatically Diagnosing Integration Test Failures
As LLMs are known to work well on text processing and summarization tasks, we employed them on the task of processing integration test failure logs and producing a diagnosis summary for developers. In this section, we discuss the elements of the system, named Auto-Diagnose, shown in Figure 5. Upon a test failure, Auto-Diagnose gets notified and runs automatically, and test driver and SUT component logs at levels INFO and higher, spread across different data centers, processes, threads and logging levels, are joined and sorted by timestamp into a single log stream to be passed to the LLM. Listing 1 shows examples of log lines collected from the server-a component.
server - a . info : 2025 -09 -17 -14:12:32 | dc7 | p41 | t -2 | file . py :444 | Server is starting server - a . error : 2025 -09 -17 -16:59:41 | dc3 | p13 | t -7 | file2 . py :41 | Server encountered an error , shutting down
Listing 1: Example log lines from the server-a component.
Then, Auto-Diagnose constructs an LLM prompt to be sent to Gemini [6, 51] using the prompt template, shown in Figure 7, concatenated with logs following the <LOGS=> section, and component metadata under the <CONTEXT=> section. The prompt is developed over several iterations of observing the LLM’s outputs on real-world failures to include guided, step-by-step reasoning with strict negative constraints and precise output formatting, to avoid speculative, incomplete, or irrelevant diagnoses. The LLM parameters used are: • LLM = Gemini 2.5 Flash [6, 51] so that Auto-Diagnose is fast and cost effective. • 𝑡𝑒𝑚𝑝𝑒𝑟𝑎𝑡𝑢𝑟𝑒 = 0.1 so that LLM responses are mostly deterministic for easy debugging. • 𝑡𝑜𝑝𝑝 = 0.8 so that there is room for creativity while keeping out extremely unlikely tokens. Gemini has not been trained or fine-tuned with the specific integration test failure logs that Auto-Diagnose ran on. After getting the diagnosis from the LLM, Auto-Diagnose postprocesses its response to show the output in markdown format, to convert the log lines it provides to links, and to produce a finding to show in Critique, as shown in Figure 6. Developers may find this diagnosis useful as is, or may click the links to investigate the failure further.
4
Evaluation
In this section, we discuss how we evaluate Auto-Diagnose, describe our case studies, and report the results of our interviews on the usability of our system.
4.1
Manual Evaluation
To evaluate the effectiveness of Auto-Diagnose, we first conducted a case study where we ran Auto-Diagnose on 71 randomly selected integration test failures from 39 distinct teams across Google, summarized in Table 1. We asked 3 expert developers with at least five years of experience each from the integration test infrastructure team to assess the diagnoses, similar to the one in Figure 6, and report whether the Conclusion or any of the reported Most Relevant Log Lines provide accurate context for the root cause of the failure. We then held a meeting to go over their reports and align on any disagreements, to obtain final agreement on the assessment for each failure. Based on the experts’ reporting, Auto-Diagnose was successful in diagnosing the root cause on 64 of the failures, resulting in a 90.14% success rate. We also asked the three expert developers to investigate where Auto-Diagnose may have failed to diagnose the root cause on the remaining 7 failures. Based on deeper investigation, two distinct issues were discovered, in 4 of the cases, test driver log files were
LLM-Based Automated Diagnosis Of Integration Test Failures At Google
Test failure pub/sub queue
Process logs
prompt
Prompt Gemini
*.info *.error
diagnosis
Post process
finding
Critique
developer
Figure 5: System overview of automatically generating findings for integration test failure diagnosis with Auto-Diagnose. Table 2: Aggregate statistics about Auto-Diagnose findings posted to Critique and the properties of the failing integration tests it diagnosed. # total code changes # total authors of the code changes # distinct tests analyzed by Auto-Diagnose # total executions of Auto-Diagnose p50 time to post Auto-Diagnose finding to Critique p90 time to post Auto-Diagnose finding to Critique Mean # log files per failing test Median # log files per failing test Mean # log lines per failing test Median # log lines per failing test Mean # input tokens per Auto-Diagnose execution Mean # output tokens per Auto-Diagnose execution
Figure 6: The LLM-based diagnosis result posted as a finding to Critique. Table 1: Aggregate statistics about the manual evaluations. # teams that owned the failures evaluated # developers that conducted evaluations # failures evaluated # accurate diagnoses on the failures evaluated
39 3 71 64 (90.14%)
not properly saved when it crashed, and in 3 of the cases, SUT component log files were not properly saved when the component crashed, both bugs discovered in the test infrastructure and reported to the relevant teams.
4.2
Production Usage and User Feedback
After the manual evaluation, we launched Auto-Diagnose to automatically run on all integration test failures during code changes across Google’s code repository [37] starting May 2025, with detailed statistics listed in Table 2. We showed Critique findings, as shown in Figure 6, on 91, 130 code changes from 22, 962 distinct authors for 224, 782 executions of 52, 635 distinct tests. For test failures, it is important to post any diagnostic help findings to Critique quickly so that developers do not change context, or
91, 130 22, 962 52, 635 224, 782 56 sec 346 sec 26 16 11, 058 2, 801 110, 617 5, 962
start manually debugging the failure themselves. Auto-Diagnose took 56 seconds in the median (p50), and 346 seconds in the 90th percentile (p90) to post its findings to Critique, significantly faster than the time developers typically spend on debugging integration tests, discussed in Figure 1b. The failing integration tests had a mean of 26 log files and a median of 16 log files, and a mean of 11, 058 log lines and a median of 2, 801 log lines, in line with the difficulties of diagnosing integration test failures discussed in Section 2.4. Executions of Auto-Diagnose had a mean of 110, 617 input tokens and 5, 962 output tokens, making it highly cost effective. On code changes, as shown in the bottom section of Figure 6, developers can click on several buttons to interact with the findings: • Please fix – PF: A reviewer can click this to ask the author of the code change to fix it. Note that the same "Please fix" button exists on the test failure finding itself, as shown in Figure 3. Typically, a reviewer would click on "Please fix" on that finding if they want the test to be fixed, while they would click "Please fix" on the Auto-Diagnose finding if they intend to ask the author to make use of the diagnosis. • m Helpful – H: An author can click this to give feedback to the tool owner that the diagnosis was helpful. • l Not helpful – N: An author can click this to give feedback to the tool owner that the diagnosis was not helpful.
Celal Ziftci, Ray Liu, Spencer Greene, and Livio Dalloro
You are helping developers at Google understand the root cause of a failed integration test. Available log lines about the test are listed under <LOGS=>, and context about SUT components are listed under <CONTEXT=>. Your goal is to find the root cause from the log lines under <LOGS=> and potentially <CONTEXT=>. Let’s think about the analysis step-by-step and show your thought process after each step: 1. Scan all sections in <LOGS=> as subsection headers. No need to print all of them out at this point, but you will need to refer to them later. 2. If <CONTEXT=> is provided, read it and treat it as context and potentially instructions to help you understand the log lines and debug test failures better. 3. Inspect the section that contains test failures. 4. Inspect the logs section if it exists. Summarize the errors, and what component failed. 5. Inspect the other sections and summarize what those lines are signaling and print the most likely cause of the error. Also take into consideration of the command line arguments passed to the component, and see if the error is related to the arguments. For example, if the command line argument contains a keyword that’s also mentioned in the error message, it’s likely that the argument is a contributing factor to the error. 6. Try to reach a conclusion on the root cause of the failure. You must not skip the rest of the steps. 7. Judge if you have enough information to reach a conclusion in step 6. You must ruthlessly adhere to the following rules: - If the logs clearly point out the SUT component that failed to start up, you MUST locate the corresponding component log lines and *ONLY* use those to reach a conclusion. - If the logs do not contain any log lines from the component that failed to become healthy, you *MUST NOT* draw any conclusion from the information you have. - You MUST not make any assumptions about the SUT infrastructure: if specific SUT components are pointed out, you MUST *ONLY* use those components’ log lines to reach a conclusion. - Any conclusion about a SUT component must be based on its log lines, and you MUST NOT draw any conclusions by guessing. 8. From step 7, if you don’t understand the root cause of the failure, think about what other information you might need. For example, if there are errors complaining about processes not mentioned in the log lines, you should mention in your response that you need access to those logs and you must not draw any conclusion from the information you have. Things to keep in mind: - Every test will have a log about the test exiting due to SIGINT. This is normal and not the cause of the failure. - The response should be in human readable sentences. The format depends on the conclusion of your investigation (i.e. if step 7 is satisfied and if step 8 is performed). If you reached a conclusion in step 6 and verified that you have enough information to reach that conclusion in step 7, start with the conclusion of your investigation under a "==Conclusion==" header. This is the most important part of the response. Finally, list the steps you have taken in the process under a "==Investigation Steps==" header. After that, print the most relevant log lines under a "==Most Relevant Log Lines==" header. These log lines must be from the sections printed in step-5. For each log line, print it in the following format: - log-file-name: <log-file-name> (e.g. foo-bar.info, must be the same as the log-file-name in the section header) - timestamp: <timestamp-in-the-format-of-YYYY-MM-DD-hh:mm:ss> (e.g. 2025-03-27-06:00:000, if you can’t find the timestamp, leave this empty) - callsite: <callsite-file-name:callsite-line> (e.g. foo/bar/baz.cc:123, if callsite-filename is long, only include the suffix) **content**: <root-cause-relevant-part-of-the-log-line> (e.g. The server encountered an error: the root cause of the failure) This log line format is important and should be followed exactly. It’s VERY IMPORTANT to only include the interesting part of the log line content. <LOGS=> %s <CONTEXT=> %s
Figure 7: The prompt template used to construct the prompt sent to the LLM.
LLM-Based Automated Diagnosis Of Integration Test Failures At Google
Table 3: Statistics about the feedback posted to Critique by different tools and Auto-Diagnose. # tools that post findings to Critique Mean feedback-rate across all findings Median feedback-rate across all findings Feedback-rate for Auto-Diagnose findings Feedback-rate rank of Auto-Diagnose among all tools
370 0.1042 0.0012 0.0023 #134
Table 4: Aggregate statistics about the user feedback for Auto-Diagnose. # total feedback # total code changes with feedback # distinct developers that reported feedback Please fix – PF # total "Please fix" # distinct developers that clicked "Please fix" m Helpful – H # total "Helpful" # distinct developers that clicked "Helpful" Helpfulness-rate ( 𝐻𝐻+𝑁 ) Helpfulness-rate rank across all tools l Not Helpful – N # total "Not helpful" # distinct developers that clicked "Not Helpful" 𝑁 Not-helpful-rate ( 𝑃 𝐹 +𝐻 +𝑁 )
517 458 437 436 (84.3%) 370 51 (9.9%) 43 62.96% #14 out of 370 30 (5.8%) 28 5.8%
To better understand user engagement and feedback, we compared Auto-Diagnose with other tools that post findings in Critique across Google on several dimensions. Feedback-rate: This metric measures user engagement, i.e. how often developers report feedback upon a finding on a code change in Critique. Listed in Table 3, among 370 tools that posted at least 100 findings to Critique, Auto-Diagnose ranked #134 with a feedbackrate of 0.0023, compared to a median of 0.0012 and a mean of 0.1042 across all findings. Developers were engaged with Auto-Diagnose in line with other tool findings posted to their code changes. Table 4 lists statistics of feedback reports we received from developers: a total of 517 feedback were reported by 437 distinct users, 436 of these reports were "Please fix" reported by 370 reviewers, 51 of the reports were "Helpful" reported by 43 distinct authors, and 30 of the reports were "Not helpful" reported by 28 distinct authors. 𝑁 Not-helpful-rate: This metric measures 𝑃 𝐹 +𝐻 +𝑁 , the percentage of "Not helpful" feedback reports across all reports. To avoid noisy tools and annoying developers with bad findings, for a tool to continue posting findings in Critique, its not-helpful-rate has to stay below 10% as a general guideline [47]. Auto-Diagnose has a 5.8% not-helpful-rate, in line with the accuracy results of our manual evaluation.
Helpfulness-rate: This metric measures 𝐻𝐻+𝑁 , the ratio of "Helpful" feedback reports to all code change author feedback reports. This ratio is important as it reflects that the diagnoses produce good findings. For Auto-Diagnose, this metric is currently 62.96%, with more "Useful" feedback than "Not useful". The reported helpfulness-rate of 62.96% is lower compared to our manual evaluation with 90.14% accurate diagnoses. It is a wellestablished phenomenon in psychology that users are more motivated to report negative experiences than positive ones [43]. Therefore, in feedback mechanisms such as thumbs up/down buttons, common in modern developer tools [48], it is expected that negative feedback will be disproportionately represented, as positive or ’as-expected’ interactions often go unreported. We propose that this phenomenon plays a role in the Auto-Diagnose user feedback. Furthermore, many of the tools that post findings to Critique do not use AI or LLMs, they are deterministic checks on various aspects of code such as lints, tests and style checks. Across 370 different types of tools that post findings to Critique, at the time of writing, Auto-Diagnose ranks as #14 (in the top 3.78%). Given the high rank, we propose that developers have been frustrated with diagnosing integration test failures as identified with surveys, and Auto-Diagnose diagnoses resonate with them when they are helpful.
4.3
User interviews
We conducted 11 in-person interviews with code change authors that reported "Useful" and "Not useful", and code change reviewers that reported "Please fix" feedback, with excerpts in Figure 8. In this section, we summarize key learnings based on the participants’ responses. First, several participants (e.g. P-1, P-2, P-3) discussed the difficulty of diagnosing failing integration tests, with reasons similar to those discussed in Section 2.4. Several participants (e.g. P-4, P-1, P-7) reported that they found the diagnoses useful. P-1 highlighted that they already expected LLMs to help with finding the root cause of failures, while P-7 focused on the pleasant user experience provided with the integration of Auto-Diagnose into Critique for automatic diagnosis upon test failures. Integrating such tools into developers’ daily workflows seamlessly and providing them findings automatically, without effort from them to run tools, is typically welcomed by developers. Participant P-9 pointed to an Auto-Diagnose diagnosis that reported "more information is needed to diagnose the root cause of the failure". Upon investigation, we determined that this was due to one of the infrastructure bugs discussed in Section 4.1, and there have been around 20 such diagnoses left by Auto-Diagnose on various code changes to date. While it helped determine issues in infrastructure, we observe that developers are highly sensitive to unhelpful Auto-Diagnose findings, especially when such findings are surfaced automatically and inside their workflow, without any action from the developer, reinforcing the guidelines on limits for unhelpful findings discussed in Section 4.2. Several participants (e.g. P-3) discussed that, even though the summaries are helpful, their expectation is receiving a fix automatically. While Auto-Diagnose may be perceived as useful, some
Celal Ziftci, Ray Liu, Spencer Greene, and Livio Dalloro
5
Threats To Validity
[P-1]: I don’t really use test logs a lot for debugging integration tests, I find it quite hard to find the actual point of failure in those.
In this section, we discuss the threats to the validity of our work and case studies.
[P-2]: Debugging [integration tests] is definitely a learning curve; there is a large list of log files, I sometimes need to dive deep and jump between them to find the actual error, very tricky. [...] I can imagine it will be very challenging for people who just start integration testing.
Manual evaluation: Our manual evaluation was conducted by three expert developers in diagnosing integration tests. However, these developers did not own the production or test code of the failing tests, and may have made mistakes in their reporting of helpfulness of Auto-Diagnose. Additionally, the failures used in this evaluation were randomly chosen from across Google, and may not be representative of all failures.
[P-3]: I wish it were clearer where to find the actual error message, often multiple layers, actual error message might be hidden deep down in some log file [...] [P-4]: The [diagnosis] information is helpful; I usually have to check logs, but now it’s putting all the information in front of me, no need to go into logs, filter by error, no need to scan all of that, I have the summary. [P-1]: I always wondered, why couldn’t an LLM just have told me about this [error] before? Today’s debugging took 10-15 minutes, but we’ve all had those issues that take over an hour and maybe another engineer’s time. [...] This is a great step forward in debugging integration tests faster! I am looking forward to using this tool more going forward. [P-7]: [...] It is great that the conclusive debug information shows up in Critique. Now that I have seen it during code review, I reopened my editor to simply make the change. [P-9]: Auto-Diagnose is telling me that it needs access to more logs to find the root cause. Doesn’t it have all the logs? [P-3]: Auto-Diagnose should not just show a summary, it should give me the fix.
Figure 8: Excerpts of feedback on Auto-Diagnose Critique findings from interviews conducted with 11 participants.
developers are now expecting more from LLM-based tooling, e.g. fixes instead of diagnoses, as they get a better understanding of the capabilities of LLMs and as they use them more often in their daily workflows. Based on the manual evaluations, user feedback and user interviews, we conclude that LLMs are highly successful in diagnosing integration test failures, given their capacity to process and summarize text. Integrating such tooling to automatically run and help developers in their workflows is perceived positively by developers, although the accuracy of the tool plays a critical role in this perception.
Developer Bias: The user feedback on code changes was open to all developers across Google. However, we got feedback from a small subset of developers, which may have introduced a selection bias and biased the accuracy we obtained from user feedback. Additionally, the interviews were conducted with code change authors who reported "Useful" or "Not useful" feedback, and reviewers who reported "Please fix". This constitutes a self-selected group, users who did not provide any feedback or who had neutral experiences might have different perspectives, which could alter the qualitative insights gained from the interviews. Developer infrastructure: Our case study was conducted within the specific context of Google’s development environment and infrastructure, which may limit the generalizability of the findings to other organizations. LLM: Our study uses Gemini 2.5 Flash. Although popular large LLMs have been shown to perform well on tasks such as summarization, the generalizability of our results to other LLMs may be limited. Additionally, LLMs are constantly evolving. The performance and characteristics of Gemini 2.5 Flash may have changed during the course of our studies, which may have impacted our case study results. Finally, Auto-Diagnose relies on a specific prompt template for Gemini. Small changes in the prompt wording, structure, or ordering of instructions could potentially lead to different or degraded LLM performance. The robustness of the prompt engineering to minor variations or future LLM updates is a potential threat to the generalizability of our findings. Logs: As they are implemented by developers, the quality and detail of logs from different components may vary significantly, and this may have influenced the accuracy of Auto-Diagnose on different tests and the results of our case studies.
6
Related Work
There is a rich body of existing work on program diagnosis and repair. Statistical debugging: Early automated debugging methods use statistical analysis of program behaviors to pinpoint bugs. For example, Liblit et al. introduce Scalable Statistical Bug Isolation [24],
LLM-Based Automated Diagnosis Of Integration Test Failures At Google
which monitors sampled boolean predicates during many runs to find those strongly correlated with failures. Similarly, Liu et al. present SOBER [25], a hypothesis-testing approach that models each predicate’s distribution in passing vs. failing executions and ranks predicates whose failure-time patterns diverge from normal. These methods assume lightweight instrumentation and rely on correlating events with test outcomes to highlight likely faults. They handle multiple bugs by separating their effects and ranking fault-relevant predicates probabilistically. Spectrum-based fault localization (SBFL): A long line of work uses execution spectra to compute suspiciousness scores for program elements. These techniques rank each statement or entity by combining its execution frequency in failing vs. passing tests. A prominent example is Tarantula [16], which scores a statement based on the fraction of failing runs relative to all executions. Abreu et al. proposed additional metrics like Ochiai and Jaccard to improve localization [1]; empirical studies show Ochiai often outperforms Tarantula in accuracy. De Souza et al.’s survey [7] highlights that SBFL techniques pinpoint program elements more likely to contain faults using these coverage-based metrics. In recent years, researchers have developed many variants and combinations of SBFL metrics to boost precision. Delta debugging: Zeller and Hildebrandt [61] propose delta debugging techniques to automatically isolate minimal failure inducing differences in inputs or program changes. Given a failing and a passing execution, delta debugging applies a systematic search to remove irrelevant parts of the input or change set until only the smallest failure-inducing chunk remains. This algorithm repeatedly tests subsets of changes until it finds a minimal failure-inducing scenario. Delta debugging has been applied not only to inputs (e.g., HTML, command-line options) but also to code changes, configurations, and other causal factors. Failure sketching: A related research area focuses on summarizing failures through program execution analysis rather than solely relying on logs, originating from thin slicing [50], a program analysis technique that generates a minimal set of statements pertinent to a specific value by prioritizing value-flow relevance while excluding indirect dependencies. Expanding upon this, Kasikci et al. [19] propose failure sketches, which produce concise root-cause summaries by highlighting only the statements that actually cause a failure and the program-state differences between failing and successful runs. Compared to traditional diagnosis techniques, our approach is inherently more robust and maintainable, as it can semantically process heterogeneous log formats without the need for componentspecific parsing rules and distributed dynamic analysis tools, thus avoiding significant and continuous engineering overhead. Machine learning for fault localization (FL): More recent work applies statistical learning to improve localization. For instance, Sohn and Yoo [49] use supervised learning on features derived from past bug fixes and test data to re-rank suspicious code. Other work explored deep-learning and graph-based models for FL [23, 63].
These methods integrate information like code complexity, textual similarity, and historical bug patterns into neural models. Notably, graph neural networks (GNNs) have been used to capture structural code context: Lou et al. [27] and Qian et al. [40] show GNN-based FL models achieving state-of-the-art localization accuracy by encoding program dependency graphs. Such learning-based approaches can improve coverage-only SBFL rankings, but they often require large training datasets and are primarily evaluated on unit-level faults. LLM and agent-based debugging: A recent wave of work leverages LLMs for debugging tasks. Studies find that LLMs trained on code (e.g. GPT-4 [36], CodeLlama [42]) excel at understanding code and suggesting fixes [18, 22, 39, 52]. For example, Wu et al. [52] evaluate ChatGPT-3.5/4 [34, 35] on fault localization and report that GPT-4 achieves much higher accuracy than prior SBFL tools on Defects4J [17], especially in small contexts. Kang et al. [18] introduce AutoFL, an LLM-based FL method that takes a failing test as input and outputs the faulty method plus a natural-language rationale for the fault. LLM and agent-based repair: There is a rich body of work on using LLMs and AI agents for program repair, with recent advancements demonstrating better performance than earlier techniques. Lee et al. present FixAgent [21], which coordinates specialized LLM agents ("Tester" and "Debugger") via prompt chaining to perform end-to-end fault localization and repair. Similarly, LLM4FL [33] splits large coverage data into manageable chunks and uses two LLM agents to iteratively analyze code to rank suspicious methods. Yang et al. [57] proposed SWE-Agent, an AI agent to solve bugs in SWE-Bench [15] and HumanEvalFix [32] benchmarks. RepairAgent [3] uses a state machine to control its agent’s actions, limiting certain operations. AutoCodeRover [62] leverages explicit program information like class and method definitions, alongside test-based localization. Extending this, SpecRover [44] introduces a naturallanguage specification for desired behavior at each potential repair site and includes a patch review agent. CodeR [4] breaks down the task of program repair into various sub-tasks, managed and reviewed by a "manager" agent through a task graph. MarsCode Agent [26] integrates a dynamic, iterative repair process with a traditional generate-and-validate approach within a multi-agent framework. AlphaRepair [55] offers a zero-shot solution, enabling LLMs to rectify bugs without the need for supplementary training data. Building upon this, FitRepair [54] enhances AlphaRepair by integrating the plastic surgery hypothesis [2], which posits that the necessary bug-fixing code often resides within the existing project. Another approach, ChatRepair [56], facilitates error correction through an interactive dialogue with the engineer. In contrast to these methods, Agentless [53] employs a streamlined, agentless strategy, bypassing intricate tools and decisionmaking for a three-stage process: localization, repair, and patch validation. Agentless has achieved state-of-the-art results with notably reduced costs, surpassing the performance of current open-source software agents. Rondon et al. propose Passerine [41] to fix bugs at Google using LLM agents. Maddila et al. discuss Engineering Agent [28] to automatically fix failing unit-tests at Meta.
Celal Ziftci, Ray Liu, Spencer Greene, and Livio Dalloro
Benchmarks for debugging and repair: To systematically assess LLM debugging and repair, new benchmarks have emerged. SWE-Bench [15] is a benchmark sourced from GitHub issues across 12 popular open source projects containing Python bugs and fixes. HumanEvalFix [32] is a short-form code debugging benchmark. DebugEval [59] defines multiple tasks (bug localization, bug classification, code review, code repair) to evaluate LLM debugging ability across languages. DebugBench [29] is a large dataset of LeetCode coding problems with injected bugs to compare several open-source LLMs on automated bug fixing. SWE-smith [58] is a pipeline for generating software engineering tasks at scale, allowing the automatic synthesis of changes that break existing tests in a codebase. SWE-Lancer [31] is a benchmark that consists of over 1,400 freelance software engineering tasks. Almost all of these benchmarks focus on unit-test related debugging and repair tasks. Previous studies discussed above generally show that large models (70B+ parameters) have convincing localization, debugging and repair capabilities. Overall, LLM-based techniques currently outperform many traditional methods for unit-level bug localization and repair, but they face new challenges (e.g. limited token context, integration of dynamic traces) when scaling to larger systems [10, 13, 52]. Most of the above work, both benchmarks and research, concentrate on small-scale or unit-test debugging and repair scenarios. Integration tests, where faults emerge from interactions among components, pose additional challenges, e.g. complex environment setup, non-deterministic dependencies, significant log output, and more complex failure modes that remain under-explored. Our approach builds on these foundations but focuses on leveraging LLMs for the difficult problem of diagnosing integration test failures, aiming to address scenarios that traditional and even recent LLM-based techniques have not fully tackled.
7
Conclusion
Integration testing is a fundamental phase in ensuring the quality and reliability of complex, distributed software systems, yet diagnosing their failures presents significant challenges for developers. Developers consistently report spending substantially more time on diagnosing integration test failures compared to unit test failures. To overcome these, this paper introduced Auto-Diagnose, a novel failure diagnosis tool powered by LLMs. The system determines the root cause of integration test failures, produces concise summaries and highlights the specific log lines most relevant to the diagnosis. It is integrated directly into developer code review workflows, providing contextual and in-time assistance to streamline the debugging process. Auto-Diagnose’s effectiveness is evaluated through quantitative and qualitative methods. A manual evaluation on 71 real-world integration test failures demonstrated that the tool achieved a 90.14% accuracy in detecting and summarizing the root cause diagnosis. This evaluation also proved beneficial in uncovering underlying infrastructure bugs related to log saving. Following its launch, Auto-Diagnose processed a large number of test failures. User feedback indicated a "Not helpful" rate of 5.8%,
which is well within the acceptable guideline for tools posting findings in Critique. Although the reported helpfulness-rate was 62.96%, this disparity compared to the manual evaluation’s 90.14% accuracy is attributed to the well-established negativity bias, where users are more inclined to report negative experiences. Auto-Diagnose’s high ranking of #14 (top 3.78%) among 370 tools in Critique further suggests that its diagnoses resonate with developers who have historically been frustrated with integration test failure diagnosis. User interviews confirmed Auto-Diagnose’s perceived usefulness and the positive reception of its integration into existing workflows. Developers appreciated the automatic, in-context diagnostic assistance, with some even beginning to expect more advanced capabilities, such as automated fixes, from LLM-based tools. In conclusion, LLMs are highly successful in diagnosing integration test failures, primarily due to their capacity to process and summarize complex textual data. The seamless integration of such AI-powered tooling into developers’ daily workflows is perceived positively, although the accuracy of the tool remains a critical factor in shaping developer perception and adoption.
8
Future Work
We observed two major improvement opportunities during our case studies of Auto-Diagnose. First, in failing integration test logs, several log lines are output at the ERROR level even though the reported issue is later recovered during the test run. Such logs are not only confusing for developers, but also for LLMs. We plan to filter such logs by using the logs from a previous passing execution of the test, and dropping a log if a similar log exists in a passing run. We predict that this will improve the accuracy of the diagnoses the LLM produces. Second, we plan to extend Auto-Diagnose to provide not only diagnoses but also fixes when possible, aligned with developer expectations.
Acknowledgments We thank Elvira Djuraeva for her help on the user surveys to understand the unit and integration test usage across Google, James McClure and Eric Beerman on their help with the manual evaluations, and Jessica Tan, Maria Arguello and Elaine Thai for their help on conducting user interviews. Google Gemini [9] was utilized to generate sections of this work, specifically several tables in Latex [20] format, and several graphs inside Google Colab [8].
References [1] Rui Abreu, Peter Zoeteweij, Rob Golsteijn, and Arjan JC Van Gemund. 2009. A practical evaluation of spectrum-based fault localization. Journal of Systems and Software 82, 11 (2009), 1780–1792. [2] Earl T Barr, Yuriy Brun, Premkumar Devanbu, Mark Harman, and Federica Sarro. 2014. The plastic surgery hypothesis. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. 306–317. [3] Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2024. Repairagent: An autonomous, llm-based agent for program repair. arXiv preprint arXiv:2403.17134 (2024). [4] Dong Chen, Shaoxin Lin, Muhan Zeng, Daoguang Zan, Jian-Gang Wang, Anton Cheshkov, Jun Sun, Hao Yu, Guoliang Dong, Artem Aliev, et al. 2024. Coder: Issue resolving with multi-agent and task graphs. arXiv preprint arXiv:2406.01304 (2024).
LLM-Based Automated Diagnosis Of Integration Test Failures At Google
[5] Lan Cheng, Emerson Murphy-Hill, Mark Canning, Ciera Jaspan, Collin Green, Andrea Knight, Nan Zhang, and Elizabeth Kammer. 2022. What improves developer productivity at google? code quality. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1302–1313. [6] Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261 (2025). [7] Higor A de Souza, Marcos L Chaim, and Fabio Kon. 2016. Spectrum-based software fault localization: A survey of techniques, advances, and challenges. arXiv preprint arXiv:1607.04347 (2016). [8] Google Inc. 2025. Google Colab. https://colab.research.google.com. Accessed: 2025-09-24. [9] Google Inc. 2025. Google Gemini: Google’s AI Assistant. https://gemini.google. com. Accessed: 2025-09-24. [10] Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muhammad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, et al. 2023. Large language models: a comprehensive survey of its applications, challenges, limitations, and future prospects. Authorea preprints 1, 3 (2023), 1–26. [11] Shilin He, Pinjia He, Zhuangbin Chen, Tianyi Yang, Yuxin Su, and Michael R Lyu. 2021. A survey on automated log analysis for reliability engineering. ACM computing surveys (CSUR) 54, 6 (2021), 1–37. [12] Shilin He, Jieming Zhu, Pinjia He, Zibin Li, and Rui Liu. 2017. Log-based anomaly detection: A survey. In 2017 IEEE 24th International Conference on Software Analysis, Evolution and Reconfiguration (SANER). IEEE, 520–529. [13] Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology 33, 8 (2024), 1–79. [14] Jez Humble and David Farley. 2010. Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation. Addison-Wesley Professional. [15] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770 (2023). [16] James A Jones, Mary Jean Harrold, and John Stasko. 2002. Visualization of test information to assist fault localization. In Proceedings of the 24th international conference on Software engineering. 467–477. [17] René Just, Darioush Jalali, and Michael D. Ernst. 2014. Defects4J: a database of existing faults to enable controlled testing studies for Java programs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis (ISSTA). ACM, 437–440. doi:10.1145/2610384.2628055 Accessed: 2025-09-22. [18] Sungmin Kang, Gabin An, and Shin Yoo. 2024. A quantitative and qualitative evaluation of LLM-based explainable fault localization. Proceedings of the ACM on Software Engineering 1, FSE (2024), 1424–1446. [19] Baris Kasikci, Cristiano Pereira, Gilles Pokam, Benjamin Schubert, Malandal Musuvathi, and George Candea. 2015. Failure sketches: A better way to debug. In 15th Workshop on Hot Topics in Operating Systems (HotOS XV). [20] Leslie Lamport. 1994. LaTeX: A Document Preparation System (2nd ed.). AddisonWesley. [21] Cheryl Lee, Chunqiu Steven Xia, Longji Yang, Jen-tse Huang, Zhouruixin Zhu, Lingming Zhang, and Michael R Lyu. 2024. A unified debugging approach via llm-based multi-agent synergy. arXiv preprint arXiv:2404.17153 (2024). [22] Jierui Li, Szymon Tworkowski, Yingying Wu, and Raymond Mooney. 2023. Explaining competitive-level programming solutions using llms. arXiv preprint arXiv:2307.05337 (2023). [23] Yi Li, Shaohua Wang, and Tien Nguyen. 2021. Fault localization with code coverage representation learning. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 661–673. [24] Ben Liblit, Mayur Naik, Alice X Zheng, Alex Aiken, and Michael I Jordan. 2005. Scalable statistical bug isolation. Acm Sigplan Notices 40, 6 (2005), 15–26. [25] Chao Liu, Long Fei, Xifeng Yan, Jiawei Han, and Samuel P Midkiff. 2006. Statistical debugging: A hypothesis testing-based approach. IEEE Transactions on software engineering 32, 10 (2006), 831–848. [26] Yizhou Liu, Pengfei Gao, Xinchen Wang, Jie Liu, Yexuan Shi, Zhao Zhang, and Chao Peng. 2024. Marscode agent: Ai-native automated bug fixing. arXiv preprint arXiv:2409.00899 (2024). [27] Yiling Lou, Qihao Zhu, Jinhao Dong, Xia Li, Zeyu Sun, Dan Hao, Lu Zhang, and Lingming Zhang. 2021. Boosting coverage-based fault localization via graphbased representation learning. In Proceedings of the 29th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering. 664–676. [28] Chandra Maddila, Adam Tait, Claire Chang, Daniel Cheng, Nauman Ahmad, Vijayaraghavan Murali, Marshall Roch, Arnaud Avondet, Aaron Meltzer, Victor Montalvao, et al. 2025. Agentic Program Repair from Test Failures at Scale: A Neuro-symbolic approach with static analysis and test execution feedback. arXiv
preprint arXiv:2507.18755 (2025). [29] Yacine Majdoub and Eya Ben Charrada. 2024. Debugging with open-source large language models: An evaluation. In Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. 510–516. [30] Atif Memon, Zebao Gao, Bao Nguyen, Sanjeev Dhanda, Eric Nickell, Rob Siemborski, and John Micco. 2017. Taming Google-Scale Continuous Testing. In Proceedings of the 39th International Conference on Software Engineering: Software Engineering in Practice Track. IEEE Press, 233–242. [31] Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke. 2025. SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? arXiv preprint arXiv:2502.12115 (2025). [32] Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. 2023. Octopack: Instruction tuning code large language models. In NeurIPS 2023 workshop on instruction tuning and instruction following. [33] Md Nakhla Rafi, Dong Jae Kim, Tse-Hsun Chen, and Shaowei Wang. 2024. Enhancing Fault Localization Through Ordered Code Analysis with LLM Agents and Self-Reflection. arXiv e-prints (2024), arXiv–2409. [34] OpenAI. 2022. ChatGPT-3.5. Large language model. https://chat.openai.com/ Accessed: 2025-09-22. [35] OpenAI. 2023. ChatGPT-4. Large language model. https://chat.openai.com/ Accessed: 2025-09-22. [36] OpenAI. 2023. GPT-4 Technical Report. Technical Report. OpenAI. https: //cdn.openai.com/papers/gpt-4.pdf Accessed: 2025-09-22. [37] Rachel Potvin and Josh Levenberg. 2016. Why Google stores billions of lines of code in a single repository. Commun. ACM 59, 7 (2016), 78–87. [38] Roger S. Pressman and Bruce R. Maxim. 2014. Software Engineering: A Practitioner’s Approach. McGraw-Hill Education. [39] Xiao Pu, Mingqi Gao, and Xiaojun Wan. 2023. Summarization is (almost) dead. arXiv preprint arXiv:2309.09558 (2023). [40] Jie Qian, Xiaolin Ju, and Xiang Chen. 2023. GNet4FL: effective fault localization via graph convolutional neural network. Automated Software Engineering 30, 2 (2023), 16. [41] Pat Rondon, Renyao Wei, José Cambronero, Jürgen Cito, Aaron Sun, Siddhant Sanyam, Michele Tufano, and Satish Chandra. 2025. Evaluating agent-based program repair at google. arXiv preprint arXiv:2501.07531 (2025). [42] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2023. Code Llama: Open Foundation Models for Code. arXiv preprint arXiv:2308.12950 (2023). https://doi.org/10.48550/arXiv.2308.12950 Accessed: 2025-09-22. [43] Paul Rozin and Edward B Royzman. 2001. Negativity bias, negativity dominance, and contagion. Personality and social psychology review 5, 4 (2001), 296–320. [44] Haifeng Ruan, Yuntong Zhang, and Abhik Roychoudhury. 2024. Specrover: Code intent extraction via llms. arXiv preprint arXiv:2408.02232 (2024). [45] Caitlin Sadowski, Emma Söderberg, Luke Church, Michal Sipko, and Alberto Bacchelli. 2018. Modern code review: a case study at Google. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice. ACM, 181–190. [46] Caitlin Sadowski, Emma Söderberg, Luke Church, Michal Sipko, and Alberto Bacchelli. 2018. Modern code review: a case study at Google. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice. ACM, 181–190. [47] Caitlin Sadowski, Jeffrey Van Gogh, Ciera Jaspan, Emma Soderberg, and Collin Winter. 2015. Tricorder: Building a program analysis ecosystem. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Vol. 1. IEEE, 598–608. [48] Kensen Shi, Deniz Altınbüken, Saswat Anand, Mihai Christodorescu, Katja Grünwedel, Alexa Koenings, Sai Naidu, Anurag Pathak, Marc Rasi, Fredde Ribeiro, et al. 2025. Natural language outlines for code: Literate programming in the llm era. In Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering. 150–161. [49] Jeongju Sohn and Shin Yoo. 2017. Fluccs: Using code and change metrics to improve fault localization. In Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis. 273–283. [50] Manu Sridharan, Stephen J Fink, and Rastislav Bodik. 2007. Thin slicing. In Proceedings of the 28th ACM SIGPLAN conference on programming language design and implementation. 112–122. [51] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023). [52] Yonghao Wu, Zheng Li, Jie M Zhang, Mike Papadakis, Mark Harman, and Yong Liu. 2023. Large language models in fault localisation. arXiv preprint
Celal Ziftci, Ray Liu, Spencer Greene, and Livio Dalloro
arXiv:2308.15276 (2023). [53] Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2024. Agentless: Demystifying llm-based software engineering agents. arXiv preprint arXiv:2407.01489 (2024). [54] Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023. Automated program repair in the era of large pre-trained language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 1482–1494. [55] Chunqiu Steven Xia and Lingming Zhang. 2022. Less training, more repairing please: revisiting automated program repair via zero-shot learning. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 959–971. [56] Chunqiu Steven Xia and Lingming Zhang. 2024. Automated program repair via conversation: Fixing 162 out of 337 bugs for $0.42 each using chatgpt. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 819–831. [57] John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems 37 (2024), 50528–50652. [58] John Yang, Kilian Lieret, Carlos E Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. 2025. Swe-smith: Scaling data for software engineering agents. arXiv preprint
arXiv:2504.21798 (2025). [59] Weiqing Yang, Hanbin Wang, Zhenghao Liu, Xinze Li, Yukun Yan, Shuo Wang, Yu Gu, Minghe Yu, Zhiyuan Liu, and Ge Yu. 2024. Enhancing the code debugging ability of llms via communicative agent based data refinement. language 30 (2024), 31. [60] Dixin Yuan, Sunghun Park, and Yuanyuan Zhou. 2012. Characterizing logging practices in open-source software. In Proceedings of the 34th International Conference on Software Engineering (ICSE). IEEE Press, 1–11. [61] Andreas Zeller and Ralf Hildebrandt. 2002. Simplifying and isolating failureinducing input. IEEE Transactions on software engineering 28, 2 (2002), 183–200. [62] Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Autocoderover: Autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1592– 1604. [63] Zhuo Zhang, Yan Lei, Xiaoguang Mao, and Panpan Li. 2019. CNN-FL: An effective approach for localizing faults using convolutional neural networks. In 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 445–455. [64] Yintong Zhao, Shilin He, Pinjia He, Zhekang Chen, Hongyu Zhang, and Renzhi Duan. 2023. Log-based Anomaly Detection and Diagnosis for Software Systems: A Survey. Comput. Surveys 56, 4 (2023), 1–37.