arXiv CSopen access

PracRepair: LLM-Empowered Automated Program Repair Inspired by Human-Like Debugging Practices

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 14, NO. 8, AUGUST 2021

1

P RAC R EPAIR: LLM-Empowered Automated Program Repair Inspired by Human-Like Debugging Practices

arXiv:2606.17612v1 [cs.SE] 16 Jun 2026

Yu Cheng, Zhongxin Liu, Zhenchang Xing, Chao Ni, Qing Huang, Xiaoxue Ren

Abstract—As software systems grow in scale and complexity, debugging and repair remain costly and time-consuming. Large language models (LLMs) have advanced automated program repair (APR), but existing LLM-based APR approaches still largely rely on static or retrieved context, error messages, and coarse-grained validation outcomes. As a result, they underutilize dynamic information for failure understanding and repair, including failure-execution dynamics and patch-validation dynamics. Effectively leveraging such information, however, is challenging: failure-execution traces are large and noisy, raw staticdynamic context is not self-explanatory, and patch-validation dynamics are often reduced to coarse feedback. To address these challenges, we propose P RAC R EPAIR, a fully automated LLM-based APR framework inspired by human-like debugging practices. P RAC R EPAIR constructs an on-demand static-dynamic context from buggy programs and failure executions, performs question-driven failure diagnosis to formulate explicit repair hypotheses, and iteratively refines candidate patches using validation diagnostics and trace-level behavioral changes. Experimental results on Defects4J V1.2 and V2.0 show that P RAC R EPAIR consistently outperforms state-of-the-art baselines. Specifically, under GPT-3.5, P RAC R EPAIR correctly fixes 139/136 bugs on Defects4J V1.2/V2.0, while under GPT-4o it further improves to 162/171. Moreover, P RAC R EPAIR generalizes effectively to RWB (Real-World Bugs), achieving the best performance across multiple foundation models. Index Terms—Automated program repair, large language model.

I. I NTRODUCTION As modern software systems continue to grow in scale and complexity, defects have become increasingly common in real-world development [1], [2]. Fixing these defects is often challenging in practice, because in real-world software systems, the causes and effects of a defect often extend beyond a single function and require reasoning over nonlocal contextual information, such as call relationships, data dependencies, and execution logic [3]. Developers typically debug in IDE-like environments [4], [5], where they leverage richer information and follow a structured workflow to understand failures, formulate repair hypotheses, and iteratively refine fixes [6]–[11]. More specifically, developers first gather both static and dynamic evidence Y. Cheng, Z. Liu, C. Ni, and X. Ren are with Zhejiang University, China. E-mail: {yucheng1127, liu zx, chaoni, xxren}@zju.edu.cn. Z. Xing is with CSIRO’s Data61, Australia. E-mail: [email protected]. Q. Huang is with Jiangxi Normal University, China. E-mail: [email protected]. X. Ren is the corresponding author.

by inspecting the buggy method and failing tests, navigating to relevant implementations, and tracing execution through interactive operations such as step into and step over. Through this process, they recover implicit execution knowledge, including call relationships, and observe fine-grained runtime behaviors such as executed paths, variable states, branch outcomes, and intermediate values [6], [9]–[12]. Based on such evidence, developers then diagnose failures in a questiondriven manner [7], [8], [13], asking targeted questions such as what happened here? or why is x null at this point?, and progressively narrowing down the root cause while identifying what additional evidence is needed to better understand the buggy behavior [12]. After completing the failure diagnosis, developers often return to the debugging environment to reexecute the patched program and compare its behavior with the original failing execution. If the patch does not fully resolve the bug, they further analyze the remaining failure and refine the repair accordingly. As a result, failure understanding and patch construction co-evolve through continuous feedback and refinement [10]–[12]. Although such a debugging workflow is effective in practice, it is also expensive and time-consuming. Software developers spend roughly 35% to 50% of their time, and 50% to 75% of project budgets, on testing, verification, and debugging, costing over 100 billion dollars each year [14]–[16]. This high cost has motivated extensive research on automated program repair (APR), which aims to automatically generate patches for buggy programs [17]–[28]. Early APR approaches mainly relied on manually designed fix patterns or bugfixing datasets [18]–[23], but their effectiveness was often constrained by limited pattern coverage, strong data dependence, and weak generalization ability [18], [29]. Recently, large language models (LLMs) have demonstrated stronger code understanding and generation capabilities for APR [24], [30], [31]. Building on this progress, recent LLM-based APR approaches, such as ChatRepair [25], ThinkRepair [27], RepairAgent [26], and ReInFix [28], further incorporate richer repair context and iterative interaction, achieving stronger repair performance on benchmarks such as Defects4J [3]. However, a key limitation is that prior approaches underutilize dynamic information for failure understanding and repair, while overestimating LLMs’ ability to precisely infer complex program behavior from static context alone. Although recent methods such as ChatRepair [25], ThinkRepair [27], RepairAgent [26], and ReInFix [28] incorporate test feedback, richer repair context, or iterative interaction, their

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 14, NO. 8, AUGUST 2021

repair processes are still largely driven by static or retrieved context, error messages, and coarse-grained validation outcomes. In particular, they do not systematically exploit two types of dynamic information that are critical in practical debugging: failure-execution dynamics, which reveal how the original failure is triggered through executed paths, runtime states, and branch outcomes, and patch-validation dynamics, which reveal how a candidate patch changes program behavior during validation. Without these dynamic signals, LLMs may miss root causes or generate incomplete and overfitted fixes, especially for bugs whose root causes depend on runtime states and value evolution. Effectively leveraging such dynamic information introduces three challenges. C1: Failure-execution dynamics are large and noisy. Directly exposing complete traces to the LLM may overwhelm the repair context rather than help identify failurerelevant behavior. C2: Raw static-dynamic context is not self-explanatory. Even when execution traces are available, the LLM still needs to determine which runtime states matter and how they relate to the faulty logic; otherwise, it may make incorrect behavioral inferences. C3: Patch-validation dynamics are often underused. They are frequently reduced to coarse validation outcomes, such as pass/fail results or error messages, leaving subsequent repair iterations without finegrained evidence about what behavior has changed and why the current patch still fails. To address the above challenges, we design and implement P RAC R EPAIR, an LLM-empowered APR framework inspired by human-like debugging practices. Specifically, P RAC R EPAIR consists of three stages. (1) Static-dynamic context construction addresses C1 by combining static program context with selectively organized execution traces collected from triggering test runs. Instead of directly exposing complete traces to the LLM, P RAC R EPAIR indexes and structures dynamic evidence through a unified interface, allowing the LLM to access relevant code context, call relationships, executed paths, and runtime states on demand. (2) Question-driven failure diagnosis addresses C2 by guiding the LLM to ask and answer targeted diagnostic questions about what happens during execution, why the failure occurs, and how the faulty logic should be corrected. By retrieving the evidence needed to answer these questions, P RAC R EPAIR progressively narrows down the root cause and formulates an explicit repair hypothesis. (3) Feedback-guided patch refinement addresses C3 by extracting validation diagnostics, code diffs, and trace diffs from failed candidate patches, and feeding these patchvalidation dynamics back into diagnosis for iterative refinement. This enables more evidence-grounded repair and helps avoid incomplete or overfitted fixes. Experimental results on real-world benchmarks demonstrate the effectiveness of P RAC R EPAIR’s human-like debugginginspired design. On Defects4J V1.2 and V2.0, P RAC R EPAIR consistently outperforms state-of-the-art APR baselines [18]– [20], [22], [24]–[28]. Specifically, under GPT-3.5, P RAC R E PAIR correctly fixes 139 bugs on Defects4J V1.2 and 136 bugs on Defects4J V2.0; under GPT-4o, it further improves to 162 and 171 bugs, respectively. It also fixes many previously unsolved bugs, including 75 unique correct fixes achieved

2

by P RAC R EPAIR with GPT-3.5 and 93 unique correct fixes under GPT-4o when compared with ReInFix. Across repair scenarios, P RAC R EPAIR performs strongly from single-line to multi-function bugs, with particularly notable advantages on more challenging cases. Ablation studies further verify the effectiveness of all three stages, showing that these gains come from enriching repair with failure-aware information, questiondriven diagnosis, and feedback-guided refinement inspired by human-like debugging practices. Beyond Defects4J, P RAC R E PAIR also generalizes well to RWB V1.0/V2.0 [27], achieving the best performance across multiple foundation models. Overall, this work makes the following main contributions: • We identify and formulate a critical gap between practical debugging workflows and existing LLM-based APR techniques. Specifically, we show that current approaches have not fully exploited three debugging practices that are widely used by developers: static-dynamic evidence gathering, question-driven failure diagnosis, and feedbackguided patch refinement. • We introduce P RAC R EPAIR , a fully automated LLMbased APR framework that operationalizes these debugging practices. P RAC R EPAIR constructs an on-demand static-dynamic context from buggy programs and failure executions, performs question-driven failure diagnosis to formulate explicit repair hypotheses, and iteratively refines candidate patches using validation diagnostics and trace-level behavioral changes. • We conduct extensive experiments on Defects4J and RWB. Results show that P RAC R EPAIR consistently outperforms SOTA APR baselines, remains effective across different repair scenarios from single-line to multifunction bugs, and generalizes well across multiple foundation models. Ablation studies further confirm the effectiveness of all three modules. The code and experimental results are available at [Link] II. M OTIVATION We present a real-world example in Figure 1, based on a simplified code snippet excerpted from the Java project commons-compress, to illustrate why APR should move beyond direct patch generation and instead follow a debuggingoriented repair process. This example mirrors how developers debug in practice: they inspect static code context, observe concrete runtime states, ask targeted diagnostic questions, and refine incomplete fixes based on changed program behavior. Accordingly, it motivates the three key designs of P RAC R EPAIR: static-dynamic context construction, questiondriven failure diagnosis, and feedback-guided patch refinement. The bug is located in the writeBits method, which packs bits into a temporary buffer cache and flushes them according to shift. The defect is an off-by-one error: shift is initialized to 7 and decremented after each bit is written, but the buggy code flushes the buffer when shift == 0, causing only seven bits to be written and triggering java.io.IOException: Unknown property 128. C1: Failure-execution dynamics are large and noisy. Existing LLM-based APR methods already use failure-related signals, but these signals are often coarse-grained. For example, ChatRepair [25] and ThinkRepair [27] use buggy code,

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 14, NO. 8, AUGUST 2021

Bug Information Test Suite: public void testSevenEmptyFiles() throws Exception { testCompress252(7, 0); } Failure Info: java.io.IOException: Unknown property 128 Buggy Code: 01 private void writeBits(final DataOutput header, final BitSet bits, final int length) throws IOException { 02 int cache = 0; 03 int shift = 7; 04 for (int i = 0; i < length; i++) { 05 cache |= ((bits.get(i) ? 1 : 0) << shift); 06 --shift; 07 if (shift == 0) { 08 header.write(cache); 09 shift = 7; 10 cache = 0; 11 } 12 } 13 if (length > 0 && shift > 0) { 14 header.write(cache); 15 } 16 }

Static Program Context private void testCompress252(int numberOfFiles, int numberOfNonEmptyFiles) throws Exception { import java.io.DataOutput; ...... import java.io.DataOutputStream; private void writeFileEmptyFiles(final DataOutput header) verifyCompress252(output, numberOfFiles, import java.io.File; throws IOException { numberOfNonEmptyFiles);} ...... import org.apache.commons.compress.archivers. writeBits(out, emptyFiles, emptyStreamCounter); out.flush(); ......}}

A. Repairing with Static Context ## Task: generate a corrected patch... ## Input: Bug_Info, Program_Context Writing cache directly may produce invalid data, so use cache & 0xFF to keep only the lower 8 bits. ...... if (shift == 0) { header.write(cache & 0xFF); ...... Unknown property 128

Dynamic Execution Trace <testSevenEmptyFiles, writeBites> ...... Line6: {'i': 6, ......, 'cache': 254, 'shift': 0} Line7: {'type': 'branch', 'executed': True} Line6: {'i': 7, ......, 'cache': 128, 'shift': 6} ...... Line13: {'length': 8, 'shift': 6, 'cache': 128} Line13: {'type': 'branch', 'executed': True}

B. Repairing with Static + Dynamic Context ## Task: generate a corrected patch... ## Input: Bug_Info, Program_Context , Execution_Trace Change `shift > 0` to `shift < 8` to ensure any remaining cached bits are flushed after writing. ...... if (length > 0 && shift < 8 ) { header.write(cache); ...... Unknown property 128

3

Question-driven Diagnosis What are cache and shift when if (shift == 0) is entered? When branch is entered,cache = 254 and shift = 0 after i = 6 How should the in-loop flush condition be changed? Change if (shift == 0) to if (shift < 0).

Validation Feedback Failure Info: java.io.IOException: Badly terminated header Code Diff: - if ( shift == 0 ){ + if ( shift < 0 ){ Trace Diff: - Line7: {'type': 'branch', 'executed': True} + Line6: {'i': 7, ......, 'cache': 258, 'shift': -1} + Line7: {'type': 'branch', 'executed': False} - Line13: {'shift': 6, 'cache': 128} + Line13: {'shift': 7, 'cache': 0}

Re-diagnosis Why does the initial patch cause a header-termination failure? After the initial patch, Line 14 writes 0 instead of 128, causing the header to terminate incorrectly.

C. Feedback-guided Patch Refinement ## Task: generate a corrected patch.. ## Input: Bug_Info Root Cause: Off-by-one flush logic Modification Suggestion: Flush only when shift < 0 Apply the modification to the in-loop flush condition. ...... if(shift < 0) { header.write(cache); ...... if (length > 0 && shift > 0) {

Badly terminated header ## Task: generate a corrected patch.. ## Input: Bug_Info, Validation_FeedBack Root Cause: The final flush condition is inconsistent with the updated inloop flush logic. Modification Suggestion: Flush remaining bits only when shift < 7. The refined patch correctly flushes the remaining bits after the loop. ...... if (shift < 0) { header.write(cache); ...... if (length > 0 && shift < 7 ) ...... All Tests Pass

Fig. 1: A motivating example based on Compress-21, illustrating the need for dynamic execution trace, question-driven diagnosis, and feedback-guided refinement in APR.

failing tests, and validation feedback, but do not expose finegrained execution traces such as executed paths, variable states, and branch outcomes. As shown in the A. Repairing with Static Context panel, given only bug information and static context, the model changes header.write(cache) to header.write(cache & 0xFF). This patch appears to address the symptom suggested by Unknown property 128, but still fails with the same error. In contrast, the Dynamic Execution Trace panel reveals how cache and shift evolve across loop iterations and exposes the runtime state where the failure is triggered. However, complete execution traces in real programs may contain many irrelevant calls, branches, and state changes, and directly exposing them to the LLM may overwhelm the repair context. This motivates staticdynamic context construction in Stage I of P RAC R EPAIR. C2: Raw static-dynamic context is not self-explanatory. Recent agentic APR methods, such as RepairAgent [26] and ReInFix [28], allow the model to interact with external tools, retrieve additional context, or refine patches iteratively. However, richer context alone does not guarantee that the model will identify the failure-relevant behavior. To illustrate this issue, the B. Repairing with Static + Dynamic Context panel shows what may happen when the model is provided with additional dynamic evidence without explicit diagnostic guidance. This suggests that raw context is useful but not selfexplanatory: the model still needs to determine which runtime states matter and how they explain the faulty logic. In the Question-driven Diagnosis panel, targeted questions such as What are cache and shift when if (shift == 0) is entered? and How should the in-loop flush condition be changed? guide the model to focus on the premature flush and formulate a more precise repair hypothesis. This motivates question-driven failure diagnosis in Stage II of P RAC R EPAIR. C3: Patch-validation dynamics are often underused. Iterative APR methods commonly use validation results to refine patches [25]–[27], but validation feedback is often reduced to coarse outcomes such as pass/fail results or error messages.

As shown in the Validation Feedback panel, the initial patch changes if (shift == 0) to if (shift < 0), which removes the original failure Unknown property 128 but introduces a new failure, Badly terminated header. If validation is treated only as a pass/fail signal, the model receives limited guidance for the next repair attempt. Instead, P RAC R EPAIR extracts structured validation feedback, including the validation diagnostic, code diff, and trace diff between the original and patched executions. The trace diff reveals that after the initial patch, the post-loop write becomes inconsistent with the updated in-loop flush behavior, localizing the remaining issue to the final flush condition. This leads to the refined patch if (length > 0 && shift < 7), which passes all tests. This motivates feedback-guided patch refinement in Stage III of P RAC R EPAIR. III. A PPROACH Figure 2 shows the overall workflow of P RAC R EPAIR, which aims to improve automated program repair by drawing inspiration from human-like debugging practices. To achieve this goal, P RAC R EPAIR first extracts static program context from the project and collects dynamic execution trace from triggering test runs to build a context basis for repair, while providing a uniform interface for the LLM to access the needed information on demand; instead of directly using this evidence for patch generation, it then guides the LLM to diagnose faulty program behaviors by incrementally raising and answering diagnostic questions, thereby formulating an explicit repair hypothesis; finally, it generates and validates candidate patches, analyzes the code-level and behavioral differences introduced by each patch, and feeds these diagnostic signals back into failure diagnosis to iteratively refine the repair. Specifically, P RAC R EPAIR contains three main stages: Staticdynamic Context Construction (Stage I), Question-driven Failure Diagnosis (Stage II), and Feedback-guided Patch Refinement (Stage III). During repair, P RAC R EPAIR maintains three intermediate artifacts: the diagnostic QA history, the

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 14, NO. 8, AUGUST 2021

4

Stage III: Feedback-guied Patch Refinement

Stage II: Question-driven Failure Diagnosis

Stage I: Static-dynamic Context Construction

Feedback Extracting and Re-diagnosing

CPG Constructing Code Property Graph (CPG) Diagnostic Answer

（

Project

）

QA Pairs Buggy Code Test Suite Failure Info Feedback (Optional) Optional

Inputs to Diagnosis

Unified Context Access Interface Diagnosis Loop for MAX Rounds Bytecode Instrumenting ByteCode

Stopping Criteria met

Tests Running Execution Trace Table

Trace Collecting

Answering

Diagnostic Question

Asking

Formulating

Repair Hypothesis

All Tests Pass?

Y

Refinement Loop for MAX Rounds N Patch Generating Patch Candidate Validating Validation Plausible Patch Result Patch

Fig. 2: The Overall Framework of P RAC R EPAIR.

repair hypothesis, and the validation feedback. The diagnosis loop updates the QA history by asking and answering one diagnostic question at a time, and terminates when no further question is needed or the diagnosis budget is exhausted. The refinement loop updates the repair hypothesis using feedback from failed candidate patches, and terminates when a plausible patch is found or the refinement budget is exhausted. In our implementation, the diagnosis and refinement budgets are set to 10 and 3, respectively, and both serve as upper bounds rather than mandatory numbers of rounds. A. Static-dynamic Context Construction As shown in Figure 2, the goal of Static-dynamic Context Construction is to build a failure-relevant context basis for subsequent diagnosis and repair. To simulate how developers debug in practice, P RAC R EPAIR must support the LLM in understanding both where failure-relevant logic resides in the program and how the faulty behavior is actually triggered during execution. This requires two complementary sources of evidence. Static information is needed to expose the structural and semantic context of the bug, such as surrounding implementations, control structures, call relationships, and valueflow dependencies. Dynamic information is needed to reveal the concrete failure behavior at runtime, including executed paths, branch outcomes, variable states, and failure-triggering execution conditions. To unify these two complementary sources for diagnosis and repair, P RAC R EPAIR provides a uniform interface to access the required information. 1) Static context construction via CPG construction.: To support diagnosis of failure-relevant program structure and semantics, P RAC R EPAIR first performs static program analysis on the input project and constructs a Code Property Graph (CPG). We use Joern [32] to parse the project source code and build the CPG [33], which unifies the abstract syntax tree (AST), control-flow graph (CFG), and data-dependence relations into a single representation. Based on this representation, P RAC R EPAIR can access not only syntactic entities such as classes, methods, and statements, but also semantic relations such as control branches, call edges, and variable definition– use chains. This static evidence is important for understanding the structural context of the buggy code, locating related program entities, tracing inter-procedural dependencies, and reasoning about how values and control decisions propagate to failure-relevant locations. 2) Dynamic context construction via trace collection.: To support the diagnosis of faulty behavior, P RAC R EPAIR further

collects runtime execution evidence from triggering test executions. Since the goal is to observe actual failing behavior without modifying source code semantics, we adopt non-intrusive bytecode instrumentation [34]. Specifically, P RAC R EPAIR instruments Java bytecode using JavaAgent [35] and ASM [36], and then executes the triggering tests to record runtime states. Considering that dynamic execution information can be extremely large in real-world programs, P RAC R EPAIR focuses trace collection on the buggy function under triggering test executions, so as to capture failure-relevant runtime behavior while controlling trace noise and token overhead. We apply statement-level instrumentation to capture execution evidence with sufficient granularity for diagnosis while controlling trace noise. P RAC R EPAIR records the executed statement sequence within the buggy function, the values of in-scope variables after each executed statement, and the outcomes of conditional branches. For object-type variables, fields are recursively serialized up to a depth of 3 to balance contextual richness and token efficiency. The collected runtime evidence is organized into an Execution Trace Table, where each table corresponds to a specific <triggering test, buggy function> pair and records the executed statements, their associated runtime states, and branch outcomes. 3) Unified context access interface.: The static and dynamic context constructed above is not provided to the LLM all at once. Instead, P RAC R EPAIR exposes it through a uniform interface that supports on-demand retrieval during diagnosis. This design avoids overwhelming the LLM with the full project context and long execution traces, while allowing context retrieval to be guided by the current diagnostic need, similar to how developers inspect code and execution behavior during debugging. For static evidence, the interface supports three forms of access: (1) dependency and entity localization, which helps identify relevant program entities and resolve referenced types; (2) structured definitions and code inspection, which helps inspect classes, methods, and implementations to understand surrounding logic and identify candidate modification points; and (3) structural and semantic relation inspection, which helps reason about control constructs, caller relationships, and variable definition–use chains. For dynamic evidence, the interface supports three common debugging needs: (1) execution-path inspection, which helps understand what actually happens during failing execution; (2) runtime-value inspection, which helps track variable evolution and identify abnormal state changes; and (3) statement-level state inspec-

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 14, NO. 8, AUGUST 2021

5

TABLE I: Unified interface for accessing static and dynamic context in P RAC R EPAIR Access Capability

Function Calls

Diagnostic Use

Dependency & entity localization Structured definitions & code inspection Structural & semantic relation inspection Execution-path inspection Runtime-value inspection Statement-level state inspection

get_imports_of_path; find_class; find_method get_definition_of_class; get_definition_of_method; get_code_of_method get_structure_of_method;get_callers_of_method;get_def_use_of_variable get_execution_path get_runtime_values get_state_at_statement

Support why- and how-type questions by locating referenced entities and dependencies Support how-type questions by inspecting surrounding logic and candidate modification points Support why-type questions by analyzing control flow, caller relations, and value propagation Support what-type questions by revealing what actually happens during failing execution Support what- and why-type questions by tracking variable evolution and abnormal states Support what-, why-, and how-type questions by examining concrete states and checking repair hypotheses

tion, which helps examine concrete program states at specific locations and check whether a repair hypothesis is consistent with the observed execution. Table I summarizes the function calls that implement these context access capabilities. Together, the static and dynamic evidence constructed in this stage, along with the uniform access interface, provide the information basis for the next stage, Question-driven Failure Diagnosis (Section III-B). B. Question-driven Failure Diagnosis As illustrated in Stage II of Figure 2, P RAC R EPAIR does not directly generate a patch from the context constructed in Section III-A. Instead, it first transforms the collected evidence into diagnostic understanding through question-driven failure diagnosis. This stage takes as input the buggy code, test suite, failure information, the accumulated diagnostic QA history, and optionally the validation feedback returned from Stage III. At each round, P RAC R EPAIR raises one diagnostic question, retrieves the evidence needed to answer it, and appends the resulting QA pair to the diagnosis history. Each QA pair records the question, retrieved evidence, diagnostic answer, and repair implication. The loop terminates when the diagnosis budget is exhausted or the accumulated QA history is sufficient to formulate a repair hypothesis. Finally, P RAC R EPAIR summarizes the diagnostic findings into an explicit repair hypothesis. 1) Question Asking.: Prior studies show that questions are central to debugging and program understanding [7], [8]. Accordingly, P RAC R EPAIR reduces diagnostic uncertainty through three question types, grounded in the context access capabilities in Table I. • What-type questions establish factual understanding of the failing execution, such as executed statements, branch outcomes, variable evolution, and deviations from expected behavior, mainly using dynamic evidence. • Why-type questions explain the failure by connecting abnormal runtime behavior to underlying program logic, such as incorrect control flow, abnormal state transitions, or invalid data dependencies, using both static and dynamic evidence. • How-type questions determine how to change the faulty logic to restore the intended semantics, mainly using the diagnosed root cause and static code context. To avoid unnecessary diagnostic overhead, P RAC R EPAIR does not ask all questions at once. Instead, it requires the LLM to make a structured diagnostic decision at each step. The decision either raises one new diagnostic question or returns a stopping signal. When raising a question, the LLM must specify the question type, the target program entity or runtime behavior to inspect, and the evidence needed to answer the

question. For example, a what-type question may target the value of a variable at a suspicious statement, while a whytype question may target the control or data dependency that explains an abnormal state. This constrained format makes the diagnosis process traceable and prevents the model from asking multiple unrelated questions in one round. The diagnosis loop terminates when either the diagnosis budget is reached or the LLM returns the stopping signal. 2) Question Answering.: To answer each diagnostic question, P RAC R EPAIR lets the LLM retrieve failure-relevant context through the unified interface in Section III-A. Depending on the question type, the LLM may inspect static evidence, such as dependencies, implementations, and structural relations, or dynamic evidence, such as execution paths, runtime values, and statement-level states. Inspired by ReAct [37], this process interleaves reasoning and retrieval until enough evidence is collected to answer the question. The resulting answer is paired with the question and appended to the QA history for subsequent diagnosis. When Stage III returns validation feedback, the same loop incorporates it to re-diagnose the current patch behavior. 3) Repair Hypothesis Formulating.: When the diagnosis loop terminates, P RAC R EPAIR formulates an explicit repair hypothesis based on the accumulated QA history and the currently available failure-relevant evidence. The hypothesis is represented in a structured form with four fields: faulty behavior, which describes the observed abnormal execution; supporting evidence, which records the key QA findings and retrieved context; suspected root cause, which explains why the failure occurs; and modification suggestion, which specifies how the faulty logic should be changed. For example, in Figure 1, the hypothesis identifies premature flushing at shift == 0 as the faulty behavior, uses the observed values of cache and shift as supporting evidence, and suggests changing the in-loop flush condition. This structured hypothesis serves as the output of Stage II and the input to Stage III, bridging diagnosis and patch generation. C. Feedback-guided Patch Refinement As illustrated in Stage III of Figure 2, P RAC R EPAIR turns the repair hypothesis produced by Question-driven Failure Diagnosis into an iterative loop. Rather than treating validation as a simple pass/fail check, this stage explicitly analyzes the behavioral differences before and after patching and uses them as new evidence for subsequent diagnosis. In this way, Stage III closes the loop between repair and diagnosis: a repair hypothesis guides patch generation, patch validation reveals how the patched execution differs from the original failing execution, and unsuccessful validation produces feedback that is fed back into Stage II to refine the diagnosis and the next

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 14, NO. 8, AUGUST 2021

repair hypothesis. This refinement loop continues until the maximum number of refinement rounds is reached (i.e., 3), or terminates earlier once a plausible patch is found. 1) Patch Generating.: Given the current repair hypothesis, P RAC R EPAIR prompts the LLM to generate a candidate patch for the buggy function. The patch-generation prompt is constructed from three parts: (1) bug context, including the buggy function, triggering tests, and failure information; (2) repair hypothesis, including the suspected root cause of the failure and the corresponding modification suggestions; and (3) generation instruction, which directs the LLM to produce a corrected implementation of the buggy function. In this way, patch generation is guided not only by the observed symptom, but also by the explicit diagnostic understanding accumulated in Stage II. To preserve input clarity and minimize prompt bias, P RAC R EPAIR adopts a zero-shot prompting strategy, with patch generation relying solely on the structured prompt rather than in-context examples. Due to page limits, all AI prompt templates are provided in the artifact [38]. 2) Patch Validating.: After generating a candidate patch, P RAC R EPAIR applies it to the original program and validates the patched program through compilation and test execution. During this process, P RAC R EPAIR also collects execution traces from the patched program using the same trace collection procedure described in Section III-A, so that patched behaviors can later be compared with the original failing execution. If the patched program compiles successfully and passes all tests within the maximum execution time (i.e., 10 minutes), the patch is regarded as a plausible patch. 3) Feedback Extracting and Re-diagnosing.: If a candidate patch does not pass validation, P RAC R EPAIR does not treat the result as a simple failure signal. Instead, it first determines how the current repair attempt fails, because different validation outcomes provide different high-level directions for the next diagnosis round. For example, a compilation failure indicates that the patch itself is syntactically or semantically invalid. To provide such high-level guidance, P RAC R EPAIR first categorizes invalid validation results into four outcomes: (1) compilation failures, where the patched program cannot be compiled; (2) runtime failures, where the patched program compiles successfully but triggers runtime exceptions or timeouts during testing; (3) remaining failures, where the originally failing test(s) are still not fully fixed; and (4) regression failures, where the original failure is resolved but previously passing tests become failing. After establishing this coarsegrained diagnosis direction, P RAC R EPAIR further extracts three complementary forms of feedback to understand why the patch fails. First, it collects validation diagnostics, such as compiler errors, runtime exceptions, timeout messages, or updated failing tests, to describe the observed failure outcome. Second, it computes a code diff between the generated patch and the original buggy function to identify which statements or conditions have been changed. Third, it computes a trace diff between the original and patched executions. To compute the trace diff, P RAC R EPAIR executes the same triggering tests on both versions, collects traces using the same instrumentation procedure, aligns trace records by executed statement and execution order, and extracts changed branch outcomes, added or

6

removed statement executions, and divergent runtime values. The resulting feedback therefore explains not only whether the patch fails, but also how the patch changes the failing behavior. This extracted feedback is then fed back into Questiondriven Failure Diagnosis as optional diagnostic input, as shown in Figure 2. Based on the original bug context, the accumulated QA history, and the new feedback, the LLM rediagnoses the current patch failure. For instance, if the trace diff shows that a branch outcome changes but the failing value remains abnormal, the next diagnosis round can ask why the changed branch still does not restore the expected state. The resulting QA pairs are used to refine the repair hypothesis, which then guides the next round of patch generation and validation. Through this feedback-guided loop, P RAC R EPAIR progressively improves candidate patches until a plausible fix is found or the refinement budget is exhausted. IV. E XPERIMENT D ESIGN To evaluate our approach, we design experiments to answer the following research questions (RQs): • RQ1 (Repair Effectiveness): How effective is P RAC R E PAIR compared with existing APR tools under the standard perfect fault localization setting, and does it remain effective when exact fault locations are unavailable? • RQ2 (Repair Scenarios): How well does P RAC R EPAIR perform across different repair scenarios? • RQ3 (Ablation Study): What are the individual contributions of each component of P RAC R EPAIR to the overall improvement in repair effectiveness? • RQ4 (Generalizability Study): How effectively does P RAC R EPAIR generalize to unseen datasets when deployed with different underlying foundation models? A. Datasets Since our approach is implemented and evaluated in the Java APR setting, we use two Java bug-repair benchmarks, Defects4J [3] and RWB (Real-World Bugs) [27]. We therefore do not include datasets in other programming languages, such as SWE-Bench, in this study. For the Defects4J dataset, following prior studies [25], [27], [28], we split it into V1.2 (391 bugs after removing 4 deprecated ones) and V2.0 (438 new bugs). We also follow [25], [27], [28] to categorize bugs into four repair scenarios: multi-function (MF), where a fix involves multiple functions; single-function (SF), where a fix is confined to one function; single-hunk (SH), where a fix modifies one contiguous code region; and single-line (SL), where a fix changes only one line. Note that SH ⊆ SF and SL ⊆ SH. Table II shows the statistics. For the generalizability study, we use the recent benchmark RWB (Real-World Bugs) introduced by ThinkRepair [27], which consists of two versions. RWB V1.0 comprises bug-fixing commits after October 2021, while RWB V2.0 includes bug-fixing commits after March 2023, resulting in 44 and 29 single-function bugs, respectively. For fault information, to eliminate potential bias introduced by different fault localization (FL) tools, we follow recent APR studies [25]–[28] and use perfect fault localization as the default setting, where the repair system is provided

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 14, NO. 8, AUGUST 2021

TABLE II: Statistics of Studied Datasets. Dataset

#Total Bugs #MF Bugs #SF Bugs #SH Bugs #SL Bugs

Defects4J 1.2 Defects4J 2.0

391 438

136 210

255 228

154 159

80 78

#Sum

909

346

563

390

235

with the exact buggy statement location(s). In addition, to examine whether the effectiveness of P RAC R EPAIR depends on this idealized assumption, we further include a relaxed fault-localization setting in RQ1, where exact buggy statement locations are not provided. B. Implementation For the base models, we use gpt-3.5-turbo [39] and gpt-4o [40] in our main experiments to maintain direct comparability with prior APR studies. Following [41], we set the sampling temperature to 1.0. To further evaluate whether P RAC R EPAIR remains effective with newer foundation models, we additionally study its generalizability in RQ4 (Section V-D) using gpt-4 [42], Llama-3 [43], and DeepSeek-v3 [44]. We set the maximum number of repair sessions to 3 per bug, where each session is independent and starts from the original bug context. Within each session, the diagnosis loop is allowed to run for at most 10 rounds, although in practice the average number of rounds is no more than 5, since the loop terminates once no further diagnostic questions are raised. The refinement loop is allowed to run for at most 3 rounds, since this setting achieves a better balance between repair effectiveness (cf. Section V-C). All experiments were conducted on a workstation running Ubuntu 20.04, with a 16-core Intel Xeon processor, 192GB of RAM, and eight NVIDIA A800 GPUs. C. Baselines In our comparative evaluation, we evaluate P RAC R EPAIR against nine state-of-the-art baselines. These baselines include one traditional APR method, TBar [18]; three learning-based APR methods, SelfAPR [19], KNOD [22], and Tare [20]; and five recent LLM-based APR methods, including Codex [45], AlphaRepair [24], ChatRepair [25], ThinkRepair [27], RepairAgent [26], and ReinFix [28]. Since our evaluation adopts the same benchmark split, fault-localization setting, and repair metrics as these studies, we follow common practice in the APR community [19], [24], [25], [27], [28] and reuse the repair results reported in their original papers [18]–[20], [22], [24]–[28] instead of directly running these APR tools. To conduct the ablation study and investigate the contribution of different components of P RAC R EPAIR, we design the following variants by removing or replacing components of the framework. • w/o SDC+QFD+FPR: This variant is designed to evaluate the overall contribution of the proposed three-stage framework. Specifically, it removes all three stages and directly prompts the underlying LLM to generate a patch from the given bug information. • w/o SDC+QFD: This variant is designed to investigate the contribution of Static-dynamic Context Construction

7

and Question-driven Failure Diagnosis. Specifically, it retains only Feedback-guided Patch Refinement, allowing the model to iteratively improve patches based on feedback. • w/o SDC: This variant evaluates the contribution of Staticdynamic Context Construction. It preserves Question-driven Failure Diagnosis and Feedback-guided Patch Refinement, but removes the structured repair context built from static and dynamic evidence. • w/o DI: This variant evaluates the contribution of dynamic execution information in Static-dynamic Context Construction. Specifically, it removes dynamic traces and retains only static program context with failure information. • P RAC R EPAIR CoT : This variant evaluates the contribution of the proposed diagnosis strategy in Question-driven Failure Diagnosis. It replaces question-driven diagnosis with Chain-of-Thought prompting, in which the model formulates a repair hypothesis from the buggy code, failure information, and execution traces, similar to ThinkRepair [27]. • P RAC R EPAIR ReAct : This variant is designed to investigate the contribution of question-driven diagnosis. Specifically, it allows the LLM to use function calls, but replaces the proposed diagnosis strategy with direct interleaving of reasoning and tool use, similar to ReInFix [28]. D. Metrics Following prior work [25]–[28], we report two widely adopted metrics to evaluate repair effectiveness: • Number of plausible patches: The number of bugs for which at least one generated patch passes all developerwritten test cases [21], [22]. A plausible patch satisfies the test oracle but is not necessarily semantically correct. • Number of correct patches: The number of bugs for which at least one generated patch is semantically correct. To determine correctness, we first check whether a generated patch matches the developer-provided fix; otherwise, we manually assess its semantic equivalence. A patch is considered correct if it passes either of these checks [19]. V. E VALUATION A. RQ1: Repair Effectiveness We evaluate the repair effectiveness of P RAC R EPAIR on Defects4J under both the standard perfect fault localization setting and a relaxed setting where exact buggy statement locations are unavailable. Under the standard setting, we first compare P RAC R EPAIR with existing APR tools on repair results, and then further analyze its unique repair capability. Under the relaxed setting, we examine whether P RAC R EPAIR remains effective without perfect fault localization. Effectiveness under Perfect Fault Localization. Following the standard setting used in prior APR studies, we instantiate P RAC R EPAIR with two foundation models, GPT-3.5 [39] and GPT-4o [40], referred to as P RAC R EPAIRGPT-3.5 and P RAC R E PAIR GPT-4o , respectively. As shown in Table III, P RAC R E PAIR GPT-3.5 generates plausible fixes for 332 bugs and correct fixes for 275 bugs. With GPT-4o, P RAC R EPAIRGPT-4o further improves to 413 plausible fixes and 333 correct fixes. Since plausible patches pass all test cases but are not necessarily

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 14, NO. 8, AUGUST 2021

8

TABLE III: Repair results (correct patches / plausible patches) of APR tools under perfect fault localization on Defects4J. APR Tool

P RAC R EPAIRGPT-4o

P RAC R EPAIRGPT-3.5

ReInFixGPT-4o

ReInFixGPT-3.5

ChatRepair

ThinkRepair

RepairAgent

AlphaRepair

KNOD

Tare

SelfRepair

TBar

Chart Closure Lang Math Mockito Time

19/22 40/52 44/49 42/70 10/11 7/9

18/20 34/40 36/39 38/54 9/9 4/5

18/20 40/50 33/47 39/68 10/11 6/11

16/17 30/37 26/33 35/52 8/9 3/4

15/– 37/– 21/– 32/– 6/– 3/–

11/– 31/– 19/– 27/– 6/– 4/–

11/14 25/25 17/17 29/29 6/6 2/3

9/– 23/– 13/– 21/– 5/– 3/–

10/11 23/29 11/13 20/25 5/5 2/2

11/– 25/– 14/– 22/– 2/– 3/–

7/– 17/– 10/– 18/– 3/– 3/–

11/– 16/– 13/– 22/– 3/– 3/–

#Total (D4J V1.2) #Total (D4J V2.0)

162/213 171/200

139/167 136/165

146/207 145/190

118/152 123/147

114/– 48/–

98/– 107/–

90/94 74/92

74/109 36/–

71/85 50/85

77/– –/–

58/– 42/–

68/95 8/–

#Sum

333/413

275/332

291/397

241/299

162/–

205/–

164/186

110/109

121/170

77/–

100/–

76/95

Note: “–” indicates that no result was reported in the original work. TABLE IV: Repair results (correct patches / plausible patches) without perfect fault localization on Defects4J V1.2.

APR Tool

(a) vs. GPT-3.5-based result

(b) vs. GPT-4o-based result

Fig. 3: Venn diagram of correct patches of P RAC R EPAIR vs. LLM-based baselines on Defects4J V1.2 and V2.0.

P RAC R EPAIRNo-PFL ThinkRepairNo-PFL Codex

Chart Closure Lang Math Mockito Time

12/16 28/32 23/31 32/43 7/8 3/3

9/– 19/– 15/– 27/– 7/– 3/–

–/– –/– –/– –/– –/– –/–

#Total (D4J V1.2)

105/133

80/–

63/–

Note: “–” indicates that no result was reported in the original work.

semantically correct, these results indicate that P RAC R EPAIR not only satisfies the test oracle on a large number of bugs, but also achieves strong repair accuracy. In addition, P RAC R EPAIR successfully fixes bugs across all Defects4J projects, including Chart, Closure, Lang, Math, Mockito, and Time, demonstrating its effectiveness across projects from different domains. Compared with prior work, P RAC R EPAIR consistently outperforms the strongest baseline, ReInFix, on both Defects4J V1.2 and V2.0. On Defects4J V1.2, P RAC R EPAIRGPT-4o improves over ReInFixGPT-4o by 16 correct fixes, while P RAC R E PAIR GPT-3.5 exceeds ReInFixGPT-3.5 by 21 fixes. Similar gains are observed on the more challenging Defects4J V2.0 benchmark, where P RAC R EPAIRGPT-4o and P RAC R EPAIRGPT-3.5 outperform their ReInFix counterparts by 26 and 13 bugs, respectively. In addition, under GPT-3.5, P RAC R EPAIR also surpasses other recent LLM-based APR approaches, including ChatRepair, ThinkRepair, and RepairAgent. Unique Fix Analysis. We further analyze the unique repair capability of P RAC R EPAIR on Defects4J V1.2 and V2.0. Specifically, we compare the sets of correctly repaired bugs produced by P RAC R EPAIR and recent LLM-based APR baselines under the same base model setting. As shown in Figure 3, under GPT-3.5, P RAC R EPAIRGPT-3.5 achieves 75 unique correct fixes, compared with 29 for ThinkRepair, 26 for RepairAgent, and 12 for ChatRepair. Under GPT-4o, P RAC R EPAIRGPT-4o achieves 93 unique correct fixes, while ReInFix achieves 51. We exclude ReInFix from the GPT-3.5-based comparison because its public results are only available under GPT-4o. These results show that P RAC R EPAIR maintains stronger unique repair capability than existing LLM-based APR baselines, suggesting that its repair process complements prior methods. Effectiveness without Perfect Fault Localization. The above comparisons assume perfect fault localization, where exact buggy statement locations are provided. To examine whether P RAC R EPAIR remains effective without this assumption, we

TABLE V: Repair results (correct fixes) under different repair scenarios. Benchmark

Defects4J V1.2

Defects4J V2.0

Repair Scenario

MF

SF

SH

SL

MF

SF

SH

SL

ChatRepair ThinkRepair RepairAgent ReInFixGPT-3.5 ReInFixGPT-4o P RAC R EPAIRGPT-3.5 P RAC R EPAIRGPT-4o

– – 7 14 22 19 27

76 98 83 104 124 120 135

– 78 71 78 93 90 97

– 52 51 53 57 55 57

– – 6 14 15 15 18

– 107 68 109 130 121 153

– 81 65 85 103 92 108

48 47 48 47 56 51 57

Note: “–” indicates that no result was reported in the original work.

further evaluate it under GPT-3.5 without providing exact fault locations, referred to as P RAC R EPAIRNo-PFL . We compare it with available baselines under the same setting, including ThinkRepairNo-PFL [27] and Codex [45], on Defects4J V1.2. As shown in Table IV, P RAC R EPAIRNo-PFL fixes 105 bugs correctly and generates 133 plausible patches. Although this is lower than P RAC R EPAIRGPT-3.5 under perfect fault localization (139 correct and 167 plausible patches), it outperforms ThinkRepairNo-PFL and Codex by 25 and 42 correct fixes, respectively. These results show that fault locations are helpful, but P RAC R EPAIR remains effective when they are unavailable. Answer to RQ1: P RAC R EPAIR achieves the best overall repair effectiveness under perfect fault localization with both GPT-3.5 and GPT-4o. Without exact buggy statement locations, its performance decreases but still surpasses the only comparable baseline, showing that its effectiveness does not solely rely on perfect fault localization.

B. RQ2: Repair Scenarios While RQ1 evaluates the overall repair effectiveness of P RAC R EPAIR, analyzing its performance under specific repair

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 14, NO. 8, AUGUST 2021

scenarios provides a more fine-grained understanding of its capabilities across different levels of repair complexity. Following prior work [26]–[28], we further examine P RAC R EPAIR under four commonly scenarios: single-line (SL), single-hunk (SH), single-function (SF), and multi-function (MF). Repair Scenarios Analysis. As shown in Table V, P RAC R E PAIR achieves strong performance across all studied repair scenarios on both Defects4J V1.2 and V2.0. In the singlefunction (SF) setting, it delivers the best overall results among all compared approaches. On Defects4J V1.2, P RAC R E PAIR GPT-3.5 and P RAC R EPAIR GPT-4o repair 120 and 135 bugs, respectively, while on Defects4J V2.0 the corresponding numbers further increase to 121 and 153, consistently exceeding all baselines. In the single-hunk (SH) and single-line (SL) settings, P RAC R EPAIR continues to match or surpass recent LLM-based baselines, demonstrating its effectiveness across simpler and more complex repair scenarios. More importantly, P RAC R EPAIR shows clear advantages in the multi-function (MF) setting. Compared with ReInFix, the only other baseline explicitly supporting MF repair, P RAC R EPAIR achieves higher repair counts on both dataset versions. For example, P RAC R EPAIRGPT-4o repairs 27 and 18 MF bugs on Defects4J V1.2 and V2.0, compared with 22 and 15 for ReInFixGPT-4o . Overall, these results indicate that P RAC R EPAIR performs robustly across diverse repair scenarios, with particularly strong advantages on challenging multi-function bugs. Answer to RQ2: P RAC R EPAIR consistently outperforms prior methods across different repair scenarios, including SL, SH, SF, and MF bugs. Its advantage is especially clear on the more challenging multi-function setting. C. RQ3: Ablation Study To assess the impact of individual components in P RAC R E PAIR, we conduct an ablation study using the variants defined in Section IV-C. These variants are designed by systematically removing or replacing key parts of the framework. Based on this design, we evaluate the contribution of each stage, as well as the effects of dynamic execution traces, diagnosis strategy, and refinement rounds. Due to computational budget constraints, all ablation experiments are conducted on Defects4J V1.2 with GPT-3.5. Impacts of the Three Stages. Table VI reports the performance of several ablated variants, each designed to isolate the contribution of one stage in P RAC R EPAIR. The w/o SDC+QFD+FPR variant performs the worst, achieving 84 correct patches and 98 plausible patches. Adding only Feedback-guided Patch Refinement in w/o SDC+QFD increases the number of correct patches to 105, showing the benefit of iterative refinement. Further adding Question-driven Failure Diagnosis in w/o SDC raises the number of correct patches to 115, indicating that diagnosis improves repair beyond refinement alone. The full P RAC R EPAIR configuration achieves the best results. Compared with w/o SDC, adding Static-dynamic Context Construction increases the number of correct patches from 115 to 139 and plausible patches from 126 to 167. Overall, the results show that all three stages

9

TABLE VI: Repair results (correct patches / plausible patches) of the ablation study on Defects4J V1.2. Variant

Result

MF

SF

SH

SL

w/o SDC+QFD+FPR w/o SDC+QFD w/o SDC w/o DI P RAC R EPAIRCOT P RAC R EPAIRReAct P RAC R EPAIRGPT-3.5

84/98 105/113 115/126 120/137 107/123 121/149 139/167

7 12 16 17 9 15 19

77 93 99 103 98 106 120

55 74 77 80 68 79 90

38 46 48 49 52 53 55

Note: “–” indicates that no result was reported in the original work.

Fig. 4: The performance of P RAC R EPAIR with different refinement rounds

contribute to repair effectiveness, and their combination yields the strongest performance. Impacts of Dynamic Execution Traces. Table VI shows the contribution of dynamic execution traces to P RAC R EPAIR. The w/o DI variant removes dynamic execution traces from Static-dynamic Context Construction, leaving only static program context and failure information. Compared with the full system, this change reduces the number of correct patches from 139 to 120 and the number of plausible patches from 167 to 137. These results indicate that dynamic execution traces provide important failure-relevant evidence that cannot be fully recovered from static context alone. By exposing runtime behaviors, such as executed paths, branch outcomes, and variable state changes, they help the model perform more grounded diagnoses and repairs. Impacts of Reasoning Strategy. Table VI shows that, compared with plain CoT and ReAct, the proposed question-driven diagnosis mechanism in Question-driven Failure Diagnosis improves repair effectiveness. The one-shot P RAC R EPAIRCoT variant, which does not use the designed function calls, produces 107 correct patches. Allowing on-demand retrieval of failure-relevant evidence in P RAC R EPAIRReAct increases this number to 121, which suggests that tool-assisted diagnosis can be more effective than reasoning over a fixed input context alone. The full P RAC R EPAIR further improves the result to 139. Since both P RAC R EPAIRReAct and the full P RAC R EPAIR support function-call interaction, this additional gain indicates that the proposed question-driven diagnosis provides benefits beyond tool use alone. By organizing diagnosis around targeted questions, P RAC R EPAIR appears to help the model systematically inspect failure-relevant evidence and formulate repair hypotheses. Impacts of Refinement Interaction Number. According to Figure 4, repair performance improves as the number of refinement interactions increases. Without refinement, P RAC R EPAIR produces 89 correct patches, which increases to 107 and 119 after one and two refinement rounds, respectively, indicating that iterative feedback helps correct early patch errors. Per-

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 14, NO. 8, AUGUST 2021

TABLE VII: Repair results (correct fixes) of the generalizability study on RWB.

APR Tool

P RAC R EPAIR

ReInFix

ThinkRepair

LLM

G4 DS-v3 L3 G3.5 DS-C G4 G3.5 DS-C G3.5 DS-C

Cli Codec Collections Compress Csv Jsoup Lang

4 4 1 3 1 7 3

4 3 1 3 1 8 3

4 3 1 3 1 7 3

4 3 1 2 1 7 3

– – – – – – –

4 3 1 2 1 7 3

4 3 1 2 1 6 3

– – – – – – –

4 3 1 1 1 6 3

– – – – – – –

# RWB V1.0 23 # RWB V2.0 15

23 14

22 13

21 –

– 13

21 –

20 –

– 12

19 –

– 10

Note: “–” indicates that no result was reported in the original work. G4/G3.5 = GPT-4/GPT-3.5; DS-v3/DS-C = DeepSeek-v3/DeepSeek-Coder; L3=Llama-3.

formance further improves to 139 correct patches after three rounds, but remains unchanged with additional interactions, showing diminishing returns beyond this point. Considering both repair effectiveness and interaction cost, we adopt three refinement rounds as the default setting. Answer to RQ3: P RAC R EPAIR is well designed, and all three stages, i.e., Static-dynamic Context Construction, Question-driven Failure Diagnosis, and Feedback-guided Patch Refinement, can be effectively integrated to improve the correct repair effectiveness of P RAC R EPAIR.

D. RQ4: Generalizability Study In the generalizability study, we evaluate P RAC R EPAIR on the RWB benchmark under the perfect fault localization setting, following ThinkRepair [27], and further instantiate P RAC R EPAIR with multiple foundation models, including GPT-4, GPT-3.5, DeepSeek-v3, DeepSeek-Coder, and Llama3. We also report the published results of ThinkRepair [27] and ReInFix [28] on the same benchmark for comparison. Notably, RWB V1.0 and RWB V2.0 consist of bug-fixing commits collected after the training cutoff dates of GPT-3.5 and DeepSeek-Coder, respectively [27]. Result Analysis. As shown in Table VII, P RAC R EPAIR achieves the best or tied-best repair performance across both RWB datasets and all evaluated model settings. On RWB V1.0 (44 bugs), P RAC R EPAIR repairs 23 bugs with GPT-4 and DeepSeek-v3, outperforming ReInFix (21 bugs with GPT-4) and ThinkRepair (19 bugs with GPT-3.5). Similar trends hold under other models: P RAC R EPAIR repairs 22 bugs with Llama3 and 21 bugs with GPT-3.5, indicating stable effectiveness across different model backbones. On the more recent RWB V2.0 benchmark (29 bugs), P RAC R EPAIR again achieves the strongest results, repairing 13 bugs with DeepSeek-Coder, compared with 12 and 10 repaired by ReInFix and ThinkRepair, respectively. Moreover, when instantiated with opensource models such as GPT-4, DeepSeek-v3, and Llama-3, P RAC R EPAIR still maintains competitive repair effectiveness. These results suggest that the proposed approach generalizes well across both benchmarks and foundation models, rather than depending on a specific dataset or model family.

10

Answer to RQ4: P RAC R EPAIR generalizes well across both unseen benchmarks and different foundation models. These results show that its repair framework is robust and not tied to a specific dataset or model. VI. D ISCUSSION A. Repair Costs Using LLMs may raise concerns about repair costs. To address this, we follow prior work [25], [26], [28] and report the average monetary cost per repaired bug based on the RQ1 results on Defects4J. For prior methods, we compare against the costs reported in their original papers. In our setting, P RAC R EPAIRGPT-3.5 costs $0.04 per repaired bug, while P RAC R EPAIRGPT-4o costs $1.13. Compared with prior LLMbased APR methods, P RAC R EPAIR remains cost-efficient: under GPT-3.5, its cost is lower than ReInFixGPT-3.5 ($0.06), RepairAgent ($0.14), and ChatRepair ($0.42), while under GPT-4o it also costs less than ReInFixGPT-4o ($1.45). These results show that P RAC R EPAIR improves repair effectiveness while maintaining competitive repair cost. B. Threats to Validity Internal Validity. One internal threat comes from the manual validation of plausible patches. Since passing all test cases does not guarantee semantic correctness, we first check whether a plausible patch exactly matches the developerprovided fix; otherwise, we manually assess its semantic equivalence, following prior APR work. Another threat comes from potential data leakage, as some benchmark bugs or reference patches may have appeared in the pre-training data of the evaluated LLMs. To mitigate this concern, we additionally evaluate P RAC R EPAIR on the RWB benchmark, whose bugfixing commits were collected after the training cutoff dates of widely used LLMs. P RAC R EPAIR still achieves strong results on this benchmark under multiple foundation models, suggesting that its gains are not merely due to memorization. External Validity. To reduce the risk of an unrepresentative evaluation, we assess P RAC R EPAIR on Defects4J and RWB, two widely used real-world Java bug benchmarks. However, both datasets are limited to Java and may not fully represent other programming languages or much larger codebases. Evaluating P RAC R EPAIR on additional languages and broader repair settings remains future work. VII. R ELATED W ORK Existing APR approaches can be broadly categorized from three perspectives: non-learning-based approaches, learningbased approaches, and LLM-based approaches. Non-learning-based Approaches. Automated program repair has been widely studied for more than a decade [17]. Early approaches formulate repair as a search problem with manually designed mutation operators or fix patterns [18], [46]. Other techniques learn transformation templates or repair patterns from human-written patches [47], [48], or synthesize repairs using symbolic execution, constraints, and SMT solving [49], [50]. Additional work integrates repair into static analysis

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 14, NO. 8, AUGUST 2021

pipelines or retrieves similar code fragments as repair ingredients [51], [52]. Beyond functional bugs, prior studies have also addressed syntax errors, performance bugs, vulnerabilities, type errors, and build failures [53], [54]. Learning-based Approaches. With the development of machine learning, repair methods increasingly rely on data-driven models. Early learning-based methods use machine learning to rank or prioritize candidate patches [55]. More recent approaches adopt neural machine translation models to directly transform buggy code into fixed code [56], [57], or design neural architectures that predict tree-level or syntax-aware code transformations [58], [59]. Some methods further train repair-specific models on curated bug-fix datasets [19], [60]. Unlike these task-specific learning approaches, recent LLMbased APR methods use general-purpose foundation models without explicit repair-specific training. LLM-based Approaches. With the emergence of large language models, APR has increasingly shifted toward promptbased and agentic repair paradigms. Early LLM-based approaches mainly rely on prompt engineering to perform oneshot repair, where the model directly generates a candidate patch from buggy code and related inputs in a single interaction [24], [45], [61]. Later methods introduce iterative repair by repeatedly querying the LLM with validation feedback and refining patches across multiple rounds [12], [25], [27]. More recent agent-based approaches further extend this paradigm by allowing the LLM to invoke external tools during repair [26], [28]. Our work is most closely related to this line of research, but differs in that it is inspired by practical debugging behaviors and structures repair around static-dynamic information integration, question-driven failure diagnosis, and feedbackguided patch refinement. VIII. C ONCLUSION In this work, we present P RAC R EPAIR, a fully automated program repair framework inspired by real-world debugging practices. Specifically, P RAC R EPAIR constructs static and dynamic context, performs question-driven failure diagnosis to formulate explicit repair hypotheses, and iteratively refines candidate patches using validation feedback. Extensive experiments, including comparisons with state-of-the-art baselines, scenario-based analysis, and ablation studies, show that P RAC R EPAIR consistently outperforms existing methods. These results suggest that developer-inspired debugging workflows can substantially improve APR effectiveness. In future work, we plan to extend P RAC R EPAIR to more languages for stronger generalizability. R EFERENCES [1] Sicong Cao, Xiaobing Sun, Xiaoxue Wu, David Lo, Lili Bo, Bin Li, Xiaolei Liu, Xingwei Lin, and Wei Liu. Snopy: Bridging sample denoising with causal graph learning for effective vulnerability detection. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ASE ’24, page 606–618, New York, NY, USA, 2024. Association for Computing Machinery. [2] Zimin Chen, Steve Kommrusch, and Martin Monperrus. Neural transfer learning for repairing security vulnerabilities in c code. IEEE Transactions on Software Engineering, 49(1):147–165, 2023.

11

[3] René Just, Darioush Jalali, and Michael D. Ernst. Defects4j: a database of existing faults to enable controlled testing studies for java programs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis, ISSTA 2014, page 437–440, New York, NY, USA, 2014. Association for Computing Machinery. [4] JetBrains. Intellij idea: The leading ide for professional java and kotlin development. https://www.jetbrains.com/idea/. Accessed: 2026-03-21. [5] Microsoft. Visual studio code: The open source ai code editor. https: //code.visualstudio.com/. Accessed: 2026-03-21. [6] John D. Gould. Some psychological evidence on how people debug computer programs. International Journal of Man-Machine Studies, 7(2):151–182, 1975. [7] Amy J. Ko and Brad A. Myers. Designing the whyline: a debugging interface for asking questions about program behavior. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’04, page 151–158, New York, NY, USA, 2004. Association for Computing Machinery. [8] Jonathan Sillito, Gail C. Murphy, and Kris De Volder. Asking and answering questions during a programming change task. IEEE Trans. Softw. Eng., 34(4):434–451, July 2008. [9] Lucas Layman, Madeline Diep, Meiyappan Nagappan, Janice Singer, Robert Deline, and Gina Venolia. Debugging revisited: Toward understanding the debugging needs of contemporary software developers. In 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement, pages 383–392, 2013. [10] Zack Coker, David Gray Widder, Claire Le Goues, Christopher Bogart, and Joshua Sunshine. A qualitative study on framework debugging. In 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 568–579, 2019. [11] Yiling Lou, Jun Yang, Samuel Benton, Dan Hao, Lin Tan, Zhenpeng Chen, Lu Zhang, and Lingming Zhang. When automated program repair meets regression testing—an extensive study on two million patches. ACM Trans. Softw. Eng. Methodol., 33(7), September 2024. [12] Kyla H. Levin, Nicolas van Kempen, Emery D. Berger, and Stephen N. Freund. Chatdbg: Augmenting debugging with large language models. Proc. ACM Softw. Eng., 2(FSE), June 2025. [13] Abdulaziz Alaboudi and Thomas D. Latoza. Hypothesizer: A hypothesis-based debugger to find and test debugging hypotheses. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST ’23, New York, NY, USA, 2023. Association for Computing Machinery. [14] Devon H. O’Dell. The debugging mind-set. Commun. ACM, 60(6):40–45, May 2017. [15] Hadeel Eladawy, Claire Le Goues, and Yuriy Brun. Automated program repair, what is it good for? not absolutely nothing! In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ICSE ’24, New York, NY, USA, 2024. Association for Computing Machinery. [16] Herb Krasner. The cost of poor software quality in the us: A 2020 report. Proc. Consortium Inf. Softw. QualityTM (CISQTM), 2(3), 2021. [17] Claire Le Goues, Michael Pradel, and Abhik Roychoudhury. Automated program repair. Commun. ACM, 62(12):56–65, November 2019. [18] Kui Liu, Anil Koyuncu, Dongsun Kim, and Tegawendé F. Bissyandé. Tbar: revisiting template-based automated program repair. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2019, page 31–42, New York, NY, USA, 2019. Association for Computing Machinery. [19] He Ye, Matias Martinez, Xiapu Luo, Tao Zhang, and Martin Monperrus. Selfapr: Self-supervised program repair with test execution diagnostics. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, ASE ’22, New York, NY, USA, 2023. Association for Computing Machinery. [20] Qihao Zhu, Zeyu Sun, Wenjie Zhang, Yingfei Xiong, and Lu Zhang. Tare: Type-Aware Neural Program Repair . In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 1443– 1455, Los Alamitos, CA, USA, May 2023. IEEE Computer Society. [21] Nan Jiang, Thibaud Lutellier, and Lin Tan. Cure: Code-aware neural machine translation for automatic program repair. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), pages 1161–1173, 2021. [22] Nan Jiang, Thibaud Lutellier, Yiling Lou, Lin Tan, Dan Goldwasser, and Xiangyu Zhang. Knod: Domain knowledge distilled tree decoder for automated program repair. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 1251–1263, 2023. [23] He Ye and Martin Monperrus. Iter: Iterative neural repair for multilocation patches. In Proceedings of the IEEE/ACM 46th International

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 14, NO. 8, AUGUST 2021

Conference on Software Engineering, ICSE ’24, New York, NY, USA, 2024. Association for Computing Machinery. [24] Chunqiu Steven Xia and Lingming Zhang. Less training, more repairing please: revisiting automated program repair via zero-shot learning. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, page 959–971, New York, NY, USA, 2022. Association for Computing Machinery. [25] Chunqiu Steven Xia and Lingming Zhang. Automated program repair via conversation: Fixing 162 out of 337 bugs for $0.42 each using chatgpt. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2024, page 819–831, New York, NY, USA, 2024. Association for Computing Machinery. [26] Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. Repairagent: An autonomous, llm-based agent for program repair. In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering, ICSE ’25, page 2188–2200. IEEE Press, 2025. [27] Xin Yin, Chao Ni, Shaohua Wang, Zhenhao Li, Limin Zeng, and Xiaohu Yang. Thinkrepair: Self-directed automated program repair. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2024, page 1274–1286, New York, NY, USA, 2024. Association for Computing Machinery. [28] Jiayi Zhang, Kai Huang, Jian Zhang, Yang Liu, and Chunyang Chen. Repair ingredients are all you need: Improving large language modelbased program repair via repair ingredients search, 2025. [29] Kai Huang, Zhengzi Xu, Su Yang, Hongyu Sun, Xuejun Li, Zheng Yan, and Yuqing Zhang. Evolving paradigms in automated program repair: Taxonomy, challenges, and opportunities. ACM Comput. Surv., 57(2), October 2024. [30] Sophia D Kolak, Ruben Martins, Claire Le Goues, and Vincent Josua Hellendoorn. Patch generation with language models: Feasibility and scaling behavior. In Deep Learning for Code Workshop, 2022. [31] Julian Aron Prenner, Hlib Babii, and Romain Robbes. Can openai’s codex fix bugs?: An evaluation on quixbugs. In 2022 IEEE/ACM International Workshop on Automated Program Repair (APR), pages 69–75, 2022. [32] joernio. Joern: The bug hunter’s workbench. https://github.com/joernio/ joern, 2024. Accessed: 2026-03-06. [33] Fabian Yamaguchi, Nico Golde, Daniel Arp, and Konrad Rieck. Modeling and discovering vulnerabilities with code property graphs. In 2014 IEEE Symposium on Security and Privacy, pages 590–604, 2014. [34] Raffi Khatchadourian, Yiming Tang, Mehdi Bagherzadeh, and Syed Ahmed. Safe automated refactoring for intelligent parallelization of java 8 streams. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pages 619–630, 2019. [35] Oracle. Package java.lang.instrument. Java Platform, Standard Edition API Specification. Accessed: 2026-03-06. [36] Romain Lenglet. Asm: a code manipulation tool to implement adaptable systems. Adaptable and extensible. . . , 2002. [37] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023. [38] Anonymous Authors. Artifact: Source code, prompts, datasets, and experimental results for this paper. https://doi.org/10.5281/zenodo. 19336422, 2026. Anonymous research artifact. [39] OpenAI. gpt-3.5-turbo-0125. https://developers.openai.com/api/docs/ models#gpt-3-5-turbo, September 2023. Accessed: 2025-09-29. [40] OpenAI. Gpt-4o-2024-05-13: Openai’s next-generation language model. https://developers.openai.com/api/docs/models#gpt-4-turbo-and-gpt-4, September 2023. Accessed: 2025-09-29. [41] Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30):e2305016120, 2023. [42] OpenAI. GPT-4. https://developers.openai.com/api/docs/models/gpt-4, 2023. OpenAI API documentation. [43] Ollama. Meta Llama 3: The Most Capable Openly Available LLM to Date. https://ollama.com/library/llama3, 2024. [44] DeepSeek AI. DeepSeek Coder: Let the Code Write Itself. https:// github.com/deepseek-ai/DeepSeek-Coder, 2023. GitHub repository. [45] Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. Automated program repair in the era of large pre-trained language models, 2023. [46] Claire Le Goues, ThanhVu Nguyen, Stephanie Forrest, and Westley Weimer. Genprog: A generic method for automatic software repair. IEEE Transactions on Software Engineering, 38(1):54–72, 2012.

12

[47] Dongsun Kim, Jaechang Nam, Jaewoo Song, and Sunghun Kim. Automatic patch generation learned from human-written patches. In Proceedings of the 2013 International Conference on Software Engineering, ICSE ’13, page 802–811. IEEE Press, 2013. [48] Rohan Bavishi, Hiroaki Yoshida, and Mukul R. Prasad. Phoenix: automated data-driven synthesis of repairs for static analysis violations. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2019, page 613–624, New York, NY, USA, 2019. Association for Computing Machinery. [49] Hoang Duong Thien Nguyen, Dawei Qi, Abhik Roychoudhury, and Satish Chandra. Semfix: program repair via semantic analysis. In Proceedings of the 2013 International Conference on Software Engineering, ICSE ’13, page 772–781. IEEE Press, 2013. [50] Xusheng Xiao, Sihan Li, Tao Xie, and Nikolai Tillmann. Characteristic studies of loop problems for structural test generation via symbolic execution. In Proceedings of the 28th IEEE/ACM International Conference on Automated Software Engineering, ASE ’13, page 246–256. IEEE Press, 2013. [51] Yu Liu, Sergey Mechtaev, Pavle Subotić, and Abhik Roychoudhury. Program repair guided by datalog-defined static analysis. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2023, page 1216–1228, New York, NY, USA, 2023. Association for Computing Machinery. [52] Alexandru Marginean, Johannes Bader, Satish Chandra, Mark Harman, Yue Jia, Ke Mao, Alexander Mols, and Andrew Scott. Sapfix: Automated end-to-end repair at scale. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pages 269–278, 2019. [53] Rahul Gupta, Aditya Kanade, and Shirish Shevade. Deep reinforcement learning for syntactic error repair in student programs. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI’19/IAAI’19/EAAI’19. AAAI Press, 2019. [54] Daniel Tarlow, Subhodeep Moitra, Andrew Rice, Zimin Chen, PierreAntoine Manzagol, Charles Sutton, and Edward Aftandilian. Learning to fix build errors with graph2diff neural networks. In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops, ICSEW’20, page 19–20, New York, NY, USA, 2020. Association for Computing Machinery. [55] Fan Long and Martin Rinard. Automatic patch generation by learning correct code. In Proceedings of the 43rd Annual ACM SIGPLANSIGACT Symposium on Principles of Programming Languages, POPL ’16, page 298–312, New York, NY, USA, 2016. Association for Computing Machinery. [56] Rahul Gupta, Soham Pal, Aditya Kanade, and Shirish Shevade. Deepfix: fixing common c language errors by deep learning. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI’17, page 1345–1351. AAAI Press, 2017. [57] Michele Tufano, Jevgenija Pantiuchina, Cody Watson, Gabriele Bavota, and Denys Poshyvanyk. On learning meaningful code changes via neural machine translation. In Proceedings of the 41st International Conference on Software Engineering, ICSE ’19, page 25–36. IEEE Press, 2019. [58] Qihao Zhu, Zeyu Sun, Yuan-an Xiao, Wenjie Zhang, Kang Yuan, Yingfei Xiong, and Lu Zhang. A syntax-guided edit decoder for neural program repair. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2021, page 341–353, New York, NY, USA, 2021. Association for Computing Machinery. [59] Yi Li, Shaohua Wang, and Tien N. Nguyen. Dlfix: context-based code transformation learning for automated program repair. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, ICSE ’20, page 602–614, New York, NY, USA, 2020. Association for Computing Machinery. [60] He Ye, Matias Martinez, and Martin Monperrus. Neural program repair with execution-based backpropagation. In Proceedings of the 44th International Conference on Software Engineering, ICSE ’22, page 1506–1518, New York, NY, USA, 2022. Association for Computing Machinery. [61] Nan Jiang, Kevin Liu, Thibaud Lutellier, and Lin Tan. Impact of code language models on automated program repair. In Proceedings of the 45th International Conference on Software Engineering, ICSE ’23, page 1430–1442. IEEE Press, 2023.

Related documents

• Experimental Settings in LLM-Based Program Repair: A Study of Inputs, Tool Access, Feedback, and Validation#965480
• A11yLTLNav: Automatic Detection of Accessibility Navigation Failures#965481
• Detecting Argument-Swap Bugs Using Context-Enhanced Code Representations#965482
• A Large-Scale Empirical Study of Quality Assurance Practices and Gaps in AI Agents#965483
• CARE: A Responsibility-Oriented Architecture for Domain-Scoped Resolution in Modular and Upgradeable Smart Contracts#965484