Rethinking Automated Program Repair: The Impact of Bug Complexity, Fault Localization, and LLM Cost-efficiency

Rethinking Automated Program Repair: The Impact of Bug Complexity, Fault Localization, and LLM Cost-efficiency Junchi Liu # Colorado State University, United States

Ali Bigdeli # Colorado State University, United States

Roya Daneshi # Colorado State University, United States

Atu Ambala # Colorado State University, United States

Sudipto Ghosh # Colorado State University, United States

arXiv:2608.14065v1 [cs.SE] 14 Aug 2026

Fabio Santos # Colorado State University, United States

Abstract Background: Software bugs remain a critical challenge in development, necessitating effective Automated Program Repair (APR) techniques. While Large Language Model (LLM)-based APR systems have shown promise, prior studies primarily focus on overall repair effectiveness. The effects of bug complexity, fault localization, reasoning settings, and repair cost-effectiveness remain insufficiently explored. Aims: This study presents a comprehensive empirical analysis of LLM-based APR, focusing on how repair performance is shaped by bug complexity, fault localization, reasoning settings, and costs. Method: We construct a curated dataset by collecting algorithmic bugs from AtCoder, a competitive programming platform. We evaluate two APR techniques (ChatRepair and CodeCorrector) using three LLMs (DeepSeek, GPT, and Llama), with multiple model variants and reasoning settings, and examine their performance across diverse levels of bug complexity and localization strategies through a multi-dimensional empirical framework and statistical analysis. Results: Although structurally complex bugs and imprecise fault localization make repair more challenging, LLM-based APR techniques still achieve competitive repair effectiveness. Imprecise fault localization can substantially enlarge the performance gap between APR techniques. Furthermore, higher-cost LLMs and stronger reasoning settings do not consistently yield better cost-efficiency, revealing a nontrivial trade-off between repair effectiveness and computational cost. We further observe that the impact of reasoning strategies varies considerably across different LLM families, affecting both repair effectiveness and cost-efficiency. Conclusions: Over 50% of moderately complex bugs can be repaired by low-cost LLM-based APR techniques. The repair effectiveness gap between APR techniques becomes larger as fault localization becomes less precise. GPT-5 repairs 7 and 39 more complex bugs than DeepSeek-V4-pro and DeepSeek-V3.2, respectively; whereas the total repair cost of DeepSeek-V3.2 across the nonreasoning and reasoning stages is only approximately 4.6% and 7.7% of GPT-5 and DeepSeek-V4-pro, respectively. 2012 ACM Subject Classification Software and its engineering Keywords and phrases Automated Program Repair, Large Language Models, Fault Localization, Bug Complexity, Cost-efficiency Digital Object Identifier 10.4230/LIPIcs.ESEM.2026.40 © Junchi Liu, Ali Bigdeli, Roya Daneshi, Atu Ambala, Sudipto Ghosh, and Fabio Santos; licensed under Creative Commons License CC-BY 4.0 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Editors: Robert Feldt, Maria Paasivaara, Daniel Mendez, Stefan Wagner, and Marvin Muñoz Barón; Article No. 40; pp. 40:1–40:20 Leibniz International Proceedings in Informatics Schloss Dagstuhl – Leibniz-Zentrum für Informatik, Dagstuhl Publishing, Germany

40:2

Rethinking Automated Program Repair

Category Technical Track Paper

1

Introduction

In recent years, the rapid advancement of Generative AI, driven by intense competition among technology companies, has significantly accelerated the development of Large Language Models (LLMs) [30]. Current LLMs have shown substantial improvements in their ability to understand, reason, and generate programs [3, 42, 48]. As a result, they have been increasingly applied to a wide range of software engineering tasks, including code generation, code summarization, and bug detection [6, 19, 23]. In the context of Automated Program Repair (APR), LLM-based approaches have demonstrated significant advantages over traditional techniques, achieving strong performance across widely used benchmarks such as Defects4J [20], Bugs.jar [41], and QuixBugs [25]. Notably, many APR techniques achieve near-perfect performance on QuixBugs and IntroClass [13], small-scale algorithmic datasets primarily composed of simple, single-hunk bugs, suggesting that these datasets may be insufficiently challenging for evaluating the effectiveness of modern LLM-based APR approaches. Also, as LLMs exhibit strong performance in code understanding and reasoning, recent studies have increasingly explored end-to-end or self-directed APR approaches that reduce reliance on externally provided fault localization (FL) information [15, 18, 22]. Nevertheless, the impact of the absence of FL on repair effectiveness has not been systematically evaluated in the agent era. To address these unexplored gaps, our study introduces a more challenging algorithmic dataset, featuring bugs with varied complexity levels, and uses the tools of Git [44], srcML [7], and GumTreeDiff [12] to systematically and automatically categorize bugs based on their complexity, enabling a fine-grained empirical analysis of repair performance across different bug complexity levels as well as under accurate, vague, or no fault localization settings. Additionally, the reliance on commercial LLM APIs produces non-negligible computational costs [45]. APR tasks often require generating numerous candidate patches [24, 52], resulting in substantial output token usage, particularly under reasoning-intensive settings [14, 50]. Consequently, different LLM choices and reasoning settings can lead to significant cost discrepancies. Such costs are often overlooked in prior APR evaluations, despite being critical for practical deployment. In our study, we incorporate cost-efficiency metrics and standardize the unit to $1 for fair comparison. Specifically, we define Repaired Bugs/$ and Attempts/$ as the numbers of successfully repaired bugs and repair attempts divided by the total API cost, respectively, where the cost is calculated from input/output token usage and the corresponding API prices. We present a comprehensive evaluation of two LLM-based APR techniques across three LLMs, including multiple versions and reasoning settings. We employ multi-dimensional metrics (number of compilable patches, plausible patches, and repaired bugs, Top − k, consistency, and cost-efficiency metrics), and statistical tests to address the following research questions (RQs): RQ1. How does bug complexity affect the repair performance of LLM-based APR? RQ2. To what extent does fault localization influence the effectiveness of LLM-based APR? RQ3. How do LLM choices and reasoning settings impact the repair performance–cost trade-off in APR? The results of this study show that:

J. Liu et al.

40:3

Repair performance varies substantially across bug complexity, with the simplest complexity bugs being consistently the easiest to fix, as expected, but repair performance remains at a relatively considerable level as complexity increases, suggesting that LLM-based APR approaches maintain a stable reasoning ability even on structurally complex bugs. FL plays a critical role in improving both repair effectiveness and patch consistency. Although more precise localization generally leads to better repair performance, appropriate APR frameworks can still maintain relatively stable repair effectiveness in settings with vague or missing localization, suggesting potential for practical end-to-end APR scenarios. Also, the impact of FL precision on repair effectiveness decreases as bug complexity increases. More costly LLMs and reasoning strategies do not always produce proportional repair improvements; the increased cost should be considered seriously, and reasoning strategies do not consistently yield ideal outcomes, revealing a nontrivial trade-off between performance and efficiency. Our contributions include: i- We empirically analyze the impact of bug complexity and fault localization on LLM-based APR performance, and incorporate unit-standardized cost-efficiency metrics to quantify the trade-off between repair effectiveness and computational cost. We also provide an extensible experimentation infrastructure to support future studies, replication efforts, and systematic evaluation of LLM-based APR techniques. ii- Our study offers novel insights into how repair effectiveness varies with bug complexity and reveals the nontrivial cost implications of different LLM choices and reasoning settings. In particular, by systematically evaluating settings with accurate, vague, and no fault localization, we demonstrate the feasibility and potential of end-to-end LLM-based APR. iii- Additionally, our results across multiple LLMs and reasoning strategies provide practical guidance for selecting appropriate LLMs under the trade-off between the repair performance and economic cost, highlighting the importance of balancing repair performance and computational cost.

2

Background and Related Work

This section provides a relevant review of the literature and highlights the limitations of existing studies.

2.1

Previous Evaluations of LLM-based APR

Recent evaluations of APR are primarily based on the Defects4J Benchmark [20, 27], which has become the dominant benchmark for APR evaluation. However, heavy reliance on a single benchmark can limit the diversity of bug types and weaken the evaluation of APR approaches’ generalizability. Nevertheless, several variants of Defects4J have been proposed, including DEFECTS4J-TRANS [21], and MuBench [34]. These benchmarks still primarily contain software-engineering–oriented bugs [40]. Meanwhile, algorithm-oriented datasets such as QuixBugs [25], IntroClass, and ManyBugs [13] have been widely utilized to evaluate the generalizability of APR techniques. However, the difficulty of these datasets is often insufficient to challenge recent LLM-based APR approaches [47], thereby leading to an overestimation of their effectiveness. In addition, the continued reuse of long-standing benchmark datasets introduces data leakage issues [40], potentially leading to biased evaluation results.

ESEM 2026

40:4

Rethinking Automated Program Repair

Furthermore, Bean et al. [4] point out that the lack of statistical testing in the majority of benchmarks reduces the reliability of empirical findings. We collect recent real-world bugs from AtCoder [16] as an extension of algorithm-oriented datasets. Then, Git, srcML, and GumTreeDiff are utilized to categorize bugs based on complexity. Finally, we conduct an empirical evaluation of LLM-based APR under different bug complexity and FL settings, with a cost-efficiency and statistical analysis.

2.2

APR Techniques

APR techniques have advanced significantly with the development of LLMs’ code-reasoning and code-generation capabilities [53]. With the success of LLMs in code reasoning and generation [17], recent studies have increasingly explored their applications in APR. Xia and Zhang [47] introduced ChatRepair, which iteratively generates and refines patches by leveraging feedback from both failed and successful repair attempts, and has been widely viewed as a representative baseline in subsequent studies. For instance, Yin et al. [49] proposed ThinkRepair, which uses Chain-of-Thought (CoT) prompting [46] to enable reasoning prior to patch generation. Bouzenia et al. [5] introduced RepairAgent, the first autonomous agent-based framework for APR. Li et al. [22] further proposed CodeCorrector, an end-to-end framework that first analyzes programs to infer repair directions and then generates patches, achieving the best performance on the Defects4J v1.2 dataset. In this study, we select ChatRepair as a representative LLM-based APR approach and CodeCorrector as a recent LLM-based method that demonstrates strong performance in prior studies.

2.3

Evaluation Metrics

Existing APR evaluation metrics primarily focus on repair effectiveness, including the number of repaired bugs, Top − k, compilable patches, and plausible patches [11]. To further assess patch quality, Dai et al. [8] introduced the consistency metric, which measures the extent of changes between patches and buggy programs. However, these metrics fail to capture cost-efficiency, which has become increasingly important in the empirical study of LLM-based APR. Although recent studies have begun to report costs [5, 22, 47], they use inconsistent units, such as cost per bug, cost per successful repair, or cost per attempt, making it difficult to perform fair, direct comparisons across approaches. This lack of a unified cost metric limits the systematic evaluation of performance–cost trade-offs. Thus, we standardize costs to a common monetary unit, enabling a direct trade-off between repair performance and cost.

3

Research Design In this section, we discuss the research methodology.

3.1

Experimental Infrastructure

The experimental infrastructure is deployed on computing servers with isolated execution environments to ensure controlled access and protected communication. Shared storage and distributed computing resources are utilized to support large-scale APR experiments across multiple machines, improving the overall experimental efficiency. The experimental infrastructure consists of five components: the dataset, APR techniques, LLMs, additional tools, and Python scripts.

J. Liu et al.

40:5

Figure 1 Flow of the research method.

The dataset is constructed from AtCoder and includes problem descriptions, pairs of incorrect and correct C++ programs, and corresponding test cases (Section 3.3). The tools include the APR tools, along with Git, srcML, and GumTreeDiff. The AtCoder Library (ACL) is included as an external dependency to support compilation. Custom Python scripts are used for data analysis and dataset filtering. All the above components are maintained within our isolated environment. We utilize multiple LLMs in our study: gpt-5-2025-08-07 (GPT-5) [33], gpt-4o-mini2024-07-18 (GPT-4o-mini) [32], DeepSeek-V3.2 [9], DeepSeek-V4-pro [10], Llama-3.1-8B [2], and Llama-3.1-405B [1]. GPT-5 and GPT-4o-mini use the OpenAI API. DeepSeek-V3.2 and DeepSeek-V4-pro use the DeepSeek API. Llama-3.1 is deployed in our experimentation infrastructure.

3.2

Experimental Workflow

Figure 1 illustrates the overall experimental infrastructure and workflow of our study. We first collect buggy and corrected programs from AtCoder and construct an automated benchmark and evaluation pipeline for LLM-based APR experiments (details are provided in subsection 3.3). Next, we analyze bug complexity from multiple perspectives. Git-based differencing is utilized to extract hunk-line complexity information, while srcML [7] and GumTreeDiff [12] are employed to measure structural complexity at the Abstract Syntax Tree (AST) level (details are provided in subsection 3.4). Based on the constructed benchmark and complexity analysis, we deploy APR frameworks and integrate multiple LLM APIs with different reasoning settings to conduct automated repair experiments under FL settings (details are provided in subsection 3.5). Finally, we perform a comprehensive empirical evaluation to investigate the impact of bug complexity, fault localization, and cost-efficiency trade-offs on the performance of LLM-based APR techniques. The corresponding findings and experimental analyses are presented in section 4.

ESEM 2026

Rethinking Automated Program Repair

Number of Bugs

200

100

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 4 50 9 +

40:6

Fixed Lines Figure 2 Distribution of Fixed Size Across Raw Data

3.3

Dataset

Existing APR benchmarks have been widely used, raising concerns about data contamination in the evaluation of LLM-based APR. To mitigate this risk, we construct our dataset from the recent AtCoder submissions rather than relying on existing benchmarks. We focus on C++ programs because comparing FL settings and bug complexity across programming languages with different characteristics could introduce additional bias. This design enables a more controlled evaluation of LLM-based APR. In this study, we use AtCoder, a Japanese competitive programming platform, and collect 1,914 C++ source code submissions in October, 2025. Our data comes from regular contests, which typically include algorithmically challenging problems designed to assess intermediate-to-advanced programming skills. We pair every incorrect program and correct program from the same participant as a pair of a bug and a correct patch (bug-patch pair). For each bug-patch pair, we use the git diff command to generate a diff file that captures the corresponding code changes. Before generating diff files using Git, we remove all comments and redundant blank lines. The diff files reflect line-level code changes, where ‘-‘ denotes deleted (buggy) lines N− , and ‘+‘ denotes added (patch) lines N+ . For each hunk, the number of fixed lines is defined as Equation 1. ( N+ , if N+ > N− , Nfixed = (1) N− , otherwise The distribution of bug-fix sizes across the original 1,914 C++ files is shown in Figure 2. Based on preliminary experiments comparing different fixed line thresholds on samples drawn from the full dataset, we observe that APR becomes considerably more difficult and computationally expensive as the number of fixed lines increases. Considering both repair feasibility and computational cost, we set a threshold of five fixed lines to focus on small-scope program repair. Finally, 640 bugs remain for our empirical study.

3.4

Bug Complexity

Buggy hunk-line-level complexity: The diff files generated by Git not only identify buggy and patched lines but also organize code changes into hunks. We use Git to generate diff files for every bug-patch pair. Based on the number of hunks and the number of fixed lines within each hunk, we categorize bugs into four categories: Single-Hunk-Single-Line

J. Liu et al.

40:7

Figure 3 AST operation-type complexity.

(SHSL), Single-Hunk-Multi-Line (SHML), Multi-Hunk-Single-Line (MHSL), and Multi-HunkMulti-Line (MHML). The categorization rule is defined in Equation 2. The number of hunks can be obtained directly from Git diff files. If any hunk contains more than one fixed line, the bug is classified as a multi-line bug.   SHSL, H = 1 ∧ ∀hi ∈ H, L(hi ) = 1     SHML, H = 1 ∧ ∃hi ∈ H, L(hi ) > 1 (2)   MHSL, H > 1 ∧ ∀hi ∈ H, L(hi ) = 1    MHML, H > 1 ∧ ∃hi ∈ H, L(hi ) > 1 where H denotes the total number of hunks in a bug, hi represents the i-th hunk (1 ≤ i ≤ H), and L(hi ) denotes the number of fixed lines in hunk hi . AST operation-type complexity: We employ srcML to construct XML-based structural representations of programs [7] and further utilize GumTreeDiff to extract AST-level edit operations between buggy and corrected program pairs [12]. Unlike hunk-line-level complexity, this metric measures structural differences at the AST node level, providing a finer-grained view of bug complexity. The AST edit distance [43] between a buggy and a correct program is defined by four edit operations: insert, delete, relabel, and move. In our study, bugs are further categorized based on the diversity of edit operations required for repair. Figure 3 illustrates how the AST edit-operation-type complexity is categorized. Table 1 and Table 2 present the number of bugs in each type under the hunk-line-level and AST operation-type complexity, respectively. We conduct empirical evaluations to analyze how bug complexity affects LLM-based APR performance.

3.5

Fault Localization And APR Settings

We use three FL strategies to explore their impact on LLM-based APR: buggy line-level FL, buggy hunk-level FL, and no FL. Figure 4 presents the examples of line-level FL and hunk-level FL according to the diff file.

ESEM 2026

40:8

Rethinking Automated Program Repair

Table 1 Hunk-line complexity.

Table 2 AST operation-type complexity

Bug Type

# Bugs

Bug Type

# Bugs

Single Hunk Single Line Single Hunk Multi Line Multi Hunk Single Line Multi Hunk Multi Line

251 226 67 96

Single Operation Two Operations Three Operations Four Operations

278 158 121 83

Total

640

Total

640

@@ -58 ,7 +58 ,7 @@ int main (){ } long long ans =0; for ( int i = x ;i <= k ; i ++){ ans =( ans + f [ n %2][0][ i ])% mo ; + ans =( ans + f [0][0][ i ])% mo ; } cout < < ans < <"\ n "; return 0;

// ### Buggy Hunk Start ### } long long ans =0; for ( int i = x ;i <= k ; i ++){ ans =( ans + f [ n %2][0][ i ])% mo ; } cout < < ans < <"\ n "; return 0; // ### Buggy Hunk End ###

for ( int i = x ;i <= k ; i ++){ // ### Buggy Line Start ### ans =( ans + f [ n %2][0][ i ])% mo ; // ### Buggy Line End ### }

(c) Line-level localization (a) Diff file

(b) Hunk-level localization

Figure 4 Fault Localization Strategy Examples

Different LLMs provide different mechanisms for controlling reasoning behavior. (a) DeepSeek-V3.2 provides two model variants: DeepSeek-V3.2-chat and DeepSeek-V3.2reasoner. For DeepSeek-V4-pro, the reasoning mode can be enabled or disabled through the thinking_type parameter in the DeepSeek API. Additionally, the reasoning_effort parameter can be configured as high or max. (b) GPT models do not provide separate chat or reasoner variants; instead, reasoning intensity can only be controlled through the reasoning.effort parameter, with settings such as medium, high, and xhigh. For Llama models, there is currently no explicit mechanism to directly control reasoning settings. Furthermore, the temperature for all LLMs is set to 1.0 [39] for all experiments. For ChatRepair, we set the maximum number of conversation rounds to 3 [47]. For both ChatRepair and CodeCorrector, we provide the problem background, the buggy code, the error message, and at most 3 failed test cases. We retain the original prompts reported in the corresponding studies [22, 47], and we do not perform model-specific parameter tuning, allowing the comparison to focus on the influence of the APR frameworks, FL strategies, and LLM effectiveness.

3.6

Experimental Design

To balance experimental comprehensiveness and computational cost, we adopt a threestage experimental design. First, we evaluate two APR tools with three different low-cost LLMs on the complete set of 640 bugs under line-level FL: DeepSeek-V3.2-chat, GPT-4o-mini, and Llama-3.1-8B. The maximum number of repair rounds is set to 10 as in [35]. We conduct a comprehensive empirical analysis of various APR techniques and LLM combinations with respect to repair effectiveness and cost-efficiency. Next, we select the best-performing LLM from the first stage and further evaluate it on all 640 bugs at line-level, hunk-level, and no-FL. The results are then compared across three FL strategies to analyze the impact of different FL precision on two LLM-based APR techniques. For the first two stages, reasoning mode is disabled across all LLMs, allowing us to identify bugs that can already be repaired under non-reasoning configurations before

J. Liu et al.

40:9

Table 3 Comparison of repair effectiveness between CodeCorrector and ChatRepair frameworks using three base LLMs on the full benchmark. CodeCorrector DeepSeekV3.2-chat

GPT-4omini

ChatRepair

Llama-3.1- DeepSeek8B V3.2-chat

GPT-4omini

Llama-3.18B

Repaired Bugs 295/640 126/640 11/640 292/640 140/640 50/640 Attempts 4003 5373 6305 6400 6400 6400 Unique Patches 3117(77.9%) 4425(82.4%) 5833(92.5%) 2623(41.0%) 3774(59.0%) 5111(80.0%) Compilable Patches 2830(70.7%) 3899(72.6%) 1466(23.3%) 2523(39.4%) 3511(54.9%) 3122(48.8%) Plausible Patches 295 126 11 421 247 76 Repaired Bugs / $ 47.62 17.86 71.43 20.41 Attempts / $ 646.17 761.48 1565.56 932.94 -

evaluating more computationally expensive reasoning-enabled models. To further investigate the repair effectiveness and cost-efficiency of advanced LLMs and reasoning settings under more challenging repair scenarios, we adopt a filtering strategy to focus the evaluation on bugs with moderate repair difficulty while controlling the computational cost of evaluating advanced LLMs. Specifically, we exclude SHSL bugs, single-operation-type bugs, and four-operation-type bugs. In addition, bugs that can already be successfully repaired by at least five APR techniques–LLM combinations in the first-stage experiments are removed. Consequently, the remaining benchmark focuses on bugs with moderate structural complexity and lower repairability under low-cost APR configurations. After filtering, 211 bugs remain for empirical analysis. We further select the better-performing APR technique from the previous stages as the representative APR method for subsequent experiments. In this stage, we evaluate four advanced LLMs: DeepSeek-V4-pro, DeepSeek-V3.2, GPT-5, and Llama-3.1-405B. The maximum number of repair rounds is increased to 13. During the first 10 rounds, reasoning mode is disabled. If a bug remains unfixed, reasoning mode is enabled for the final three rounds. For GPT-5, DeepSeek-V3.2, and DeepSeek-V4-pro, the reasoning effort is configured as high. Experimental Token Consumption. To facilitate the reproduction of our experiments, we report the total input and output tokens consumed by each evaluated LLM that incurred API costs. DeepSeek-V3.2 consumed 116.5M input tokens and 42.7M output tokens. GPT4o-mini used 49.5M input tokens and 14.4M output tokens. DeepSeek-V4-pro consumed 6.3M input tokens and 9.9M output tokens, GPT-5 consumed 3.4M input tokens and 7.7M output tokens, and Llama-3.1-405B consumed 6.3M input tokens and 1.7M output tokens. API Response Time. We also report the approximate response time per repair request. Without reasoning mode, all evaluated LLMs require approximately 1–3 minutes per request. With reasoning enabled, DeepSeek-V3.2 and DeepSeek-V4-pro required approximately 40–60 minutes per request, while GPT-5 required about 30 minutes.

4

Results and Empirical Analysis In this section, we show the results of our empirical experiment.

4.1

APR Performance With Different LLMs

Table 3 presents the repair performance of CodeCorrector and ChatRepair with three base LLMs under the non-reasoning setting. In this experiment, line-level FL is provided

ESEM 2026

40:10

Rethinking Automated Program Repair

Table 4 Pairwise 2×2 contingency tables for chi-square tests. (a) Hunk-line Complexity Comparisons SHSL vs SHML

Type

Fixed

SHSL vs MHSL

Unfixed

SHSL 197(78.5%) SHML 113(50.0%)

54 113

Type

Fixed

54 28

Type

113 28

Type

Fixed

Unfixed 54 48

MHSL vs MHML

Unfixed

SHML 113(50.0%) MHML 48(50.0%)

Fixed

SHSL 197(78.5%) MHML 48(50.0%)

SHML vs MHML

Unfixed

SHML 113(50.0%) MHSL 39(58.2%)

Unfixed

SHSL 197(78.5%) MHSL 39(58.2%)

SHML vs MHSL

Type

Fixed

SHSL vs MHML

113 48

Type

Fixed

Unfixed

MHSL 39(58.2%) MHML 48(50.0%)

28 48

(b) AST Operation Complexity Comparisons Single vs Two

Type

Fixed

Single vs Three

Unfixed

Single 209(75.2%) Two 90(57.0%)

69 68

Type

Fixed

Two 90(57.0%) Three 66(54.5%)

Single vs Four

Unfixed

Single 209(75.2%) Three 66(54.5%)

Two vs Three

Type

Fixed

69 55

Type

Fixed

Unfixed

Single 209(75.2%) Four 32(38.6%)

Two vs Four

69 51

Three vs Four

Unfixed

Type

Fixed

Unfixed

68 55

Two Four

90(57.0%) 32(38.6%)

68 51

Type

Fixed

Three 66(54.5%) Four 32(38.6%)

Unfixed 55 51

Table 5 Pairwise Pearson Chi-square test results across different bug complexity categories. SHSL SHSL SHML MHSL MHML

– – – –

SHML

MHSL

MHML

p < 0.0001 p = 0.0013 p < 0.0001 p = 0.30 p = 1.000 – p = 0.38 – – – – –

1-op 1-op 2-op 3-op 4-op

– – – –

2-op

3-op

4-op

p = 0.0001 p < 0.0001 p < 0.0001 p = 0.81 p = 0.013 – p = 0.041 – – – – –

(a) Hunk-line complexity (b) AST operation type complexity Darker cells indicate smaller p-values.

within the incorrect programs, and the maximum repair round is set to 10. CodeCorrector stops after generating the first plausible patch, whereas ChatRepair continues generating candidate patches. Overall, DeepSeek-V3.2-chat achieves substantially stronger repair effectiveness than GPT-4o-mini across both APR frameworks. When integrated with DeepSeek-V3.2-chat, both CodeCorrector and ChatRepair repair more than twice as many bugs as they do with GPT-4o-mini (295, 292 vs. 126, 140), whereas Llama-3.1-8B demonstrates very limited repair behavior: only 11 bugs were corrected with CodeCorrector and 50 with ChatRepair. In addition, DeepSeek-V3.2-chat exhibits noticeably better cost-efficiency. DeepSeekV3.2-chat repairs 47.62 bugs per dollar under CodeCorrector, compared with only 17.86 bugs per dollar for GPT-4o-mini, representing approximately 2.7× higher cost-efficiency. Similarly, under ChatRepair, DeepSeek achieves 71.43 repaired bugs per dollar, whereas GPT-4o-mini repairs only 20.41 bugs per dollar, corresponding to approximately 3.5× higher cost-efficiency. Finding 1: Among relatively low-cost LLMs, DeepSeek-V3.2-chat demonstrates the strongest overall APR effectiveness, achieving significantly better repair results and costefficiency than GPT-4o-mini, whereas Llama-3.1-8B exhibits limited repair effectiveness despite its negligible inference cost.

J. Liu et al.

40:11

Bug Complexity: We then analyze how bug complexity affects overall repairability, where a bug is considered repaired if any evaluated APR technique and LLM configuration can fix it. Table 4 and Table 5 present the pairwise 2 × 2 contingency tables and Pearson chisquare test results for the numbers of fixed and unfixed bugs across different bug complexity categories. Pairwise chi-square tests are conducted to assess whether differences in repair rates across complexity categories are statistically significant. Following [28], we consider differences statistically significant when p < 0.05, while p < 0.005 indicates sufficient evidence against the null hypothesis. SHSL and single-operation-type bugs exhibit the lowest structural complexity and consequently achieve the highest repair rates among all categories, reaching 78.5% and 75.2%, respectively. Furthermore, all pairwise comparisons between these two categories and the remaining complexity categories yield p < 0.005, providing strong statistical evidence that SHSL and single-operation-type bugs are substantially easier for APR techniques to repair. However, the SHML, MHSL, MHML, two-operation, and three-operation bugs also achieve repair rates above 50%. For hunk-line-level complexity, no statistically significant difference in repair is observed among multi-line bug categories (p > 0.05). In contrast, although no statistically significant difference is observed between two-operation-type and three-operation-type bugs (p = 0.81), four-operation-type bugs exhibit significantly lower repair effectiveness than other operationtype categories (p < 0.05). Thus, AST operation complexity further distinguishes highly complex bugs, with statistical analysis showing that four-operation bugs exhibit the most complex repair structure. Finding 2: SHSL and single-operation-type bugs are the easiest to repair, whereas fouroperation-type bugs remain the most challenging. The repair rates of moderate-complexity categories exceed 50%, indicating that LLM-based APR techniques maintain strong repair effectiveness. Although four-operation-type bugs exhibit lower repair performance, their repair rates remain considerable. These results suggest that low-cost LLM-based APR techniques retain substantial repair effectiveness even for structurally complex bugs. RQ1 Summary. Bug complexity significantly affects repair effectiveness. Chi-square test results suggest three approximate levels of repair difficulty across different bug complexity categories. Overall, repair effectiveness decreases with increasing bug complexity, while the simplest bugs remain substantially easier to repair. Although four-operation bugs are the hardest to repair, APR techniques still maintain considerable repair capability for structurally complex bugs. Over 50% of moderate-complexity bugs can still be repaired successfully. These results demonstrate the potential of low-cost LLM-based APR techniques for handling challenging real-world repair scenarios.

4.2

Fault Localization Evaluation

Given DeepSeek-V3.2-chat’s best performance in the previous experiment, we further evaluate the impact of different FL strategies on APR performance using this LLM with both APR techniques. In this experiment, the maximum repair round is still set to 10. Table 6 shows that CodeCorrector and ChatRepair achieve nearly identical repair effectiveness under line-level FL, with CodeCorrector even repairing three more bugs. However, when only hunk-level FL or no FL is provided, ChatRepair demonstrates consistently greater stability and generalizability. Under hunk-level FL, ChatRepair repairs 58 more bugs than CodeCorrector, and this gap further increases to 93 bugs when FL is unavailable.

ESEM 2026

40:12

Rethinking Automated Program Repair

Table 6 Comparison of repair effectiveness between CodeCorrector and ChatRepair frameworks using three Fault Localization strategies on the full benchmark. CodeCorrector line-level

hunk-level

ChatRepair w/o FL

line-level

hunk-level

w/o FL

Repaired Bugs 295/640 179/640 112/640 292/640 237/640 205/640 Attempts 4003 4986 5480 6400 6400 6400 Unique Patches 3117(77.9%) 4346(87.2%) 4012(73.2%) 2623(41.0%) 4741(74.1%) 3261(51.0%) Compilable Patches 2830(70.7%) 3503(70.3%) 3756(68.5%) 2523(39.4%) 4073(63.6%) 3174(49.6%) Plausible Patches 295 179 112 421 542 299 Repaired Bugs / $ 47.62 16.60 18.87 71.43 24.41 47.62 Attempts / $ 646.17 462.52 923.18 1565.56 659.13 1486.64

Table 7 Comparison of repair consistency under different FL strategies. FL Strategy

CodeCorrector

ChatRepair

Line-level FL Hunk-level FL w/o FL

0.85 0.75 0.54

0.92 0.83 0.85

Finding 3: The performance gap between APR techniques increases markedly as FL becomes less precise, indicating that repair stability and generalizability are critical for realworld APR scenarios. Different APR techniques may achieve similar repair rates under precise line-level FL, but their repair rates diverge substantially under imprecise FL settings. We further introduce the code consistency rate (CCR) to measure how well repaired programs preserve the original code’s structure. Equation 3 [8] defines the consistency metric, where k denotes the total number of code lines in the repaired program, and r denotes the number of preserved code lines after repair.

CCR (Code Consistency Rate) =

r k

(3)

According to Table 7, line-level FL achieves the highest patch consistency across all FL settings for both APR techniques. ChatRepair generally generates patches with higher structural consistency than CodeCorrector, particularly without FL (0.85 vs. 0.54). Finding 4: FL not only improves repair effectiveness but also helps generate patches that remain more structurally consistent with the original programs. The design of APR techniques also significantly impacts the structural consistency of generated patches. We use logistic regression [29] to examine the extent to which reduced FL precision affects the likelihood of successful repair. We set line-level FL as the reference category and analyze each complexity level independently. The results are reported using odds ratios (ORs) and corresponding p-values. Smaller OR values reflect a greater reduction in the odds of successful repair. Figure 5 shows that repair rates consistently decrease across all complexity categories as FL becomes less precise (blue vs. orange vs. red). According to Table 8, both hunk-level FL and no-FL settings yield overall p-values smaller than 0.001, indicating that FL precision has a statistically significant effect on overall repair effectiveness. We further observe that the simplest bugs (SHSL and single-operation bugs) appear sensitive to reductions in FL precision, as both hunk-level FL and no-FL settings produce very

J. Liu et al.

Line-Level

74.1

Hunk-Level w/o FL

59

60% 40%

48.2 38.5

49

37.3

47.4

32.3 30.2

30.1

20% 0%

52.2

25.4

SHSL

SHML MHSL MHML Hunk-line Complexity

Repair Rate (%)

Repair Rate (%)

80%

40:13

Line-Level

80%

70.9

60%

54.3

Hunk-Level w/o FL

53.8

53.7

44.3 38

40%

28.9 32.3

20% 0%

36.1

45 30.6

24.1

Single-o

Two-o

Three-o

Four-o

AST-operation Type Complexity

Figure 5 Repair rate under different bug complexity categories. Table 8 Logistic regression results of FL effects on repair success under different complexity categories. (a) Hunk-line complexity categories Scope Overall SHSL SHML MHSL MHML

Hunk-level FL

(b) AST-operation complexity categories

No FL

Scope

OR

p

OR

p

0.582 0.502 0.672 0.544 0.497

< .001 < .001 0.037 0.084 0.019

0.399 0.315 0.462 0.311 0.451

< .001 < .001 < .001 0.002 0.008

Overall single-op two-op three-op four-op

Hunk-level FL

No FL

OR

p

OR

p

0.582 0.489 0.683 0.528 0.719

< .001 < .001 0.092 0.015 0.321

0.399 0.336 0.409 0.379 0.561

< .001 < .001 < .001 < .001 0.092

low odds ratios, also with p < 0.001. In contrast, the highest complexity bugs (four-operation bugs) exhibit the least noticeable repair degradation under hunk-level FL and no-FL settings, as the odds ratios remain relatively higher and the FL degradation effects are not statistically significant (p > 0.05). Finding 5: Repair rates consistently decrease as FL becomes less precise across all bug complexity categories, indicating that FL precision remains important regardless of bug complexity. However, as bug complexity increases, repair effectiveness becomes less dependent on precise FL, suggesting it is more likely reliant on structural understanding and reasoning of program logic. RQ2 Summary. FL still plays a critical role in the bug repair process. More precise FL not only improves overall repair rates but also helps APR techniques generate more consistent patches. Furthermore, APR techniques with similar repair performance under the same FL strategy can exhibit substantially different effectiveness as FL becomes increasingly imprecise, resulting in progressively larger performance gaps among APR approaches. These results highlight the importance of the stability and generalizability of APR techniques for handling realistic bug-repair scenarios. Moreover, as bug structures become increasingly complex, the relative importance of FL precision decreases, indicating the growing significance of reasoning and program understanding.

4.3

APR With Advanced LLMs

ChatRepair demonstrates overall better performance than CodeCorrector in sections 4.1 and 4.2, so this study chooses ChatRepair to evaluate performance and cost-efficiency for

ESEM 2026

40:14

Rethinking Automated Program Repair

Table 9 The Advanced LLMs Performance Repaired Bugs

Attempts

Repaired Bugs / $

Attempts / $

GPT-5 GPT-5-reasoning

175/211 3/36

666 106

2.94 0.15

11.19 5.24

DeepSeek-V3.2-chat DeepSeek-V3.2-reasoner

75/211 64/136

1550 318

75.75 23.97

1565.56 119.10

DeepSeek-V4-pro DeepSeek-V4-pro-reasoner

90/211 81/121

1475 232

4.17 3.16

68.29 9.05

Llama-3.1-405B

49/211

1801

-

-

advanced LLM-based approaches. To reduce API cost, ChatRepair stops when generating the first plausible patch. Subsection 3.6 describes the experimental environment, and Table 9 presents the results of ChatRepair combined with four LLMs and two reasoning settings. Cost analysis shows that inference costs increase substantially when more advanced LLMs are adopted. DeepSeek-V3.2-chat supports approximately 1,565.56 repair attempts per dollar in non-reasoning mode and 119.10 in reasoning mode, whereas DeepSeek-V4-pro decreases to 68.29 in non-reasoning mode, and 9.05 in reasoning mode. GPT-5 supports only 11.19 attempts per dollar in non-reasoning mode and 5.24 attempts per dollar in reasoning mode. These results show that the cost increases considerably as larger and reasoning-oriented LLMs are used. Compared with the advanced LLMs, DeepSeek-V3.2 is much cheaper than GPT-5 and DeepSeek-V4-pro. Therefore, both the choice and reasoning settings of LLMs heavily impact the cost-efficiency. The performance of different LLMs varies substantially under non-reasoning and reasoning settings. Figure 6 (a) presents the number of additionally repaired bugs achieved by each LLM when reasoning mode is enabled. GPT-5 repairs 175 bugs under non-reasoning mode, achieving the highest number of repaired bugs among all evaluated models, while DeepSeek-V4-pro and DeepSeek-V3.2 repair 90 and 75 bugs, respectively. However, when the reasoning effort is increased to high, GPT-5 repairs only 3 additional bugs. In contrast, enabling high-reasoning-mode effort allows DeepSeek-V3.2 and DeepSeek-V4-pro to repair 64 and 81 additional bugs, respectively. Finding 6: Both LLM selection and reasoning settings have significant impacts on repair effectiveness and cost, while the effectiveness of reasoning strategies differs across LLM families. Enabling reasoning mode provides only limited repair improvement for GPT-5, whereas DeepSeek-V3.2 and DeepSeek-V4-pro benefit considerably more from reasoning-based settings, repairing many bugs that cannot be fixed under non-reasoning configurations. Figure 6 (b) presents the repair rates of different LLMs and reasoning settings under the cost metric of attempts per dollar (Attempts/$). Combined with Figure 6 (a), GPT-5 repairs 175 bugs under the non-reasoning setting, which is higher than all other evaluated LLMs under both reasoning and non-reasoning settings (171 for DeepSeek-V4-pro and 139 for DeepSeek-V3.2). Moreover, GPT-5 under the non-reasoning setting is even cheaper than DeepSeek-V4-pro under the reasoning setting in terms of attempts per dollar (11.19 vs. 9.05). Although GPT-5 under the non-reasoning setting is more expensive than DeepSeek-V3.2 under the reasoning setting, the repair rate is markedly higher. Figure 6 (b) shows that under the non-reasoning setting, DeepSeek-V3.2 achieves competitive repair performance compared with DeepSeek-V4-pro, while maintaining a substantially lower cost. Therefore, using DeepSeek-V3.2 in non-reasoning mode for overall bug repair, and subsequently applying GPT-5 in non-reasoning mode to repair the remaining unresolved

J. Liu et al.

40:15

1 150 81

100

175

90

50

75 49

0 5 TGP

GPT5

0.8

64

Repair Rate

Fixed Bugs Number

3

DSV4Prore 0.6 DSV4Pro

0.4

DSV3.2re DSV3.2chat

0.2

e

epS

3.2 k-V e e epS De

V4 ek-

De

ma Lla

GPT-5

GPT-5-reasoning

DeepSeek-V4-pro

DeepSeek-V4-pro-reasoner

DeepSeek-V3.2-chat

DeepSeek-V3.2-reasoner

Llama-3.1-405B

GPT5re 0

0

5 10 15

1565.56

Attempts / $ Note: re = reasoning mode. DS = DeepSeek. (b) Cost efficiency

(a) Repair Effectiveness

Figure 6 Repair effectiveness and cost efficiency of advanced LLMs.

bugs, provides a more effective trade-off between repair performance and cost. Finding 7: GPT-5 achieves the strongest overall repair effectiveness, whereas DeepSeekV3.2-chat demonstrates significantly better cost-efficiency. Combining cost-efficient LLMs with more advanced LLMs for unresolved bugs can therefore achieve a better balance between repair effectiveness and cost-efficiency in LLM-based APR techniques. Table 10 presents the Top − k metric for both APR techniques across three LLM families and three different FL strategies. The first five repair rounds usually resolve the largest proportion of bugs across evaluated APR techniques, LLMs, and FL settings. However, the marginal repair gain decreases considerably as additional repair rounds are introduced, with later rounds contributing only a limited number of newly repaired bugs. Finding 8: The majority of bugs are repaired within the first five repair rounds, whereas the marginal repair effectiveness decreases significantly in later rounds. RQ3 Summary. Using advanced LLMs and reasoning modes improves repair rates and enables APR techniques to fix more structurally complex bugs. However, these improvements come with substantially higher costs, making the trade-off between repair performance and cost increasingly important. Moreover, the benefits of reasoning modes vary across different LLM families. In our experiments, GPT-5 shows the smallest gain from reasoning, whereas the DeepSeek series repairs many additional bugs that cannot be fixed in non-reasoning mode. We further find that applying cost-efficient LLMs first, and then using advanced LLMs to repair the remaining unresolved bugs, can achieve a better balance between repair effectiveness and cost.

5

Discussion

We investigated the effectiveness of LLM-based APR under different bug complexity categories and fault localization strategies. The results reveal several insights regarding the repair effectiveness and cost-efficiency of the evaluated LLM-based APR techniques, providing useful guidance for future APR research and practical deployment. Impacts of bug complexity: Our results show that bug complexity substantially affects

ESEM 2026

40:16

Rethinking Automated Program Repair

Table 10 Repair performance (Top-k) across APR configurations. Values are reported as incremental fixes and cumulative total in parentheses. APR

LLMs

FL

Top-1

Top-2

Top-3

Top-4

Top-5

Top-10

CC

L3.1-8B 4o-mini DS-V3.2 DS-V3.2 DS-V3.2

ll ll ll hl w/o FL

8 80 191 106 80

2 (10) 23 (103) 49 (240) 32 (138) 15 (95)

1 (11) 8 (111) 18 (258) 12 (150) 6 (101)

0 (11) 6 (117) 14 (272) 12 (162) 1 (102)

0 (11) 4 (121) 6 (278) 2 (164) 2 (104)

0 (11) 5 (126) 17 (295) 15 (179) 8 (112)

CR

L3.1-8B 4o-mini DS-V3.2 DS-V3.2 DS-V3.2

ll ll ll hl w/o FL

15 67 194 93 128

5 (20) 16 (83) 30 (224) 38 (131) 30 (158)

1 (21) 9 (92) 8 (232) 22 (153) 14 (172)

9 (30) 14 (106) 19 (251) 26 (179) 11 (183)

6 (36) 9 (115) 8 (259) 16 (195) 7 (190)

14 (50) 25 (140) 33 (292) 42 (237) 15 (205)

CR adv.LLM

L3.1-405B GPT-5 DS-V3.2 DS-V4-pro GPT-5(re) DS-V3.2(re) DS-V4-pro(re)

ll ll ll ll ll ll ll

15 103 45 31 1 33 58

12 (27) 43 (146) 8 (53) 26 (57) 0 (1) 19 (52) 15 (73)

0 (27) 16 (162) 4 (57) 5 (62) 2 (3) 12 (64) 8 (81)

5 (32) 4 (166) 3 (60) 5 (67) – – –

3 (35) 5 (171) 3 (63) 7 (74) – – –

14 (49) 4 (175) 12 (75) 16 (90) – – –

Note: CC=CodeCorrector, CR=ChatRepair, L3.1 = Llama-3.1, 4o-mini = GPT-4o-mini, DS = DeepSeek, re = reasoning, w/o = without, ll = line-level FL, hl = hunk-level FL.

the effectiveness of LLM-based APR techniques. The simplest bugs achieve much higher repair rates, while structurally complex bugs containing multiple buggy lines or diverse AST edit operations remain more challenging. The repair performance across different bug complexity levels suggests that future APR evaluations could pay more attention to complex repair scenarios rather than to single-line or single-operation bugs. Structurally complex bugs may better reflect some of the challenges encountered in practical APR tasks. New benchmarks and standard performance metrics could consider evaluations across different bug categories, domains, and complexity levels to better characterize tool performance [11, 37, 40]. Impacts of fault localization: Precise FL not only improves repair effectiveness, but also helps APR techniques generate patches that are more consistent with the original program logic. Such consistency may facilitate the understanding, validation, and maintenance of repaired programs. Additionally, existing LLM-based APR methods are affected differently by less precise FL information. This difference highlights robustness to imprecise FL as an important requirement for practical APR techniques [11]. Since real-world FL is often noisy and incomplete, methods that maintain stable repair performance when the precision of FL decreases are more likely to be effective in practice. For complex bugs, the relatively smaller effect of FL precision indicates that FL may not be the only bottleneck. Repairing these bugs may also depend on reasoning ability and deeper program-understanding capabilities. The Trade-off Between Repair Effectiveness and Cost-efficiency: Using more advanced LLMs and reasoning modes substantially increases computational cost, while often improving repair effectiveness. Based on the results presented in section 4.3, GPT-5 achieves the highest repair effectiveness, while DeepSeek-V3.2-chat provides the most cost-efficient repair performance. Our results suggest that such a two-stage strategy could be a promising direction for future investigation. The marginal repair efficiency drops as the number of repair rounds increases [35, 51], which further impacts overall cost, especially when using advanced LLMs such as GPT5 and DeepSeek-V4-pro in reasoning mode. Our results also suggest that selecting an

J. Liu et al.

40:17

appropriate maximum number of repair rounds is important for balancing repair effectiveness and cost-efficiency.

6

Threats to Validity

External Validity: Our dataset is collected solely from the competitive programming platform AtCoder and focuses only on C++ programs, which are typically shorter but more algorithm-oriented [13, 25] than industrial software systems. Thus, the findings of our study may not generalize fully to real-world software projects with larger codebases, complex dependencies, and practical software engineering constraints. Furthermore, we evaluate only two APR techniques and three LLM series, and thus, the best-performing approaches identified in this study should not be regarded as definitive state-of-the-art APR solutions. The evaluated LLMs have different knowledge cutoff dates, and we collect benchmark data after 2025-10-01, which is after the knowledge cutoff date of the GPT and Llama models we used. DeepSeek did not publish its knowledge cutoff date, so it remains difficult to completely rule out the possibility that some competitive programming solutions or related code patterns were exposed during model pretraining [31, 38]. Internal Validity: We set the temperature to 1.0 for all evaluated LLMs and used high reasoning-effort settings instead of the maximum settings for GPT-5, DeepSeek-V3.2-reasoner, and DeepSeek-V4-pro due to their substantial time costs. Different parameter configurations and reasoning settings may lead to variations in repair performance. In addition, API-based LLMs may produce nondeterministic outputs across different runs, which could also affect repair results. Furthermore, different prompting strategies, repair rounds, and stopping criteria may influence the effectiveness and cost-efficiency of APR. Construct Validity: We determine repaired bugs mainly based on whether the generated patches can pass all collected test cases. However, the test cases may not be sufficiently comprehensive to fully validate the semantic correctness of repaired programs. Therefore, some plausible patches may still contain hidden defects or overfitting issues.

7

Conclusion and Future Work

This work conducts an empirical study on how bug complexity and FL affect LLM-based APR techniques. We utilize Git, srcML, and GumTreeDiff to categorize bug complexity from both hunk-line-level and AST-operation-level perspectives, and to evaluate APR effectiveness under accurate, vague, and no fault-localization settings. The experimental results show that structurally complex bugs and imprecise FL generally reduce repair effectiveness. Nevertheless, well-designed LLM-based APR techniques still maintain considerable repair effectiveness under these challenging conditions. Furthermore, different LLMs exhibit clear trade-offs between repair effectiveness and cost-efficiency, highlighting the importance of balancing repair capability, robustness, and cost for practical APR techniques. Future work may develop a benchmark that can automatically collect, filter, and categorize bug complexity. Also, we will consider practical factors such as API response latency [26, 36] and overall inference time when evaluating the efficiency of LLM-based APR systems. We plan to explore additional LLM families, such as Gemini and Claude, to further evaluate the effectiveness of LLM-based APR. Finally, the agent-based APR frameworks should be further investigated, as agent-based techniques have become increasingly popular across various software engineering and reasoning tasks in recent years.

ESEM 2026

40:18

REFERENCES

8

Data Availability

To support transparency and reproducibility, the de-identified data and analysis used in this study are available as a replication package here. Dataset: <https://zenodo.org/records/20257267>; Code: <https://github.com/Leo6-sys/Rethink-APR-Empirical-Study-2026>

References 1 Meta AI. Llama-3.1-405b. https://huggingface.co/meta-llama/Meta-Llama-3.1-405B, 2024. 2 Meta AI. Llama-3.1-8b. https://huggingface.co/meta-llama/Llama-3.1-8B, 2024. 3 Aram Bahrini, Mohammadsadra Khamoshifar, Hossein Abbasimehr, Robert J. Riggs, Maryam Esmaeili, Rastin Mastali Majdabadkohne, and Morteza Pasehvar. ChatGPT: Applications, opportunities, and threats. In 2023 Systems and Information Engineering Design Symposium (SIEDS), pages 274–279. IEEE, 2023. doi:10.1109/SIEDS58326.2023.10137850. 4 Andrew M Bean, Ryan Othniel Kearns, Angelika Romanou, Franziska Sofia Hafner, Harry Mayne, Jan Batzner, Negar Foroutan Eghlidi, Chris Schmitz, Karolina Korgul, Hunar Batra, et al. Measuring what matters: Construct validity in large language model benchmarks. Advances in Neural Information Processing Systems, 38, 2026. doi:10.52202/085713-0590. 5 Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. RepairAgent: An autonomous, llm-based agent for program repair. In Proceedings of the 47th IEEE/ACM International Conference on Software Engineering, ICSE ’25, pages 2188–2200. IEEE, 2025. doi:10.1109/ICSE55347.2025.00157. 6 Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. arXiv:2107.03374. 7 Michael L. Collard, Michael J. Decker, and Jonathan I. Maletic. srcML: An infrastructure for the exploration, analysis, and manipulation of source code: A tool demonstration. In Proceedings of the 29th IEEE International Conference on Software Maintenance, ICSM ’13, pages 516–519. IEEE, 2013. doi:10.1109/ICSM.2013.85. 8 Zhenlong Dai, Bingrui Chen, Zhuoluo Zhao, Xiu Tang, Sai Wu, Chang Yao, Zhipeng Gao, and Jingyuan Chen. Less is more: Adaptive program repair with bug localization and preference learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 128–136. AAAI Press, 2025. doi:10.1609/aaai.v39i1.31988. 9 DeepSeek-AI. Deepseek-v3.2. https://api-docs.deepseek.com/news/news251201, 2025. 10 DeepSeek-AI. Deepseek-v4. https://api-docs.deepseek.com/news/news260424, 2026. 11 Sena Dikici and Turgay Tugay Bilgin. Advancements in automated program repair: a comprehensive review. Knowledge and Information Systems, 67(6):4737–4783, 2025. 12 Jean-Rémy Falleri, Floréal Morandat, Xavier Blanc, Matias Martinez, and Martin Monperrus. Finegrained and accurate source code differencing. In Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering, ASE ’14, pages 313–324. ACM, 2014. doi:10.1145/ 2642937.2642982. 13 Claire Le Goues, Neal Holtschulte, Edward K. Smith, Yuriy Brun, Premkumar Devanbu, Stephanie Forrest, and Westley Weimer. The ManyBugs and IntroClass benchmarks for automated repair of c programs. IEEE Transactions on Software Engineering, 41(12):1236–1256, 2015. doi:10.1109/TSE. 2015.2454513. 14 Tingxu Han, Zhenting Wang, Chunrong Fang, Shiyu Zhao, Shiqing Ma, and Zhenyu Chen. Tokenbudget-aware LLM reasoning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 24842–24855. Association for Computational Linguistics, 2025. doi:10.18653/v1/2025.f indings-acl.1274. 15 Soneya Binta Hossain, Nan Jiang, Qiang Zhou, Xiaopeng Li, Wen-Hao Chiang, Yingjun Lyu, Hoan Anh Nguyen, and Omer Tripp. A deep dive into large language models for automated bug localization and repair. Proceedings of the ACM on Software Engineering, 1(FSE):1471–1493, 2024. doi:10.1145/36 60773. 16 AtCoder Inc. Atcoder. https://atcoder.jp/. 17 Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation. ACM Transactions on Software Engineering and Methodology, 35(2):1–72, 2026. doi:10.1145/3747588. 18 Matthew Jin, Syed Shahriar, Michele Tufano, Xin Shi, Shuai Lu, Neel Sundaresan, and Alexey Svyatkovskiy. InferFix: End-to-end program repair with LLMs. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE ’23, pages 1646–1656. ACM, 2023. doi:10.1145/3611643.3613892. 19 Sathvik Joel, Jie JW Wu, and Fatemeh H. Fard. A survey on LLM-based code generation for lowresource and domain-specific programming languages. ACM Transactions on Software Engineering and Methodology, 2025. doi:10.1145/3770084.

REFERENCES

40:19

20 René Just, Darioush Jalali, and Michael D. Ernst. Defects4J: A database of existing faults to enable controlled testing studies for java programs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis, ISSTA ’14, pages 437–440. ACM, 2014. doi:10.1145/2610384.2628055. 21 Fengjie Li, Jiajun Jiang, Jiajun Sun, and Hongyu Zhang. Evaluating the generalizability of llms in automated program repair. In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering: New Ideas and Emerging Results, ICSE-NIER ’25, pages 91–95. IEEE, 2025. doi:10.1109/ICSE-NIER66352.2025.00024. 22 Yingling Li, Muxin Cai, Junjie Chen, Yang Xu, Lei Huang, and Jianping Li. Context-aware prompting for llm-based program repair. Automated Software Engineering, 32(2):42, 2025. doi:10.1007/s10515 -025-00512-w. 23 Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with AlphaCode. Science, 378(6624):1092–1097, 2022. doi:10.1126/science.abq1158. 24 Bo Lin, Shangwen Wang, Ming Wen, Liqian Chen, and Xiaoguang Mao. One size does not fit all: Multi-granularity patch generation for better automated program repair. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA ’24, pages 1554–1566. ACM, 2024. doi:10.1145/3650212.3680381. 25 Derrick Lin, James Koppel, Angela Chen, and Armando Solar-Lezama. QuixBugs: A multi-lingual program repair benchmark set based on the quixey challenge. In Proceedings Companion of the 2017 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity, SPLASH Companion ’17, pages 55–56. ACM, 2017. doi:10.1145/3135932.31 35941. 26 Fan Liu, Wen-Shuo Chao, Naiqiang Tan, and Hao Liu. Bag of tricks for inference-time computation of llm reasoning. Advances in Neural Information Processing Systems, 38, 2026. 27 Matias Martinez, Thomas Durieux, Romain Sommerard, Jifeng Xuan, and Martin Monperrus. Automatic repair of real bugs in java: A large-scale experiment on the Defects4J dataset. Empirical Software Engineering, 22(4):1936–1964, 2017. doi:10.1007/s10664-016-9470-4. 28 Mary L McHugh. The chi-square test of independence. Biochemia medica, 23(2):143–149, 2013. doi:10.11613/BM.2013.018. 29 Scott Menard. Applied logistic regression analysis. SAGE publications, 2001. doi:10.4135/97814129 83433. 30 Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. A comprehensive overview of large language models. ACM Transactions on Intelligent Systems and Technology, 16(5):1–72, 2025. doi:10.1145/3744746. 31 Shiwen Ni, Xiangtao Kong, Chengming Li, Xiping Hu, Ruifeng Xu, Jia Zhu, and Min Yang. Training on the benchmark is not all you need. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 24948–24956. AAAI Press, 2025. doi:10.1609/aaai.v39i23.34678. 32 OpenAI. Gpt-4o mini. https://platform.openai.com/docs/models/gpt-4o-mini, 2024. 33 OpenAI. Gpt-5. https://platform.openai.com/docs/models/gpt-5, 2025. 34 Yicheng Ouyang, Jun Yang, and Lingming Zhang. Benchmarking automated program repair: An extensive study on both real-world and artificial bugs. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA ’24, pages 440–452. ACM, 2024. doi:10.1145/3650212.3652140. 35 Mayur Amarnath Palavalli and Mark Santolucito. Using a feedback loop for llm-based infrastructure as code generation. arXiv preprint arXiv:2411.19043, 2024. 36 Gyeongmuk Park, Seonghyeon Lee, and Yeonsu Park. Minimizing response latency in LLM-based agent systems: A comprehensive survey. IEEE Access, 14:26140–26168, 2026. doi:10.1109/ACCESS.2 026.3664226. 37 Meghana Puvvadi, Sai Kumar Arava, Adarsh Santoria, Sesha Sai Prasanna Chennupati, and Harsha Vardhan Puvvadi. Coding agents: A comprehensive survey of automated bug fixing systems and benchmarks. In 2025 IEEE 14th International Conference on Communication Systems and Network Technologies (CSNT), pages 680–686. IEEE, 2025. 38 Daniel Ramos, Claudia Mamede, Kush Jain, Paulo Canelas, Catarina Gamboa, and Claire Le Goues. Are large language models memorizing bug benchmarks? In 2025 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code), pages 1–8. IEEE, 2025. doi:10.1109/LLM4Code66 737.2025.00005. 39 Matthew Renze. The effect of sampling temperature on problem solving in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7346–7356. Association for Computational Linguistics, 2024. doi:10.18653/v1/2024.findings-emnlp.432. 40 Joseph Renzullo, Pemma Reiter, Westley Weimer, and Stephanie Forrest. Automated program repair: Emerging trends pose and expose problems for benchmarks. ACM Computing Surveys, 57(8):1–18, 2025. 41 Ripon K. Saha, Yingjun Lyu, Wing Lam, Hiroaki Yoshida, and Mukul R. Prasad. Bugs.jar: A large-scale, diverse dataset of real-world java bugs. In Proceedings of the 15th International Conference on Mining Software Repositories, pages 10–13. ACM, 2018. doi:10.1145/3196398.3196473.

ESEM 2026

40:20

REFERENCES

42 Sandeep Singh Sengar, Affan Bin Hasan, Sanjay Kumar, and Fiona Carroll. Generative artificial intelligence: A systematic review and applications. Multimedia Tools and Applications, 84(21):23661– 23700, 2025. doi:10.1007/s11042-024-20016-1. 43 Yewei Song, Cedric Lothritz, Xunzhu Tang, Tegawendé Bissyandé, and Jacques Klein. Revisiting code similarity evaluation with abstract syntax tree edit distance. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 38–46. Association for Computational Linguistics, 2024. doi:10.18653/v1/2024.acl-short.3. 44 Diomidis Spinellis. Git. IEEE Software, 29(3):100–101, 2012. doi:10.1109/MS.2012.61. 45 Chi Wang, Xueqing Liu, and Ahmed Hassan Awadallah. Cost-effective hyperparameter optimization for large language model generation inference. In Proceedings of the Second International Conference on Automated Machine Learning, volume 224 of Proceedings of Machine Learning Research, pages 21/1–17. PMLR, 2023. URL: https://proceedings.mlr.press/v224/wang23b.html. 46 Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022. URL: https://proceedings.neurips.cc/p aper_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html. 47 Chunqiu Steven Xia and Lingming Zhang. Automated program repair via conversation: Fixing 162 out of 337 bugs for $0.42 each using chatgpt. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA ’24, pages 819–831. ACM, 2024. doi:10.1145/3650212.3680323. 48 Fengli Xu, Qianyue Hao, Chenyang Shao, Zefang Zong, Yu Li, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, et al. Toward large reasoning models: A survey of reinforced reasoning with large language models. Patterns, 6(10):101370, 2025. doi:10.1016/j.patter.2025.10 1370. 49 Xin Yin, Chao Ni, Shaohua Wang, Zhenhao Li, Limin Zeng, and Xiaohu Yang. ThinkRepair: Self-directed automated program repair. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA ’24, pages 1274–1286. ACM, 2024. doi: 10.1145/3650212.3680359. 50 Murong Yue, Jie Zhao, Min Zhang, Liang Du, and Ziyu Yao. Large language model cascades with mixture of thought representations for cost-efficient reasoning. In The Twelfth International Conference on Learning Representations, 2024. URL: https://openreview.net/forum?id=6okaSfANzh. 51 Yuze Zhao, Zhenya Huang, Yixiao Ma, Rui Li, Kai Zhang, Hao Jiang, Qi Liu, Linbo Zhu, and Yu Su. Repair: Automated program repair with process-based feedback. In Findings of the Association for Computational Linguistics: ACL 2024, pages 16415–16429, 2024. 52 Xin Zhou, Bowen Xu, Kisub Kim, DongGyun Han, Hung Huu Nguyen, Thanh Le-Cong, Junda He, Bach Le, and David Lo. Leveraging large language model for automatic patch correctness assessment. IEEE Transactions on Software Engineering, 50(11):2865–2883, 2024. doi:10.1109/TSE.2024.3452252. 53 Fida Zubair, Maryam Al-Hitmi, and Cagatay Catal. The use of large language models for program repair. Computer Standards & Interfaces, 93:103951, April 2025. doi:10.1016/j.csi.2024.103951.

Related documents

• Experimental Settings in LLM-Based Program Repair: A Study of Inputs, Tool Access, Feedback, and Validation#1011704
• Experimental Settings in LLM-Based Program Repair: A Study of Inputs, Tool Access, Feedback, and Validation#965480
• Rethinking Automated Program Repair: The Impact of Bug Complexity, Fault Localization, and LLM Cost-efficiency#615216
• Rethinking Automated Program Repair: The Impact of Bug Complexity, Fault Localization, and LLM Cost-efficiency#614249
• If It's Not Buggy, Don't Fix It: On the Dynamics of Iterative Bug-fixing with LLMs#673594