Is Three the Magic Number? An Empirical Evaluation of LLM-Based Repair Loops Tobias Kiecker
Eik Reichmann
Hosung Kang
Gabin An
Lars Grunske
HumboldtUniversität zu Berlin Berlin, Germany
HumboldtUniversität zu Berlin Berlin, Germany
Korea University Seoul, Korea
Korea University Seoul, Korea
HumboldtUniversität zu Berlin Berlin, Germany
arXiv:2607.05197v1 [cs.SE] 6 Jul 2026
Abstract Iterative repair loops have become a core design pattern in LLMbased software engineering systems. These workflows repeatedly generate, validate, and repair artifacts using feedback such as compiler errors or test failures. Despite their widespread use, the impact of repair-loop iteration limits remains poorly understood, as most prior work adopts fixed, often arbitrary, repair budgets. We study repair-loop effectiveness across multiple software engineering tasks, including code generation, test generation, and code translation. Across several representative workflows, datasets, and contemporary low-cost LLMs, we observe a consistent pattern of diminishing returns: the first three to four repair iterations account for most achievable gains, while later iterations contribute only marginal improvements. We further find that repair behavior is influenced more strongly by workflow orchestration and feedback design than by the underlying model itself. These results suggest that repair budgets should be treated as an explicit experimental variable, as they directly affect evaluation outcomes, computational cost, runtime, and reproducibility in LLM-based software engineering research.
1
Introduction
Large Language Models (LLMs) are far from perfect. Nevertheless, they are increasingly used in software engineering (SE) research and practice [5]. A common pattern when using LLMs for programming tasks is iterative refinement: users generate a solution, validate the output, identify remaining issues such as compiler errors, failing test cases, or missing edge cases, and feed these back to the model for repair. This process is repeated until the generated artifact either satisfies the requirements or the user abandons the attempt and reformulates the prompt. This iterative repair behavior has also been adopted by many LLM-based SE tools [9–14, 18, 20, 21, 24]. Instead of relying on manual feedback, such systems automatically validate generated artifacts using mechanisms such as compilation, static analysis, or test execution. When validation fails, the observed errors are provided to the model in order to generate a repaired version automatically. To prevent infinite repair cycles or oscillating behaviors in which one repair introduces new faults, these systems typically enforce an arbitrary fixed iteration limit [16, 23]. Furthermore, with the emergence of autonomous LLM agents [2, 19, 22] repair loops are becoming even more relevant. Such agents internally perform iterative cycles of tool calls, generation, validation, testing, and repair in order to accomplish complex SE tasks. As agent-based workflows become increasingly common, the choice of repair-loop limits directly affects the practicality and efficiency of these systems.
wrong solution + feedback
✘
LLM
? Candidate
Input
repair limit not reached not valid
Validator
repair limit reached
valid
Error
Solution
✓
Figure 1: Generalized SE workflow with LLM-repair loop
Choosing the right limit for repair loop iterations is important and impactful for several reasons. First, each additional repair attempt increases API usage costs and energy consumption, contributing both to higher economic as well as environmental impact [4]. Second, repeated inference calls increase runtime which can lead to slower development workflows. Third, iteration limits directly influence reproducibility and comparability across studies. Different repair budgets may lead to substantially different performance outcomes, making it difficult to fairly compare approaches when iteration limits are selected arbitrarily or are insufficiently documented [6]. Despite their importance, iteration limits are rarely studied systematically. Existing work often selects a small fixed number of repair attempts without providing empirical justification for this choice [10, 11, 23]. To the best of our knowledge, no prior work systematically studies the impact of repair-loop iteration limits across SE tasks and LLMs. In this work, we investigate the effectiveness of repair loops across multiple SE tasks, including code generation, test generation, and code translation. Using several low-cost LLMs and representative LLM-based SE tools, we analyze whether the commonly used limit of three or five repair attempts is sufficient or whether meaningful improvements continue to occur beyond this threshold. Our findings aim to contribute to a better understanding of how repair budgets should be selected and reported in future LLM-based SE research and systems. Our central question is: Are 3 repair attempts enough for LLM-based SE tools?
2
Repair Loops in LLM-Based SE-Tools
We define a repair-loop-based system as any LLM-based software engineering workflow, tool, or approach that iteratively generates,
Pre-Print, arXiv, Germany
validates, and repairs artifacts until either a correct solution is produced or a predefined iteration limit is reached. Figure 1 illustrates the general structure of such an approach. These systems typically receive input in the form of task-specific context and an initial generation prompt. For example, a system may receive a textual specification and is prompted to generate source code, or receive source code and is prompted to generate corresponding test cases. Based on this input, the LLM generates an initial candidate artifact. The generated output is then validated using task-dependent mechanisms. In code generation scenarios, this may involve compilation, static analysis, or test execution. In test-generation tasks, generated tests may be checked for syntactic validity, coverage, or execution behavior. If the produced artifact satisfies the validation criteria, the workflow terminates successfully and returns the generated output. Otherwise, the system extracts feedback from the failed validation step and provides this information back to the LLM to generate a repaired version of the artifact. The repaired artifact is then validated again, resulting in an iterative generate → validate → repair cycle. This process continues until either a valid artifact is produced or a predefined iteration limit is reached and an error is returned. In practice these iteration limits are commonly implemented implicitly through bounded loops or explicitly through retry counters within the orchestration logic of the system. Despite being a central configuration parameter, these limits are often selected heuristically and rarely justified empirically [11, 13, 18]. In the context of this work, repair generally refers to feedbackdriven regeneration conditioned on previous validation failures and we define a repair step as one iteration of the repair loop following a failed validation attempt.
3
Empirical Study
To answer our posed question we conduct an empirical study across a set of tools with different tasks, datasets, and programming languages. The tasks span code generation, test-case generation and code translation. To ensure that any observed results are not due to a particular model’s behavior we select three models: Google’s Gemma-41 , Qwen3.52 and OpenAI’s gpt-4o-mini3 . We study four existing repair-based tools and a baseline that simulates human-style iterative chat interactions solving coding problems. The tools include INTERVENOR [18], Large Language Model Debugger (LDB) [24], from which we use the code translation component, CASCADE [9] and OpenCodeInterpreter (OCI) [23] in two versions. First, the original version where the initial prompt contains all test-cases that should pass and a second version without test-cases in the initial prompt. Table 1 lists the tools and their datasets. Each datasets was also used in the tools original papers. For the baseline (Basic LLM Call) we use Defects4J [7]. To enable a comparison between the tools, we made two types of modifications. First, all tools were adapted to support the backbone LLMs that were hosted on a local server (Gemma-4 and
1 huggingface.co/google/gemma-4-31B-it 2 huggingface.co/Qwen/Qwen3.5-27B 3 platform.openai.com/docs/models/gpt-4o-mini
Tobias Kiecker, Eik Reichmann, Hosung Kang, Gabin An, and Lars Grunske
Table 1: Tools, datasets, and tasks used in our empirical study. Tool
Dataset
Task
Basic LLM Call OCI [23] INTERVENOR [18] CASCADE [9] LDB [24]
Defects4J [7] MBPP [1] HumanEval [3] CASCADE [9] TransCoder [15]
Test Generation Code Generation Code Generation Test Generation Code Translation
Qwen3.5). For INTERVENOR this entailed a change from the OpenAI completions to the chat.completions API to enable the usage of the contemporary models since the completions API is deprecated. OCI was restructured to use a multi-turn conversation format rather than rebuilding the prompt from scratch each iteration, as in the original. These changes slightly alter the semantics of OCI and INTERVENOR, however, the change to the API call would be needed anyways to use contemporary models and were not specific to our study. Second modification was to LDB, where the original implementation only evaluated correctness when the final repair loop is finished. We added per-iteration evaluation against the full held-out test suite at every step. For CASCADE, whose primary purpose is detecting inconsistent documentation-method pairs, we extracted only its test-case generation stage and discarded the later inconsistency analysis stages, since the initial test-generation component is the only part that implements a complete iterative repair loop suitable for our study. Crucially, the repair loop control logic and correctness criteria remained unchanged in all tools. The only changes were with respect to measurement and model compatibility. Furthermore, we fixed the temperature for all experiments at 0.2. For each tool with every backbone LLM we compute the percentage of completed tasks on the respective dataset after the initial generation, step 0, and at each subsequent repair step up to a maximum of ten repairs. Those results are shown in Figure 2, where each panel corresponds to one tool and each curve to a model. Across all tools the curves share a consistent concave shape. From the initial generation to the first few repair steps the completion rate rises steeply and then flattens, with each additional step resulting in smaller improvements than the last. Table 2 shows this in concrete numbers, where we calculated the average relative improvements from each repair step to the next. Notably, the largest relative improvement occurs at step 1 and that by step 3 the marginal gains have dropped to single digit percentages or below. This pattern holds across all three backbone models and all tasks including outliers such as Qwen on LDB starting with only 30% completion rate and then improving drastically. This suggests that diminishing returns are a recurring characteristic of the evaluated repair-loop workflows. When examining where tools differ, the main distinction lies in how quickly the return on repair diminishes. OCI shows only marginal improvements after the first repair step, while the Basic LLM Call and CASCADE are still rising at step 3. Furthermore, within each panel, models may start at different absolute levels but follow the same curve shape. The gap between models is largest at step 0 and narrows with each repair step, visible most clearly in LDB
Is Three the Magic Number? An Empirical Evaluation of LLM-Based Repair Loops
Basic LLM Call - Defects4J
70
Pre-Print, arXiv, Germany
OCI(w/out tests) - MBPP
100
INTERVENOR - HumanEval 95
50 40 30 20 10
% of correct code
80
% of correct code
% of passing tests
60
60 40 20
85 80 75 70 65
0 0
1
2
3
4
5
6
7
8
9
10
0
1
2
# of repair steps
3
4
5
6
7
8
9
10
50
45 3
4
5
6
7
8
9
4
5
6
7
8
9
10
9
10
LDB - TransCoder
96 94 92 90
10
3
90 80 70 60 50 40 30
88 2
2
# of repair steps
% of correct code
% of correct code
55
1
1
OCI - MBPP
98
60
0
0
# of repair steps
CASCADE % of compilable tests
90
0
1
2
# of repair steps
3
4
5
6
7
8
9
10
0
1
2
# of repair steps
Models google/gemma-4-31B-it
gpt-4o-mini
3
4
5
6
7
8
# of repair steps
Qwen/Qwen3.5-27B
Figure 2: Completion rate as a function of repair steps across six tool–dataset combinations and three LLM backbones. where Qwen starts near 30% and Gemma near 85% yet both curves flatten at roughly the same point. Furthermore, we can observe that each tool, independent of the model backbone, provides a different performance ceiling and a different rate at which repairs approach it.
4
Discussion
The first three to four repair steps contribute the vast majority of the total achievable gain. Following iterations can still repair additional cases, however, improvements drop to near zero shortly after steps 5 to 7 across all evaluated tasks and tools. Hence we argue that the costs that come with these improvements require a careful trade-off analysis with respect to parameters such as API costs and energy usage. This consideration is increasingly relevant for agent systems, where repair loops are often deeply integrated Table 2: Mean relative improvement per repair step.
1 Basic LLM Call INTERVENOR CASCADE OCI OCI (w/out tests) LDB
266.7 6.7 14.3 4.6 871.4 31.7
Percentage increase to previous step 2 3 4 5 6 7 8 92.2 1.7 3.1 0.7 11.0 1.5
11.5 1.2 3.0 0.3 2.8 1.2
12.7 1.9 0.4 0.3 1.0 0.5
5.9 0.2 0.0 0.0 0.8 0.3
2.0 0.7 0.0 0.1 0.4 0.2
1.0 0.7 0.0 0.1 0.0 0.1
1.5 0.7 0.4 0.0 0.2 0.1
9
10
1.0 0.0 0.4 0.0 0.0 0.0
0.0 0.2 0.0 0.0 0.1 0.0
into the execution workflow and may trigger multiple expensive inference calls per task or even result in infinite loops. Interestingly, our results indicate that factors such as feedback design and content, orchestration logic, and the validation strategy substantially influence how effective additional repair attempts are at improving generated artifacts. This comes from the observation that completion curves for the same tool have similar shapes across all models, even when the absolute performance levels differ. The comparison between the two OCI variants particularly highlights this effect. While both variants share nearly identical repair behavior in later iterations, they differ substantially in their initial success rates. The primary difference between the two approaches is that one variant already includes tests during the initial generation step, whereas the other only incorporates the output of the test suite as repair feedback during subsequent iterations. Despite this relatively small architectural difference, the resulting repair trajectories differ noticeably during the early repair stages before converging toward similar trends. At the same time, model-specific performance characteristics remain visible across tools. For almost all evaluated workflows, the Gemma model achieves the strongest overall performance, while the older GPT variant generally performs worst. However, CASCADE represents a notable exception to this trend. One possible explanation is that CASCADE was originally evaluated and probably optimized around the specific output of GPT-based models. This
Pre-Print, arXiv, Germany
observation further reinforces that the interaction between tool repair logic and model behavior needs to be examined carefully. Furthermore, our replication effort revealed that many existing LLM-based SE tools are not easily transferable across models. In practice, substantial modifications to prompts, parsing logic, validation mechanisms, or orchestration strategies are often required when replacing the originally intended backbone model. This raises broader concerns regarding comparability and reproducibility in LLM-based SE research. While a detailed investigation of reproducibility challenges is outside the scope of this work, our findings indicate that LLM tools and specifically repair-loop behavior cannot always be studied independently of the surrounding tool architecture and model assumptions and therefore cannot always be compared directly.
5
Threats to Validity
Internal Validity. Our study does not use the original models employed by the evaluated tools and prior work. In addition, due to computational and financial constraints, we do not perform repeated executions for every experimental configuration. Since LLMs exhibit non-deterministic behavior, repeated runs may produce variance in absolute performance values. However, the goal of this work is not to reproduce exact performance numbers, but to analyze repair-loop trends across different iteration limits. We therefore use the same set of models consistently across all evaluated tools and tasks. Furthermore, the evaluation generally relies on task-specific automated validation procedures such as compilation and test execution. While these mechanisms provide scalable correctness signals, they may not fully capture semantic correctness or overall software quality, however, they ensure consistency with existing evaluation methodologies. External Validity. Our experiments focus primarily on cost-efficient LLMs, and the observed results may therefore not generalize directly to larger, newer models. More capable models may require fewer repair attempts due to stronger reasoning and generation capabilities. Furthermore, the selected tasks may not represent the full spectrum of repair-loop-based SE workflows. Nevertheless, we evaluate three substantially different SE tasks across five datasets, providing diversity in both generation objectives and validation mechanisms. Code generation in particular represents one of the most widely studied and practically relevant applications of LLMs in SE, increasing the practical relevance of the evaluated tools and benchmarks. The consistency of observed trends across multiple tasks and datasets partially mitigates this threat even further.
6
Related Work
Prior work on LLM-based repair can be categorized into three types. The first category is feedback driven generation that only uses an LLM and no information from the external executions [10, 20]. The next category characterizes tools that utilize some form of execution based feedback [11], including runtime and compilation errors [13, 16, 23], natural language [14] and execution graphs [24]. Another category distributes the repair process across multiple agents or structured components. INTERVENOR [18] separates a
Tobias Kiecker, Eik Reichmann, Hosung Kang, Gabin An, and Lars Grunske
Code Teacher, which constructs a Chain-of-Repair from compiler feedback, from a Code Learner, which executes the revisions, within three turns. PairCoder [21] combines planning and implementation agents with execution-guided repair within ten iterations. Finally, our work is also related to empirical studies of repair and self-correction. Critical surveys note that self-correction effectiveness depends strongly on the feedback source and evaluation design, and that unclear protocols can overestimate gains [8]. Closest to our motivation, Olausson et al. [12] compare repair with drawing additional independent samples under comparable compute budgets, while Tang et al. [17] frame repair as an exploration–exploitation tradeoff. In contrast, we ask a complementary question: Given that many LLM-based SE tools already implement repair loops, how much does each additional iteration help? We therefore measure per-iteration completion rates across existing tools, tasks, datasets, and model backbones, shifting the focus from whether to use repair to how iteration limits should be selected, justified, and reported.
7
Conclusion
Repair loops have become a fundamental component of modern LLM-based software engineering systems, ranging from code-generation tools to autonomous agent workflows. Despite their widespread use, the choice of repair-loop iteration limits is often treated as an arbitrary implementation detail and rarely justified empirically. In this work, we conducted an empirical study across multiple repair-loop-based software engineering tools, tasks, datasets, and backbone models to analyze the effectiveness of additional repair iterations. Across all evaluated settings, we observe a consistent pattern of diminishing returns: the majority of achievable gains are obtained within the first three to four repair steps, while later iterations can still contribute marginal improvements. Our results suggest that repair-loop behavior depends more strongly on orchestration logic, validation strategy, and feedback design than on the selected backbone model itself. At the same time, the observed interaction effects between tools and models indicate broader reproducibility and comparability challenges in LLM-based software engineering research. Overall, our findings suggest that repair budgets should be explicitly reported and justified in future work, as they directly influence evaluation outcomes, computational cost, runtime, and environmental impact. As repair loops become increasingly integrated into autonomous agent systems, understanding and standardizing repair-loop behavior will become even more important for the development of reliable and reproducible LLM-based software engineering workflows. For future work we plan to extend on our results by adding more tools, models and conducting a larger scale reproducibility study for LLM-based SE tools.
Is Three the Magic Number? An Empirical Evaluation of LLM-Based Repair Loops
References [1] Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. CoRR abs/2108.07732 (2021). arXiv:2108.07732 https://arxiv.org/abs/2108.07732 [2] Islem Bouzenia, Premkumar T. Devanbu, and Michael Pradel. 2025. RepairAgent: An Autonomous, LLM-Based Agent for Program Repair. In 47th IEEE/ACM International Conference on Software Engineering, ICSE 2025, Ottawa, ON, Canada, April 26 - May 6, 2025. IEEE, 2188–2200. doi:10.1109/ICSE55347.2025.00157 [3] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. CoRR abs/2107.03374 (2021). arXiv:2107.03374 https://arxiv.org/abs/2107.03374 [4] Kuen Sum Cheung, Mayuri Kaul, Gunel Jahangirova, Mohammad Reza Mousavi, and Eric Zie. 2025. Comparative Analysis of Carbon Footprint in Manual vs. LLM-Assisted Code Development. In Proceedings of the 1st International Workshop on Responsible Software Engineering, ResponsibleSE 2025, Trondheim, Norway, June 23-28, 2025. ACM, 13–20. doi:10.1145/3711919.3728678 [5] Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang. 2023. Large language models for software engineering: Survey and open problems. In 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE). IEEE, 31–53. [6] Cuiyun Gao, Xing Hu, Shan Gao, Xin Xia, and Zhi Jin. 2025. The Current Challenges of Software Engineering in the Era of Large Language Models. ACM Trans. Softw. Eng. Methodol. 34, 5 (2025), 127:1–127:30. doi:10.1145/3712005 [7] René Just, Darioush Jalali, and Michael D. Ernst. 2014. Defects4J: a database of existing faults to enable controlled testing studies for Java programs. In International Symposium on Software Testing and Analysis, ISSTA ’14, San Jose, CA, USA July 21 - 26, 2014, Corina S. Pasareanu and Darko Marinov (Eds.). ACM, 437–440. doi:10.1145/2610384.2628055 [8] Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. 2024. When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of SelfCorrection of LLMs. Trans. Assoc. Comput. Linguistics 12 (2024), 1417–1440. doi:10.1162/TACL_A_00713 [9] Tobias Kiecker, Jan Arne Sparka, Martin Reuter, Albert Ziegler, and Lars Grunske. 2026. Cascade: Detecting Inconsistencies between Code and Documentation with Automatic Test Generation. Proceedings of the ACM on Software Engineering 3, FSE, Article FSE168 (July 2026), 23 pages. doi:10.1145/3808175 [10] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. SELF-REFINE: Iterative Refinement with Self-Feedback. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NeurIPS 2023, New Orleans, LA, USA, December 10-16, 2023 (NIPS ’23). Curran Associates Inc., Red Hook, NY, USA, 46534–46594. doi:10.5555/3666122.3668141 [11] Thu-Trang Nguyen, Thanh Trong Vu, Hieu Dinh Vo, and Son Nguyen. 2025. An empirical study on capability of Large Language Models in understanding code semantics. Information and Software Technology 185 (2025), 107780. doi:10.1016/j. infsof.2025.107780 [12] Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. 2024. Is Self-Repair a Silver Bullet for Code Generation?. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. https://openreview.net/forum? id=y0GJXRungR [13] Rangeet Pan, Myeongsoo Kim, Rahul Krishna, Raju Pavuluri, and Saurabh Sinha. 2025. ASTER: Natural and Multi-Language Unit Test Generation with LLMs. In 47th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice, SEIP@ICSE 2025, Ottawa, ON, Canada, April 27 - May 3, 2025 (ICSE-SEIP 2025). IEEE Computer Society, Los Alamitos, CA, USA, 413–424. doi:10.1109/ICSE-SEIP66354.2025.00042 [14] Tal Ridnik, Dedy Kredo, and Itamar Friedman. 2024. Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering. CoRR abs/2401.08500 (2024). arXiv:2401.08500 doi:10.48550/ARXIV.2401.08500 [15] Baptiste Rozière, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. 2020. Unsupervised Translation of Programming Languages. In Advances
Pre-Print, arXiv, Germany
in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.). https://proceedings.neurips.cc/paper/2020/hash/ ed23fbf18c2cd35f8c7f8de44f85c08d-Abstract.html [16] Arkadii Sapozhnikov, Mitchell Olsthoorn, Annibale Panichella, Vladimir Kovalenko, and Pouria Derakhshanfar. 2024. TestSpark: IntelliJ IDEA’s Ultimate Test Generation Companion. In Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings, ICSE-Companion 2024, Lisbon, Portugal, April 14-20, 2024 (ICSE-Companion ’24). Association for Computing Machinery, New York, NY, USA, 30–34. doi:10.1145/3639478.3640024 [17] Hao Tang, Keya Hu, Jin Zhou, Sicheng Zhong, Wei-Long Zheng, Xujie Si, and Kevin Ellis. 2024. Code Repair with LLMs gives an ExplorationExploitation Tradeoff. (2024). http://papers.nips.cc/paper_files/paper/2024/ hash/d5c56ec4f69c9a473089b16000d3f8cd-Abstract-Conference.html [18] Hanbin Wang, Zhenghao Liu, Shuo Wang, Ganqu Cui, Ning Ding, Zhiyuan Liu, and Ge Yu. 2024. INTERVENOR: Prompting the Coding Ability of Large Language Models with the Interactive Chain of Repair. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024 (Findings of ACL), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, 2081–2107. doi:10. 18653/V1/2024.FINDINGS-ACL.124 [19] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. (2024). http://papers.nips. cc/paper_files/paper/2024/hash/5a7c947568c1b1328ccc5230172e1e7c-AbstractConference.html [20] Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 2471–2484. doi:10.18653/v1/2023.emnlp-main.151 [21] Huan Zhang, Wei Cheng, Yuhan Wu, and Wei Hu. 2024. A Pair Programming Framework for Code Generation via Multi-Plan Exploration and Feedback-Driven Refinement. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ASE 2024, Sacramento, CA, USA, October 27 November 1, 2024 (ASE ’24). Association for Computing Machinery, New York, NY, USA, 1319–1331. doi:10.1145/3691620.3695506 [22] Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. AutoCodeRover: Autonomous Program Improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2024, Vienna, Austria, September 16-20, 2024, Maria Christakis and Michael Pradel (Eds.). ACM, 1592–1604. doi:10.1145/3650212.3680384 [23] Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. 2024. OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024 (Findings of ACL), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, 12834–12859. doi:10. 18653/V1/2024.FINDINGS-ACL.762 [24] Li Zhong, Zilong Wang, and Jingbo Shang. 2024. Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step by Step. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024 (Findings of ACL), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, 851–870. doi:10.18653/V1/2024.FINDINGS-ACL.49