How Many Tries Does It Take? Iterative Self-Repair in LLM Code Generation Across Model Scales and Benchmarks Johin Johny Arimbur
arXiv:2604.10508v1 [cs.SE] 12 Apr 2026
Independent Researcher [email protected] Abstract—Large language models frequently fail to produce correct code on their first attempt, yet most benchmarks evaluate them in a single-shot setting. We investigate iterative self-repair (feeding execution errors back to the model for correction) across seven models spanning three families and both openweight and proprietary providers: Llama 3.1 8B, Llama 3.3 70B, Llama 4 Scout (MoE, 16 experts), Llama 4 Maverick (MoE, 128 experts), Qwen3 32B, Gemini 2.5 Flash, and Gemini 2.5 Pro. On HumanEval (164 problems) and MBPP Sanitized (257 problems) with up to five attempts, self-repair universally improves pass rates: +4.9 to +17.1 pp on HumanEval and +16.0 to +30.0 pp on MBPP. Gemini 2.5 Flash achieves the highest final pass rates (96.3% HumanEval, 93.8% MBPP). Most gains concentrate in the first two rounds. Error-type analysis shows assertion errors (logical mistakes) are the hardest to repair at ∼45%, while syntax and name errors are repaired at substantially higher rates, connecting to broader findings on the limits of LLM selfcorrection. Prior work found that weaker models fail at self-repair or require fine-tuning; we show that modern instruction-tuned models succeed with prompting alone, even at 8B scale. We also provide the first comparison of dense and MoE architectures for self-repair, and extend the repair-vs-resampling tradeoff analysis to modern models. A prompt ablation reveals chain-of-thought repair yields up to +5.5 pp additional self-repair gain (measured as improvement in repair delta) over minimal prompting for capable models. Index Terms—large language models, code generation, selfrepair, iterative refinement, HumanEval, MBPP, self-debugging
I. I NTRODUCTION The ability of large language models (LLMs) to generate functionally correct code from natural-language specifications has improved dramatically in recent years [1]–[3]. Benchmarks such as HumanEval [1] and MBPP [4] have become standard yardsticks, typically reporting a pass@1 metric: the fraction of problems solved correctly on a single attempt. While informative, this single-shot evaluation protocol diverges sharply from how human programmers actually write code. A professional developer routinely writes an initial draft, runs it, reads the error message, and iterates, often multiple times, before arriving at a correct solution. Iterative self-repair replicates this workflow within an LLM pipeline. When a generated program fails its test suite, the model receives the error traceback and is asked to produce a corrected version. This cycle can repeat for a fixed number of rounds. Prior work has explored self-repair with proprietary
models such as GPT-3.5 and GPT-4 [5], [6], yielding mixed conclusions about its effectiveness. Olausson et al. [5] notably argued that self-repair is “not a silver bullet,” observing that weaker models may introduce new errors during repair attempts. However, the landscape has changed significantly since those studies. A new generation of open-weight models (2024–2025) exhibits substantially stronger instruction-following and errorcomprehension capabilities, and novel architectures such as mixture-of-experts (MoE) have become prevalent. No prior work has evaluated self-repair on these modern models or compared dense vs. MoE architectures for repair. These developments motivate a fresh empirical investigation: does self-repair now work universally, even for smaller models? How do dense and MoE architectures differ in their repair dynamics? And what practical guidelines can we derive for deployment? In this paper, we address these questions with a systematic evaluation across seven models from three families: Llama 3.1 8B (8B dense) [7], Llama 3.3 70B (70B dense) [8], Llama 4 Scout 17B (17B active, MoE with 16 experts) [9], Llama 4 Maverick 17B (17B active, MoE with 128 experts) [9], Qwen3 32B (32B dense) [10], Gemini 2.5 Flash [11], and Gemini 2.5 Pro [11]. The five open-weight models are accessed via the Groq free-tier API; the two Gemini models are accessed via Google Cloud Vertex AI. All use greedy decoding (temperature = 0.0) for reproducibility. We evaluate on two benchmarks, HumanEval (164 problems) and MBPP Sanitized (257 problems), with up to five attempts per problem. Our contributions are as follows: 1) We demonstrate that prompt-based self-repair is now effective across seven models from three families, even at 8B scale, updating the finding by Olausson et al. [5] that weaker models are harmed by self-repair, and contrasting with Chen et al. [12] who found limited gains for a 7B model. Prior work required fine-tuning to enable small-model self-repair [13], [14]; we show that modern instruction-tuned models succeed without it. 2) We present, to our knowledge, the first direct comparison of dense and mixture-of-experts architectures for selfrepair across multiple model families, revealing distinct interaction patterns: on HumanEval, Scout (16 experts) achieves the highest repair gain among open-weight
models (+14.0 pp), while the Gemini models establish new ceiling performance levels. 3) We provide cross-benchmark validation on both HumanEval and MBPP Sanitized, showing consistent model rankings and diminishing-returns patterns, strengthening generalizability beyond single-benchmark studies. 4) We present fine-grained error-type analysis revealing that error category is a strong predictor of repair success: name errors are repaired at ∼77% rates, while assertion errors succeed at only ∼45%, connecting to the broader literature on LLM self-correction limits [15]. 5) We quantify the diminishing-returns curve and derive practical deployment guidelines: two repair rounds capture the majority (76–95%) of achievable gains. Our token-efficiency comparison against independent resampling extends the exploration-exploitation framework of Tang et al. [16] to modern models, confirming that selfrepair is increasingly advantageous for capable models. 6) We conduct a repair prompt ablation comparing minimal, explain-then-fix, and chain-of-thought strategies, finding that CoT prompting yields +5.5 pp additional repair gain over minimal prompting for the 70B model.
C. Iterative Refinement Beyond Code The principle of iterative self-improvement has been explored in broader contexts. Madaan et al. [20] introduced Self-Refine, a general framework in which LLMs iteratively critique and revise their own outputs across diverse tasks. Shinn et al. [21] proposed Reflexion, using verbal reinforcement signals to improve agent performance over multiple episodes. However, Huang et al. [15] cautioned that LLMs cannot reliably selfcorrect reasoning without external feedback, an observation consistent with our finding that assertion errors (which require reasoning corrections) are the hardest to repair. Tang et al. [16] formalized the repair-vs-resampling decision as an explorationexploitation tradeoff, proposing a Thompson Sampling strategy (REx) that dynamically allocates between repair and resampling. Our resampling comparison extends this line of work with modern 2024–2025 models, confirming that the tradeoff is model-dependent. D. Multi-Turn and Collaborative Code Generation
Recent work has also investigated multi-turn interactions for code generation. Key et al. [22] explored multi-turn code generation with instruction-tuned models, while Zhang et al. [23] investigated self-collaboration patterns with ChatGPT. II. R ELATED W ORK Zhong et al. [24] proposed LDB, a debugger that verifies A. Code Generation with LLMs runtime execution step-by-step. SelfEvolve [25] used LLMs to The emergence of Codex [1] demonstrated that LLMs fine- iteratively evolve code through self-generated feedback. Our tuned on code repositories could generate functionally correct work differs in its focus on the simplest possible self-repair programs from docstrings. Subsequent work scaled this capa- protocol (feeding back raw error messages) and its systematic bility through larger models [2], code-specific pre-training [3], comparison across seven models from three families and two and reinforcement learning from execution feedback [17]. The benchmarks. HumanEval benchmark [1] and the Mostly Basic Programming III. M ETHODOLOGY Problems (MBPP) dataset [4] have become standard evaluation suites, while more challenging benchmarks such as APPS [18] A. Self-Repair Protocol and SPoC [19] test competition-level reasoning. Our iterative self-repair protocol operates as follows. Given a programming problem specified by a function signature and B. Self-Debugging and Self-Repair Chen et al. [6] proposed Self-Debug, demonstrating that docstring (as in HumanEval and MBPP), the model generates LLMs can identify and fix bugs in their own code when an initial code solution at round R0 . This solution is extracted provided with execution results. Their approach included from the model’s response, combined with the corresponding several feedback strategies: simple error messages, code test cases, and executed in a sandboxed Python environment. explanations, and unit test traces. Olausson et al. [5] conducted If all test cases pass, the problem is marked as solved. If a thorough investigation of self-repair, concluding that while execution produces an error, we capture the error type and GPT-4 benefited substantially, weaker models like GPT-3.5 traceback message, then construct a repair prompt containing: often failed to improve or even degraded in performance. More (1) the original problem specification, (2) the model’s previous recently, Chen et al. [12] revisited self-debugging with self- code attempt, and (3) the error message. The model is then generated tests, finding that even a 7B model (Qwen2.5-Coder) asked to generate a corrected solution. This cycle repeats for showed limited or negative gains from prompt-based self- up to four repair rounds (R1 through R4 ), for a maximum of debugging on HumanEval. Two concurrent lines of work have five total attempts per problem. Formally, for a problem p and model M , let c0 = M (p) be shown that fine-tuning can unlock self-repair for smaller models: the initial code generation. If exec(c0 ) succeeds, we record CYCLE [13] trained 350M–3B models to self-refine through success at R0 . Otherwise, let e0 = error(c0 ) be the captured iterative code generation, and LeDex [14] used supervised fineerror. For each subsequent round i ∈ {1, 2, 3, 4}: tuning and reinforcement learning to enable self-debugging in CodeLlama-7B/13B. Our work differs from all of these in demonstrating that modern 2024–2025 instruction-tuned models benefit from prompt-based self-repair without any fine-tuning, even at the 8B scale.
ci = M (p, ci−1 , ei−1 )
(1)
where the model receives the problem, its previous attempt, and the error message. If exec(ci ) succeeds at any round,
the problem is marked as solved at round Ri and no further attempts are made. B. Code Extraction LLM outputs frequently contain natural-language explanations, markdown formatting, and other non-code content surrounding the actual solution. We employ a code extraction pipeline that: (1) removes chain-of-thought traces (e.g., <think>...</think> tags) that reasoning models may prepend; (2) extracts code from markdown fences (‘‘‘python ... ‘‘‘), including handling of unclosed fences from truncated responses; and (3) if no complete function definition is found, prepends the function signature to body-only responses with appropriate indentation. C. Error Classification When a solution fails, we classify the error into one of the following categories based on the Python exception type: AssertionError: The code runs without crashing but produces incorrect output, failing one or more test assertions. These errors indicate logical mistakes. • SyntaxError: The generated code is not valid Python. • TypeError / ValueError: Runtime type mismatches or invalid values, often indicating misunderstanding of the expected input/output types. • NameError: References to undefined variables or functions, typically caused by incomplete code generation. • IndexError / KeyError: Out-of-bounds access or missing dictionary keys, indicating edge-case handling failures. • Timeout: The code does not terminate within the allotted time, suggesting infinite loops or excessive computational complexity. •
D. Metrics
1) Open-Weight Models (Groq API): 1) Llama 3.1 8B [7]: An 8-billion-parameter dense transformer from Meta. This serves as our smallest baseline and represents the class of efficient, instruction-tuned models commonly deployed in resource-constrained settings. 2) Llama 3.3 70B [8]: A 70-billion-parameter dense transformer from Meta. This is the largest dense open-weight model in our evaluation and provides a strong baseline for comparing against mixture-of-experts architectures. 3) Llama 4 Scout 17B [9]: A mixture-of-experts (MoE) model from Meta with 17 billion active parameters distributed across 16 experts. The MoE architecture allows the model to maintain a large total parameter count while activating only a fraction of parameters per token. 4) Llama 4 Maverick 17B [9]: A mixture-of-experts model from Meta with 17 billion active parameters and 128 experts. Compared to Scout, Maverick uses significantly more experts, providing an interesting comparison point for how MoE granularity affects code generation and self-repair. 5) Qwen3 32B [10]: A 32-billion-parameter dense transformer from Alibaba. Qwen3 supports both reasoning (thinking) and non-reasoning modes. Thinking mode was disabled via the /no_think setting to ensure fair comparison with non-reasoning models. 2) Proprietary Models (Vertex AI): 6) Gemini 2.5 Flash [11]: A lightweight model from Google optimized for speed and cost-efficiency. Gemini 2.5 Flash is designed as a fast-inference model suitable for high-throughput applications, making it an interesting comparison for self-repair efficiency. 7) Gemini 2.5 Pro [11]: Google’s flagship reasoning model with extended “thinking” capabilities. Unlike Qwen3’s optional thinking mode, Gemini 2.5 Pro integrates reasoning natively. This model represents the frontier of proprietary model capability in our evaluation.
Our primary metric is cumulative pass@1 at each round Ri , defined as the fraction of problems solved by round Ri (inclusive of all preceding rounds). Because each problem The inclusion of Gemini models extends our evaluation from receives exactly one attempt per round with greedy decoding two model families (Meta, Alibaba) to three (Meta, Alibaba, (temperature = 0.0), pass@1 is deterministic and does not Google), and from open-weight-only to a mix of open-weight require the unbiased estimator used for stochastic sampling [1]. and proprietary models. This broadens the generalizability of We also report the self-repair gain ∆, defined as the difference our findings. between the final cumulative pass@1 (at R4 ) and the initial All models are queried with greedy decoding (temperature = pass@1 (at R0 ), measured in percentage points (pp). We note that on HumanEval (n = 164), a single problem corresponds 0.0) to ensure deterministic, reproducible outputs. A minimal to ≈0.6 pp; differences smaller than ∼2 pp (3 problems) should system prompt is used for all models: “You are an expert Python programmer. Complete the given function. Return ONLY be interpreted cautiously. the Python code, no explanations, no markdown formatting.” No few-shot examples are provided. For initial generation, IV. E XPERIMENTAL S ETUP each model receives only the function signature and docstring. For repair rounds, the model receives the original problem, A. Models its previous code attempt, and the error message. The repair We evaluate seven models from three families, spanning prompt is minimal: it presents the error and asks for a corrected open-weight and proprietary providers: function (see Section III).
Model
Family
R0
R1
R2
R3
R4
∆
Llama 3.1 8B Llama 3.3 70B Scout 17B (16E) Maverick 17B (128E) Qwen3 32B
Meta Meta Meta Meta Alibaba
67.1 82.9 75.6 87.2 87.8
73.2 89.6 84.8 91.5 90.2
75.6 90.9 87.2 92.7 92.1
76.2 92.7 89.0 93.9 92.7
76.8 93.3 89.6 93.9 92.7
+9.8 +10.4 +14.0 +6.7 +4.9
Gemini 2.5 Flash Gemini 2.5 Pro
Google Google
86.6 73.2
95.7 82.9
95.7 86.6
96.3 89.6
96.3 90.2
+9.8 +17.1
Cumulative Pass@1 by Repair Round (HumanEval) 100
80
Cumulative Pass Rate (%)
TABLE I C UMULATIVE PASS @1 (%) ON H UMAN E VAL (164 PROBLEMS ) BY REPAIR ROUND . R0 IS THE INITIAL ATTEMPT; R1 –R4 ARE REPAIR ROUNDS . ∆ DENOTES THE TOTAL SELF - REPAIR GAIN (R4 − R0 ) IN PERCENTAGE POINTS .
60
40 Llama 3.1 8B Llama 3.3 70B Scout 17B (16E) Maverick 17B (128E) Qwen3 32B Gemini 2.5 Flash Gemini 2.5 Pro
20
0
R0
R1
R2
R3
R4
Repair Round
B. Benchmarks We evaluate on two standard code generation benchmarks: 1) HumanEval [1]: 164 hand-written Python programming problems, each with a function signature, docstring, and test cases. Problems range from simple string manipulations to moderately complex algorithmic tasks. All 164 problems were evaluated for all seven models. 2) MBPP Sanitized [4]: 257 problems from the Mostly Basic Programming Problems dataset (sanitized subset). MBPP problems tend to be shorter and more focused than HumanEval, but cover a broader range of programming concepts. All 257 problems were evaluated for all seven models. We acknowledge that these benchmarks are considered relatively easy by 2025 standards. We discuss the implications of this choice and the need for evaluation on harder benchmarks in Section VII.
Fig. 1. Cumulative pass@1 rate on HumanEval across repair rounds for all seven models. Gemini 2.5 Flash achieves the highest final rate (96.3%), while most gains concentrate in the first two rounds. TABLE II C UMULATIVE PASS @1 (%) ON MBPP S ANITIZED (257 PROBLEMS ) BY REPAIR ROUND . Model
Family
R0
R1
R2
R3
R4
∆
Llama 3.1 8B Llama 3.3 70B Scout 17B (16E) Maverick 17B (128E) Qwen3 32B
Meta Meta Meta Meta Alibaba
55.6 67.7 65.4 72.0 70.8
66.9 86.4 77.4 85.6 81.7
69.6 89.5 80.5 88.7 85.6
70.8 89.9 82.9 91.8 87.9
71.6 90.7 83.3 92.6 88.3
+16.0 +23.0 +17.9 +20.6 +17.5
Gemini 2.5 Flash Gemini 2.5 Pro
Google Google
63.8 66.5
86.8 85.2
91.8 89.1
93.4 92.2
93.8 92.2
+30.0 +25.7
to 95.7% (+9.1 pp in a single round) is among the largest first-round gains (comparable to Pro’s +9.8 pp and matching Scout’s +9.1 pp), suggesting strong error comprehension. C. Infrastructure Gemini 2.5 Pro shows the largest repair gain. Pro Open-weight model experiments were conducted using the achieves +17.1 pp, the highest ∆ on HumanEval. However, Groq free-tier API, which provides rate-limited access to its lower initial pass rate (73.2%) reflects overhead from hosted models at no monetary cost. Gemini experiments were its native reasoning traces, which embed extended chain-ofconducted using Google Cloud Vertex AI, with total Gemini thought within the response and occasionally interfere with API costs under $20 USD (including initial debugging runs). code extraction despite our extraction pipeline (see Section III). Code execution was performed in isolated Python subprocesses A similar pattern appears with Qwen3’s thinking mode (see with a 15-second timeout per test case. All code and results Section VII). Pro’s final rate (90.2%) is competitive but are publicly available.1 below Flash, illustrating that stronger reasoning does not always translate to better code generation in a simple promptV. R ESULTS completion paradigm. Diminishing returns from higher baselines. Among the A. HumanEval Results open-weight models, the pattern holds: Qwen3 32B starts at Table I presents the cumulative pass@1 rates on HumanEval 87.8% and gains only +4.9 pp, while Scout starts lower at 75.6% across all repair rounds for all seven models. Several key and gains +14.0 pp. Flash is a notable exception: it starts high observations emerge: (86.6%) yet still achieves a substantial +9.8 pp gain, suggesting Universal improvement. Self-repair improves pass rates for that its repair capability exceeds the diminishing-returns trend. every model tested, across all three families and both openFigure 1 visualizes these trajectories. The curves illustrate weight and proprietary models. Every model gains at least the universal upward trend, with Flash breaking through the +4.9 pp. ∼94% ceiling observed among the open-weight models. Gemini 2.5 Flash achieves the highest final rate. At 96.3%, Flash surpasses all open-weight models, including Maverick B. MBPP Results (93.9%) and Llama 3.3 70B (93.3%). Its R1 jump from 86.6% Table II presents results on MBPP Sanitized. The MBPP 1 https://github.com/Johin2/iterative-code-repair results reinforce and extend the findings from HumanEval:
Error Type Distribution at R0 (HumanEval)
TABLE III C ROSS - BENCHMARK COMPARISON OF SELF - REPAIR GAINS (∆, IN PERCENTAGE POINTS ) ON H UMAN E VAL AND MBPP S ANITIZED .
assertion index_key_error name_error recursion runtime_other
40
syntax timeout type_error value_error
Family
HumanEval ∆
MBPP ∆
Llama 3.1 8B Llama 3.3 70B Scout 17B (16E) Maverick 17B (128E) Qwen3 32B
Meta Meta Meta Meta Alibaba
+9.8 +10.4 +14.0 +6.7 +4.9
+16.0 +23.0 +17.9 +20.6 +17.5
Gemini 2.5 Flash Gemini 2.5 Pro
Google Google
+9.8 +17.1
+30.0 +25.7
Count
30
Model
20
10
0
B .1 8 a3
Llam
0B .3 7 a3
7B ut 1
Llam
E) (16
Sco
8E (12
7B ck 1
)
veri
Ma
en3
Qw
32B
ini
Gem
2.5
sh Fla
Gem
ini
2.5
Pro
Model
Fig. 3. Distribution of error types at the initial attempt (R0 ) on HumanEval for each model.
Cross-Benchmark Self-Repair Comparison HumanEval
Self-Repair Success Rate by Initial Error Type
MBPP
100
n=1
n=1
n=3
80
60
40 Llama 3.1 8B Llama 3.3 70B Scout 17B (16E) Maverick 17B (128E) Qwen3 32B Gemini 2.5 Flash Gemini 2.5 Pro
20
0
R0
Llama 3.1 8B Llama 3.3 70B Scout 17B (16E) Maverick 17B (128E) Qwen3 32B Gemini 2.5 Flash Gemini 2.5 Pro
R1
R2
Repair Round
R3
R4
R0
R1
R2
R3
R4
Repair Round
Repair Success Rate (%)
Cumulative Pass Rate (%)
100
80
n=13 n=6
n=38
60 n=157
n=9
40
20
Fig. 2. Cross-benchmark comparison of self-repair gains for all seven models. MBPP consistently yields larger gains than HumanEval, with Gemini 2.5 Flash showing the largest MBPP gain (+30.0 pp).
n=1
0
s
as
on
ti er
ey _k ex ind
r rro _e
na
m
r ro er e_
er
on
si ur
c re
e im
t un
r
th _o
sy
x
a nt
ut eo tim
typ
r ro er e_
r rro _e
lue
va
Error Type
Larger absolute gains. Self-repair gains on MBPP are substantially larger than on HumanEval for every model. Fig. 4. Repair success rate by error type across all models. Assertion errors Among open-weight models, gains range from +16.0 pp to (logical mistakes) are the hardest to repair. +23.0 pp. The Gemini models show even larger gains: Flash achieves +30.0 pp and Pro +25.7 pp, the two highest in our Figure 2 visualizes the cross-benchmark comparison, highstudy. lighting the consistent pattern of larger MBPP gains across all Flash dominates on MBPP. Gemini 2.5 Flash achieves models and families. the highest final pass rate (93.8%), despite starting at only 63.8%, lower than four of the five open-weight models. Its D. Error Distribution Analysis Figure 3 shows the distribution of error types at the initial +30.0 pp repair gain exceeds the best open-weight model on MBPP (+23.0 pp) by 7 pp, suggesting qualitatively different attempt (R0 ) on HumanEval. Across all seven models, assertion repair behavior. Flash’s R1 jump (+23.0 pp in a single round) errors dominate, indicating that most failures produce syntactically valid but logically incorrect code. Higher-capability accounts for the majority of its gain. Consistent model ranking. The relative ordering of models models have fewer total failures but the same error-type profile, is largely consistent across benchmarks. Llama 3.1 8B has confirming that assertion errors are the irreducible core of the lowest final rate on both benchmarks, while Flash leads failure regardless of model scale or family. on both. On MBPP (n = 257), one problem corresponds to E. Repair Success by Error Type ∼0.4 pp, providing finer resolution for ranking comparisons. Figure 4 analyzes repair success rates broken down by error type. A clear hierarchy emerges: C. Cross-Benchmark Comparison • Name errors are repaired at high rates (∼77%). These Table III summarizes the self-repair gains across both errors provide specific diagnostic information (undefined benchmarks. MBPP consistently yields higher repair gains than variable or function names) that guides the model toward HumanEval for every model. This difference likely reflects the fix. two factors: (1) lower initial pass rates on MBPP provide • Syntax errors are repaired at moderate rates (∼66%). more room for improvement, and (2) MBPP problems, while While many syntax errors are straightforward to fix, some numerous, tend to be shorter and more focused, making their reflect deeper structural issues in code generation (e.g., errors potentially easier to diagnose and repair. The Gemini incomplete functions or mismatched indentation) that are models show the largest MBPP gains (+30.0 and +25.7 pp), harder to resolve. suggesting that their error comprehension scales particularly • Assertion errors are the hardest to repair, with success well on shorter, focused problems. rates around 45%. An assertion error indicates that the
New Problems Solved Per Round (HumanEval) Llama 3.1 8B Llama 3.3 70B Scout 17B (16E) Maverick 17B (128E) Qwen3 32B Gemini 2.5 Flash Gemini 2.5 Pro
Newly Passing (%)
80
TABLE IV T OTAL TOKEN USAGE ( PROMPT + COMPLETION ) AND SELF - REPAIR GAIN ON H UMAN E VAL (164 PROBLEMS ). G EMINI MODELS ’ INTERNAL REASONING TOKENS ARE EXCLUDED FOR COMPARABILITY; SEE TEXT.
60
Total Tokens
∆ (pp)
Tokens/pp
Llama 3.1 8B Llama 3.3 70B Scout 17B (16E) Maverick 17B (128E) Qwen3 32B
195K 112K 155K 93K 102K
+9.8 +10.4 +14.0 +6.7 +4.9
19.9K 10.8K 11.1K 13.9K 20.8K
Gemini 2.5 Flash Gemini 2.5 Pro
121K 162K
+9.8 +17.1
12.3K 9.5K
Model 40
20
0
R0
R1
R2
R3
R4
Repair Round
Fig. 5. Marginal improvement in pass@1 per repair round on HumanEval. The first repair round (R0 → R1 ) yields the largest gain for all models.
code ran successfully but produced the wrong output. The error message provides minimal diagnostic information about what the code computed incorrectly or why, requiring the model to re-examine its logic. This finding is consistent with Huang et al.’s observation [15] that LLMs struggle to self-correct reasoning without external feedback. On MBPP, assertion errors are repaired at higher rates (∼63%) than on HumanEval (∼45%), likely because MBPP’s shorter problems make logical errors easier to diagnose. F. Per-Round Improvement
TABLE V P ER - ROUND REPAIR BREAKDOWN ON H UMAN E VAL : NUMBER OF PROBLEMS FIRST SOLVED AT EACH ROUND , AND THE REPAIR SUCCESS RATE ( FRACTION OF INITIALLY- FAILED PROBLEMS EVENTUALLY REPAIRED ). Model
R0
R1
R2
R3
R4
Never
Rep. %
Llama 3.1 8B Llama 3.3 70B Scout 17B (16E) Maverick 17B (128E) Qwen3 32B
110 136 124 143 144
10 11 15 7 4
4 2 4 2 3
1 3 3 2 1
1 1 1 0 0
38 11 17 10 12
29.6 60.7 57.5 52.4 40.0
Gemini 2.5 Flash Gemini 2.5 Pro
142 120
15 16
0 6
1 5
0 1
6 16
72.7 63.6
at 9.5K tokens per percentage point of improvement, followed by Llama 3.3 70B at 10.8K tokens/pp. Table V provides a fine-grained breakdown of when problems are first solved. The first repair round (R1 ) accounts for the majority of all repairs across every model. Repair success rate (the fraction of initially-failed problems that are eventually fixed) correlates positively with model capability: Gemini 2.5 Flash repairs 72.7% of its failures (the highest), followed by Gemini 2.5 Pro at 63.6% and Llama 3.3 70B at 60.7%, while Llama 3.1 8B repairs only 29.6%.
Figure 5 shows the marginal improvement at each repair round on HumanEval. The pattern of diminishing returns is consistent across all models: • R0 → R1 (first repair): The largest single-round gain for all seven models. Gemini 2.5 Pro gains +9.8 pp (the largest first-round gain), followed by Scout and Flash (both +9.1 pp), while Qwen3 gains +2.4 pp (the smallest, reflecting its high baseline). • R1 → R2 (second repair): Still meaningful but reduced for all models. H. Repair Prompt Ablation • R2 → R3 and R3 → R4 : Minimal additional gains. For Our main experiments use a minimal repair prompt that Qwen3, Maverick, and Gemini 2.5 Flash, R4 yields no provides only the error message and asks for a corrected additional improvement over R3 , indicating that these function. To assess sensitivity to prompt design, we compare up models have solved all problems amenable to self-repair to three repair prompt strategies on four models: Llama 3.1 8B by round 3. (dense, 8B), Llama 3.3 70B (dense, 70B), Llama 4 Scout (MoE, The concentration of gains in early rounds has practical 17B active), and Qwen3 32B (dense, 32B). We evaluate all implications: for most applications, two repair rounds capture 164 HumanEval problems with up to two repair rounds (R0 – the bulk of the benefit, and additional rounds offer diminishing R ). All three strategies were evaluated for three models; for 2 marginal returns relative to their computational cost. Scout, only CoT and Explain-then-fix are available because G. Token Cost Analysis the Minimal run was interrupted by Groq API rate limits and Table IV reports total token usage for each model on could not be re-run before the evaluation window closed. HumanEval. Token counts reflect prompt and completion tokens 1) Minimal: The baseline strategy used throughout this only. For Gemini models, internal reasoning (“thinking”) tokens paper. The repair prompt presents the error message and are excluded from this count; including them would increase requests a corrected function. Flash’s total to ∼489K and Pro’s to ∼994K. Since thinking 2) Explain-then-fix: The model is asked to first explain tokens are not directly comparable to standard input/output the bug in 1–2 sentences, then provide the corrected tokens, we report prompt+completion for fair cross-model code. This follows the explain-before-fix paradigm from comparison. Gemini 2.5 Pro achieves the best cost-effectiveness Self-Debug [6].
TABLE VI R EPAIR PROMPT ABLATION ON H UMAN E VAL . C UMULATIVE PASS @1 (%) AT EACH ROUND FOR UP TO THREE PROMPT STRATEGIES ON FOUR MODELS . S COUT ’ S M INIMAL RUN IS UNAVAILABLE ( SEE TEXT ). D ELTAS COMPUTED FROM RAW PROBLEM COUNTS BEFORE ROUNDING . R0 AND SUBSEQUENT ROUNDS MAY VARY SLIGHTLY ACROSS RUNS DUE TO API- LEVEL NON - DETERMINISM ; ∆ COMPUTED FROM EACH RUN ’ S OWN R0 .
Model
Strategy
R0
R1
R2
∆
Llama 3.1 8B
Chain-of-thought Explain-then-fix Minimal
65.9 65.9 65.9
72.6 72.0 73.2
77.4 76.2 75.6
+11.6 +10.4 +9.8
Llama 3.3 70B
Chain-of-thought Explain-then-fix Minimal
84.1 82.9 81.1
93.9 90.2 86.6
96.3 93.9 87.8
+12.2 +11.0 +6.7
Scout 17B (16E)
Chain-of-thought Explain-then-fix
79.3 78.0
87.2 87.2
89.6 87.2
+10.4 +9.1
Qwen3 32B
Chain-of-thought Explain-then-fix Minimal
89.6 88.4 87.8
97.0 93.3 90.2
97.0 94.5 91.5
+7.3 +6.1 +3.7
Llama-3.3-70B Chain-of-thought Minimal (error only)
Chain-of-thought Explain-then-fix Minimal (error only)
Cumulative Pass@1 (%)
84.1
80
77.4
93.9
Llama-4-Scout-17B 96.3
90.2
93.9
89.6
82.9
90.9
Chain-of-thought Explain-then-fix Minimal (error only) 79.3
75.6
73.2
72.6
82.9
78.0
Qwen3-32B 97.0
87.2
87.2
89.6 84.8
87.2
87.2
89.6
88.4
93.3
87.8
97.0
94.5
90.2
92.1
75.6
67.1
65.9
60
40
20
0
Chain-of-thought Explain-then-fix Minimal (error only)
R0
R1
Round
R2
R0
R1
Round
R2
R0
R1
Round
R2
R0
R1
Model
Llama 3.1 8B Llama 3.3 70B Scout 17B (16E) Qwen3 32B
Prompt Strategy Ablation on HumanEval Llama-3.1-8B 100
TABLE VII S ELF - REPAIR ( GREEDY, 5 ROUNDS ) VS . INDEPENDENT RESAMPLING ( TEMPERATURE 0.8, 5 SAMPLES ) ON H UMAN E VAL (164 PROBLEMS ). B OLD INDICATES HIGHER PASS RATE . N OTE : REPAIR USES GREEDY DECODING (T = 0) WHILE RESAMPLING USES T = 0.8; SEE TEXT.
R2
Round
Fig. 6. Prompt strategy ablation on HumanEval. CoT consistently achieves the highest final pass rate across all models, with the benefit scaling with model capability.
Self-Repair
Resampling
Final %
Tokens
pass@5 %
Tokens
76.8 93.3 89.6 92.7
195K 112K 155K 102K
79.9 90.9 86.0 92.7
219K 231K 255K 223K
+3.6 pp for Qwen3. In absolute terms, CoT achieves final R2 rates 1.8–8.5 pp higher than minimal, though part of this gap reflects R0 variation across runs (see above). This confirms the interaction between model capability and prompt strategy across the three models with complete ablation data, spanning 8B to 70B parameters. The practical implication is clear: for capable models (≥32B), chain-of-thought repair prompting is strongly recommended and can yield gains comparable to an additional 1–2 repair rounds. For smaller models, the benefit is present but modest, and the simpler minimal prompt remains a competitive choice. I. Self-Repair vs. Independent Resampling
A key question is whether the token budget spent on iterative repair could be better used by simply drawing multiple 3) Chain-of-thought (CoT): The model is prompted to independent samples. Tang et al. [16] formalized this as an reason step-by-step: (1) what does the error tell us, (2) exploration-exploitation tradeoff and proposed a Thompson what is the root cause, (3) what is the fix. The corrected Sampling strategy (REx) to dynamically allocate between code follows the analysis. repair and resampling. We extend this analysis to modern Table VI and Figure 6 present the results across four models. 2024–2025 models by comparing self-repair (greedy decoding, Note that each ablation configuration involves a complete T = 0, up to 5 sequential attempts with error feedback) against re-execution of all 164 problems, including initial generation independent resampling (T = 0.8, 5 independent samples) on at R0 . Since the repair strategy does not affect the initial HumanEval. This comparison involves two differences: error generation prompt, any R0 variation across strategies for the feedback (present in repair, absent in resampling) and decoding same model reflects non-determinism in the Groq API’s served temperature (T = 0 vs. T = 0.8). A fully controlled study would model weights across sessions. For example, Llama 3.3 70B also require stochastic repair (T > 0 with feedback); we leave shows R0 values of 84.1%, 82.9%, and 81.1% across the three this to future work. Nevertheless, this comparison addresses strategies. The ∆ column is computed relative to each run’s the practical question: given a token budget, should one invest own R0 , making repair gain measurements independent of this in sequential repair or parallel resampling? For resampling, we baseline variation. However, run-to-run variation is non-trivial report the unbiased pass@k estimator [1]. (e.g., 70B’s R2 ranges from 87.8% to 96.3%), and some of Table VII and Figure 7 compare self-repair against indepenthis gap may reflect session-level variance. Crucially, the key dent resampling on HumanEval for four models.2 The results finding holds regardless of which R0 baseline is used: CoT reveal a clear, model-dependent pattern: achieves the highest final R2 rate for every model. Self-repair is always more token-efficient. Across all four CoT consistently wins across all models. Chain-of-thought models, self-repair uses fewer tokens than resampling to achieve achieves the highest final pass rate for every model: 77.4% comparable or better pass rates, with savings ranging from (8B), 96.3% (70B), 89.6% (Scout), and 97.0% (Qwen3). Qwen3 11% (8B) to 54% (Qwen3). The savings are largest for models with CoT achieves the highest pass rate of any model-strategy combination in our ablation (97.0% at R2 ). 2 Maverick was unavailable on the Groq free-tier API during the resampling The benefit of richer prompts scales with model capability. experiments; Gemini models were excluded because stochastic resampling on Measured by the additional self-repair gain (∆CoT − ∆minimal ), Vertex AI would have required additional paid API calls beyond our budget. The four evaluated models span the full capability range (8B–70B) and both the advantage of CoT is modest for the 8B model (+1.8 pp) but architecture types (dense and MoE), providing sufficient evidence for the widens for stronger models: +5.5 pp for Llama 3.3 70B and observed patterns.
Pass Rate: Self-Repair vs Resampling (HUMANEVAL)
Token Usage Comparison
Self-Repair (final) Resampling (pass@5)
100
80
200
Total Tokens (K)
Pass Rate (%)
Self-Repair Resampling
250
60
40
20
150
100
50
0
0
70B
8B
a-3.1-
Llam
Llam
a-3.3-
Llam
t-17B
Scou
a-4-
B
3-32
Qwen
8B
a-3.1Llam
70B
a-3.3Llam
7B
t-1 Scou
a-4Llam
B 3-32
Qwen
Fig. 7. Self-repair vs. independent resampling on HumanEval. Self-repair matches or exceeds resampling for all models except the 8B, while consistently using fewer tokens.
whose strong error comprehension enables efficient singleattempt repairs, while the 8B model’s longer repair traces narrow the gap. This efficiency advantage arises because selfrepair generates only the corrected function at each round, while resampling generates a complete solution from scratch each time. Self-repair becomes increasingly competitive with model capability. For the weakest model (Llama 3.1 8B), resampling achieves a modestly higher pass@5 (79.9% vs. 76.8%); the diversity of independent samples compensates for the model’s limited error comprehension. For the three stronger models, self-repair matches or exceeds resampling: Scout (89.6% vs. 86.0%), Llama 3.3 70B (93.3% vs. 90.9%, a difference of 4 problems), and Qwen3 32B (92.7% tied) all favor repair. The 70B result is particularly striking: self-repair achieves +2.4 pp higher pass rate while using less than half the tokens (112K vs. 231K). This pattern has a clear interpretation: as models become more capable, their ability to diagnose and fix errors from feedback improves, making the informational signal from error messages more valuable than the diversity from independent samples. We note that this comparison does not fully disentangle the value of error feedback from the effect of decoding temperature; a stochastic repair condition would be needed to isolate these factors. The practical implication is that selfrepair is the preferred strategy for capable models, while a hybrid approach (repairing diagnosable errors and resampling for assertion errors) may be optimal for weaker models. VI. D ISCUSSION A. Self-Repair as a Universal Improvement Strategy Our results demonstrate that iterative self-repair with error feedback is a universally effective strategy for improving LLM code generation across all seven models tested, spanning three families (Meta, Alibaba, Google), both open-weight and proprietary models, and both dense and MoE architectures. This finding updates the conclusions of Olausson et al. [5], who found that weaker models could be harmed by self-repair, and contrasts with Chen et al. [12], who found limited gains for Qwen2.5-Coder-7B with prompt-based self-debugging. Prior work demonstrated that fine-tuning can enable small-model self-repair [13], [14]; our results suggest that the 2024–2025 generation of instruction-tuned models has closed this gap
without task-specific training. The discrepancy with earlier findings likely reflects improvements in instruction-following and error-comprehension capabilities: all models in Olausson et al.’s study were released in 2022–2023, while ours cover 2024–2025. The cross-benchmark consistency of our findings strengthens this conclusion. Self-repair improves every model on both HumanEval and MBPP, with largely consistent model rankings across benchmarks. The inclusion of Gemini models from a third family further strengthens generalizability: self-repair effectiveness is a robust property of modern LLMs rather than an artifact of a particular model family or benchmark. B. Dense vs. Mixture-of-Experts Architectures Our evaluation includes dense models (Llama 8B, 70B, Qwen3 32B) and MoE models (Scout 16E, Maverick 128E) among the open-weight models, as well as two proprietary Gemini models whose architectures have not been publicly disclosed, enabling comparisons across three families. Several observations are noteworthy: • Among open-weight models, Scout (16 experts) achieves the highest repair gain on HumanEval (+14.0 pp), suggesting that its MoE architecture particularly benefits from the opportunity to repair its errors. • Gemini 2.5 Flash achieves the highest final pass rates on both benchmarks (96.3% HumanEval, 93.8% MBPP), surpassing all open-weight models. Its combination of high initial accuracy and strong repair capability sets a new ceiling. • Gemini 2.5 Pro, despite being a frontier reasoning model, underperforms Flash on both benchmarks (90.2% vs. 96.3% on HumanEval). This suggests that reasoning overhead can hurt in a simple prompt-completion paradigm, echoing our findings with Qwen3’s thinking mode. • The 70B dense model approaches Maverick’s performance (93.3% vs. 93.9% on HumanEval; 90.7% vs. 92.6% on MBPP), suggesting that a large dense model can compete with a smaller MoE model. These comparisons are suggestive but must be interpreted cautiously: models differ not only in architecture but also in training data, parameter counts, and other confounds that we cannot isolate. We report these observations as correlations rather than causal claims. C. Qualitative Analysis of Repair Outcomes To complement our quantitative findings, we examine representative cases from HumanEval. Successful repairs. On HumanEval/38, Llama 3.3 70B produced code that failed at R0 with a NameError and repaired it in a single round. More interestingly, on HumanEval/26, Llama 3.1 8B failed with assertion errors for three consecutive rounds before finally producing a correct solution at R3 , demonstrating that persistence can resolve even logical errors. On HumanEval/77, Scout exhibited error-type mutation across rounds (type error → assertion → type error → pass), showing
TABLE VIII L IVE C ODE B ENCH RESULTS (50 PROBLEMS ). S ELF - REPAIR IMPROVES ALL FOUR MODELS EVALUATED . Model
R0
Final
∆
Llama 3.1 8B Scout 17B (16E) Qwen3 32B Gemini 2.5 Flash
10.0 6.0 8.0 0.0
12.0 14.0 10.0 16.0
+2.0 +8.0 +2.0 +16.0
the model genuinely restructuring its approach rather than making superficial edits. Persistent failures. On HumanEval/10, Llama 3.1 8B produced assertion errors on all five attempts, never resolving the underlying logic error. On HumanEval/32, Maverick exhibited error thrashing: assertion → name error → assertion → assertion → timeout, the failure mode described by Olausson et al. [5]. We identified several “universally hard” problems (e.g., HumanEval/132, HumanEval/145) that defeated all models even with four repair rounds. D. Preliminary Evaluation on Harder Benchmarks To assess whether self-repair generalizes beyond the relatively easy HumanEval and MBPP benchmarks, we conducted a preliminary evaluation on LiveCodeBench [26], a contamination-free benchmark of competitive programming problems. We evaluated four models on 50 LiveCodeBench problems (18 easy, 23 medium, 9 hard) with the same 5-round self-repair protocol. The four models (8B, Scout, Qwen3, Flash) span the capability range; the remaining three were omitted due to API availability constraints. We emphasize that this evaluation is exploratory: the small sample size (50 problems) limits statistical power. Table VIII shows that self-repair improves all four models on LiveCodeBench. Gemini 2.5 Flash achieves the largest gain (+16.0 pp), reaching 16% from an initial pass rate of 0%; all solved problems come entirely from repair rounds. Flash’s 0% R0 reflects the difficulty of competitive programming problems combined with stdin/stdout format requirements, yet its repair capability recovers 8 problems. Scout achieves the secondlargest gain (+8.0 pp), more than doubling its pass rate from 6% to 14%. While the absolute pass rates are much lower than on HumanEval (as expected for competitive programming), selfrepair remains effective. The relative repair gains are actually larger than on HumanEval, suggesting that self-repair may scale to harder benchmarks where the lower baseline leaves more room for improvement. At n = 50, each problem corresponds to 2 pp, so differences smaller than ∼4 pp should be interpreted cautiously. E. Practical Implications Our findings suggest several practical guidelines for deploying LLM code generation systems: 1) Always include at least one repair round: The first repair round consistently provides the largest marginal improvement for all models tested.
2) Two rounds capture most gains: For cost-sensitive applications, two repair rounds (R0 through R2 ) capture the majority (76–95%) of the total achievable improvement. 3) Consider error type for routing: Name errors are repaired at the highest rates (∼77%), while assertion errors are the hardest (∼45%). For assertion-dominated failures, alternative strategies (e.g., sampling multiple independent solutions) may be more cost-effective. 4) Model selection matters: For applications where final pass rate is paramount, Gemini 2.5 Flash (96.3%) or Maverick (93.9%) should be preferred. For self-repair gain per token, Gemini 2.5 Pro and Llama 3.3 70B offer the best efficiency. VII. L IMITATIONS AND F UTURE W ORK We identify several limitations of this study that suggest directions for future work. Resampling comparison scope. Our resampling comparison (Section V, Table VII) covers four of seven models on HumanEval only. While the results clearly show that selfrepair is more token-efficient and increasingly advantageous for stronger models, extending this comparison to MBPP and controlling for token budget more precisely (e.g., allocating the exact same number of tokens to each strategy) would strengthen the analysis. Additionally, we compare greedy repair against stochastic resampling (temperature = 0.8), which conflates two differences: error feedback vs. diversity, and greedy vs. stochastic decoding. A fairer comparison might use stochastic repair as well. Benchmark scope. Our main evaluation uses HumanEval and MBPP, which are considered relatively easy by 2025 standards. Our preliminary LiveCodeBench evaluation (Section VI) shows that self-repair remains effective on harder competitive programming problems, but covers only four models on 50 problems. A comprehensive evaluation across all models on LiveCodeBench [26], BigCodeBench [27], or SWE-Bench [28] would more fully characterize self-repair on problems requiring deep algorithmic reasoning. Greedy decoding only. Our use of temperature = 0.0 ensures deterministic, reproducible results but eliminates sampling variance, precluding confidence intervals or statistical significance tests. On HumanEval (164 problems), a single problem corresponds to ±0.6 pp, meaning differences of ∼1.2 pp or less (e.g., Maverick at 93.9% vs. Qwen3 at 92.7%, a gap of just two problems) should not be over-interpreted. Experiments with stochastic decoding across multiple seeds would provide more robust estimates and enable statistical analysis. Ablation scope. Our prompt ablation covers four of seven models (8B, 70B, Scout, Qwen3) with two repair rounds (R0 – R2 ). Extending to all five repair rounds would characterize whether CoT’s advantage persists or converges in later rounds. The ablation uses HumanEval only; prompt sensitivity on MBPP or LiveCodeBench may differ. API-served model weights. Open-weight models were accessed via the Groq free-tier API, which may serve quantized
or otherwise optimized model variants. We did not verify that IX. C ONCLUSION the served weights exactly match the canonical model releases. We have presented a systematic study of iterative selfGemini models were accessed via Google Cloud Vertex AI, repair across seven models from three families (Meta, Alibaba, where the served model versions are controlled by Google. Google), spanning both open-weight and proprietary models, Pass rates may differ slightly across API versions or on locallydense and MoE architectures, and two benchmarks (HumanEval hosted weights. and MBPP Sanitized). Self-repair is universally effective: every Model family diversity. Our evaluation covers three model model improves on every benchmark. Error-type analysis families (Llama, Qwen, and Gemini) across two inference confirms that name errors are repaired at high rates, while providers (Groq and Vertex AI), including both open-weight and assertion errors remain the most challenging, connecting to proprietary models. Code-specialized models (e.g., DeepSeek- broader findings on the limits of LLM self-correction. Coder [29]) may exhibit different repair dynamics due to codeTwo repair rounds capture the majority of achievable gains, focused pre-training and remain a direction for future work. providing clear deployment guidance. Our prompt ablation Qwen3 reasoning mode. We evaluated Qwen3 with thinking shows that chain-of-thought repair can yield additional gains for mode enabled on all 164 HumanEval problems. Thinking mode capable models, though the benefit is model-dependent. These achieves a lower final pass rate (89.0% vs. 92.7% for no-think), results establish iterative self-repair as a practical, trainingdespite a much larger repair gain (+14.0 pp vs. +4.9 pp). The free technique for improving LLM code generation, while explanation lies in the initial pass rate: thinking mode’s R0 identifying code-specialized models and harder-benchmark drops to 75.0% (vs. 87.8% for no-think) because long reasoning evaluation as important directions for future work. traces contaminate the code output, causing syntax errors. Self-repair compensates by fixing 23 additional problems, but ACKNOWLEDGMENT 18 problems fail due to API errors or persistent extraction The author thanks Groq for providing free-tier API access issues. Our decision to disable thinking mode in the main that enabled the open-weight model experiments, and Google experiments is validated: with the current code extraction Cloud for Vertex AI credits used for the Gemini experiments. pipeline, thinking mode reduces net performance. A more All experimental code and results are publicly available at https: robust extraction approach (or native tool-use integration) could //github.com/Johin2/iterative-code-repair. The author declares potentially unlock the reasoning benefits without the extraction no conflicts of interest. overhead. R EFERENCES VIII. T HREATS TO VALIDITY Internal validity. Our use of greedy decoding ensures deterministic, reproducible results but reflects a single point in the sampling distribution. Prior work [5], [6] similarly reports single-run results under fixed decoding settings. Our prompt ablation (Table VI) shows that repair prompt design does affect outcomes, meaning the main results represent a lower bound for capable models. Additionally, minor R0 variation in the ablation table suggests the Groq API may serve slightly different model versions across sessions. External validity. Our model selection (seven models from three families) spans a meaningful range, including both openweight and proprietary models, but may not capture behaviors of all architectures. We do not evaluate code-specialized models (e.g., DeepSeek-Coder) or models with native tool use. Additionally, because HumanEval and MBPP are widely known, evaluated models may have been exposed to these problems during pre-training, which could inflate initial pass rates. However, this affects all models equally and does not invalidate relative comparisons. Construct validity. Our cumulative pass@1 metric counts a problem as solved if it passes all provided test cases. However, test suites may not cover all edge cases, meaning some “passing” solutions could contain latent bugs. Our error classification relies on Python exception types, which may conflate distinct failure modes.
[1] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021. [2] Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago et al., “Competition-level code generation with AlphaCode,” in Science, vol. 378, no. 6624, 2022, pp. 1092–1097. [3] B. Rozière, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, R. Sauvestre, T. Remez et al., “Code Llama: Open foundation models for code,” arXiv preprint arXiv:2308.12950, 2024. [4] J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton, “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732, 2021. [5] T. X. Olausson, J. P. Inala, C. Wang, J. Gao, and A. Solar-Lezama, “Is self-repair a silver bullet for code generation?” arXiv preprint arXiv:2306.09896, 2024. [6] X. Chen, M. Lin, N. Schärli, and D. Zhou, “Teaching large language models to self-debug,” arXiv preprint arXiv:2304.05128, 2024. [7] A. Grattafiori, A. Dubey, A. Jauhri et al., “Llama 3 model card,” arXiv preprint arXiv:2407.21783, 2024. [8] Meta AI, “Llama 3.3: Large language model,” Meta AI Blog, 2024, https://ai.meta.com/blog/llama-3-3/. [9] ——, “Llama 4: Maverick, scout, and behemoth,” Meta AI Blog, 2025, https://ai.meta.com/blog/llama-4-multimodal-intelligence/. [10] Qwen Team, “Qwen3 technical report,” arXiv preprint arXiv:2505.09388, 2025. [11] Google DeepMind, “Gemini 2.5: Our most intelligent AI model,” Google Blog, 2025, https://blog.google/technology/google-deepmind/ gemini-model-thinking-updates-march-2025/. [12] X. Chen, Z. Tao, K. Zhang, C. Zhou, X. Zhang, W. Gu, Y. He, M. Zhang, X. Cai, H. Zhao, and Z. Jin, “Revisit self-debugging with self-generated tests for code generation,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), 2025, pp. 18 003– 18 023.
[13] Y. Ding, M. J. Min, G. Kaiser, and B. Ray, “CYCLE: Learning to selfrefine the code generation,” Proceedings of the ACM on Programming Languages, vol. 8, no. OOPSLA1, 2024. [14] N. Jiang, X. Li, S. Wang, Q. Zhou, S. B. Hossain, B. Ray, V. Kumar, X. Ma, and A. Deoras, “LeDex: Training LLMs to better self-debug and explain code,” Advances in Neural Information Processing Systems, vol. 37, 2024. [15] J. Huang, C. Xia, P. Shenoy, and J. Zhao, “Large language models cannot self-correct reasoning yet,” arXiv preprint arXiv:2310.01798, 2024. [16] H. Tang, K. Hu, J. P. Zhou, S. Zhong, W.-L. Zheng, X. Si, and K. Ellis, “Code repair with LLMs gives an exploration-exploitation tradeoff,” Advances in Neural Information Processing Systems, vol. 37, 2024. [17] H. Le, Y. Wang, A. D. Gotmare, S. Savarese, and S. C. H. Hoi, “CodeRL: Mastering code generation through pretrained models and deep reinforcement learning,” Advances in Neural Information Processing Systems, vol. 35, pp. 21 314–21 328, 2022. [18] D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Zou, D. Song, and J. Steinhardt, “Measuring coding challenge competence with APPS,” arXiv preprint arXiv:2105.09938, 2021. [19] S. Kulal, P. Pasupat, K. Chandra, M. Lee, O. Padon, A. Aiken, and P. Liang, “SPoC: Search-based pseudocode to code,” Advances in Neural Information Processing Systems, vol. 32, 2019. [20] A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang et al., “Self-refine: Iterative refinement with self-feedback,” Advances in Neural Information Processing Systems, vol. 36, 2023. [21] N. Shinn, F. Cassano, A. Gopinath, K. Shakkottai, A. Labash, and S. Liu, “Reflexion: Language agents with verbal reinforcement learning,” Advances in Neural Information Processing Systems, vol. 36, 2023. [22] D. Key, S. Lee, J. Shin, and S. Hwang, “Multi-turn code generation with single-turn instruction tuning,” arXiv preprint arXiv:2404.11137, 2024. [23] Y. Zhang, J. Chen, D. Li, and Y. Zheng, “Self-collaboration code generation via ChatGPT,” arXiv preprint arXiv:2304.07590, 2023. [24] L. Zhong, Z. Wang, and J. Shang, “LDB: A large language model debugger via verifying runtime execution step-by-step,” arXiv preprint arXiv:2402.16906, 2024. [25] S. Jiang, Y. Wang, and Y. Wang, “Selfevolve: A code evolution framework via large language models,” arXiv preprint arXiv:2306.02907, 2023. [26] N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica, “LiveCodeBench: Holistic and contamination free evaluation of large language models for code,” arXiv preprint arXiv:2403.07974, 2024. [27] T. Y. Zhuo, M. C. Vu, J. Chim, H. Hu, W. Yu, R. Widyasari, I. N. B. Yusuf, H. Zhan, J. He, I. Paul et al., “BigCodeBench: Benchmarking code generation with diverse function calls and complex instructions,” arXiv preprint arXiv:2406.15877, 2024. [28] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “SWE-bench: Can language models resolve real-world GitHub issues?” arXiv preprint arXiv:2310.06770, 2024. [29] D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. Li et al., “DeepSeek-Coder: When the large language model meets programming – the rise of code intelligence,” arXiv preprint arXiv:2401.14196, 2024.