On the Limits of Execution Traces for Repairing Assertion Failures in LLM-Generated Code

arXiv:2611.09212v1 [cs.SE] 6 Nov 2026

On the Limits of Execution Traces for Repairing Assertion Failures in LLM-Generated Code Muntasir Adnan∗

Carlos C. N. Kuhn

Open Source Institute Faculty of Science and Technology University of Canberra Canberra, Australia

Open Source Institute Faculty of Science and Technology University of Canberra Canberra, Australia

Abstract—Iterative repair returns a failed program’s error to a large language model (LLM) and asks it to fix it. After iterative repair, the remaining failures are mostly assertion failures: with four Qwen2.5 models, they make up 79% of residual failures on BigCodeBench, all on HumanEval and 93% on LiveCodeBench. We argue, through an information view of the feedback, that an assertion failure says little about where the fault lies. We propose Breakpoint Debugging (BPD), which returns the failed program annotated inline with its variables’ runtime values. We take the tasks that iterative repair failed to fix with five repair attempts and compare two ways of spending five further generations on them: BPD, and resampling. On BigCodeBench, BPD repairs 26.7% of these tasks against 22.4%, and its advantage grows with model size. On HumanEval, BPD outperforms resampling but only slightly, and on LiveCodeBench, fresh sampling repairs more (6.5% against 4.9%). Furthermore, BPD reduces unchanged-code resubmissions from 28% to 8% on BigCodeBench. Using only the first failing test’s trace, it outperforms tracing all failing tests while using fewer tokens than both all-trace BPD and plain iterative repair. On LiveCodeBench, BPD gives no gain even on the public tests, so a visible failing input is not enough for a trace to help. These results match the information view: a trace helps only when the test exposes a failing input, and the fault is an implementation slip; otherwise, a repair loop should resample rather than trace. Index Terms—code generation, large language models, self-debugging, program repair, execution traces, assertion errors

This work is funded under the agreement with the ACT Government, Future Jobs Fund - Open Source Institute(OpenSI) - R01553; and NetApp Technology Alliance Agreement with OpenSI - R01657. Additionally, the Australian Government supported this research through the Department of Education’s National Industry PhD Program (project 36337). The views expressed herein are those of the authors and are not necessarily those of the Australian Government or the Department of Education. ∗ Corresponding author: Adnan.adnan@canberra.edu.au

I. I NTRODUCTION Programs written by large language models (LLMs) often fail their tests, and on benchmarks such as BigCodeBench and LiveCodeBench, even strong models solve only part of the tasks on the first attempt [1–3]. Iterative self-debugging executes the program, returns the error to the model and asks for a corrected version [4– 6], and the quality of this feedback is widely seen as its bottleneck [7]. Previous studies take opposite positions on how rich the feedback should be. LDB [6] segments the program into basic blocks, records intermediate variable values and asks the model to verify each block for every type of failure. PyCapsule [5], MapCoder [8] return a lightly processed error message that keeps the relevant traceback frames, including expected and actual values for assertion failures. With the same LLM, PyCapsule reported 85.2% against LDB’s 82.9% on HumanEval and 78.5% against 76.0% on MBPP [9]. These figures are not a controlled comparison, but recent studies point the same way: execution traces in repair prompts beat trace-free prompts in only two of six configurations in one study [10] and were of limited use in another [11], while trace-driven agents report gains [12, 13]. What is missing is an account of when a trace should help and when it cannot. We start from the failures that a structured-error loop leaves behind, and find that they are almost all assertion failures. Unlike crashes, which often indicate where execution first goes wrong, assertion failures identify only where an incorrect result is detected, offering little information about the underlying fault [14]. We formalise this information gap in bits and derive five hypotheses (Section III). We then propose BPD, which targets assertion failures and returns the program with the values from the failing run written as inline comments,

similar to the inline traces of NExT [15] but used at inference time without training. This paper makes three contributions. We show that assertion failures make up 79–100% of the failures left by iterative repair, and explain with an information view why they resist repair. We propose BPD, which adds inline state traces only to assertion failures, and compare it with fresh sampling across four model sizes, measuring repairs, repair behaviour, and token cost. We identify, from LiveCodeBench, when traces cannot help, and turn this into practical rules for when a repair loop should trace or resample. We organise the evaluation around five research questions. RQ1: Which failures remain after iterative repair? RQ2: Do traces repair assertion failures better than fresh sampling? RQ3: How do traces change repair behaviour? RQ4: Is one trace enough, and what do traces cost?

repair with fresh sampling at an equal budget; and we explain, rather than only report, when traces cannot help.

II. R ELATED W ORK

the number of bits by which o reduces the uncertainty about L. Under a uniform prior, n candidate lines contain log2 n bits of uncertainty. Thus, 5 bits are sufficient in principle to distinguish among 32 equally likely candidates. An observation carrying one bit of mutual information reduces the posterior entropy by one bit on average.

Iterative self-repair. Self-Debugging [4] showed that LLMs can improve their code from execution results and their own explanations, and PyCapsule [5] made the loop cheaper with structured error messages. Without external feedback, LLMs struggle to correct their own reasoning [16], and once you account for repair costs, self-repair often yields modest gains limited by feedback quality [7]. Execution information as feedback. Several methods give the model runtime information instead of an error message alone. LDB [6] splits a program into basic blocks and asks the model to verify each intermediate state, and NExT [15] trains models to reason over traces written as inline comments, the format closest to BPD. Recent agentic systems collect traces and pass them to several cooperating or debating agents [12, 13, 17]. However, empirical evidence for their benefits is mixed: traces outperform trace-free baselines in only two of six prompt configurations, with smaller gains from more complex traces [10], while another study finds limited utility for trace-based information in both fine-tuning and inference [11]. LLMs are also unreliable at predicting program execution themselves [18], which is the gap a trace is meant to fill. Repair versus resampling. Repair spends its budget refining one program, whereas resampling spends it on new ones. Tang et al. [19] frame this as an exploration– exploitation trade-off, and Zheng et al. [20] find that multi-turn repair can underperform repeated sampling on competitive programming, where problem understanding rather than repair limits performance. Our work differs in three respects. We apply traces only to assertion failures, where the information view predicts they add the most; we compare trace-guided

III. A SSERTION FAILURE : A N I NFORMATION V IEW Let P be a candidate program, x a failing test input and y ∗ its expected output. Running P on x produces a trace τ = (s0 , . . . , sT ), where st is a program state consisting of the values of all live variables at step t. The run either raises an exception at some step or returns y = sT , which the test compares with y ∗ . We model the fault as a single unknown line L with a prior spread over n lines. Its entropy is bounded by H(L) ≤ log2 n bits, with equality under a uniform prior. The model receives feedback o, and we measure how informative o is about the fault by the mutual information I(L; o) = H(L) − H(L | o)

A. Four differences between crashes and assertion failures a) Who judges the failure: A crash violates a rule that the interpreter already knows, such as an index out of range, so the interpreter acts as a judge that needs no knowledge of the programmer’s intent. An assertion failure violates no rule of the language. A function that tests for odd numbers when the task asked for even numbers runs without error and returns the wrong total, and only the expected value of the test reveals the error. b) How far the symptom is from the cause: A fault produces an observable failure only if three things happen in turn: the faulty line is executed, it corrupts the program state (infection), and the corrupted state reaches the output (propagation), as described by the PIE model [14]. A crash is usually reported at the moment of infection, close to the fault. An assertion failure appears only after propagation, and its traceback names the assertion in the test, which is the same line wherever the fault is. c) Information is lost, never created, along the chain: For a fixed program and test (x, y ∗ ), the fault location determines the execution trace, which determines the returned value and, consequently, the verdict. The feedback variants that a repair loop can return are nested functions of one another. Let b = I[y = y ∗ ] the pass/fail verdict, y = sT the returned value and τ the full trace,

all for the same fixed program and failing test. Since L → τ → y → b forms a Markov chain, I(L; b) ≤ I(L; y) ≤ I(L; τ ), The verdict b is binary and has at most 1 bit of entropy. Since mutual information cannot exceed the entropy of the observed variable, I(L; b) ≤ H(b) ≤ 1 The proposition follows directly from the dataprocessing inequality [21]. The three terms correspond to three feedback regimes in our experiments: a hidden test that only reports failure (b), a visible test whose message shows the expected and actual output (y), and (τ ). The middle inequality is often strict because the map from traces to final values is many-to-one. The proposition bounds what a trace could add; it does not say that the model can use it. The information that a computationally bounded observer can actually extract, the usable or Vinformation [22], may be much smaller than I(L; τ ), and unlike Shannon information [23] it can decrease when more data are added, for instance when a long trace distracts the model from the relevant part [24, 25]. d) Candidate-wise pass/fail feedback is information-inefficient: Suppose a repair procedure considers N mutually exclusive candidate edits, with a uniform prior over the correct edit L. Identifying the correct edit requires H(L) = log2 N bits; for N = 1000, this is approximately 10 bits. A binary verdict can convey at most one bit in a single attempt, but this upper bound is attained only when its outcomes are approximately equiprobable. If an attempt tests one particular candidate e, then B = I[L = e], so, under the uniform prior, I(L; B) = H(B) = h2 (1/N ), where h2 is the binary entropy function. For N = 1000, this is only approximately 0.011 bits before observing the outcome. Moreover, if the attempt fails, the remaining uncertainty decreases only from log2 N to log2 (N − 1), a reduction of approximately 0.0014 bits for N = 1000. Thus, verdict-only feedback is not intrinsically uninformative: a sequence of verdicts can eventually identify the correct repair. Rather, candidatewise testing is information-inefficient because each unsuccessful attempt produces a highly unbalanced partition of the remaining hypothesis space. Effective search must therefore obtain much of its guidance from the model’s prior, from the structure of the attempted edit, or from richer execution feedback. This illustrates the limitation of verdict-only feedback, consistent with the diminishing contribution of later attempts observed in

previous studies [5, 26] and with findings that multi-turn repair can underperform repeated sampling on competitive programming [20]. B. Why values might help a model Without a trace, a model that wants to locate a fault must simulate the program internally. If it simulates each step correctly with probability a, it simulates T steps correctly; with a = 0.98 and T = 50, it has only about a 36% chance of simulating all 50 steps correctly under this simplified model. A trace replaces simulation with lookup, so the model only has to judge whether each recorded value is plausible given the task. This resembles the contrast between process-level and outcome-level supervision in reasoning, where feedback on intermediate steps is more useful than feedback on the final answer alone [27, 28]. We derive five hypotheses from the argument above. Hypothesis H1 (Localisation). Where the feedback already shows the failing input, the expected output and the actual output, a trace still adds information about the location of the fault, so traces repair some failures that the baseline does not. Hypothesis H2 (Stalling). When the feedback does not localise the fault, the model often resubmits identical or near-identical code; traces should reduce this. Hypothesis H3 (Information floor). Where a failing test is hidden, the feedback carries at most one bit; no trace can be produced, so repair attempts should be no more productive than fresh samples. Hypothesis H4 (Ceiling). Traces help with implementation slips, such as a wrong condition, but not with a misread task, wrong algorithm or missing functionality. Here the trace faithfully records a program that does the wrong thing. Hypothesis H5 (Usable information). Multiple traces added at once, in a single context window, are not necessarily more useful. One trace should perform at least as well as all traces, at lower cost. IV. M ETHODOLOGY BPD extends a PyCapsule-style repair loop [5] with one change. When a program fails an assertion on a test whose input is visible, the feedback contains the program annotated with its own execution trace, showing the model what a developer would see with a breakpoint on every executed line. All other failures receive the baseline’s structured error message. We run the failing test under a line-level tracer sys.settrace and write three kinds of annotations as end-of-line comments (Fig. 1). Assignments show

TABLE I ACCURACY ON ALL TASKS (%) AND TASKS REPAIRED ON THE BPD SET ( COUNT ). “F RESH ” MAKES SIX PLAIN CALLS PER TASK . B Y CONSTRUCTION , THE BASELINE REPAIRS NONE OF THE BPD SET. B EST PERFORMANCE IN EACH ROW IS SHOWN IN BOLD .

Baseline

Fresh start

One trace

All traces

Benchmark

Model

BPD set

Acc. (%)

Acc. (%)

Repaired

Acc. (%)

Repaired

Acc. (%)

Repaired

BigCodeBench

7B 14B 32B 72B Pooled

320 258 222 214 1,014

63.5 71.3 76.3 76.3 71.9

70.9 76.9 80.3 79.3 76.8

84 64 45 34 227 (22.4%)

71.7 78.8 81.8 81.4 78.4

93 85 63 58 299 (29.5%)

70.2 77.3 82.1 80.8 77.6

76 68 66 51 261 (25.8%)

HumanEval

7B 14B 32B 72B Pooled

17 11 6 6 40

89.6 93.3 96.3 96.3 93.9

95.1 97.0 97.6 98.2 97.0

9 6 2 3 20 (50.0%)

95.1 97.0 98.2 98.2 97.1

9 6 3 3 21 (52.5%)

93.9 98.8 97.6 96.3 96.6

7 9 2 0 18 (45%)

LiveCodeBench

7B 14B 32B 72B Pooled

121 120 109 99 449

27.4 28.6 33.7 34.9 31.1

32.0 33.7 38.3 37.1 35.3

8 9 8 4 29 (6.5%)

29.1 32.6 38.3 37.1 34.3

3 7 8 4 22 (4.9%)

29.7 30.9 38.9 36.6 34.0

4 4 9 3 20 (4.5%)

the values taken by the assigned variable, loop headers show the sequence of loop-variable values, and controlflow statements show the iteration at which they were reached and which iterations were skipped as a result. Sequences over 6 elements keep the first 4 and last 2 with . . . between; values over 40 characters are clipped; and a trace is capped at 6,000 characters, with no trace if tracing runs past 20,000. The prompt also contains the failing input, and the expected and actual outputs. One trace annotates the program with the first failing test only; all traces adds a trace for every failing visible test sequentially.

1 2 3 4 5 6 7

def sum_of_evens(nums): # nums = [2, 4, 3, 2] total = 0 # total = 0 for n in nums: # n = 2, 4, 3 if n % 2 == 1: # Fx2, T break # iter 3 total += n # total = 2, 6 return total # returns 6 Input: [2, 4, 3, 2]

Expected: 8

Actual: 6

Fig. 1. An illustrative BPD prompt fragment.

BPD differs from LDB [6] in three ways. It is applied only to assertion failures, leaving crashes to tracebacks that already localise them. It does not segment the program into blocks or ask for per-block verdicts, since function-level programs fit whole in the prompt. And it places each value on the line that produced it rather than in a separate listing.

V. R ESULTS A. Experimental Setup Benchmarks. We use HumanEval [29] (164 tasks), BigCodeBench [2] (1,140 tasks, full split) and 175 LiveCodeBench [3] problems (version [30]). On HumanEval and BigCodeBench all tests are visible, and a failed assertion reports the input, the expected output and the actual output. LiveCodeBench also has hidden tests, whose failure is reported only as “a hidden test failed”, with no input or trace. Models. We use Qwen2.5 Instruct models with 7B, 14B, 32B and 72B parameters [31], all released in December 2024, before the release of the LiveCodeBench benchmark version used in our study [30]. Protocol. The baseline is a PyCapsule-style loop [5]: an initial generation (attempt 0) followed by up to five repair attempts with structured error messages, stopping when all tests pass. The BPDset comprises tasks for which the baseline exhausted all repair attempts without success, ending with an assertion failure. For each, we rerun the full loop with BPD feedback. Total accuracy with BPD is the baseline’s passes plus the BPD run’s passes on the BPD set. Since rerunning selected failures may solve some problems by chance on the first attempt, we include a fresh-start control that makes six independent plain calls per BPD-set task, without feedback or iteration, following the fresh-start protocol of Debugging Decay Index [26]. Measures. We report accuracy as the share of a benchmark’s tasks that are solved, the share of the BPD set repaired (separating attempt 0), a public-tests-only view for LiveCodeBench, repair behaviour (identical

resubmission, mention of “trace”) and token usage with the Qwen2.5 tokeniser. B. RQ1: Which failures remain? Assertion failures account for 79.0% of the failures left by the baseline on BigCodeBench, all 40 on HumanEval and 93.2% on LiveCodeBench, with little variation across model sizes (Table II). They also dominate the repair process itself. Across all attempts of earlier PyCapsule runs on BigCodeBench, assertion errors make up 47–55% of all error events, three to five times the next type, ValueError, and their share rises with capability, from 46.7% for Qwen2.5-Coder 7B to 55.0% for 32B. As models improve, crashes become rarer and wrong answers make up more of what remains. TABLE II S HARE OF RESIDUAL FAILURES THAT ARE ASSERTION FAILURES (%)

Benchmark

7B

14B

32B

72B

BigCodeBench HumanEval LiveCodeBench

76.9 100 95.3

78.9 100 96.0

82.2 100 94.0

79.3 100 86.8

C. RQ2: Do traces beat fresh sampling? Table I reports the main results. Against the baseline, one trace raises pooled accuracy from 71.9% to 78.4% on BigCodeBench, from 93.9% to 97.1% on HumanEval and from 31.1% to 34.3% on LiveCodeBench. Much of this gain, however, is available without any feedback. Six fresh samples alone repair 22.4% of the BigCodeBench BPD set, 50.0% of HumanEval and 6.5% of LiveCodeBench, tasks that the baseline loop had failed after six calls. Repair loops that keep refining an unlucky first program forgo these solutions, in line with the exploration–exploitation view of repair [19]. Attempt 0 of a BPD run is itself such a sample: it passes on 9.5% of BigCodeBench tasks, against 9.4% for a fresh call. At an equal budget, traces help only where the failing input is visible and the tasks are not algorithmic. On BigCodeBench, one trace repairs 299 tasks against 227 for fresh sampling, and it is ahead for every model. On HumanEval, fresh sampling and one trace achieve comparable repair rates, and the set is too small to separate them. On LiveCodeBench, fresh sampling is ahead of both arms (29 against 22 and 20), though not significantly (p = 0.31). H1 is therefore supported on BigCodeBench, where it is testable, and not on LiveCodeBench, as H3 predicts. The advantage of the trace on BigCodeBench grows with model size compared to fresh sampling. Fresh sampling repairs fewer tasks as models grow, from 26.2% of the set at 7B to 15.9% at 72B, while one trace stays

between 27% and 32%. Two factors likely contribute. Larger models fail fewer tasks, so the tasks they leave are harder, and their fresh samples are less productive: a single fresh call passes on 10.9% of the 7B set but only 6.9% of the 72B set. A capable model sampled again without feedback tends to reproduce the same wrong program, so resampling adds little. Feedback gives it something new to act on, and BPD’s repair rate falls much less as the remaining tasks grow harder. D. RQ3: How do traces change repair behaviour? The clearest effect of the trace is that the model stops resubmitting the same code (Table III). Identical resubmissions fall from 28% to 8% on BigCodeBench and from 23% to 6% on HumanEval, and similarity to the previous program falls from 0.96 to 0.91 and from 0.90 to 0.79, so the model makes larger edits. These edits are more often productive: attempts after which more tests pass rise from 13% to 17% and from 12% to 24%, while attempts after which fewer pass stay at 7% on BigCodeBench and rise from 11% to 14% on HumanEval. A quarter to a third of one-trace replies refer to the trace by name. This supports H2. TABLE III R EPAIR ATTEMPTS ON THE BPD SET (B IG C ODE B ENCH / H UMAN E VAL ).

Measure

Baseline One trace All traces

Identical resubmission (%) 28–23 Similarity to previous .96–.90 More tests pass (%) 13–12 Mentions “trace” (%) 0

8–6 .91–.79 17–24 25–32

9–5 .90–.80 16–16 19–15

A change in behaviour, however, does not guarantee a repair, and LiveCodeBench shows where it falls short. LiveCodeBench differs from the other benchmarks in two ways: it has hidden tests, and its problems are algorithmic. To separate the two, Table IV judges success only on the public tests, where the model sees the input, the expected output, and the trace. Under iterative repair, 27.8% of the BPD set already passes the public tests and fails only hidden ones. For these problems, each attempt returns a single bit, and no trace can be produced, which is the regime of H3. Consistent with H3, repair attempts are no more productive than fresh samples here: fresh samples solve 29 tasks, more than either BPD arm, and the trace-guided attempts after attempt 0 solve 11, the same number as attempt 0 alone. The public tests show that hidden tests are not the whole explanation. Where the trace is available, BPD still gives no gain: the BPD set clears its public tests in 27.8% of problems under iterative repair, 28.5% with one trace and 26.5% with all traces, accuracy on all 175 problems is unchanged (51.7%, 52.1%, 50.9%), and

TABLE IV L IVE C ODE B ENCH BPD SET JUDGED ON PUBLIC TESTS ONLY: SHARE CLEARED (%), WITH THE SHARE CLEARED AT ATTEMPT 0 IN BRACKETS .

Model

Set

Baseline

One trace

All traces

7B 14B 32B 72B

121 120 109 99

12.4 30.0 33.0 38.4

18.2 (12.4) 29.2 (18.3) 38.5 (15.6) 29.3 (14.1)

19.8 (10.7) 23.3 (14.2) 33.0 (15.6) 31.3 (19.2)

Pooled

449

27.8

28.5 (15.1)

26.5 (14.7)

the 14B and 72B models lose. A fresh first sample alone accounts for about half of the public tests cleared. This supports H4. A wrong answer in competitive programming more often reflects a wrong algorithm or a misread constraint than an implementation slip, and on such benchmarks LLMs are limited mainly by their understanding of the problem rather than by their ability to repair [20]. The trace then faithfully records a program that does what it was written to do, so the larger edits it prompts are edits to the wrong idea. Continuing to revise a program that already passes its public tests, in response to a one-bit hidden-test signal, may further explain the losses. E. RQ4: Is one trace enough, and what does it cost? TABLE V T OKEN USAGE ON THE BPD SET, POOLED OVER MODELS (Q WEN 2.5 TOKENISER ). BPD IS B IG C ODE B ENCH , HE IS H UMAN E VAL AND LCB IS L IVE C ODE BANCH

Bench. Run

Calls Tok./call Total (k)

Change

BCB

Baseline 6,086 One trace 5,095 All traces 5,203

1,795 2,061 2,692

10,925 10,502 −3.9% 14,007 +28.2%

HE

Baseline One trace All traces

240 167 168

943 1,139 1,604

226 190 −16.0% 270 +19.1%

LCB

Baseline 2,694 One trace 2,603 All traces 2,628

1,511 1,825 2,306

4,070 4,749 +16.7% 6,060 +48.9%

One trace repairs more tasks in total than all traces on every benchmark. Fewer replies mention the trace when all traces are shown (Table III), suggesting the model attends less to longer traces. This supports H5 and agrees with the finding that the benefit of traces falls as they grow more complex [10]. A trace lengthens every call (Table V), so the total cost depends on whether it saves calls by solving tasks sooner. Total tokens factorise into calls times tokens per call, so a format that makes each call’s token usage ρ times longer pays for itself only if it cuts calls by

at least 1 − 1/ρ. For one trace, ρ is 1.15–1.21, giving a break-even of 13–17%. One trace cut calls by 16% on BigCodeBench and 30% on HumanEval, lowering total tokens by 3.9% and 16.0%, but by only 3% on LiveCodeBench, raising them by 16.7%. All traces, with ρ of 1.50–1.70, never pay for themselves. Fresh sampling is cheaper still per call, since its prompt carries no history. VI. D ISCUSSION , L IMITATIONS AND F UTURE W ORK The results separate two conditions for a trace to help. The first, that the feedback exposes a failing input, is necessary: under hidden-test verdicts no trace exists, and fresh sampling does at least as well (H3). On LiveCodeBench’s public tests, the trace is visible and still gives no gain. Our explanation is H4: competitiveprogramming failures are more often wrong algorithms or misread constraints than implementation slips. We support it only indirectly, and labelling failure causes is the natural next step. The gain is modest: over the whole of BigCodeBench, one trace reaches 78.4% against 76.8% for fresh sampling, and much of what a repair loop recovers from its residual failures can be recovered by sampling again. Traces matter more for larger models, whose residual failures resist resampling. We offer this as a hypothesis, since we did not run LDB. Practical rules. (i) Trace assertion failures only. (ii) Under hidden-test verdicts, and on competitiveprogramming tasks, spend the budget on fresh samples. (iii) Show one trace, not all. (iv) Check the break-even rule: a format that lengthens calls by ρ must cut calls by 1 − 1/ρ. Threats to validity. The BPD set is selected for failure under iterative repair, so any rerun repairs some tasks by chance; the fresh-start control at an equal budget addresses this. Each configuration was run once, so we cannot report run-to-run variance. We classify failures by the exception raised, and resubmission, similarity and the word “trace” are coarse proxies for behaviour. We study one model family and Python function-level tasks; the HumanEval BPD set has only 40 tasks. Longer programs may need block-level summaries of the kind used by LDB. VII. C ONCLUSION This study asked whether execution traces help an LLM repair the assertion failures that iterative repair leaves behind, and whether they do so better than sampling new programs at the same budget. Assertion failures make up 79–100% of the residual failures on three benchmarks, and our information view explains why they persist. Our verdicts on the five hypotheses are as follows. Traces localise faults that the error message does not

(H1). On BigCodeBench, where one trace repaired 26.7% of the residual failures against 22.4% for fresh sampling; on HumanEval the two were within two of 40 tasks, and on LiveCodeBench fresh sampling repaired more (6.5% against 4.9%). Traces clearly change repair behaviour (H2): identical resubmissions fell from 28% to 8% on BigCodeBench and from 23% to 6% on HumanEval, and more attempts increased the number of tests passed. Where the failing test was hidden, repair attempts were no more productive than fresh samples (H3). Where the failing public test was visible on LiveCodeBench, the trace still gave no gain (28.5% against 27.8%), consistent with H4. One trace repaired at least as many tasks as all traces on every benchmark and was the only variant to lower token cost, by 3.9% on BigCodeBench and 16.0% on HumanEval (H5). The central lesson is that changing the model’s behaviour is not the same as solving the problem. A trace makes the model edit instead of stall, and on BigCodeBench those edits repair failures that resampling does not. Competitive-programming problems, however, are more often failed because the model misunderstands the problem than because it implements it incorrectly [20], and there, a trace only records a program built on the wrong idea. Repair feedback should therefore be compared with fresh sampling at an equal budget, and a repair loop should resample rather than trace when the failing tests are hidden or the task is algorithmic. Future work should label the causes of failure, extend the comparison to other model families and longer programs, and build loops that choose between tracing and resampling according to the feedback available. R EFERENCES [1] M. Chen, J. Tworek, H. Jun, Q. Yuan et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021. [2] T. Y. Zhuo, M. C. Vu, J. Chim, H. Hu, W. Yu et al., “BigCodeBench: Benchmarking code generation with diverse function calls and complex instructions,” in International Conference on Learning Representations, 2025. [3] N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica, “LiveCodeBench: Holistic and contamination free evaluation of large language models for code,” in International Conference on Learning Representations, 2025. During the preparation of this manuscript, the authors used Claude Sonnet 5.5 to debug and refine the draft text. The authors have reviewed and edited all AI-assisted output and take full responsibility for the content of this publication. The code and data are openly available at https://github.com/TheOpenSI/SYNKRASIS/tree/BPDebug

[4] X. Chen, M. Lin, N. Schärli, and D. Zhou, “Teaching large language models to self-debug,” in International Conference on Learning Representations, 2024. [5] M. Adnan, Z. Xu, and C. C. N. Kuhn, “Large language model guided self-debugging code generation,” arXiv preprint arXiv:2502.02928, 2025. [6] L. Zhong, Z. Wang, and J. Shang, “Debug like a human: A large language model debugger via verifying runtime execution step by step,” in Findings of the Association for Computational Linguistics: ACL 2024, 2024, arXiv:2402.16906. [7] T. X. Olausson, J. P. Inala, C. Wang, J. Gao, and A. Solar-Lezama, “Is self-repair a silver bullet for code generation?” in International Conference on Learning Representations, 2024. [8] M. A. Islam, M. E. Ali, and M. R. Parvez, “MapCoder: Multi-agent code generation for competitive problem solving,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Bangkok, Thailand: Association for Computational Linguistics, Aug. 2024, pp. 4912– 4944. [Online]. Available: https://aclanthology.org/ 2024.acl-long.269/ [9] J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le et al., “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732, 2021. [10] M. Haque, P. Babkin, F. Farmahinifarahani, and M. Veloso, “Towards effectively leveraging execution traces for program repair with code LLMs,” in Proceedings of the 4th International Workshop on Knowledge-Augmented Methods for Natural Language Processing (KnowledgeNLP’25). Association for Computational Linguistics, 2025, pp. 160– 179. [11] J. J. Wang, X. Xie, Q. Hu, S. Liu, and Y. Li, “Do code semantics help? a comprehensive study on execution trace-based information for code large language models,” in Findings of the Association for Computational Linguistics: EMNLP 2025. Suzhou, China: Association for Computational Linguistics, Nov. 2025, pp. 10 367– 10 385. [Online]. Available: https://aclanthology. org/2025.findings-emnlp.548/ [12] J. Huang, W. Ye, W. Sun, J. Zhang, M. Zhang, and Y. Liu, “TraceCoder: A trace-driven multiagent framework for automated debugging of LLMgenerated code,” in Proceedings of the IEEE/ACM 48th International Conference on Software Engineering (ICSE), 2026, arXiv:2602.06875. [13] J. Wu, T. Wu, M. Zhang, Y. Dong, and B. Shen,

“Runtime execution traces guided automated program repair with multi-agent debate,” arXiv preprint arXiv:2604.02647, 2026. [14] J. M. Voas, “PIE: A dynamic failure-based technique,” IEEE Transactions on Software Engineering, vol. 18, no. 8, pp. 717–727, 1992. [15] A. Ni, M. Allamanis, A. Cohan, Y. Deng, K. Shi, C. Sutton, and P. Yin, “NExT: Teaching large language models to reason about code execution,” in Proceedings of the 41st International Conference on Machine Learning, ser. PMLR, vol. 235, 2024, pp. 37 929–37 956. [16] J. Huang, X. Chen, S. Mishra, H. S. Zheng, A. W. Yu, X. Song, and D. Zhou, “Large language models cannot self-correct reasoning yet,” in International Conference on Learning Representations, 2024. [17] Z. Huang, L. Xu, C. Liu, W. Sun, X. Zhang, Y. Lei, M. Yan, and H. Zhang, “Dynafix: Iterative automated program repair driven by executionlevel dynamic information,” arXiv preprint arXiv:2512.24635, 2025. [18] A. Gu, B. Rozière, H. Leather, A. Solar-Lezama, G. Synnaeve, and S. I. Wang, “CRUXEval: A benchmark for code reasoning, understanding and execution,” in Proceedings of the 41st International Conference on Machine Learning, 2024. [19] H. Tang, K. Hu, J. P. Zhou, S. C. Zhong, W.-L. Zheng, X. Si, and K. Ellis, “Code repair with LLMs gives an exploration-exploitation tradeoff,” in Advances in Neural Information Processing Systems, 2024. [20] K. Zheng, J. Decugis, J. Gehring, T. Cohen, B. Negrevergne, and G. Synnaeve, “What makes large language models reason in (multi-turn) code generation?” in International Conference on Learning Representations, 2025. [21] T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed. Wiley, 2006. [22] Y. Xu, S. Zhao, J. Song, R. Stewart, and S. Ermon, “A theory of usable information under computational constraints,” in International Conference on Learning Representations, 2020. [23] C. E. Shannon, “A mathematical theory of communication,” The Bell System Technical Journal, vol. 27, no. 3, pp. 379–423, 1948. [24] F. Shi, X. Chen, K. Misra, N. Scales, D. Dohan, E. H. Chi, N. Schärli, and D. Zhou, “Large language models can be easily distracted by irrelevant context,” in Proceedings of the 40th International Conference on Machine Learning, 2023. [25] N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,” Transactions of the Association for Com-

putational Linguistics, vol. 12, pp. 157–173, 2024. [26] M. Adnan and C. C. Kuhn, “Measuring and mitigating debugging effectiveness decay in code language models,” Scientific Reports, vol. 15, no. 1, p. 44120, 2025. [27] J. Uesato, N. Kushman, R. Kumar, F. Song, N. Siegel, L. Wang, A. Creswell, G. Irving, and I. Higgins, “Solving math word problems with process- and outcome-based feedback,” arXiv preprint arXiv:2211.14275, 2022. [28] H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe, “Let’s verify step by step,” in International Conference on Learning Representations, 2024. [29] M. Chen, J. Tworek, H. Jun, Q. Yuan et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021. [30] Test6, HuggingFace. [Online]. Available: https://huggingface.co/datasets/livecodebench/ code_generation_lite/blob/main/test6.jsonl [31] Qwen Team, “Qwen2.5 technical report,” arXiv preprint arXiv:2412.15115, 2024.

Related documents

• Does Fault Localization Beat a Fresh Attempt? A Placebo-Controlled Study of Test-Guided Code Repair#820858
• Does Fault Localization Beat a Fresh Attempt? A Placebo-Controlled Study of Test-Guided Code Repair#627754
• From Reading Code to Reading Spec: A Verified Layer for LLM-Driven Codebase Maintenance#668138
• Validation Evidence in LLM Repair Agents: How Much of What Passes Actually Tests the Bug?#422333
• Resilience of the surface code to error bursts#1330529