Validation Evidence in LLM Repair Agents: How Much of What Passes Actually Tests the Bug? Xiaonan Xu1* and Wenjing Wu2 1*
College of Computing, Georgia Institute of Technology, Atlanta, GA, 30332, USA. 2 Department of Computer Science, University of Colorado Boulder, Boulder, CO, 80309, USA.
arXiv:2607.28871v1 [cs.SE] 30 Jul 2026
*Corresponding author(s). E-mail(s): [email protected]; Abstract When a repair agent runs a test and sees it pass, the result is treated as evidence about the reported defect. We measure how often that treatment is warranted. BSG-VA (buggy-state/candidate-state/gold-fix validation analysis) captures each validation command at its exact working-tree state, extracts a test-only patch, and replays the command on the original buggy code (B), the candidate state (S), and the developer gold fix (G). The captured outcome and the replay results assign every event an evidence role, from gold-aligned bugdiscriminating through regression-only to misleading. Across 3,730 events in 643 rollouts on 110 tasks, 46.0% of positive comparable events carry no bugdiscriminating information; 23.8% of baseline rollouts, with no feedback injected, close with a patch whose entire positive evidence base is of this kind. A three-arm experiment tests whether returning the B-replay outcome to the agent changes this pattern. Bug-contrast feedback reduces evidence-inadequate closure by 7.8 percentage points relative to an attention-matched reminder (p = 0.0029) and raises bug-discriminating evidence by 7.4 points (p = 0.011), with no detectable cost to repair success. Both estimates fall below the prespecified 10-percentagepoint smallest effect size of interest, so practical magnitude remains uncertain. Roughly a third of the improvement traces to the reminder alone; across two exploratory replications, varying the scaffold and the model, the B-replay content adds a detectable increment only with gpt-5.6-sol under the unconstrained tool-use loop. BSG-VA applies post hoc to any replayable repair trajectory that preserves the required code states and execution environment. Keywords: program repair agents, validation evidence, test adequacy, large language models, software quality, controlled experiment
1
1 Introduction Repair agents on SWE-bench-style benchmarks now resolve a substantial and growing share of real-world GitHub issues. A typical successful trajectory involves dozens of tool calls: reading files, editing code, running tests, observing outputs, and iterating. Among these actions, running a test and seeing it pass occupies a privileged position: it is the closest thing the agent has to empirical confirmation that its changes work. Benchmark evaluation reinforces the reading, since the final verdict is whether the submitted patch passes a held-out test suite. But a passing test can mean different things. Consider an agent assigned a defect in a date-formatting routine. The agent modifies the routine, then writes a test that imports the module and asserts that the output is a string. The test passes. It would also have passed on the original buggy code, because the bug was not about return types but about locale handling. The agent has confirmed that its patch does not break the import or change the output type. It has learned nothing about whether the locale bug is fixed. If this is the only positive evidence the agent collects before submitting, the submission rests on validation that is real but irrelevant to the assigned defect. How common is this pattern? No existing method answers the question, because mid-trajectory validation events have not been systematically evaluated for evidential content. Prior work on agent-generated tests has treated them as standalone artefacts: the SWT-Bench family evaluates dedicated pipelines that take an issue description and produce a test designed to fail on buggy code and pass on the fix (Ahmed et al., 2025; Khatib, Mathews, & Nagappan, 2025; Mündler, Müller, He, & Vechev, 2024). These are dedicated systems, separate from the repair process. Trajectory analyses of frontier models (Z. Chen et al., 2026) have characterised the frequency and syntactic form of in-trajectory test events, showing that agents write tests often but not evaluating what those tests prove about the assigned bug. The gap between “the agent ran a test” and “the test targeted the defect” remains unmeasured. We introduce BSG-VA (buggy-state/candidate-state/gold-fix validation analysis) to close this gap. The method intercepts every validation command the agent executes, snapshots the working tree at execution time, and extracts a test-only patch that separates the validation logic from any concurrent production edits. It then replays the same command on three code states: the original buggy version (B), the captured candidate (S), and the developer gold fix (G). The captured outcome and the replay results place each event in one of seven evidence roles. An event where the test fails on B, passes on S, and passes on G is gold-aligned bug-discriminating: it targets a property specific to the defect, in a way that would also accept the developer’s own fix. An event that passes on B as well as S is at best regression-only: it confirms the patch has not introduced a new failure, but says nothing about the bug. Between these poles sit candidate-specific events (B-fail, S-pass, G-fail), misleading events, and diagnostic negatives; two further categories set aside events whose outcomes are unstable or not evaluable. The taxonomy is exhaustive, mutually exclusive, and defined entirely by recorded, observable outcomes. We apply BSG-VA at scale in a controlled experiment using gpt-5.6-sol (hereafter sol ) on 110 tasks drawn equally from SWE-bench Verified and SWE-rebench. The measurement covers 3,730 retained post-edit validation events across 643 rollouts 2
(events before the agent’s first production edit are excluded, as discussed in Section 3). Of 2,548 positive comparable events, 46.0% are regression-only or misleading. At the rollout level, 23.8% of baseline runs exhibit what we call evidence-inadequate closure: the agent submitted a patch, collected positive validation results, and none of those results discriminated the reported bug from a regression-free alternative. The measurement enables a direct intervention. Because BSG-VA can evaluate an event’s B-replay outcome in real time (median 11.0 seconds per rollout), we test whether feeding that outcome back to the agent changes the closure pattern. In the three-arm experiment, bug-contrast feedback (BCF) takes the agent’s passing validation event, replays it on B, and reports the result in a system message. If the test also passes on B, the agent learns its evidence is non-discriminating. The comparison arms are a structure-matched generic reminder that prompts the agent to reconsider its evidence without providing B-replay information, and a baseline arm into which no messages are injected. BCF reduces evidence-inadequate closure by 7.8 percentage points relative to the reminder (p = 0.0029, 95% CI [−12.9, −2.7]) and raises bugdiscriminating evidence by 7.4 points (p = 0.011). Repair success shows no detectable change. The active-control decomposition shows the generic reminder alone accounts for roughly a third of the total BCF-versus-baseline improvement, a share that comes from prompting the agent to attend to evidence quality rather than from the B-replay content itself. Under a plan-execute-verify scaffold, baseline evidence-inadequate closure falls to 8.3%, a floor effect that leaves little room for either component. On gpt-5.6-terra (hereafter terra ) in the unconstrained loop, the generic reminder produces a 7.5-point reduction on its own, comparable to the full BCF effect on sol, while the B-replay content adds nothing detectable. The prompting effect recurs on the second model; the B-replay content adds measurable value only with gpt-5.6-sol in the unconstrained loop. The study is organised around four objectives. RQ1 (descriptive): what share of positive validation evidence produced mid-trajectory discriminates the assigned bug? RQ2 (associational): does evidence quality carry predictive signal for official resolution (the benchmark’s own pass/fail verdict) beyond task-level covariates? The confirmatory objective tests whether BCF reduces evidence-inadequate closure relative to a structure-matched generic reminder, with bug-discriminating evidence as a gated secondary outcome. Two exploratory replications (Shull, Carver, Vegas, & Juristo, 2008) then ask how these patterns change across a different scaffold and a second model. This paper contributes an event-level measurement method and evidence-role taxonomy that assign each validation event an evidence role with directly checkable content; a quantified prevalence estimate showing that nearly half of the positive comparable validation evidence in a 3,730-event study does not discriminate the assigned bug; and a fully crossed three-arm experiment that separates the prompting effect of a generic reminder, which recurs on a second model, from the additional increment of B-replay content, which does not.
3
2 Background and related work 2.1 Repository-level program repair with LLM agents Automated program repair (APR) has evolved from template-based and search-based patch generation (Le Goues, Dewey-Vogt, Forrest, & Weimer, 2012; Weimer, Nguyen, Le Goues, & Forrest, 2009) toward neural and large-language-model approaches (Jiang, Liu, Lutellier, & Tan, 2023; Xia, Wei, & Zhang, 2023). The current generation combines a large language model with an agentic tool-use loop: the model reads an issue description, navigates the repository, edits files, and runs commands, iterating until it produces a candidate patch or exhausts a budget. SWE-agent (Yang et al., 2024) established the architecture; Agentless (Xia, Deng, Dunn, & Zhang, 2025) showed that a fixed pipeline of localisation, repair, and patch validation without persistent agent state can be competitive; CodeR (D. Chen et al., 2024), Moatless (Örwall, 2024), and AutoCodeRover (Zhang, Ruan, Fan, & Roychoudhury, 2024) introduced variations on retrieval, planning, and tool design. SWE-bench (Jimenez et al., 2024) and its curated subset SWE-bench Verified (OpenAI, 2024) provide the standard evaluation protocol: the agent receives an issue, works within a repository checkout, and submits a patch that is evaluated against a held-out developer test suite. Across all these architectures, agents routinely execute validation commands during the trajectory. The evaluation protocol, however, examines only the final submitted patch. The mid-trajectory validation activity is logged but otherwise ignored by the benchmark harness.
2.2 Deliberately generated tests for patch validation A separate body of work generates tests deliberately, as a post-hoc step to validate candidate patches. SWT-Bench (Mündler et al., 2024) defines the fail-to-pass criterion: a generated test must fail on the original buggy code and pass on the developer fix. Filtering agent patches through such tests raises precision substantially (Mündler et al., 2024). The protocol has become a competitive benchmark in its own right, with systems including Otter, AssertFlip, Issue2Test, Echo, and EvoOtter optimising failto-pass success rates (Ahmed et al., 2025; Ahmed, Ganhotra, Shinnar, & Hirzel, 2026; Fei et al., 2026; Khatib et al., 2025; Nashid, Bouzenia, Pradel, & Mesbah, 2026), and with repair agents tasked to co-generate fixes and reproduction tests inside the repair loop (Cheng et al., 2026). LIBRO (Kang, Yoon, & Yoo, 2023) generates issuereproducing tests from bug reports without access to the developer fix at generation time. A valid fail-to-pass test is still not a complete oracle: it can cover a single manifestation of the issue and steer repair toward a partial patch (Guo et al., 2026). Assessment of generated suites has its own tooling, from mutation-based measures of discriminative power (Sun et al., 2026) to lifecycle evidence infrastructure that tracks execution, coverage, and flakiness (Leary, Hanuska, & Brown, 2026). The distinction between this body of work and the present study is one of context and agency. Deliberate test generation is an external pipeline step, executed after the agent’s trajectory is complete, with the explicit goal of producing a discriminating test. BSG-VA examines the agent’s own spontaneous validation behaviour during the
4
trajectory, where the agent decides what to test, when, and how, without any external instruction to produce a fail-to-pass test. The two perspectives are complementary: the former asks whether discriminating tests can be generated on demand, while the latter asks what the tests that agents already generate actually prove.
2.3 Trajectory analysis and test-value assessment Z. Chen et al. (2026) analyse six models on SWE-bench Verified and report that test-writing frequency is similar between successful and failed trajectories; that most validation commands are print-statement explorations rather than assertion-bearing tests; and that prompting models to write more or fewer tests produces no detectable change in repair outcomes. The finding that test-writing is abundant but not predictive of success is consistent with the hypothesis that much of the validation activity lacks discriminating power, but the study does not measure discriminating power directly. Lin et al. (2026) analyse the cost-effectiveness of code execution in LLM-based repair at scale and report that failed commercial-agent runs often pass their own self-validation while failing official evaluation. That mismatch is the starting point here: where their analysis works at the level of run frequency, cost, and coarse agreement between the agent’s own test results and the official evaluation, BSG-VA reconstructs the exact candidate state of each self-selected command and replays it to establish which passing checks could not have detected the defect. Broader diagnostics work instruments trajectories at a structural level: luckypass detection and production trajectory review (Podivilov et al., 2026; Sahoo et al., 2026), trajectory-structure and verification-skip diagnostics (Bouzenia & Pradel, 2025; Shu et al., 2026), traceability analyses of reproduction and regression testing across repair agents (Ceka et al., 2026), and interactive trajectory visualisation (Sajadi, Nguyen, Huynh, Parra, & Chatterjee, 2026). Cross-framework comparison warns that the same aggregate trajectory signal can carry different semantics in different frameworks (W. Ma et al., 2026). BSG-VA is narrower: it attaches an executable, issuespecific meaning to individual validation events rather than characterising trajectory structure. Overfitting to the validation signal has a long history in APR: generated patches can overfit weak test suites (Smith, Barr, Le Goues, & Brun, 2015), and test generation was proposed early to expose overfitted patches (Xin & Reiss, 2017). Konstantinou, Tambon, and Papadakis (2026) approach the agent-era version of the problem, showing that tests generated after exposure to an incorrect implementation detect fewer faults than independently generated tests, a mechanism by which non-discriminating suites arise. Controlled evidence that incorrect code context reduces generated-test fault detection supports the same account (Huang, Zhang, Harman, Du, & Cui, 2026). Hora and Robbes (2026) document a related fragility in which agent commits mock more heavily than non-agent commits, risking tests that compile and run but exercise little of the relevant behaviour. Y. Ma, Kereopa-Yorke, and Schultz (2026) show, in a controlled code-as-spec study, that agents can satisfy every visible check without making the requested artefact load-bearing. These findings identify specific mechanisms by which validation evidence can be non-discriminating, but none provides a systematic taxonomy of evidence roles or a prevalence estimate across a controlled population. 5
B: buggy base agent trajectory
validation event command + working tree
test-only patch
S: candidate
same command, pinned container
evidence role
G: gold fix
Fig. 1 BSG-VA: each self-selected validation event is captured at its exact working-tree state, reduced to a test-only patch, and replayed deterministically on the buggy base (B), the captured candidate (S), and the developer gold fix (G); the captured outcome and the replay pattern assign an evidence role
BSG-VA extends this line by operationalising the measurement. Rather than classifying tests by static syntactic signals, as in large-scale oracle-signal audits of agent-authored test code (Banik, Chowdhury, & Shamim, 2026), or by comparing agent tests with developer tests as monolithic suites, BSG-VA captures each event at its exact state, isolates the test-only component, and replays it on the buggy, candidate, and gold states. The captured outcome and the replay pattern assign each individual event an evidence role with directly checkable content, yielding an event-level taxonomy and prevalence estimate.
2.4 Feedback and steering in agentic repair Existing work on steering agent behaviour during repair has focused on tool design, retrieval augmentation, and planning structure (Örwall, 2024; Yang et al., 2024; Zhang et al., 2024). At a conceptual level, automated verification signals have been framed as imperfect proxies constrained by scalability, faithfulness, and robustness (Wang et al., 2026), and process-discipline benchmarks score whether agents verify and recover at all (Madiraju & Madiraju, 2026). Closest in spirit to an evidence-quality intervention, EviACT gates repair actions behind evidence-driven retrieval, compilation, and targettest guardrails (Meng, Zhang, Ren, & Visser, 2026). Few studies have investigated online feedback about the quality of the agent’s own validation evidence as a steering signal. The approach tested here, in which the agent receives the outcome of replaying its own test on the original buggy state, is related in spirit to self-debugging techniques (X. Chen, Lin, Schärli, & Zhou, 2023) that feed execution output back into the model, but differs in that the information provided is a counterfactual comparison (the test on a state the agent never visited) rather than an observation of the current execution.
3 The BSG-VA measurement method 3.1 Event capture BSG-VA instruments the agent’s tool loop so that every shell command the agent issues after its first production edit is intercepted and classified (Figure 1). Classification is online and identical across all experimental arms. Each command is labelled as one of four types: a project test run (an invocation of the repository’s own test runner on its own test files), a scripted explicit oracle (an agent-written script whose exit code encodes a pass/fail verdict), an observational probe (output to be inspected by 6
the model without a programmatic oracle), or a non-validation action. Non-validation actions are logged but excluded from the measurement. The classifier was audited against a sample of 111 events labelled manually by an author; it agreed on 92.8% of them, the 8 disagreements were adjudicated by the second author, and all resulting corrections were adopted into the production classifier. For each validation event the harness records the command string, the standard output and error streams, the exit status, and the working-tree state at execution time. The working-tree state is decomposed into two components: a production patch containing the agent’s changes to the repository’s source and configuration files, and a test-only patch containing everything the agent added or modified to support the validation itself (new test files, test helpers, fixture data). The decomposition matches changed paths against the repository’s existing test directory conventions and the set of files the agent created during the trajectory.
3.2 Test-only patch extraction Reducing the captured state to a test-only patch is the step that makes cross-state replay meaningful. If the full working tree were replayed on B, any test depending on the agent’s production changes would fail for trivial reasons (missing functions, changed imports), and the replay outcome would reflect dependency breakage rather than evidential content. The test-only patch retains only the validation artefacts: the test files, scripts, and fixtures the agent created or modified, applied on top of whatever code state the replay targets. An event whose test-only patch is empty indicates that the agent ran a pre-existing repository test without adding test code; these events are retained because the choice of which existing test to run is itself a validation decision. The extraction handles several boundary cases. When the agent’s test imports a function that exists only in the agent’s production patch, the replay on B will fail with an import error rather than a test-logic failure. BSG-VA flags such events as depending on a new production symbol. When the agent creates a temporary file that its test reads, the file is included in the test-only patch. When the agent modifies an existing test file, the extraction preserves the modification against the repository’s baseline version of that file. These cases are defined in advance in the event codebook; the classification is deterministic given the working-tree snapshot.
3.3 Cross-state replay Each retained event is replayed on three code states, each constructed in a fresh container from the same base image used during the agent’s original run. State B is the original buggy code with the test-only patch applied. B represents the question: would this validation event have detected the reported defect? If the test passes on B, the event carries no information that distinguishes buggy from candidate code. State S is the captured candidate code. Replaying on S checks consistency: in the common case the outcome matches what the agent observed during the trajectory, and Section 3.4 defines how the remaining cases are classified.
7
State G is the developer gold fix: the buggy code with the benchmark’s gold patch applied, plus the same test-only patch. G answers the question: is the event’s discriminating power tied to the specific candidate, or does it generalise to the correct fix? An event that fails on B and passes on S but also fails on G discriminates the bug from this particular candidate, but would reject the developer’s own solution. Every event is replayed once per state. Stability is assessed by comparing the S-replay with the captured outcome: disagreement marks the event flaky, and flaky events are excluded from all constructs. This rule favours precision over recall: only stable observations support role assignment.
3.4 Evidence-role taxonomy Two kinds of outcome attach to each event. The captured outcome is the result the agent observed during the trajectory. The replay outcomes are what the offline replays produce on B, S, and G. Roles are assigned in a fixed order that makes the seven categories mutually exclusive and exhaustive: the captured outcome anchors the assignment, the B and G replays supply the bug contrast, and the S-replay enters through the stability comparison of Section 3.3. DIAGNOSTIC NEGATIVE. The captured outcome is not positive. The event gave the agent no positive result on the candidate and carries no positive evidence, whatever its replays show. FLAKY. The captured outcome is positive but the S-replay disagrees with it. The observation is not reproducible; flaky events are excluded from all constructs. NOT COMPARABLE. The captured outcome is positive and stable, but the B-replay or the G-replay is not evaluable. Reasons include import errors from tests that depend on symbols introduced by the agent’s production edits, missing temporary artefacts, container setup failures, and timeout. Every remaining event has a stable positive captured outcome with evaluable B and G replays and receives one of four roles. GOLD ALIGNED BUG DISCRIMINATING. B fails, G passes. The event detects a property present in the buggy code and absent in both the candidate and the gold fix. This is the strongest form of validation evidence: it targets the defect and generalises beyond the specific candidate. CANDIDATE SPECIFIC. B fails, G fails. The event distinguishes the bug from the candidate, but its discriminating power depends on implementation details unique to the agent’s patch. The gold fix does not satisfy this test. REGRESSION ONLY. B passes, G passes. The event confirms only that the candidate has not introduced a failure relative to the original code. It says nothing about whether the bug is fixed. MISLEADING. B passes, G fails. The check is one the buggy code satisfies and the developer gold fix does not: the passing result contradicts the developer’s fix, so reading it as support for the candidate misleads. An event is positive comparable when its captured outcome was positive and its assigned role is one of the four roles from GOLD ALIGNED BUG DISCRIMINATING to MISLEADING; under the assignment order above, these are exactly the events that
8
reach the four outcome roles. These are the events the agent could have counted as supporting evidence, evaluated for what they in fact establish. Two rollout-level constructs aggregate the event roles. A rollout provides bugdiscriminating evidence when it contains at least one gold-aligned bug-discriminating or candidate-specific event, the two roles whose checks fail on the buggy state. Evidence-inadequate closure (EIC) is true when a rollout submitted a patch, produced at least one positive comparable event, and contains no bug-discriminating event: every positive comparable result is regression-only or misleading.
3.5 Use of AI assistants An LLM assistant (Claude, Fable 5 model, Anthropic) was used in four capacities: configuring the container-based experiment environment (Docker images, harness wiring, replay infrastructure); implementing and running the harness and orchestration code that launched experiment sessions and collected data, under designs specified by the authors (the repair agents under study are separate OpenAI models, not this assistant); writing analysis code; and improving the readability and language of text written by the authors. All experimental parameters, analysis plans, and decision rules were specified by the authors before the corresponding model requests. The analysis code was verified by the authors through code review, unit-level checks against manually computed reference values, and reproduction of every reported statistic from the released data. The authors reviewed all LLM-assisted text, revised it against the style and accuracy standards of the study, and take full responsibility for the final content.
4 Study design 4.1 Task population We drew tasks equally from two sources: SWE-bench Verified, a curated subset of SWE-bench with confirmed human solutions (Jimenez et al., 2024; OpenAI, 2024), and SWE-rebench, a dataset of GitHub issues filed after the model’s training cutoff, with lower contamination risk (Badertdinov et al., 2025). The two sources define two strata, one of curated verified tasks and one of recent tasks with lower contamination risk, weighted equally in all primary analyses. Each task specifies a repository, a commit, an issue description, and a container image. Official resolution is judged by each benchmark’s evaluator on the submitted production patch, with agent-written test artefacts stripped before evaluation.
4.2 Three-arm design Every task is run under all three arms, with 2 repetitions per task and arm: a fully crossed, within-task design. Because each task serves as its own control, no betweenarm randomisation of tasks is required, and every arm contrast is estimated within task by the estimator of Section 4.3. Baseline. The agent works with the default scaffold: an unconstrained tool-use loop in which the model freely interleaves file reads, edits, and command execution until it submits (hereafter the unconstrained loop ). No messages are injected. 9
Table 1 Intervention messages. BCF delivers exactly one of its three variants according to the deterministic B-replay outcome Condition
Message
Static Reminder
This check passed in the current candidate. A passing check may cover only part of the reported issue. Before concluding, consider whether it is sensitive to the reported behavior and whether regressions remain. This check also passes on the original buggy version. It provides regression evidence but does not distinguish the two states. Before concluding, consider whether your validation is sensitive to the reported behavior and whether regressions remain. This check fails on the original buggy version and passes in the current candidate. It distinguishes the two states, but one check does not establish complete correctness. Before concluding, consider remaining issue behavior and regressions. The original-version replay did not reach a comparable behavioral check, so the current pass is unverified evidence for the reported issue. Before concluding, consider a check that is directly sensitive to the reported behavior.
BCF, B passes
BCF, B fails
BCF, not comparable
Messages are reproduced verbatim as delivered to the agent (hence the American spelling).
Static Reminder. After each eligible positive validation event, a system message prompts the agent to reconsider whether its evidence targets the reported defect. The message carries no B-replay information. Its structure, length, and directive strength are matched to the BCF message, so this arm serves as an attention control that isolates the effect of the B-replay content from the effect of prompting the agent to attend to evidence quality. Table 1 reproduces the text. Bug-Contrast Feedback (BCF). After each eligible positive validation event, the harness replays the event’s test-only patch on B in real time and injects the outcome as a system message. If the test passes on B, the message states that the check does not distinguish the candidate from the original buggy code. If the test fails on B, the message confirms that the check detects a difference. Events where the B-replay is not evaluable receive a message stating that the comparison was not possible. Table 1 reproduces all message variants. All arms share one model configuration: gpt-5.6-sol in standard reasoning mode with high effort, 6,000 maximum output tokens per response, and no repository network access. Each rollout is capped at 200 responses and 200 tool calls. Eligibility for feedback is defined identically in all arms: the event must be a validation event with a positive captured outcome, and it must not be a duplicate of an event already triggered in the same rollout. In the baseline arm, eligible events are identified but no message is sent; this ensures that trigger-rate statistics are comparable across arms.
4.3 Confirmatory experiment The confirmatory experiment covers 110 tasks (55 per stratum) with 2 repetitions of each task under each arm, giving 660 planned cells. 16 candidate tasks failed an
10
Table 2 Confirmatory experiment population and design Tasks Repositories Arms Repetitions per task and arm Planned rollout cells Observed cells Cells lost to failures (by arm) Model Scaffold Primary contrast Primary outcome Key secondary outcome
110 (55 per stratum) 64 Baseline, Static Reminder, BCF 2 660 643 17 (6/5/6 by arm) gpt-5.6-sol, standard reasoning, high effort unconstrained tool-use loop; no timeout, no automatic retry BCF − Static Reminder EIC (favourable direction negative) bug-discriminating evidence (favourable direction positive)
automated preflight environment check at startup and were replaced by the next candidate from the same stratum before any model request, giving a final set of 110 tasks across 64 repositories. Table 2 summarises the design. The analysis plan, task list, and decision rules were specified before any confirmatory model request. The primary outcome is EIC (BCF versus Static Reminder). The secondary outcome, tested in a fixed sequence gated on the primary (Westfall & Krishen, 2001), is bug-discriminating evidence (BCF versus Static Reminder). The estimator is a tasklevel paired difference: for each task with at least one observation in both arms, we compute the within-task rate difference, average within each stratum, and take the unweighted mean of the two stratum averages (the equal-source estimate, weighting the two strata equally). Inference uses a two-stratum Welch t-test with Satterthwaite degrees of freedom (Satterthwaite, 1946; Welch, 1947). The smallest effect size of interest (SESOI) (Lakens, Scheel, & Isager, 2018) was set before execution to 10 percentage points: an effect of that size would change which of a repair agent’s closures an integrator would accept. The SESOI sized the experiment; it is not a threshold for dismissing smaller observed effects. A further secondary outcome, closure without positive machine-verifiable evidence, monitors whether the feedback suppresses validation activity. The analysis plan also includes a secondary associational analysis (RQ2) examining the relationship between evidence quality and repair success through logistic models with repository-grouped cross-validation, computed from the released data with the plan’s seed and procedure.
4.4 Exploratory replications: varying the scaffold and the model Two exploratory replications probe generality on a shared 20-task subset selected by a deterministic outcome-blind rule from the confirmatory task set. The scaffold replication was specified together with the confirmatory design; the second-model replication was designed after the scaffold results were available and specified before any request to the second model.
11
The scaffold replication replaces the unconstrained loop with a plan-execute-verify scaffold that requires the agent to submit a plan before executing and gates the final submission on a verification phase. The model remains gpt-5.6-sol. Three arms, 2 repetitions per task, 120 cells. The second-model replication replaces gpt-5.6-sol with gpt-5.6-terra in the unconstrained loop. Three arms, 2 repetitions, 120 cells. Estimates from both studies are reported with confidence intervals and descriptive p-values and carry no confirmatory weight.
4.5 Power and sample size We calibrated sample size to the prespecified SESOI of 10 percentage points rather than to the pilot point estimates, and the estimates below are reported against it rather than filtered by it. No published threshold was available to take the value from, because the construct is introduced here; Section 4.3 gives the design judgement it rests on. Two pilots preceded the experiment; their data served only to develop the measurement and intervention pipeline and to parameterise the power model below, and pilot observations enter no confirmatory estimate. Under a data-generating process using the pilot’s stratum-level base rates and within-task dependence, 110 tasks with 2 repetitions yield 96.2% power for EIC and 93.2% for bug-discriminating evidence at the SESOI, and 91.2% joint power. A supplementary five-percentage-point scenario, added before execution, yields 28.8% joint power, consistent with the expectation that an effect below the SESOI would likely go undetected.
5 Results 5.1 Evidence-role taxonomy at scale Table 3 and Figure 2 report the distribution of evidence roles across 3,730 retained events from the confirmatory experiment. Of these, 2,548 events are positive comparable: the captured outcome was positive and the event holds one of the four comparable roles. Within this set the roles divide as follows: 1,007 gold-aligned bug-discriminating, 370 candidate-specific, 1,141 regression-only, and 30 misleading. Regression-only and misleading events account for 46.0% of positive comparable events. Among the 1,377 events where the check fails on B and passes on the candidate, 26.9% are candidate-specific: they fail on the developer gold fix G. At the rollout level, 72.9% of baseline rollouts contain at least one bugdiscriminating event, while 23.8% exhibit evidence-inadequate closure. Both patterns coexist: 48.1% of baseline rollouts that submit a patch contain both discriminating and non-discriminating positive evidence. On the shared 20-task subset of the second-model replication, baseline EIC is 40.5% on gpt-5.6-terra versus 18.9% on gpt-5.6-sol.
5.2 Evidence quality and repair success Over the 643 observed rollouts in 64 repositories (ten repository-grouped folds), the covariates-only model attains an out-of-fold Brier score of 0.231 against 0.203 for the 12
1141
Regression-only 1007
Gold-aligned bug-discriminating 797
Diagnostic negative 370
Not comparable
positive comparable positive, non-discriminating (hatched) non-positive or non-evaluable
370
Candidate-specific Misleading
30
Flaky
15
0
200
400 600 800 1000 Retained validation events
1200
Fig. 2 Evidence roles of the 3,730 retained validation events in the confirmatory experiment. Dark bars are positive comparable roles; hatching marks positive roles that do not discriminate the reported bug
Table 3 Evidence roles of the 3,730 retained validation events in the confirmatory experiment Evidence role
Events
Share of events (%)
GOLD ALIGNED BUG DISCRIMINATING CANDIDATE SPECIFIC REGRESSION ONLY MISLEADING DIAGNOSTIC NEGATIVE NOT COMPARABLE FLAKY
1,007 370 1,141 30 797 370 15
27.0 9.9 30.6 0.8 21.4 9.9 0.4
All retained events Positive comparable events of which regression-only or misleading B-fail/S-pass events of which candidate-specific
3,730 2,548 1,171 1,377 370
100.0 68.3 46.0a 36.9 26.9b
Shares in the two indented rows are relative to their parent row, marked a of positive comparable events and b of B-fail/S-pass events.
covariates-plus-evidence model, a reduction of 0.028 (repository-cluster bootstrap 95% CI [0.003, 0.053]); out-of-fold AUROC rises from 0.521 to 0.694, and the augmented model recalibrates with intercept 0.22 and slope 0.63. These evidence terms carry associational predictive signal for official resolution beyond the covariates. The analysis is associational and predictive; it does not establish that evidence causes official resolution.
5.3 Confirmatory experiment Of 660 planned cells, 643 completed and 17 were lost to infrastructure failures, balanced across arms.
13
Table 4 Arm-level rates and three-arm decomposition of the confirmatory effects (task-level equal-source contrasts; Welch p values outside the confirmatory family are descriptive) Rollout-level rate (%)
Baseline Static Reminder
EIC Bug-discriminating evidence Official resolution Contrast Static − Baseline, EIC Static − Baseline, bug-discriminating evidence BCF − Static, EIC BCF − Static, bug-discriminating evidence BCF − Baseline, EIC BCF − Baseline, bug-discriminating evidence
BCF
23.8 72.9 65.9
21.4 75.8 66.0
13.6 83.2 66.8
Difference (pp)
95% CI (pp)
p
−3.22 4.12 −7.79 7.36 −10.45 10.91
[−7.58, 1.13] 0.15 [−0.94, 9.17] 0.11 [−12.87, −2.72] 0.0029 [1.73, 12.98] 0.011 [−15.78, −5.13] 0.00018 [4.82, 17.00] 0.00058
Secondary outcomes, BCF − Static Closure without positive machine-verifiable evidence Official resolution
0.44 0.48
[−1.94, 2.82] [−3.73, 4.69]
0.72 0.82
EIC (BCF minus Static Reminder) is −7.8 percentage points (95% CI [−12.9, −2.7], p = 0.0029), based on 109 paired tasks. The difference in bug-discriminating evidence is +7.4 percentage points (95% CI [+1.7, +13.0], p = 0.011). Both outcomes favour BCF in the prespecified fixed sequence. Both point estimates fall below the prespecified 10-percentage-point SESOI; the confidence intervals include the SESOI but do not exclude effects well below it, so the data do not resolve whether the true effects reach that threshold. Official resolution (BCF minus Static Reminder) is +0.5 percentage points: BCF does not detectably change repair success. Closure without positive machine-verifiable evidence shifts by +0.4 points: BCF does not suppress validation activity. Sign tests, leave-one-repository-out estimation, a repository-cluster wild bootstrap (Cameron, Gelbach, & Miller, 2008), a complete-pairs restriction, and worst-case missing-cell bounds all preserve the primary direction; Appendix A reports them together with per-arm cell accounting and trigger exposure (Table 7).
5.4 Active-control decomposition Table 4 decomposes the BCF effect into two components. The generic reminder reduces EIC by 3.2 percentage points relative to baseline and raises bug-discriminating evidence by 4.1 points. BCF adds a further 7.8 points on EIC and 7.4 points on bugdiscriminating evidence beyond the reminder. The total BCF-versus-baseline contrast is −10.5 points on EIC and +10.9 on bug-discriminating evidence.
5.5 Runtime and token overhead Table 5 reports per-rollout resource means by arm. The replay computation itself is inexpensive: BCF adds a median of 11.0 seconds of B-replay time per rollout, 3.3%
14
Table 5 Runtime and token overhead of the interventions in the confirmatory experiment (per-rollout means over observed cells) Per-rollout mean
Baseline
Static Reminder
BCF
29.8 956 10.7 390
29.1 946 10.5 381
31.4 1068 11.5 440
Model responses Total tokens (103 ) Output tokens (103 ) Wall time (s)
EIC (favourable: negative)
Bug-discriminating evidence (favourable: positive)
Confirmatory: BCF − Static Confirmatory: Static − Baseline Confirmatory: BCF − Baseline Scaffold replication: BCF − Static Second-model replication (terra): BCF − Static Second-model replication (terra): Static − Baseline
−20 0 Difference (pp) Primary contrast
20
−20
0 20 Difference (pp)
40
Descriptive comparison
Fig. 3 Task-level equal-source differences with 95% CIs across settings. Filled markers show the primary contrast family; open markers show descriptive comparisons. The dashed line separates the confirmatory experiment from the exploratory replications
of wall time at the median. The BCF-versus-Static difference in per-rollout token consumption is 99 thousand tokens, not statistically distinguishable from zero.
5.6 Exploratory replications Table 6 and Figure 3 report both exploratory replications on the shared 20-task subset; Appendix A details cell loss and trigger exposure. The scaffold replication reduces baseline evidence-inadequate closure to 8.3%, compared to 18.9% for the same tasks under the unconstrained loop, leaving little room for either component. On gpt-5.6terra in the unconstrained loop, the generic reminder reduces evidence-inadequate closure by 7.5 points relative to baseline, matching the full BCF effect on sol, while the B-replay content adds nothing detectable on top; feedback reached 97.4% of intervention rollouts, so the null is not explained by too few intervention triggers. Across the two studies, the reminder-only arm shows the more consistent favourable pattern; an incremental BCF advantage is established only with sol in the unconstrained loop.
15
Table 6 Exploratory replications on the shared 20-task subset (BCF − Static Reminder; all quantities are descriptive)
Observed / planned cells EIC difference (pp) EIC 95% CI (pp) Bug-discriminating evidence difference (pp) Bug-discriminating evidence 95% CI (pp) Baseline EIC (%) Baseline bug-discriminating evidence (%) Interaction, EIC (pp; p) Interaction, bugdiscriminating evidence (pp; p) Cells lost to failures (by arm)
Confirmatory (same 20 tasks)
Scaffold replication
Second-model replication (terra)
– −7.78 – 7.78
108/120 −3.13 [−10.51, 4.26] −1.88
113/120 2.50 [−16.08, 21.08] −2.50
–
[−11.56, 7.81]
[−21.74, 16.74]
– –
8.3 83.3
40.5 51.4
– –
5.00; 0.52 −10.00; 0.33
10.83; 0.24 −10.56; 0.33
–
12 (4/4/4)
7 (3/2/2)
Settings: the confirmatory experiment and the second-model replication run the unconstrained tooluse loop; the scaffold replication runs the plan-execute-verify scaffold. Models: gpt-5.6-sol for the confirmatory experiment and the scaffold replication; gpt-5.6-terra for the second-model replication. The confirmatory column restricts the confirmatory sample to the 19 of these 20 tasks with complete pairs. Interaction estimates compare each replication with the full confirmatory experiment and are descriptive.
6 Discussion 6.1 What the taxonomy reveals The central finding is a prevalence estimate: 46.0% of the positive validation evidence that agents produce mid-trajectory carries no information about the reported defect. This holds in a population where 72.9% of rollouts also contain at least one discriminating event. Agents frequently produce both kinds of evidence in the same run. The tool loop, however, exposes no signal that distinguishes the two kinds: nothing indicates whether a passing check would also have passed on B. A finer-grained pattern sits within the discriminating events. Among events where the check fails on B and passes on the candidate (the B-fail/S-pass pattern), 26.9% also fail on the developer gold fix G. Their discriminating power is tied to implementation details of the agent’s patch rather than to the defect itself: used as acceptance tests downstream, they would reject correct alternative fixes. This distinction is invisible without the G-replay that BSG-VA provides, and it bears on the growing practice of using agent-generated tests as patch validators (Mündler et al., 2024). On the shared subset of the second-model replication, baseline EIC is 40.5% on gpt-5.6-terra versus 18.9% on gpt-5.6-sol: the phenomenon is at least as pronounced on gpt-5.6-terra.
16
6.2 Awareness versus information We designed the three-arm experiment to distinguish two hypotheses about why BCF works. The first is that agents simply need to be reminded to evaluate their evidence. The second is that the specific B-replay content matters. The generic reminder, matched in structure and timing to BCF but carrying no replay information, serves as the attention control that separates these contributions. The results favour a mixed account. The reminder alone reduces EIC by 3.2 percentage points relative to baseline: a nonspecific prompting effect, captured by the attention-control arm. The B-replay content adds a further 7.8 points beyond it. On sol in the unconstrained loop, both components contribute. On terra, the prompting effect alone accounts for the full improvement. Under the plan-execute-verify scaffold, baseline EIC is already at 8.3%, and neither component has room to show an effect. The practical takeaway is that the cheaper intervention has the broader reach. A generic validation prompt, delivered at the right moment in the tool loop, produces a favourable shift in both models we tested and requires no replay infrastructure. The B-replay content adds incremental value with gpt-5.6-sol in the unconstrained loop. Whether that increment justifies the infrastructure cost depends on the deployment context. The measurement method is relevant either way: it provides the means to evaluate evidence quality regardless of whether a feedback loop is deployed.
6.3 Threats to validity Construct validity. Evidence roles are assigned against the developer gold fix as reference standard, which introduces noise on tasks where the gold fix is incomplete or where multiple valid fixes exist. The EIC construct depends on the validation-event definition set before data collection; alternative definitions of what counts as a validation event would shift the prevalence estimates. The measurement begins at the agent’s first production edit, so diagnostic commands issued during initial bug exploration fall outside the captured population. If agents produce discriminating checks before editing code, the reported non-discriminating share would overstate the trajectory-wide rate. Internal validity. Cells lost to infrastructure failures are balanced across arms in all three experiments, and the confirmatory worst-case bound preserves the sign of the effect; the secondmodel bound does not. External validity. Both models come from a single provider and model family, and the task population draws on two Python-ecosystem benchmarks. The exploratory replications cover a shared subset with few repetitions per arm, yielding wide confidence intervals; their results are descriptive.
17
7 Conclusion We introduce BSG-VA, a method that measures the evidential value of the validation activity repair agents perform mid-trajectory, by replaying each command on the buggy, candidate, and gold-fix code states. Applied at scale, the method reveals that 46.0% of positive comparable validation events carry no bug-discriminating information, and that 23.8% of baseline rollouts close on the basis of such evidence alone. Bug-contrast feedback, which returns the B-replay outcome to the agent in real time, reduces evidence-inadequate closure by 7.8 percentage points, without measurable cost to repair success. That estimate falls below the prespecified 10percentage-point SESOI, so the direction of the effect is established while its practical magnitude is not. The active-control decomposition attributes roughly a third of the total improvement to directing the agent’s attention to evidence quality, and the exploratory replications show the same reminder-only pattern on the second model; the richer B-replay content adds value in one configuration, so a deployment without replay infrastructure still has a measured, cheaper option in the generic reminder. What holds across every configuration we tested is the measurement itself: BSG-VA and the evidence-role taxonomy assign each passing check in any replayable trajectory an evidence role with directly checkable content, making the adequacy of an agent’s validation evidence a property that can be audited before a patch is trusted.
Statements and Declarations Funding. The authors received no funding for this work. Competing interests. The authors have no competing interests to declare. Ethics approval and consent to participate. Not applicable. This study involved no human participants and no animal subjects; all data are machine-generated execution records. Consent for publication. Not applicable. Data availability. The event-level, rollout-level, and task-level validation data supporting this study are openly available on Zenodo at https://doi.org/10.5281/ zenodo.21642576 (Xu & Wu, 2026). The release carries the three JSONL files behind every reported statistic. Replay environments are pinned by public container image references. The protocol documents, field definitions and comparability rules, and the raw trajectory logs are available from the corresponding author on reasonable request. Code availability. The BSG-VA capture, replay, and analysis code used to produce every reported statistic is not publicly released; it is available from the corresponding author on reasonable request. Author contributions. Conceptualization: Xiaonan Xu. Methodology: Xiaonan Xu. Formal analysis: Xiaonan Xu. Software: Wenjing Wu. Data curation: Wenjing Wu. Writing, original draft: Xiaonan Xu. Writing, review and editing: Xiaonan Xu and Wenjing Wu. Both authors read and approved the final manuscript. Use of AI tools. Section 3.5 details the role of an LLM assistant (Claude, Fable 5 model, Anthropic) in environment configuration, experiment implementation, analysis code, and language polishing. The authors take responsibility for all content. 18
Table 7 Confirmatory contrast (BCF versus Static Reminder, 109 paired tasks, equal-source weighting) and robustness analyses
Task-level difference Welch 95% CI Welch two-sided p Repository wild bootstrap p Wild bootstrap 95% CI Exact sign test (favourable:unfavourable) Exact sign test p Leave-one-repository-out favourable Complete-pairs difference (100 tasks) Worst-case missingness bound
EIC (pp)
Bug-discriminating evidence (pp)
−7.79 [−12.87, −2.72] 0.0029 0.00020 [−12.16, −3.43] 19:5 0.0066 63/63 −8.00 −5.00
7.36 [1.73, 12.98] 0.011 0.0018 [2.55, 12.13] 21:7 0.013 63/63 7.45 4.55
Fixed-sequence family: EIC tested first at two-sided α = 0.05; bug-discriminating evidence tested only after EIC rejection in its favourable direction. Additional analyses are reported as robustness checks.
Appendix A A.1
Robustness analyses and exposure details
Confirmatory robustness
Table 7 collects the robustness checks for the primary contrast. The exact sign test on task-level differences is 19:5 for EIC (p = 0.0066) and 21:7 for bug-discriminating evidence (p = 0.013). Leave-one-repository-out estimation is favourable in 63/63 repositories for EIC and 63/63 for bug-discriminating evidence. The repository-cluster wild bootstrap (99,999 draws) yields p = 0.00020 for EIC and p = 0.0018 for bug-discriminating evidence. Restricting to the 100 tasks with complete pairs in both intervention arms gives −8.0 points on EIC. The 17 cells lost to infrastructure failures split 6/5/6 across baseline, Static Reminder, and BCF; under worst-case imputation of all of them, the EIC difference attenuates to −5.0 points and the bug-discriminating-evidence difference to +4.5, preserving direction in both cases.
A.2
Trigger exposure
Feedback reached 97.2% of Static Reminder rollouts and 96.7% of BCF rollouts in the confirmatory experiment. In the second-model replication the trigger rate is 97.4% in both intervention arms.
A.3
Replication cell loss
The scaffold replication lost 12 of 120 cells (4/4/4 by arm); 6 of these are one task whose rollouts the scaffold’s submission gate aborted, affecting all arms equally (two cells per arm). The second-model replication lost 7 cells (3/2/2 by arm) to infrastructure failures. The second-model worst-case missingness bound can reverse the sign of the estimate. 19
References Ahmed, T., Ganhotra, J., Pan, R., Shinnar, A., Sinha, S., Hirzel, M. (2025). Otter: Generating Tests from Issues to Validate SWE Patches. International conference on machine learning (ICML). Retrieved from https://arxiv.org/abs/2502.05368 Ahmed, T., Ganhotra, J., Shinnar, A., Hirzel, M. (2026). EvoOtter: Evolutionary Reproduction Test Generator. Retrieved from https://arxiv.org/abs/2607.02854 (arXiv preprint) Badertdinov, I., Golubev, A., Nekrashevich, M., Shevtsov, A., Karasik, S., Andriushchenko, A., . . . Yangel, B. (2025). SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents. Retrieved from https://arxiv.org/abs/2505.20411 Banik, D., Chowdhury, K., Shamim, S.I. (2026). All Smoke, No Alarm: Oracle Signals in Agent-Authored Test Code. Retrieved from https://arxiv.org/abs/2606.18168 (IEEE AITest 2026 accepted) Bouzenia, I., & Pradel, M. (2025). Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories. 2025 40th IEEE/ACM international conference on automated software engineering (ASE) (pp. 2846–2857). IEEE. Cameron, A.C., Gelbach, J.B., Miller, D.L. (2008). Bootstrap-Based Improvements for Inference with Clustered Errors. Review of Economics and Statistics , 90 (3), 414–427, https://doi.org/10.1162/rest.90.3.414
Ceka, I., Mitchell, H., Pujar, S., Buratti, L., Ramji, S., Yang, J., . . . Ray, B. (2026). Understanding Automated Program Repair Agents Through the Lens of Traceability: An Empirical Study. Retrieved from https://arxiv.org/abs/2506.08311 (ISSTA 2026 accepted) Chen, D., Lin, S., Zeng, M., Zan, D., Wang, J.-G., Cheshkov, A., . . . Wang, Q. (2024). CodeR: Issue Resolving with Multi-Agent and Task Graphs. Retrieved from https://arxiv.org/abs/2406.01304 Chen, X., Lin, M., Schärli, N., Zhou, D. (2023). Teaching Large Language Models to Self-Debug. Retrieved from https://arxiv.org/abs/2304.05128 Chen, Z., Sun, Z., Shi, Y., Peng, C., Gu, X., Lo, D., Jiang, L. (2026). Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents. Retrieved from https://arxiv.org/abs/2602.07900 (arXiv v2 preprint)
20
Cheng, R., Tufano, M., Cambronero, J., Wei, R., Shi, S., Uy, G., . . . Ivančić, F. (2026). Dynamic Cogeneration of Bug Reproduction Test in Agentic Program Repair. Retrieved from https://arxiv.org/abs/2601.19066 (arXiv preprint) Fei, Z., Pan, Y., Sarro, F., Ge, J., Liu, M., Ng, V., Ye, H. (2026). Echo: Graph-Enhanced Retrieval and Execution Feedback for Issue Reproduction Test Generation. Retrieved from https://arxiv.org/abs/2603.07326 (arXiv preprint) Guo, Y., Liu, Y., Zhang, J.M., Ma, Y., Lou, Y., Chen, Z. (2026). SWE-Doctor: Guiding Software Engineering Agents with Runtime Diagnosis from Multi-Faceted Bug Reproduction Tests. Retrieved from https://arxiv.org/abs/2607.00990 (arXiv preprint) Hora, A., & Robbes, R. (2026). Are Coding Agents Generating Over-Mocked Tests? An Empirical Study. Retrieved from https://arxiv.org/abs/2602.00409 (MSR 2026) Huang, D., Zhang, J.M., Harman, M., Du, M., Cui, H. (2026). Measuring the Influence of Incorrect Code on Test Generation. Retrieved from https://conf.researchr.org/details/icse-2026/icse-2026-researchtrack/15/Measuring-the-Influence-of-Incorrect-Code-on-Test-Generation (ICSE 2026 Research Track) Jiang, N., Liu, K., Lutellier, T., Tan, L. (2023). Impact of Code Language Models on Automated Program Repair. 2023 IEEE/ACM 45th international conference on software engineering (ICSE) (pp. 1430–1442). Jimenez, C.E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., Narasimhan, K. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? International conference on learning representations (ICLR). Retrieved from https://arxiv.org/abs/2310.06770 Kang, S., Yoon, J., Yoo, S. (2023). Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction. 2023 IEEE/ACM 45th international conference on software engineering (ICSE) (pp. 2312–2323). Khatib, L., Mathews, N.S., Nagappan, M. (2025). AssertFlip: Reproducing Bugs via Inversion of LLM-Generated Passing Tests. Retrieved from https://arxiv.org/abs/2507.17542 (ICSE 2026 Research Track) Konstantinou, M., Tambon, F., Papadakis, M. (2026). On the risk of coding before testing: An empirical study on LLM-based test generation workflow. Retrieved from https://arxiv.org/abs/2607.05139 (arXiv preprint) Lakens, D., Scheel, A.M., Isager, P.M. (2018). Equivalence Testing for Psychological Research: A Tutorial. Advances in Methods and Practices in Psychological Science , 1 (2), 259–269, https://doi.org/10.1177/2515245918770963
21
Leary, H., Hanuska, L., Brown, C. (2026). TestMap: Evidence Infrastructure for Foundation-Model-Assisted Test Generation. Retrieved from https://arxiv.org/abs/2606.10211 (AIWare 2026 arXiv Track) Le Goues, C., Dewey-Vogt, M., Forrest, S., Weimer, W. (2012). A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each. 2012 34th international conference on software engineering (ICSE) (pp. 3–13). Lin, Z., Zhu, J., Zhou, M., Wang, X., Sun, Z., Yang, R., . . . Li, L. (2026). To Run or Not to Run: Analyzing the Cost-Effectiveness of Code Execution in LLM-Based Program Repair. Retrieved from https://arxiv.org/abs/2606.26978 (ISSTA 2026 accepted) Ma, W., Chen, Z., Gu, J., Li, T., Liu, S., Jiang, L. (2026). Same Signal, Different Semantics: A Cross-Framework Behavioral Analysis of Software Engineering Agents. Retrieved from https://arxiv.org/abs/2605.18332 (arXiv preprint) Ma, Y., Kereopa-Yorke, B., Schultz, B. (2026). Building to the Test: Coding Agents Deliver What You Check, Not What You Requested. Retrieved from https://arxiv.org/abs/2606.28430 (arXiv preprint / Microsoft Research) Madiraju, M.B., & Madiraju, M.S.P. (2026). RigorBench: Benchmarking Engineering Process Discipline in Autonomous AI Coding Agents. Retrieved from https://arxiv.org/abs/2606.22678 (arXiv preprint) Meng, Q., Zhang, X., Ren, Z., Visser, J. (2026). EviACT: An Evidenceto-Action Framework for Agentic Program Repair. Retrieved from https://arxiv.org/abs/2605.27238 (arXiv preprint) Mündler, N., Müller, M., He, J., Vechev, M. (2024). SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents. Advances in neural information processing systems (NeurIPS). Retrieved from https://openreview.net/forum?id=9Y8zUO11EQ Nashid, N., Bouzenia, I., Pradel, M., Mesbah, A. (2026). Issue2Test: Generating Reproducing Test Cases from Issue Reports. Retrieved from https://conf.researchr.org/details/icse-2026/icse-2026-researchtrack/87/Issue2Test-Generating-Reproducing-Test-Cases-from-Issue-Reports (ICSE 2026 Research Track) OpenAI (2024). Introducing SWE-bench Verified. https://openai.com/index/introducing-swe-bench-verified/ subset of SWE-bench)
22
Retrieved from (Human-validated
Örwall, A. (2024). Moatless Tools. Retrieved https://github.com/aorwall/moatless-tools (Software repository)
from
Podivilov, A., Lomshakov, V., Savin, S., Startsev, M., Pozharskiy, R., Parshin, M., Nikolenko, S. (2026). AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation. Retrieved from https://arxiv.org/abs/2607.06624 (arXiv preprint) Sahoo, P., Mittal, G., Li, X., Ma, S., Steenhoek, B., Lin, P., Hu, Y. (2026). AgentLens: Revealing the Lucky Pass Problem in SWE-Agent Evaluation. Retrieved from https://arxiv.org/abs/2605.12925 (arXiv v3 preprint) Sajadi, A., Nguyen, T., Huynh, K., Parra, E., Chatterjee, P. (2026). TraceView: Interactive Visualization of Agentic Program Repair Trajectories. Retrieved from https://arxiv.org/abs/2606.22110 (arXiv preprint) Satterthwaite, F.E. (1946). An Approximate Distribution of Estimates of Variance Components. Biometrics Bulletin , 2 (6), 110–114, https://doi.org/10.2307/ 3002019
Shu, R., Chong, C.Y., Zhou, X., Peng, Y., Wu, Z., Han, X., . . . Wang, Y. (2026). What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agents. Retrieved from https://arxiv.org/abs/2607.06184 (arXiv preprint) Shull, F.J., Carver, J.C., Vegas, S., Juristo, N. (2008). The role of replications in Empirical Software Engineering. Empirical Software Engineering , 13 (2), 211– 218, https://doi.org/10.1007/s10664-008-9060-1
Smith, E.K., Barr, E.T., Le Goues, C., Brun, Y. (2015). Is the cure worse than the disease? overfitting in automated program repair. Proceedings of the 2015 10th joint meeting on foundations of software engineering (pp. 532–543). Sun, Y., Zhao, Y., Wang, Y., Du, Y., Ma, Z., Wang, J., . . . Huang, Z. (2026). SWEMutation: Can LLMs Generate Reliable Test Suites in Software Engineering? Retrieved from https://arxiv.org/abs/2605.22175 (Findings of ACL 2026) Wang, B., Zhang, C., Liu, D., Zhang, J., Chen, J., Li, M., . . . Cui, Z. (2026). The Verification Horizon: No Silver Bullet for Coding Agent Rewards. Retrieved from https://arxiv.org/abs/2606.26300 (arXiv preprint) Weimer, W., Nguyen, T., Le Goues, C., Forrest, S. (2009). Automatically finding patches using genetic programming. 2009 IEEE 31st international conference on software engineering (pp. 364–374).
23
Welch, B.L. (1947). The Generalization of ‘Student’s’ Problem when Several Different Population Variances are Involved. Biometrika , 34 (1–2), 28–35, https://doi .org/10.1093/biomet/34.1-2.28
Westfall, P.H., & Krishen, A. (2001). Optimally weighted, fixed sequence and gatekeeper multiple testing procedures. Journal of Statistical Planning and Inference , 99 (1), 25–40, https://doi.org/10.1016/s0378-3758(01)00077-5
Xia, C.S., Deng, Y., Dunn, S., Zhang, L. (2025). Demystifying LLM-Based Software Engineering Agents. Proceedings of the ACM on Software Engineering , 2 (FSE), 801–824, https://doi.org/10.1145/3715754
Xia, C.S., Wei, Y., Zhang, L. (2023). Automated Program Repair in the Era of Large Pre-trained Language Models. 2023 IEEE/ACM 45th international conference on software engineering (ICSE) (pp. 1482–1494). Xin, Q., & Reiss, S.P. (2017). Identifying test-suite-overfitted patches through test case generation. Proceedings of the 26th ACM SIGSOFT international symposium on software testing and analysis (pp. 226–236). Xu, X., & Wu, W. (2026, July). Validation evidence in LLM repair agents: eventlevel, rollout-level, and task-level data. Dataset, Zenodo. Retrieved from https://doi.org/10.5281/zenodo.21642576 Yang, J., Jimenez, C.E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., Press, O. (2024). SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. Advances in neural information processing systems (NeurIPS). Retrieved from https://arxiv.org/abs/2405.15793 Zhang, Y., Ruan, H., Fan, Z., Roychoudhury, A. (2024). AutoCodeRover: Autonomous Program Improvement. Proceedings of the 33rd ACM SIGSOFT international symposium on software testing and analysis (pp. 1592–1604).
24