E VI ACT: An Evidence-to-Action Framework for Agentic Program Repair Qianru Meng LIACS, Leiden University Leiden, Netherlands [email protected]
Xiao Zhang CLCG, University of Groningen Groningen, Netherlands [email protected]
Zhaochun Ren LIACS, Leiden University Leiden, Netherlands [email protected]
Joost Visser LIACS, Leiden University Leiden, Netherlands [email protected]
arXiv:2605.27238v1 [cs.SE] 26 May 2026
Abstract LLM-based agents have moved automated program repair (APR) from fixed-context patch generation to interactive repository-level repair. However, existing agentic APR systems still struggle to use execution evidence to guide localization, patch generation, and validation. We propose E VI ACT (Evidence-to-Action), an agentic APR framework that coordinates three evidence-driven guardrails across repair stages. The retrieval scaffold grounds repair context, the compile gate filters invalid edits, and the test-driven gate checks target-test recovery before full regression. Across four benchmarks, EVIACT improves resolve rate over the strongest reported comparable baselines by 1.6– 6.0 percentage points and shows 70.1–88.6% lower reported per-bug API cost where baseline costs are available. Ablations and diagnostics suggest that these gains are associated with the coordinated evidence-to-action chain, making agentic APR more effective and efficient.
1
Localize
Setup
failure logs
E₁
code suspects
weak failure evidence
Mislocalization cascade Failure evidence Wrong suspects Wrong patch
Verify
Patch
patch edits
E₁ + E₂ wrong suspect set
test results
E₁ + E₂ + E₃ flawed patch hypothesis
Invalid patch attempts
High validation cost
Malformed or uncompilable
Repeated testing of incorrect patch
Retry
Figure 1: Problems in workflow-based agentic APR systems, including a mislocalization cascade (E1 , E2 , and E3 denote accumulated errors across different repair stages), invalid patch attempts, and high validation cost.
APR systems (Yang et al., 2024; Zhang et al., 2024; Arora et al., 2024; Xia et al., 2025) that interact with development environments through search, inspection, editing, execution, and validation tools. Existing agentic APR systems follow two broad architectural patterns. Interaction-based agents (Yang et al., 2024; Arora et al., 2024; Zhang et al., 2024; Wang et al., 2025) preserve flexibility by allowing the LLM to interleave tool use and reflection, but long trajectories can dilute the failure evidence that should guide repair, leading to localization drift, stalled execution, and repeated edits to weakly related code. Workflow-based APR systems reduce open-ended exploration by decomposing repair into stages such as localization, patch generation, and validation. However, staging alone provides an execution order but leaves the boundaries between stages under-specified, without ensuring that execution evidence is progressively transformed into later repair decisions. As illustrated in Figure 1, weak stage-boundary control can cause a mislocalization cascade, where an early wrong suspect set leads to incorrect edits and repeated retries under the same flawed hypothesis.
Introduction
Automated Program Repair (APR) aims to reduce the cost of software maintenance by automatically generating patches for real defects. Classical APR is commonly formulated as generate-and-validate search, where candidate edits are produced and accepted if they satisfy a test oracle (Weimer et al., 2009; Le Goues et al., 2012; Nguyen et al., 2013; Mechtaev et al., 2016; Long and Rinard, 2016). Learning-based and pretrained-code repair methods later improved patch generation by learning repair patterns from historical fixes or large code corpora (Tufano et al., 2019; Chen et al., 2019; Lutellier et al., 2020; Xia and Zhang, 2022). However, most of these methods operate from a fixed or bounded repair context. As software repositories grow in size and complexity, repair increasingly requires cross-file localization and iterative use of runtime feedback, motivating LLM-based agentic 1
Even with plausible localization, invalid patch attempts—including malformed or non-compilable candidates—may still reach validation. Both cases increase validation cost, because the system repeatedly tests semantically irrelevant or non-executable patches. These problems reflect an evidence-toaction gap: execution evidence is observed, but not reliably used to control what to retrieve, what to edit, and when to validate. To address this gap, we propose E VI ACT (Evidence-to-Action), an evidence-driven agentic APR framework for workflow-based repository repair. E VI ACT follows a standard Setup–Localize– Patch–Verify pipeline, but controls stage transitions through three guardrails: a retrieval scaffold that maps RED failure evidence to structurally grounded code suspects, a compile gate that filters invalid patches using diagnostics, and a testdriven (TD) gate that requires the originally failing tests to pass before full regression. Together, these guardrails turn execution evidence into repair actions across stage boundaries, keeping repair trajectories tied to observed failures, compiler feedback, and validation outcomes. We evaluate E VI ACT on four benchmarks: Defects4J 2.0, SWE-bench Verified, SWE-bench Lite, and SWE-bench Live. Under the GPT-4o setting, E VI ACT achieves higher resolve rates than the strongest reported comparable baselines on the same benchmark settings, and shows lower reported per-bug API cost where baseline cost results are available. Ablations and diagnostics further suggest that these gains are associated with coordinated evidence control, while manual analyses show that the residual failure frontier shifts from executable invalidity to semantic grounding. Our contributions are as follows:
• We will release the artifacts to support reproducibility.
2
Related Work
Classical APR is commonly formulated as generate-and-validate search over bounded repair spaces, where candidate patches are generated and accepted according to test oracles (Weimer et al., 2009; Le Goues et al., 2012; Nguyen et al., 2013; Mechtaev et al., 2016; Long and Rinard, 2016). Learning-based and execution-guided repair further extend this formulation by learning repair patterns, controlling edit spaces, and using execution feedback to guide patch search (Wong et al., 2021; Parasaram et al., 2021; Zhang et al., 2023). LLM-based agentic APR shifts repair to repositorylevel interaction, where agents inspect code, invoke tools, edit files, execute tests, and validate candidate patches (Yang et al., 2024; Wang et al., 2025; Zhang et al., 2024; Arora et al., 2024; Xia et al., 2025; Chen et al., 2024; Li et al., 2025; Yang et al., 2025; Yu et al., 2025). Recent systems differ mainly in how they organize this repair process. Interaction-based agents, such as SWE-Agent, OpenHands, AutoCodeRover, and MASAI, expose flexible tool interfaces and leave much of the repair trajectory to model-driven exploration (Yang et al., 2024; Wang et al., 2025; Zhang et al., 2024; Arora et al., 2024). Workflowbased systems, such as Agentless, PatchPilot, and CoDeR, impose more structure by separating localization, patch construction, and validation into explicit stages (Xia et al., 2025; Li et al., 2025; Chen et al., 2024). Staged workflows improve procedural control, but the interfaces between stages are weakly specified. E VI ACT addresses this limitation by coupling adjacent stages through evidence-driven control, where intermediate evidence guides retrieval, patch refinement, and validation decisions.
• We identify the evidence-to-action gap as a stage-boundary design problem in workflowbased agentic APR. • We introduce E VI ACT, an agentic APR framework that addresses this gap by coordinating three evidence-driven guardrails across workflow stages.
Stage-local constraints in agentic APR. Prior work has introduced several stage-specific mechanisms related to E VI ACT. Retrieval-based localization narrows the repair context using execution signals, repository structure, learned retrievers, or dependency relations (Xia et al., 2025; Zhang et al., 2024; Xie et al., 2025; Ma et al., 2025; Tang et al., 2025). Patch validation and selection methods filter or rank candidates through executability checks, semantic review, collaborative patching, specification reasoning, adversarial intent reasoning, or
• We evaluate E VI ACT on 4 benchmarks against reported comparable baselines and across multiple LLM backbones. • Experimental results, ablations, diagnostics, failure and patch correctness analyses provide evidence for the effectiveness of E VI ACT. 2
Guardrails (Stage-Specific Checks)
APR Workflow
Execution Evidence (Artifacts)
Compile Gate
Test-Driven Gate RED test must fail
Retrieval Scaffold Structural AST code graph
Syntax check Compilation check
1. Setup
2. Localize
3. Patch
• configure environment • build retrieval index • reproduce RED tests
• read failure evidence • retrieve relevant context • rank suspicious locations
• synthesize candidate patch • filter invalid edits • apply patch
failure logs (RED test)
ranked code suspects
candidate patch edits
Test-Driven Gate GREEN test must pass
✓ ✓ ✓
4. Verify
• run GREEN tests • run regression tests • collect test results
✓ ✓ ✓
No Repair Success Yes
Pass?
Tool use
GREEN & full test results
Agent
Figure 2: Overview of E VI ACT, a four-stage agentic APR pipeline (Setup → Localize → Patch → Verify) that converts traceable execution evidence into repair decisions. The retrieval scaffold derives suspects based on RED failure logs (produced by TD gate); the compile gate filters invalid patch candidates; and the TD gate applies GREEN test to pre-verify the valid patch before full regression. The Localize and Patch agents invoke tools respectively to investigate code suspects and refine edits.
multi-candidate selection (Li et al., 2025; Pabba et al., 2025; Tang et al., 2026; Ruan et al., 2024; Liu et al., 2024; Ye et al., 2025). Related work on patch safety, vulnerability repair, and human-grounded patch triage further shows that test-passing patches can still be overfitted, unsafe, or semantically insufficient (Kim et al., 2025; Hu et al., 2025; Cambronero et al., 2025). Test-based workflows such as TDFlow use RED–GREEN–REFACTOR feedback for repository-scale software engineering (Han et al., 2025). These works establish the value of retrieval, executability checking, and test-based validation, but typically apply such signals as local checks within individual stages.
3
Methodology
3.1
Overview
As shown in Figure 2, E VI ACT follows the common staged structure of workflow-based APR systems, using Setup, Localize, Patch, and Verify as its execution backbone. Its key distinction lies in using execution evidence as a control signal across stage transitions, where repair trajectories are most likely to drift. (1) Setup initializes the repair environment, constructs a structure-preserving retrieval index, and reproduces the target failure to obtain RED evidence. (2) Localize converts this evidence into failure-grounded queries, expands over structural code relations, and ranks suspicious locations. (3) Patch synthesizes candidate edits over the localized suspect set and filters invalid candidates before validation. (4) Verify reruns the target tests before full regression and accepts a patch only when both validation stages pass. The workflow is controlled by three guardrails. The retrieval scaffold bridges Setup and Localize by mapping RED evidence to a focused suspect set over AST spans and code-graph relations, providing structurally grounded localization beyond unconstrained keyword retrieval. The compile gate operates in Patch by rejecting malformed, unappliable, or non-compilable edits before test validation. The TD gate spans Setup and Verify: RED reproduction anchors localization, while GREEN validation requires the originally failing tests to pass
E VI ACT instead organizes these local signals into an evidence-to-action chain: the retrieval scaffold constructs focused suspects from failure and code-structure evidence, the compile gate rejects or refines invalid candidates using diagnostics, and the TD gate adapts RED/GREEN flow to APR. The RED test is the pre-repair execution of the originally failing target test, which reproduces the bug and provides failure evidence for localization; the GREEN test is the post-patch re-execution of the same target test, which checks target-failure recovery before full regression. As a result, intermediate evidence constrains successive repair decisions: what to retrieve, which edits to refine or reject, and when to proceed to validation. 3
before full regression. Overall, these guardrails anchor repair actions in traceable evidence. 3.2
E VI ACT performs deterministic code-graph retrieval rather than neural embedding retrieval. It first maps RED evidence to seed spans by matching failure-grounded signals from the failing test name, stack trace, error message, assertion content, and available file or line references against structural span fields, including file path, line range, symbol name, signature, local code content, and structural descriptor. It then expands from these seed spans over the code graph. Under a bounded search budget, the scaffold traverses containment, call, import, and inheritance edges to recover code regions structurally related to the directly observed failure location. Finally, E VI ACT ranks candidate spans using deterministic evidence alignment. For each candidate span s, the ranking key is: ρ(s) = mSYM (s), mLOC (s), −dG (s), rG (s), −|s| , (2) where mSYM (s) denotes exact trace-symbol match, mLOC (s) denotes file or line proximity when available, dG (s) is the graph distance from REDmatched seeds, rG (s) measures relation support in the expanded subgraph, and |s| is the span length. Candidates are sorted in descending lexicographic order by ρ(s), and the top-K spans form the suspect set UK , with K = 3 as a fixed compact default. Appendix A discusses the rationale.
Stage 1: Setup
Setup produces two artifacts used throughout the repair process: a structure-preserving retrieval index I and RED failure evidence from the originally failing target tests. E VI ACT first initializes a clean workspace s0 at the target repository revision. The retrieval index preserves executable code structure rather than treating the repository as flat text. It is constructed in three steps. (S1) Structural parsing. E VI ACT parses source files into ASTs using Tree-sitter1 and extracts structural spans at file, class, method, and function granularity. Each span stores its file path, line range, symbol name, signature when available, and local code content. (S2) Cascaded compression. Following the cascaded-compression design of ABCoder,2 E VI ACT performs offline, LLM-free deterministic compression: lower-level spans retain executable code, while higher-level spans store structural descriptors derived from child symbols, signatures, imports, and containment metadata. (S3) Codegraph construction. E VI ACT extracts containment, call, import, and inheritance edges between spans, forming a lightweight code graph for structurally grounded localization. Formally, for a repository R, E VI ACT constructs:
3.4
S0 = S PANS(ASTPARSE(R)), S = C OMPRESS(S0 ),
Conditioned on the suspect set UK and its supporting evidence, the patch agent generates candidate diffs through bounded tool use. At each attempt, it receives the localized working set, RED evidence, and current diagnostic context. Nearby code may be inspected when necessary, but patch generation remains anchored to the evidence-backed suspect set unless re-localization is triggered. Before test validation, E VI ACT applies the compile gate to each candidate patch p:
(1)
I = (S, E), where S is the set of compressed structural spans and E contains the code-graph edges between spans. Setup also executes the RED phase of the TD gate by reproducing the originally failing target tests in the clean workspace. The resulting RED evidence includes the failing test identifier, error message, stack trace, and assertion failure when available. This evidence serves as the executable anchor for localization. 3.3
gCMP (p) = IDIFF (p)·IAPPLY (p)·ICOMP (s0 ⊕p). (3) Here, IDIFF (p), IAPPLY (p), and ICOMP (s0 ⊕ p) indicate whether the patch is a well-formed diff, applies cleanly, and passes syntax or build checks, respectively. Thus, gCMP (p) = 1 only when all checks pass. When the compile gate fails, E VI ACT returns structured diagnostics, including the failure type, affected file or hunk when available, and relevant syntax or build output, as refinement context for
Stage 2: Localize
Given RED evidence and the retrieval index I = (S, E), the retrieval scaffold constructs a focused suspect set for patch generation. 1 2
Stage 3: Patch
https://tree-sitter.github.io/tree-sitter/ https://github.com/cloudwego/abcoder
4
the next patch attempt. Only candidates that pass the compile gate proceed to verification. 3.5
Stage 4: Verify
The Verify stage applies the GREEN phase of the TD gate before full regression. For a compiled patch p, E VI ACT first reruns the originally failing target tests on the patched workspace s0 ⊕ p. If these GREEN tests fail, the resulting failure evidence is returned to the patch agent for refinement. Repeated GREEN failures under the same suspect set trigger re-localization when budget remains. Only GREEN-passing patches proceed to full regression validation. This ordering separates targetfailure repair from regression preservation. Formally, the final acceptance condition is:
4.1
Datasets and Metrics
Java Python Python Python
4.3
835 500 300 100
17 12 11 55
Implementation
E VI ACT is implemented in Python and interfaces with benchmark-specific repository, build, and test environments. Each run stops when a patch is accepted, after 45 minutes, or after 56 tool calls, allocated as 36 localization calls and 20 patching calls. This budget was fixed before evaluation and was not tuned on benchmark validation outcomes; Appendix A discusses the allocation rationale. All LLMs are accessed through vendor APIs. When decoding controls are available, we set temperature = 1.0 and top-p = 1.0; otherwise, we use provider defaults. Prompt details are provided in Appendix B.
Datasets. We evaluate E VI ACT under four benchmark settings: Defects4J 2.0 (Just et al., 2014), SWE-bench Verified (Chowdhury et al., 2024), SWE-bench Lite (Jimenez et al., 2024), and SWEbench Live (Zhang et al., 2025). Since SWE-bench Live is designed to reduce benchmark leakage from public training and evaluation artifacts, we randomly sample 100 instances for evaluation, covering 55 repositories. Table 1 summarizes the dataset statistics. The target-test protocol used by E VI ACT is detailed in Appendix C. Metrics. We report resolve rate as the primary effectiveness metric, defined as the percentage of instances for which the accepted patch passes the benchmark validation suite. We report average perbug API cost in USD as the primary efficiency metric. Additional diagnostic metrics for ablation and error analysis are defined in Appendix C. 4.2
Defects4J 2.0 SWE-bench Verified SWE-bench Lite SWE-bench Live
et al., 2025). On SWE-bench, we compare with SWE-Agent (Yang et al., 2024), Agentless (Xia et al., 2025), and OpenHands (Wang et al., 2025) when results on the same split are available. Baseline results are obtained from the corresponding papers under comparable benchmark settings. Models. To align with prior GPT-4o-based baselines, we evaluate E VI ACT with GPT-4o (OpenAI, 2024) as the primary backbone. We further evaluate E VI ACT with GPT-5.2 (OpenAI, 2025), Gemini-2.5-Pro (Team, 2025), and DeepSeekV3.2 (DeepSeek-AI, 2025).
where IGREEN indicates whether the originally failing target tests pass and IREG indicates whether the full regression suite passes. A patch is accepted only when gTD (p) = 1.
Experimental Setup
Language Number Coverage
Table 1: Benchmark settings used in our evaluation, including the programming language, number of instances, and repository/project coverage.
gTD (p) = IGREEN (s0 ⊕ p) · IREG (s0 ⊕ p), (4)
4
Dataset
5
Results and Analysis
5.1
Overall Results
Comparison with Baselines. Table 2 compares E VI ACT with representative baselines on Defects4J 2.0 and SWE-bench. Under the common GPT-4o backbone, E VI ACT achieves the best resolve rate among evaluated systems on all four datasets: 25.0% on Defects4J 2.0, 40.4% on SWE-bench Verified, 38.0% on SWE-bench Lite, and 16.0% on SWE-bench Live. Compared with the strongest available GPT-4o baseline on each dataset, these results correspond to gains of 5.4, 1.6, 6.0, and 4.0 percentage points, respectively.
Baselines and Models
Baselines. Because prior APR systems report results on different benchmark splits, we compare against the available baselines for each setting where comparable results are reported. On Defects4J 2.0, we compare with RepairAgent (Bouzenia et al., 2024) and AdverIntent (Ye 5
Framework
Model
Defects4J 2.0
SWE-bench
835 bugs
Verified 500
Lite 300
Live 100
RepairAgent AdverIntent SWE-Agent OpenHands Agentless E VI ACT
GPT-4o GPT-4o GPT-4o GPT-4o GPT-4o GPT-4o
19.6% / $1.49 16.9% / $2.41 – – – 25.0% / $0.17
– – 23.2% / – – 38.8% / – 40.4% / $0.20
– – 18.3% / $2.53 22.0% / $1.72 32.0% / $0.70 38.0% / $0.19
– – 10.0% / $2.42 8.0% / $1.89 12.0% / $0.77 16.0% / $0.23
E VI ACT E VI ACT E VI ACT
GPT-5.2 DeepSeek-V3.2 Gemini-2.5-Pro
47.3% / $0.34 38.3% / $0.04 32.3% / $0.44
70.2% / $0.38 58.4% / $0.05 53.6% / $0.46
64.0% / $0.43 55.7% / $0.03 51.2% / $0.42
36.0% / $0.45 28.0% / $0.04 21.0% / $0.48
Table 2: Main results on Defects4J 2.0 and SWE-bench. Each cell reports resolve rate (%) / average per-bug cost ($). “–” indicates that the corresponding result is unavailable (Note: Defects4J 2.0 and SWE-bench have different baselines). Bold marks the best resolve rate in each dataset column. Blue highlights E VI ACT using the same GPT-4o backbone as prior baselines; green highlights E VI ACT with stronger alternative backbones.
Where baseline costs are reported, E VI ACT shows 70.1–88.6% lower reported per-bug API cost than the corresponding GPT-4o baselines. Model Scaling. Among LLM variants, GPT-5.2 provides the highest-performance setting, achieving the best resolve rates on all four datasets: 47.3% on Defects4J 2.0, 70.2% on SWE-bench Verified, 64.0% on SWE-bench Lite, and 36.0% on SWEbench Live. DeepSeek-V3.2 provides the lowestcost setting, ranking second in resolve rate on all four datasets while reducing repair cost to $0.03– $0.05 per bug. Gemini-2.5-Pro is less competitive, with lower resolve rates than GPT-5.2 and DeepSeek-V3.2 despite higher repair cost. As a model-level diagnostic, Figure 3 compares the four E VI ACT backbones on Defects4J 2.0. The results suggest that higher resolve rates tend to coincide with stronger file-level localization and fewer compile-gate rejections, indicating that E VI ACT is sensitive to both suspect construction and executable patch generation. GPT-5.2 achieves the highest resolve rate while maintaining moderate runtime, token usage, and tool-call counts. DeepSeek-V3.2 performs more extensive exploration, but its lower API price yields the lowest average per-bug cost. Gemini-2.5-Pro shows weaker localization and more compile-gate rejections, corresponding to lower cost-effectiveness. Overall, GPT-5.2 offers the highest-performance configuration, while DeepSeek-V3.2 offers the most costeffective configuration. 5.2
instance stratified sample from the four evaluation datasets, following the sampling protocol in Appendix E. We fix the backbone to DeepSeek-V3.2 to control for model variation and keep evaluation cost manageable. We compare five configurations: no guardrails, +Retrieval, +Compile, +TD, and full E VI ACT. The stratified bootstrap procedure used for ablation uncertainty estimation is described in Appendix E. Figure 4 summarizes the aggregate effect of the guardrails. All guarded configurations improve resolve rate over the no-guardrail baseline, while full E VI ACT achieves the largest gain: +13.0 percentage points, with 84.1K fewer tokens and 195.3s lower runtime. These gains align with the three failure modes in Figure 1: the retrieval scaffold reduces localization drift by focusing the suspect set, the compile gate filters malformed, unappliable, and non-compilable candidates before validation, and the TD gate avoids premature full regression by requiring the originally failing tests to turn GREEN first. Figure 5 shows the corresponding stage-level effects. In Figure 5(a), +Retrieval and full E VI ACT achieve the best Hit@3/token trade-off, indicating improved localization without broader exploration. In Figure 5(b), guarded variants reduce validation failures and patch-token usage, showing that invalid or target-failing candidates are filtered before expensive validation. The full system is both the most effective and efficient, consistent with the sequential dependence of the controls: retrieval stabilizes the repair context, compile gating enforces executability, and the TD gate reduces premature validation cost.
Ablation Studies
To quantify the contribution of each guardrail, we conduct a component ablation study on a 2006
40 20
30 40 Resolve rate (%)
1.5
50
20
(a) Localization Performance
30 40 Resolve rate (%)
200 150 GPT-5.2
100
GPT-5.2
1.0
0 20
2.0
DeepSeek-V3.2
Gemini-2.5-pro
300
DeepSeek-V3.2
Tokens (k)
CompFail
GPT-4o
250
Gemini-2.5-pro
2.5
GPT-5.2
60
Hit@3 (%)
GPT-4o
DeepSeek-V3.2
Tokens (k)
80
50
GPT-4o
200
(b) Patch Performance
600
DeepSeek-V3.2
200 150 100
Gemini-2.5-pro
300 400 500 Runtime (s)
250
GPT-5.2 Gemini-2.5-pro GPT-4o
5
(c) Efficiency
10 15 20 Tool Calls
25
(d) Tool Use
Figure 3: Cross-LLM comparison of E VI ACT on Defects4J 2.0 from four perspectives. (a) Localization performance, measured by Hit@3 against resolve rate. (b) Patch performance, measured by C OMP FAIL, the average number of candidates rejected by the compile gate per bug, against resolve rate. (c) Runtime–token efficiency, measured by average runtime and token usage. (d) Tool-use behavior, measured by tool-call count and token usage.
0
Tokens (k)
0
Runtime (s)
0
Localization
+2.5 +7.5 +7.5 +13 -19.1 -41.0
0
-84.1
Tool calls
Resolve (%)
Patch
Dataset
40
Base Ret. Comp. TD EviACT
Defects4J 2.0 SWE-bench Verified SWE-bench Lite SWE-bench Live
42.0 34.0 54.0 56.0
42.0 40.0 40.0 32.0
16.0 22.0 6.0 8.0
0.0 4.0 0.0 4.0
(b) Tool Use Distribution
Overall (avg.)
46.5
38.5
13.0
2.0
20
-130 +33 -112 -195
0
Base Ret. Comp. TD EviACT
(a) Resolution Rate and Efficiency
Figure 4: Overall component ablation of E VI ACT. (a) Changes in resolve rate, token usage, and runtime relative to the no-guardrail baseline. (b) Tool-call distribution across the Localize and Patch stages. EviACT
50
+Compile
25
10
Baseline
75 +Retrieval
ValFail
Hit@3 (%)
100
5
100
150
200
Localization Tokens (K)
(a) Localization Performance
0
Table 3: Failure modes of E VI ACT with DeepSeekV3.2. Each metric reports (%). Misloc. = missed repair context; Mis-edit = correct context retrieved but wrong edit; Partial = partially addresses the observed failure but remains incomplete; Timeout = budget or tool exhaustion.
Baseline +Compile EviACT
+TD
100
+Retrieval
a single patch in the correct repair context; the patch passes the compile gate, turns the target failing test GREEN under the TD gate, and succeeds under full validation with 50.9K tokens and 153.6s runtime. This case highlights the diagnostic value of E VI ACT’s evidence-to-action design: the trace makes clear that the full system succeeds not by additional search, but by keeping the repair hypothesis aligned with the observed failure, recovering the configuration-related repair context from stack-frame and symbol evidence, and validating a compact executable patch only after it resolves the original failing behavior.
+TD
150
200
Patch Tokens (K)
(b) Patch Performance
Figure 5: Stage-level ablation analysis of E VI ACT. (a) Localization performance, comparing Hit@3 against localization-token usage. (b) Patch performance, comparing validation failures against patch-token usage.
5.3
Misloc. Mis-edit Partial Timeout
Diagnostic Analysis
We compare the no-guardrail variant and full E VI ACT on JacksonDatabind-18 from Defects4J 2.0, where a timezone configuration error causes TestConfig.testDateFormatConfig to fail. The full trace comparison is provided in Appendix F. Given the same failure evidence, the noguardrail variant follows a generic keyword-search path, mislocalizes the repair context to MappingIterator.java, and repeatedly patches an unrelated file. This early localization error cascades into 19 failed patch attempts, consuming 239.3K tokens and 1129s before exhausting the budget. In contrast, E VI ACT uses stack-frame and symbol evidence, including ObjectMapper#setDateFormat, to retrieve BaseSettings.java. It then generates
5.4
Failure Analysis
Aggregate resolve rates do not reveal where E VI ACT still fails. We therefore manually analyze 200 failed runs under DeepSeek-V3.2, with 50 instances from each benchmark, following the stratified sampling protocol in Appendix E. Two authors independently assign the earliest traceable failure in the evidence-to-action chain using the criteria in Appendix G; disagreements are resolved by discussion, with substantial agreement (κ = 0.82). Table 3 reports the resulting distribution. 7
Dataset
ACT with DeepSeek-V3.2. We sample 100 resolved instances in total, with 25 patches from each dataset, using the stratified sampling protocol in Appendix E. Two authors independently annotate the sampled patches (κ = 0.86), with disagreements resolved by discussion. The annotation criteria are provided in Table 8. As shown in Table 4, 78.0% of audited patches are labeled as Correct, indicating that most sampled benchmark-resolved E VI ACT patches are semantically consistent with the developer fix or issue requirement. However, 11.5% are labeled as Overfitting: these patches pass the full benchmark test suite but do not implement a general repair. This exposes a limitation beyond executability and target-test success: a patch that survives the benchmark oracle may still exploit incomplete tests, hard-code observed behavior, weaken program logic, or cover only the exercised branch. The remaining 10.5% are labeled as Unknown: these patches differ from the developer fix, but the available issue description, tests, and local code context are insufficient to determine whether they are incorrect or valid alternative repairs. Overall, the analysis suggests that E VI ACT filters nonexecutable candidates before final validation, but full benchmark validation can still leave overfitted or uncertain patches. This finding motivates stronger semantic validation beyond executability and passing the target tests.
Correct Overfitting Unknown
Defects4J 2.0 SWE-bench Verified SWE-bench Lite SWE-bench Live
81.0 76.0 72.0 83.0
12.0 14.0 12.0 8.0
7.0 10.0 16.0 9.0
Overall (avg.)
78.0
11.5
10.5
Table 4: Manual patch correctness assessment of E VI ACT with DeepSeek-V3.2. Each metric reports (%). Correct = semantically valid repair; Overfitting = passes the full test suite but does not implement a general repair; Unknown = insufficient evidence to determine correctness.
The remaining failures are dominated by mislocalization and mis-edit. Mislocalization accounts for 46.5% of failed runs on average, confirming that repository-level repair remains highly sensitive to whether the correct repair context is surfaced. Manual inspection further separates mislocalization into two recurring causes: budget-limited graph expansion, where the fix file is structurally reachable but not explored within the budget, and stack trace mismatch, where failure evidence points to the failure manifestation rather than the repair location. Misedit accounts for another 38.5% of failures: the correct context is available, but the agent modifies the wrong location or implements an incorrect semantic change. Therefore, these two categories cover 85.0% of failures, showing that the main remaining bottleneck is not tool access or patch executability, but evidence-aligned semantic action. Partial repairs account for 13.0% of failures and expose a later-stage limitation: a patch may address the observed target failure but still violate broader invariants revealed by full regression. Timeouts are rare overall at 2.0%, mainly reflecting repositoryscale execution cost rather than patch construction failure. Notably, malformed or non-compilable patches do not appear as terminal failure modes in the sampled unresolved set after compile-gated filtering. Thus, E VI ACT largely removes executable invalidity as a final failure mode; future gains could focus on robust retrieval, more reliable evidencegrounded editing, and stronger semantic validation. 5.5
6
Conclusion
In this paper, we presented E VI ACT, an evidencedriven agentic APR framework that turns execution evidence into repair decisions through three guardrails. Across four benchmarks, E VI ACT achieves higher resolve rates than the strongest reported comparable baselines and shows lower reported per-bug API cost where baseline costs are available. Ablations, diagnostic analysis, and manual inspection suggest that these gains are associated with coordinated evidence-driven guardrails, which ground repair context, filter invalid patches before validation, and shift the remaining failure frontier toward semantic grounding. More broadly, E VI ACT provides practical evidence-to-action control for agentic APR, making repair decisions more effective and reliable. Future work should extend this design toward adaptive repair systems that refine localization, editing, and validation from accumulated execution feedback.
Patch Correctness Analysis
While the failure analysis examines unresolved runs, benchmark-resolved patches may still be semantically incorrect due to incomplete test oracles. We therefore conduct a manual patch correctness analysis on resolved instances generated by E VI 8
Limitations
are evaluated on the same sampled instances with the same backbone and budget, allowing us to examine whether the guardrails reduce computational and interaction overhead relative to ablated variants.
Benchmark contamination. As with prior APR evaluations on public benchmarks (Xia et al., 2025; Yang et al., 2024), benchmark contamination remains a potential threat. Defects4J 2.0, SWE-bench Verified, and SWE-bench Lite are publicly available, and the training data of closed-source LLMs cannot be fully audited. We mitigate this risk by emphasizing same-backbone framework comparisons and by including SWE-bench Live, which is designed to reduce reliance on static benchmark data and public evaluation artifacts. However, SWE-bench Live reduces rather than eliminates contamination risk, especially for proprietary models whose training corpora are not observable.
References Daman Arora, Atharv Sonwane, Nalin Wadhwa, Abhav Mehrotra, Saiteja Utpala, Ramakrishna Bairi, Aditya Kanade, and Nagarajan Natarajan. 2024. Masai: Modular architecture for software-engineering ai agents. arXiv preprint arXiv:2406.11638. Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2024. Repairagent: An autonomous, llmbased agent for program repair. arXiv preprint arXiv:2403.17134.
Programming-language and open-weight model coverage. Our evaluation covers Defects4J 2.0 and SWE-bench datasets, which primarily exercise Java and Python repository repair. Although these benchmarks cover different project structures, issue types, and validation settings, they do not fully represent the diversity of modern software ecosystems, such as C/C++, JavaScript/TypeScript, Rust, Go, mobile projects, or large polyglot systems. Since E VI ACT relies on repository indexing, compilation or build feedback, target-test execution, and full validation, adapting it to languages or ecosystems with weaker test infrastructure, nonstandard build systems, flaky tests, or limited execution feedback may require additional engineering. We include open-weight model results and their implementation details in Appendix D, but broader evaluation across open-weight backbones and programming ecosystems remains future work.
José Cambronero, Michele Tufano, Sherry Shi, Renyao Wei, Grant Uy, Runxiang Cheng, Chin-Jung Liu, Shiying Pan, Satish Chandra, and Pat Rondon. 2025. Abstain and validate: A dual-llm policy for reducing noise in agentic program repair. arXiv preprint arXiv:2510.03217. Dong Chen, Shaoxin Lin, Muhan Zeng, Daoguang Zan, Jian-Gang Wang, Anton Cheshkov, Jun Sun, Hao Yu, Guoliang Dong, Artem Aliev, Jie Wang, Xiao Cheng, Guangtai Liang, Yuchi Ma, Pan Bian, Tao Xie, and Qianxiang Wang. 2024. Coder: Issue resolving with multi-agent and task graphs. Preprint, arXiv:2406.01304. Zimin Chen, Steve Kommrusch, Michele Tufano, LouisNoël Pouchet, Denys Poshyvanyk, and Martin Monperrus. 2019. Sequencer: Sequence-to-sequence learning for end-to-end program repair. IEEE Transactions on Software Engineering, 47(9):1943–1959. Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, Carlos E Jimenez, John Yang, Leyton Ho, Tejal Patwardhan, Kevin Liu, and Aleksander Madry. 2024. Introducing swe-bench verified. https://openai.com/index /introducing-swe-bench-verified/.
External baseline comparability and efficiency metrics. Our comparison with prior APR systems is based on the same benchmark settings and reported results when available, but not all systems can be rerun under identical tool budgets, prompts, execution environments, retry policies, or cost-accounting rules. Thus, external comparisons should be interpreted as reported same-setting comparisons rather than fully controlled head-tohead reruns. This limitation is particularly relevant for monetary API cost, which depends on provider pricing and token accounting as well as framework design. For this reason, we complement API cost with price-independent efficiency metrics, including runtime, token usage, and tool-call counts. In the controlled internal ablations, all configurations
DeepSeek-AI. 2025. Deepseek-v3.2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556. Kevin Han, Siddharth Maddikayala, Tim Knappe, Om Patel, Austen Liao, and Amir Barati Farimani. 2025. Tdflow: Agentic workflows for test driven software engineering. arXiv e-prints, pages arXiv–2510. Yiwei Hu, Zhen Liu, Kedie Shu, Shenghua Guan, Deqing Zou, Shouhuai Xu, Bin Yuan, and Hai Jin. 2025. {SoK}: Automated vulnerability repair: Methods, tools, and assessments. In 34th USENIX Security Symposium (USENIX Security 25), pages 4421–4440.
9
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations.
Sergey Mechtaev, Jooyong Yi, and Abhik Roychoudhury. 2016. Angelix: Scalable multiline program patch synthesis via symbolic analysis. In Proceedings of the 38th international conference on software engineering, pages 691–701. Hoang Duong Thien Nguyen, Dawei Qi, Abhik Roychoudhury, and Satish Chandra. 2013. Semfix: Program repair via semantic analysis. In 2013 35th International Conference on Software Engineering (ICSE), pages 772–781. IEEE.
René Just, Darioush Jalali, and Michael D Ernst. 2014. Defects4j: A database of existing faults to enable controlled testing studies for java programs. In Proceedings of the 2014 international symposium on software testing and analysis, pages 437–440.
OpenAI. 2024. GPT-4o System Card. OpenAI. Available at https://openai.com/index/gpt-4o-sys tem-card/.
Youngjoon Kim, Sunguk Shin, Hyoungshick Kim, and Jiwon Yoon. 2025. Logs in, patches out: Automated vulnerability repair via {Tree-of-Thought}{LLM} analysis. In 34th USENIX Security Symposium (USENIX Security 25), pages 4401–4419.
OpenAI. 2025. Introducing GPT-5.2. OpenAI. Available at https://openai.com/index/introducing -gpt-5-2/.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.
Anvith Pabba, Alex Mathai, Anindya Chakraborty, and Baishakhi Ray. 2025. Semagent: A semantics aware program repair agent. arXiv preprint arXiv:2506.16650. Nikhil Parasaram, Earl T Barr, and Sergey Mechtaev. 2021. Trident: Controlling side effects in automated program repair. IEEE Transactions on Software Engineering, 48(12):4717–4732.
Claire Le Goues, Michael Dewey-Vogt, Stephanie Forrest, and Westley Weimer. 2012. A systematic study of automated program repair: Fixing 55 out of 105 bugs for 8 each. In 2012 34th international conference on software engineering (ICSE), pages 3–13. IEEE.
Qwen Team. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. Haifeng Ruan, Yuntong Zhang, and Abhik Roychoudhury. 2024. Specrover: Code intent extraction via llms. arXiv preprint arXiv:2408.02232.
Hongwei Li, Yuheng Tang, Shiqi Wang, and Wenbo Guo. 2025. Patchpilot: A stable and cost-efficient agentic patching framework. arXiv e-prints, pages arXiv–2502.
Xunzhu Tang, Jiechao Gao, Jin Xu, Tiezhu Sun, Yewei Song, Saad Ezzini, Wendkûuni C Ouédraogo, Jacques Klein, and Tegawendé F Bissyandé. 2025. Synfix: Dependency-aware program repair via relationgraph analysis. In Findings of the Association for Computational Linguistics: ACL 2025, pages 4878– 4894.
Yizhou Liu, Pengfei Gao, Xinchen Wang, Jie Liu, Yexuan Shi, Zhao Zhang, and Chao Peng. 2024. Marscode agent: Ai-native automated bug fixing. arXiv preprint arXiv:2409.00899.
Yuheng Tang, Hongwei Li, Kaijie Zhu, Michael Yang, Yangruibo Ding, and Wenbo Guo. 2026. Co-patcher: Collaborative software patching with componentspecific small reasoning models. Advances in Neural Information Processing Systems, 38:42041–42069.
Fan Long and Martin Rinard. 2016. Automatic patch generation by learning correct code. In Proceedings of the 43rd annual ACM SIGPLAN-SIGACT symposium on principles of programming languages, pages 298–312.
Gemini Team. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261.
Thibaud Lutellier, Hung Viet Pham, Lawrence Pang, Yitong Li, Moshi Wei, and Lin Tan. 2020. Coconut: combining context-aware neural translation models using ensemble for program repair. In Proceedings of the 29th ACM SIGSOFT international symposium on software testing and analysis, pages 101–114.
Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys Poshyvanyk. 2019. An empirical study on learning bugfixing patches in the wild via neural machine translation. ACM Transactions on Software Engineering and Methodology (TOSEM), 28(4):1–29.
Yingwei Ma, Qingping Yang, Rongyu Cao, Binhua Li, Fei Huang, and Yongbin Li. 2025. Alibaba lingmaagent: Improving automated issue resolution via comprehensive repository exploration. In Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering, pages 238– 249.
Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang Tran, Fuqiang Li,
10
Ren Ma, Mingzhang Zheng, Bill Qian, Daniel Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, and 5 others. 2025. Openhands: An open platform for ai software developers as generalist agents. In International Conference on Learning Representations.
Wang, Yufan Huang, Shengyu Fu, Elsie Nallipogu, Qingwei Lin, Yingnong Dang, Saravan Rajmohan, and Dongmei Zhang. 2025. Swe-bench goes live! arXiv preprint arXiv:2505.23419. Quanjun Zhang, Chunrong Fang, Yuxiang Ma, Weisong Sun, and Zhenyu Chen. 2023. A survey of learningbased automated program repair. ACM Transactions on Software Engineering and Methodology, 33(2):1– 69.
Westley Weimer, ThanhVu Nguyen, Claire Le Goues, and Stephanie Forrest. 2009. Automatically finding patches using genetic programming. In 2009 IEEE 31st International Conference on Software Engineering, pages 364–374. IEEE.
Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Autocoderover: Autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 1592–1604.
Chu-Pan Wong, Priscila Santiesteban, Christian Kästner, and Claire Le Goues. 2021. Varfix: balancing edit expressiveness and search effectiveness in automated program repair. In Proceedings of the 29th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, pages 354–366.
A
Default Configuration Rationale
This appendix explains the default localization and budget settings used in E VI ACT. These settings were fixed before evaluation and were not selected by optimizing benchmark validation outcomes. We do not claim that they are globally optimal. Instead, they define a compact and reproducible operating point that bounds repository exploration, patch refinement, and validation cost across benchmarks.
Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2025. Demystifying llm-based software engineering agents. Proceedings of the ACM on Software Engineering, 2(FSE):801–824. Chunqiu Steven Xia and Lingming Zhang. 2022. Less training, more repairing please: revisiting automated program repair via zero-shot learning. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 959–971.
A.1
Chengxing Xie, Bowen Li, Chang Gao, He Du, Wai Lam, Difan Zou, and Kai Chen. 2025. Swefixer: Training open-source llms for effective and efficient github issue resolution. arXiv preprint arXiv:2501.05040.
Suspect-Set Size
E VI ACT uses K = 3 suspect spans as the default localized working set. This choice reflects a precision–recall trade-off in repository-level repair. A smaller suspect set gives the patch agent a compact context and reduces the risk of editing loosely related code, but it may miss the repair location when RED evidence points only to a failure manifestation. A larger suspect set increases the chance of including the correct repair context, but it also dilutes the working set and can encourage the agent to spread edits across weakly related files. We therefore use K = 3 as a compact default rather than as a tuned optimum. It allows the retrieval scaffold to expose multiple structurally related candidates while keeping patch generation anchored to a small evidence-backed context. This setting also aligns with the intended role of the retrieval scaffold: to constrain the patch agent to a focused suspect set, not to provide a broad repository summary.
Boyang Yang, Zijian Cai, Fengling Liu, Bach Le, Lingming Zhang, Tegawendé F Bissyandé, Yang Liu, and Haoye Tian. 2025. A survey of llm-based automated program repair: Taxonomies, design paradigms, and applications. arXiv preprint arXiv:2506.23749. John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528– 50652. He Ye, Aidan ZH Yang, Chang Hu, Yanlin Wang, Tao Zhang, and Claire Le Goues. 2025. Adverintentagent: Adversarial reasoning for repair based on inferred program intent. Proceedings of the ACM on Software Engineering, 2(ISSTA):1398–1420. Zheng Yu, Ziyi Guo, Yuhang Wu, Jiahao Yu, Meng Xu, Dongliang Mu, Yan Chen, and Xinyu Xing. 2025. {PATCHAGENT}: A practical program repair agent mimicking human expertise. In 34th USENIX Security Symposium (USENIX Security 25), pages 4381– 4400.
A.2
Ranking Order
The lexicographic ranking key prioritizes direct executable evidence before weaker structural signals. Exact trace-symbol matches are ranked first because stack frames, failing test names, and
Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan
11
assertion-related symbols provide the strongest anchors when they are available. File or line proximity is ranked next because failure logs often identify the manifestation site even when they do not identify the repair site. Graph distance and relation support are then used to recover structurally connected contexts through containment, call, import, and inheritance relations. Span length is used only as a final tie-breaker, preferring compact spans when other evidence signals are equivalent. This ordering is intentionally conservative. It favors candidates that are directly grounded in RED evidence before expanding to more indirect graphneighborhood evidence. The design reduces unconstrained keyword search while still allowing the retrieval scaffold to move beyond the immediate failure frame when the repair location is structurally related but not directly named in the failure log. A.3
precise failure evidence may benefit from even tighter localization and fewer tool calls. The main experiments therefore evaluate E VI ACT under a single pre-specified operating point. Controlled ablations in the main paper isolate the contribution of the three guardrails under this fixed protocol. A full search over suspect-set sizes and budget allocations is orthogonal to the main claim and is left for future work on adaptive evidence budgeting.
B
This appendix reports the prompt templates used by E VI ACT. Placeholders such as <BENCHMARK>, <LANGUAGE>, and <RED_LOG_PATH> are filled automatically at runtime. The prompts enforce the same gated workflow used in the main method: RED evidence is first used for localization, candidate edits are then checked by the compile gate, and only compilable patches proceed to GREEN target-test validation and full regression.
Tool-Call Budget
E VI ACT uses a fixed 56-call budget per instance, divided into 36 localization calls and 20 patching calls. The purpose of this budget is to make runs reproducible and to prevent long open-ended agent trajectories. The allocation gives more budget to localization because early repair drift is difficult to recover from: if the suspect set is wrong, additional patch attempts tend to refine an incorrect repair hypothesis rather than fix the underlying localization error. The patching budget is kept bounded because candidate edits are filtered by the compile gate and then by the TD gate. In other words, E VI ACT is designed to improve patch quality by controlling the evidence flow into patch generation, rather than by allowing many unconstrained patch retries. This budget allocation is therefore part of the framework’s stage-boundary design: localization is allowed enough interaction to establish an evidencebacked working set, while patching is constrained to produce executable and target-recovering edits within that working set. A.4
Prompt Templates
B.1
System Prompt
System Prompt You are an agentic program repair system for <BENCHMARK>. Global rules: - Do not modify, skip, delete, or disable tests. - Do not hardcode behavior only for the observed failing test. - Produce minimal and localized patches. - Preserve the syntax, style, and idioms of <LANGUAGE>. - Prefer changes that explain the RED failure rather than broad rewrites. Workflow: 1. Setup: check out the buggy revision and prepare the environment. 2. RED: run the originally failing target test and collect failure evidence. 3. Localize: use RED evidence and the repository index to identify suspicious files, symbols, and line ranges. 4. Patch: generate one minimal candidate edit over the localized working set. 5. Compile: apply the edit and run syntax or build checks. 6. GREEN: rerun the originally failing target test and require it to pass. 7. Validate: run full regression validation before accepting the patch. When asked to localize, output only the requested localization JSON. When asked to patch, output only the requested edit JSON or unified diff. Do not include explanations unless explicitly requested by the controller.
Scope of the Default Settings
These default settings should be interpreted as a fixed evaluation protocol, not as evidence that the chosen values are optimal for all repositories, languages, or models. Larger repositories, weaker stack traces, flaky tests, or nonstandard build systems may require different suspect-set sizes or budget allocations. Conversely, smaller projects with
B.2
Localization Prompt
Localization Prompt Goal: localize the bug using RED failure evidence and the repository index. Inputs: - Workdir: <WORKDIR>
12
}
- RED log: <RED_LOG_PATH> - Repository index: <INDEX_PATH> - Target language: <LANGUAGE>
] The controller also accepts a unified diff if structured edits cannot express the change. In that case, output only the unified diff beginning with "diff --git". Do not use markdown fences or explanations.
Instructions: 1. Read the RED log first. 2. Extract the failing test, assertion or exception message, and relevant stack frames. 3. Derive primary symbols from stack frames, test names, assertion text, and error messages. 4. Query the repository index using symbol lookup before textual search. 5. Read matched AST spans before selecting suspects. 6. Inspect caller, callee, import, inheritance, or reference spans only when they are structurally connected to the RED evidence. 7. Keep the working set small and evidence-grounded.
C
Metric details. Table 5 defines the effectiveness and efficiency metrics used throughout the paper. We use file-level Hit@3 because the patch agent operates over a small top-K suspect set, and repository-level repair typically requires surfacing at least one file modified by the benchmark reference patch before finer-grained editing can be attempted. This metric is therefore used as a coarse localization proxy rather than a complete measure of line- or symbol-level edit precision.
Output only valid JSON in the following schema: { "red_test": "name of the failing target test", "red_failure_summary": "one-sentence failure summary", "failure_contract": [ "behavior that should hold but is violated" ], "primary_symbols": [ "symbol names derived from RED evidence" ], "suspects": [ { "file": "relative/path/to/file", "symbol": "class, method, function, or field name", "start_line": 1, "end_line": 20, "evidence": "why this span is connected to the RED failure" } ], "working_set": [ { "file": "relative/path/to/file", "symbol": "symbol to inspect or edit" } ] }
B.3
Evaluation Details
Target-test protocol. E VI ACT is designed for TD-gated APR. For Defects4J 2.0, the target tests are the benchmark-provided triggering tests. For SWE-bench Verified, SWE-bench Lite, and SWEbench Live, the target tests are the benchmarkprovided FAIL_TO_PASS tests. E VI ACT uses these target tests in the TD gate: it first executes them on the original buggy revision to obtain RED failure evidence, and reruns the same tests after patch generation to check whether the observed failure turns GREEN. The agent does not access reference patches or hidden oracle annotations; target tests are used only through their execution outcomes and failure logs. Final validation follows the benchmark validation protocol, including regression or PASS_TO_PASS checks where applicable.
Patch Prompt
Patch Prompt Goal: fix the localized bug with a minimal candidate edit. Inputs: - RED evidence: <RED_EVIDENCE> - Suspect spans: <SUSPECT_SPANS> - Diagnostic context: <DIAGNOSTIC_CONTEXT> - Target language: <LANGUAGE>
D
Instructions: 1. Patch only files that are supported by the suspect spans unless additional context is necessary. 2. Preserve unrelated behavior and avoid broad rewrites. 3. Do not modify tests. 4. Do not hardcode behavior only for the observed failing test. 5. Ensure that the edit applies cleanly and preserves syntax. 6. If the previous candidate failed to apply, compile, or pass GREEN, use the diagnostic context to revise only the relevant part of the patch.
Open-weight Model Results
To examine whether E VI ACT can operate with open-weight backbones, Table 6 reports results with two Qwen variants (Qwen Team, 2024) on Defects4J 2.0 and SWE-bench Verified. For implementation details, two Qwen variants are served with vLLM (Kwon et al., 2023); Qwen2.5-Coder32B-Instruct runs on 2×A100 GPUs and Qwen2.572B-Instruct runs on 4×H100 GPUs. On Defects4J 2.0, Qwen2.5-72B-Instruct reaches 18.0%, which is lower than E VI ACT with stronger proprietary models but still comparable to GPT-4o baselines such as AdverIntent and RepairAgent. This suggests that E VI ACT can remain competitive on structured Java repair settings even with an open-weight backbone, especially when the benchmark provides executable failures
Output only valid JSON in the following schema: [ { "path": "relative/path/to/file", "ops": [ { "type": "replace", "start_line": 10, "end_line": 12, "text": "replacement code with preserved indentation\n" } ]
13
Table 5: Definitions of evaluation metrics for assessing the effectiveness of E VI ACT and its efficiency.
Failure mode Mislocalization Localize
Metric
Definition Effectiveness
Resolve (%) Hit@3 (%)
CompFail
ValFail
Mis-edit Patch
Percentage of instances solved by the benchmark oracle. Standard Top-3 file-level localization accuracy; an instance is counted as hit if any reference-modified file appears in the top three localized files. Number of attempted patches rejected by the compile gate, including malformed diffs, failed hunk application, syntax errors, and build failures. Number of attempted patches that compile but fail target-test or full-regression validation.
Partial repair Full regression
Timeout Any stage
Criterion The correct fix file or symbol is absent from the retrieved suspect set, so the agent lacks the necessary repair context. The correct fix file or symbol is retrieved, but the executable patch modifies the wrong location or implements an incorrect semantic change, causing the GREEN target tests to fail. The patch passes the GREEN target tests but fails full regression, indicating an incomplete repair, missed edge case, or regression risk. The run exhausts the time or tool budget before reaching a stable validation outcome, including patch generation, GREEN test, or full regression.
Efficiency Cost ($) Runtime (s) Tokens (k)
Tool Calls
Table 7: Root-cause criteria used in failure-mode annotation. The stage shown under each label indicates where the failure is primarily assigned.
Average API cost per bug computed from token usage and vendor pricing. End-to-end wall-clock time per bug from localization to verification. Total LLM tokens consumed per bug, including input and output tokens; also reported by stage when available. Repository, search, edit, build, or test invocations per instance, reported by stage when available.
Qwen2.5-Coder-32B Qwen2.5-72B
Correct
The patch is semantically consistent with the developer fix or satisfies the issue requirement without weakening existing behavior. Syntactic differences from the developer patch are allowed. The patch passes benchmark validation but does not implement a general repair, e.g., by hard-coding observed behavior, weakening logic, or covering only the exercised branch or input pattern. The patch differs from the developer fix or follows a different repair strategy, but the issue description, tests, and local context are insufficient to determine whether it is incorrect or a valid alternative repair.
Defects4J 2.0 SWE-bench Verified 14.3% 18.0%
Unknown
15.2% 20.5%
and relatively stable repair environments. On SWE-bench Verified, however, the Qwen variants remain behind GPT-4o baselines, indicating that repository-scale issue repair places higher demands on localization, tool use, and semantic patch generation. Overall, these results show that E VI ACT is compatible with open-weight models, but its effectiveness remains sensitive to the underlying model’s ability to follow retrieved evidence and produce executable, semantically appropriate edits.
E
Criterion
Overfitting
Table 6: Resolve rates of E VI ACT using open-weight Qwen models. Model
Label
Table 8: Manual patch correctness criteria.
placement using a fixed random seed of 42. The sampled instance IDs are released with the artifact. Allocation. For a target sample size N on a dataset, we first assign each non-empty stratum at least one instance whenever possible. The remaining quota is allocated approximately proportional to stratum size. If a stratum contains fewer eligible instances than its allocated quota, all eligible instances from that stratum are included and the remaining quota is redistributed to larger strata.
Stratified Sampling Protocol
We use benchmark-wise stratified sampling for all sampled analyses to ensure coverage over project or repository groups. For Defects4J 2.0, strata are defined by project. For SWE-bench Verified, SWEbench Lite, and SWE-bench Live, strata are defined by repository. Sampling is performed without re-
Ablation studies. For the component ablation, we sample 200 instances in total, with 50 instances from each benchmark. Within each benchmark, instances are sampled to preserve project or repos14
H Patch Correctness Annotation Criteria
itory coverage as much as possible. The same sampled instances are used for all five configurations: no guardrails, +Retrieval, +Compile, +TD, and full E VI ACT. This paired design ensures that differences across configurations are attributable to the ablated guardrails rather than different instance composition. For uncertainty estimation, we use 10,000 stratified bootstrap replicates: each replicate resamples instances with replacement within each benchmark, preserves the 50-instance-perbenchmark allocation, recomputes each ablation metric, and takes the 2.5th and 97.5th percentiles as the 95% confidence interval.
Patch correctness is annotated only for resolved instances. For each sampled patch, annotators inspect the generated diff, issue description, original failure log, full-validation result, and developer/reference fix when available. The goal is to distinguish semantically valid repairs from patches that pass the benchmark tests but do not implement a general fix. Table 8 defines the annotation labels of patch correctness.
Failure analysis. For failure-mode analysis, we sample failed runs only. We draw 50 failed instances from each benchmark under the DeepSeekV3.2 setting, yielding 200 failed runs in total. The sampling follows the same project/repository stratification described above. When a stratum has fewer failed runs than its allocated quota, all failed runs from that stratum are included and the residual quota is redistributed. Patch correctness analysis. For patch correctness analysis, we sample resolved instances only. We draw 25 full-validation-passing patches from each benchmark under the DeepSeek-V3.2 setting, yielding 100 patches in total. The same stratification and redistribution rules are applied to preserve project or repository coverage.
F
Diagnostic Case Study
Table 9 illustrates the evidence chain on JacksonDatabind-18. The baseline follows a generic exception search and repeatedly edits an unrelated file. E VI ACT uses the failure evidence to retrieve the relevant configuration path, applies a single patch, and validates it through the compile and TD gates.
G
Failure Mode Annotation Criteria
For failure analysis, each failed run is assigned a primary failure label at the earliest stage where the evidence-to-action chain breaks. Annotators inspect the retrieved candidates, edited files, generated diffs, developer/reference fix when available, Compile-gate outputs, GREEN-test logs, fullregression logs, and timeout/tool records. When later failures are downstream effects of an earlier error, we label the earliest traceable cause. Table 7 defines the labeling criteria for four failure modes. 15
Table 9: Diagnostic case study on JacksonDatabind-18 (Defects4J 2.0) comparing the Baseline and E VI ACT execution traces across stages (Setup/Localize/Patch/Verify). Red highlights incorrect stack-trace hits, and blue highlights correct stack-trace hits. Stage
Baseline
E VI ACT
Stack Trace junit.framework.AssertionFailedError : expected:<sun.util.calendar.ZoneInfo[id=" America/Los_Angeles ",...]> but was:<sun.util.calendar.ZoneInfo[id=" GMT ",...]> at TestConfig. testDateFormatConfig ( TestConfig.java:221 )
1. Setup
2. Localize
Process: Run test suite Output: test.trigger.log
Process: Run test suite Output: red.log [TD RED]
Process: • Read log file • Search for "Exception" keyword • Read MappingIterator.java
Process: • Read red.log [TD RED] • Lookup ObjectMapper#setDateFormat •
Output: Located file: MappingIterator.java × API calls: 11, Tokens: 35.4K
3. Patch
4. Verify
Process: • Read MappingIterator.java • Generate 1 attempt • Generate 2 attempt • Generate 3 attempt • Generate 4 attempt • ... Output: 19 attempts API calls: 20, Tokens: 203.9K, Compile: ∅
Read
span
of
BaseSettings.java
[Retrieval] Output: Located file: BaseSettings.java ✓ API calls: 6, Tokens: 20.6K Process: • Read span of
TestConfig.java
and
BaseSettings.java [Retrieval] • Generate 1 patch • Compile check [Compile] • Verify with green.log [TD GREEN] Output: 1 attempt, success API calls: 4, Tokens: 30.3K, Compile: ✓
Patch Diff:
Patch Diff:
File: MappingIterator.java - if (t == null || t == JsonToken.END_ARRAY) { + if (t == JsonToken.END_ARRAY) { ... } + if (t == null) { ... } . . .
File: BaseSettings.java - TimeZone tz = (df == null) ? _timeZone : df.getTimeZone(); + // Should NOT change TimeZone when DateFormat changes return new BaseSettings (..., - tz, _defaultBase64); + _timeZone, _defaultBase64);
Process: • Run full test suite • 19 failures remain Output: Result: Failed, Time: 1129s, Total tokens: 239.3K
Process: • Run full test suite • All tests passed Output: Result: Success, Time: 153.6s, Total tokens: 50.9K
16