Securing Agent-Generated Code with Its Own Development History

Securing Agent-Generated Code with Its Own Development History Prateek Kumar Rajput∗ , Shahzeb Khan† , El-Hacen Diallo∗ , Yewei Song∗ , Jacques Klein∗ , and Tegawendé F. Bissyandé∗ ∗ University of Luxembourg, Luxembourg

arXiv:2610.34098v1 [cs.SE] 25 Oct 2026

{prateek.rajput, el-hacen.diallo, yewei.song, jacques.klein, tegawende.bissyande}@uni.lu † Nostrum Biodiscovery S.L., Barcelona, Spain shahzeb.khan@nostrumbiodiscovery.com

Abstract—Pull requests written by coding agents can pass a project’s test suite while omitting a guard or a validation step. To find the missing condition, a reviewer currently relies on static analysis of the final code. An agent-generated change, however, is accompanied by a development history that a human commit does not provide. The agent’s trajectory records each edit together with the agent’s stated assumptions, and structural differencing shows how each changed function evolved. We present T RACER, a workflow that uses this development history as evidence for security review. T RACER tracks changes at the symbol level, treats the agent’s recorded rationale as claims to be verified against the current code, accepts a repair only after assessment and functional validation, and attaches to each accepted repair an executable check that later changes must satisfy. On the SusVibes benchmark, symbol-level tracking places the vulnerable symbol in the review scope for 94.6% of agent patches, compared with 13.4% for the primary scanner warning. With two investigator models, the configuration that includes the agent’s rationale yields the largest number of verified security repairs while preserving functionality, and adding AST history to it does not increase that number. A funnel analysis identifies where the remaining candidates are lost. In a study of structural-biology software, the combination of repair memory and a boundary check allows a subsequent feature request to succeed in nine of ten tasks without reintroducing the repaired vulnerability, whereas functional feedback alone succeeds in four and reintroduces the vulnerability in three. Index Terms—Agentic Software Engineering, Vulnerability Repair, Provenance and Reasoning Traces, Silent Security Fixes, Continuous Software Evolution, Static and Dynamic Analysis.

I. I NTRODUCTION Coding agents now navigate repositories, edit source files, and run test suites through interfaces such as SWE-agent [1], and agent-based repair systems apply the same tool-driven loop to debugging [2]. A patch that passes the project’s tests may still be insecure. Studies of generated code and of developers working with coding assistants report recurring security weaknesses [3]–[5], and SusVibes measures the gap at repository scale by grading functionality and security with separate test suites [6], [7]. In the cohort studied in this paper, 350 of 354 agent patches pass their functional tests and 69 pass their security tests. A team that merges a functionally correct patch therefore merges the missing security condition with it. Reviewing such a patch requires reconstructing what the agent assumed and which condition it omitted. We refer to this

reconstruction as code archaeology. Static application security testing (SAST) tools such as CodeQL [8] and Semgrep [9] report locations and flows in the final snapshot, and industrial deployments have established that diagnostics must be actionable at review time to be used [10]–[12]. LLM-assisted taint and dataflow analysis [13]–[15], analyzer-guided repair [16], and analysis-guided proof-of-concept generation [17] connect model judgments to source and execution evidence. All of these techniques operate on the current source. They do not explain why the agent omitted a guard or what it assumed about upstream inputs, and in our cohort the primary SAST warning is located on the symbol touched by the reference security fix in 13.4% of patches. In the remaining cases, the reviewer must first determine which symbol to examine. An agent-generated change is accompanied by evidence that a human commit does not provide. The trajectory records each edit, each tool call, and the agent’s ReAct-style explanation of its actions [18]. Structural differencing [19] shows how each changed function evolved, for instance a guard added at one turn and removed at a later one. The recorded rationale states assumptions about inputs, callers, and libraries. These statements are not faithful accounts of the model’s internal reasoning [20], [21], and we treat them as claims to be verified. A claim that an input is already validated, for example, creates an obligation to inspect the validator and its callers. The thesis of this paper is that this development history constitutes security evidence, and that review of agent-generated code should be organized around the changed symbols and the recorded claims in addition to scanner warnings. A second problem is that repairs are not preserved over time. Security fixes are often committed without being labelled as such [22], and a later change can undo them [23]. Section II describes an observed case in which a request for legacy checkpoint compatibility re-enabled the unrestricted deserialization that an earlier repair had removed, while every functional test passed. The evidence gathered during investigation, that is the invariant, its rationale, and an executable boundary check, can remain attached to the repaired symbol and be evaluated against every later proposal. We present T RACER (Trace-based Runtime Analysis for Continuous Evolution and Release), a workflow that addresses both problems and is packaged as a portable skill that a coding

Development evidence

Investigate + repair

Post-Repair Maintenance

Persistent security memory

Task + current source + SAST

Decision ledger

Changed symbols + related callers

External repair note

Hypothesis + source evidence

Invariant + rationale Symbol / source identity Saved boundary check

Temporal AST evolution

f

f

Call

Guard

t0

f Call

Guard

t1

Saved check

Call

Current-source assessment

f Guard

f Call

Guard

Call

t2 >_ Check if needed

Qualified repair

Feature proposal

Fail: one retry

Recorded rationale

Repair gate Supported edit Functional validation

Evidence configurations

C0

Snapshot + SAST

C1

C0 + temporal AST

C2

C0 + rationale

C3

C0 + both

Boundary gate Ordinary Feature Saved boundary Pass

</>

Validated repair

Accepted update

Fig. 1. T RACER links symbol-level development evidence to guarded security repair, then preserves repair invariants and executable checks for subsequent feature evolution. C0 to C3 vary the investigation evidence.

agent or a continuous-integration pipeline can load. Given a Git range or an agent trajectory, T RACER tracks changes at symbol level, keeps typed AST evolution and recorded rationale as separate evidence channels, and admits a repair only through a gate that requires a supported assessment of the current source, review of downstream consumers, and public functional validation. We call this guarded investigation. After a repair, T RACER writes a note that binds the invariant to the symbol’s structural hash and to an independently owned boundary check that later changes must pass. We evaluate T RACER in two settings. On SusVibes, 387 agent patches from seven coding generators are reviewed in four evidence configurations by two investigator models, DeepSeek-V3.2 and Gemini-3.8-Flash, for 3,096 repair runs. Symbol-level tracking brings the reference-vulnerable symbol into the review scope for 94.6% of patches, against 13.4% for the primary SAST location. Adding the agent’s recorded rationale gives the highest verified security count with both investigators, 76 and 75 passes from 68 on the 345 patches evaluated in every configuration, with at most one functional regression. The gain is modest and its confidence interval includes zero, so we report it as an ordering that holds across two independent investigators rather than as a large effect. Adding AST history to the rationale configuration lowers the count with both investigators. In a second study, five components of structural-biology software relevant to the in silico drug discovery pipeline of Nostrum Biodiscovery, our industrial partner [24]–[29], are repaired and then followed through ten feature-evolution tasks under a one-retry budget. Functional feedback alone delivers both the feature and the repaired property in 4 of 10 tasks and accepts three contract violations. Repair memory with a boundary gate reaches 9 of 10 and accepts none. The paper makes three contributions. • A provenance-aware workflow and a portable skill. T RACER connects symbol-level change tracking, separate AST-

evolution and rationale channels, guarded static and dynamic investigation, and persistent repair memory, and runs alongside an agent-generated change in an agent host or in CI. • A two-investigator evaluation on 387 agent patches. The SusVibes study compares four evidence configurations across seven coding generators, reporting review coverage, verified security outcomes, and functional preservation together. The structural-biology study isolates repair memory from boundary feedback in a 2×2 factorial under silent-fix feature evolution. • A four-step repair funnel and a failure analysis. Following every attempt from hypothesis to retained patch separates losses in diagnosis, turn scheduling, and semantic repair from operational errors, and yields design lessons for agentic review systems. II. BACKGROUND AND M OTIVATION This section states the review problem that T RACER addresses, illustrates it with a case from the industrial study, and describes the two evidence channels that an agent trajectory makes available. A. Code Archaeology in Agent-Generated Pull Requests When an agent-generated pull request works, the reviewer still has to decide which implementation assumptions deserve scrutiny and which security conditions must survive later maintenance. CodeQL and Semgrep report locations and flows in the final snapshot [8], [9], and connecting those findings to the submitted change, its callers, and the intended contract is left to the reviewer. The industrial study provides a concrete instance. Nostrum Biodiscovery, a spin-off of IRB Barcelona and the Barcelona Supercomputing Center, provides in silico drug discovery and enzyme engineering services built on molecular simulation, and the structure-prediction, design, and docking tools examined in Section VII are relevant to that

pipeline. These tools load model checkpoints, parse structure files, extract archives, and write output paths, and each of these operations is a boundary that a compatibility change can weaken. The following case from that study illustrates both the review problem and the maintenance problem. RFdiffusion’s Sampler.load_checkpoint loaded PyTorch checkpoints with unrestricted pickle deserialization, which under PyTorch 2.5.1 permits code execution from an untrusted checkpoint [30], [31]. The repair sets weights_only=True with a scoped safe_globals allowlist. A later feature request asked for compatibility with legacy argparse.Namespace configurations. With functional feedback only, DeepSeek-V3.2 implemented the compatibility branch with weights_only=False inside the safe_globals context, and all ordinary and feature tests passed. The test suite does not record the reason for the restriction. With the repair note and the saved boundary check, the same model extended the allowlist to argparse.Namespace and retained restricted loading. Section VII reports the full factorial. Preserving review evidence as reusable checks is also what secure-development guidance asks for when it calls for retaining vulnerability knowledge across releases [32]. B. Evidence Available in an Agent Trajectory Two channels of the trajectory carry information that the final snapshot does not. The first is structural. Source lines shift, definitions are added or renamed, and refactoring moves function boundaries, so a line-based git diff does not say which function-level contract changed, and intermediate fragments may not parse. CoLeFunDa’s GumTree-based functionchange descriptions [22] are the model for our compact summaries of symbol edits, which avoid carrying full ASTs across versions [19], [33]. The second channel is the recorded reasoning. ReAct-style trajectories interleave explanations with actions [18], so the task description, the stated assumptions, and the observed edits can be read together. Neither channel supplies the requested behavior or the domain assumptions behind the code, and the explanations are auditable claims rather than guaranteed accounts of the model’s reasoning [20], [21]. A function can run correctly while trusting a caller without justification or implementing only part of the task’s contract. III. TRACER W ORKFLOW AND A RCHITECTURE Figure 1 gives an overview of the workflow. T RACER builds a symbol-level view of the change with scanner evidence attached, adds the two history channels, drives investigation and repair through a decision ledger and a repair gate, and emits the persistent note and boundary check that later maintenance is tested against. A. Symbol-Level Change Tracking and Static Analysis T RACER indexes code at the symbol level with the standardlibrary ast.parse parser [34] instead of reviewing raw diff lines. For each snapshot, the index records functions, methods,

and classes by repository path, qualified name, line span, and structural hash. When a patch is submitted, T RACER maps modified lines to their enclosing symbols, excluding test and documentation files. It runs Semgrep [9] and CodeQL [8] on the snapshot and attaches warnings and flow evidence to the affected symbols. Guard conditions and possible callers in the same file are collected as context, and scanner evidence keeps its tool attribution and its link to the changed function. B. Temporal AST Evolution via GumTree To show how a modified symbol changed during development, T RACER parses before-and-after source pairs into typed Python ASTs and computes structural edit scripts with GumTree 3.0.0 [19], [33]. It restricts the operations to the target symbols and highlights added, modified, or removed guards and calls. Whether the suspected boundary is enforced is always decided on the current code. If a fragment fails to parse or a symbol cannot be linked across versions, T RACER records the gap rather than assuming continuity through a rename or a refactor. C. Symbol-Linked Reasoning Traces Alongside the task description and the code changes, T RACER extracts the coding agent’s original ReAct-style explanations [18], excluding fenced code blocks so that naturallanguage reasoning stays separate from source edits. Excerpts are linked to symbols through recorded file paths, symbol mentions, and matching edit fragments. Because stated rationale does not always match what the code does [20], [21], each explanation is treated as a hypothesis to test. When an explanation asserts an assumption about inputs or callers, the quotation and a verification plan are attached to the target symbol, and the investigator checks whether the current source supports or contradicts it. The quotation stays attached to its assessment so that a later reviewer can inspect both. D. Decision Ledger, Repair Gate, and Post-Repair Maintenance The investigator records each security hypothesis, its target symbol, input boundary, and suspected flaw in a decision ledger saved as JSON. It checks static warnings against the current source and reads whatever historical evidence its configuration provides. When the evidence is inconclusive, T RACER runs targeted Python checks in the analysis-guided testing pattern of PAGENT [17], executing the target on ordinary and boundary inputs to observe whether the suspected flaw can be triggered. The repair gate accepts a candidate only after a supported assessment, a review of the relevant downstream consumers, and public functional validation. After a repair, T RACER saves a persistent note that binds the security invariant to the repaired symbol’s hash and to an independently owned executable boundary check. Later feature proposals are checked against the ordinary tests, the new feature tests, and the saved boundary check. On failure, the maintenance gate returns the failing check as feedback for one bounded retry, and otherwise retains the secure starting

code. The portable tracer-security skill1 packages this workflow for host coding agents. The host supplies editing and test interfaces, and CI can run the saved checks on future commits without the benchmark harness.

TABLE I C OHORT INVENTORY, LOCALLY VERIFIED ORIGINAL OUTCOMES , AND REFERENCE - SYMBOL REVIEW COVERAGE . V ERIF. COUNTS COMPLETED ORIGINAL EVALUATIONS . F UNC AND S EC REFER TO THAT SUBSET. C OVERAGE USES ALL 387 ADMITTED PATCHES .

IV. E XPERIMENTAL S ETUP

Coding generator

This section states the research questions, describes the SusVibes cohort and the two investigator models, defines the four evidence configurations, and fixes the evaluation protocol.

Claude 4 Sonnet Gemini 2.5 Pro Gemini 3 Pro GLM-4.7-Flash GLM-5 Kimi K2 Qwen3-Coder-Next

186 186 186 186 186 186 186

88 29 58 38 100 33 41

83 24 55 30 98 28 36

81 24 54 30 98 27 36

15 6 16 5 17 4 6

Total

1302

387

354

350

69

A. Research Questions Five research questions follow an agent-generated change from localization to later maintenance. RQ1 (Localization Coverage). How often does symbol-level change tracking bring the vulnerable symbol into the review scope, compared with stock SAST locations? RQ2 (End-to-End Verified Repair). Does guarded investigation improve verified security outcomes without degrading functionality? RQ3 (Provenance Channels). How do temporal AST history, recorded rationale, and their combination affect verified repair? RQ4 (Repair Funnel Bottlenecks). Where are candidates lost between symbol localization and verified repair? RQ5 (Persistent Memory Under Silent-Fix Evolution). Can repair memory and boundary gates preserve a repaired property while a subsequent feature change succeeds? B. The SusVibes Benchmark and Study Dataset We use the 186-task SusVibes v1.0 release. The original paper reports statistics over 200 tasks [6], [7], with repositories averaging 162K lines across 867 files, 108 projects, 77 CWE categories, and on average 68.8 functional tests and 3.7 security tests per task, so agents work against existing interfaces and callers rather than isolated functions. Table I lists 1,302 archived patches from seven coding generators across the 186 Python tasks, taken from the public SWE-agent trajectories and patches on the SusVibes leaderboard [35]. Of these, 472 carry a publisher-reported functional success. Source screening and local environment reproduction admit 387 patches across 118 tasks. Original evaluations complete for 354 of them, of which 350 pass the functional tests and 69 pass the security tests. Admission follows the publisher’s functional label, and the table reports our locally verified outcomes. C. Evaluated Configurations and Investigator Models The two security investigators are DeepSeek-V3.2, accessed through OpenRouter’s SiliconFlow FP8 endpoint, and Gemini3.8-Flash, accessed through the native Gemini API. Both run at temperature zero with a 22-turn cap and a 100,000byte message budget, and Gemini uses LOW thinking. Every configuration receives the functional request, the modified symbols, the scanner findings, and the same repair protocol. 1 https://github.com/pkrajput/TRACER_skill

Source Admitted Verified Func

Sec

Reference-symbol review coverage among 387 admitted patches SAST reference file 228 58.9% SAST primary reference symbol 52 13.4% SAST symbol including flow endpoints 63 16.3% Change-informed plan reference symbol 366 94.6%

C0 (Snapshot + SAST). Current source of the modified symbols and static warnings, without historical AST edits or recorded reasoning. • C1 (C0 + Temporal AST). Adds the structural edits showing how those symbols evolved. • C2 (C0 + Recorded Rationale). Adds the coding agent’s original development explanations. • C3 (C0 + Both). Combines temporal AST edits and recorded explanations. The seven models in Table I are the coding generators that wrote the original patches, and the two investigators review every generator’s outputs. C1, C2, and C3 also receive audit cues that C0 does not. Comparisons against C0 therefore evaluate complete evidence bundles, while comparisons among C1, C2, and C3 isolate the contribution of each history channel. All configurations use Semgrep CE 1.101.0 with the Python rules at commit 495df89c, including correctness and maintainability rules, and CodeQL 2.20.0 with Python queries 1.3.4 and the security-extended suite. Investigation inputs scope warnings to changed files and to at most twelve symbol groups, and coverage audits use the full scanner output. SAST-seeded patches receive at least one scoped warning, and unseeded patches receive none. The benchmark hides the repository’s prior Git history, so the history available to the investigators is the coding agent’s recorded trajectory after the reference commit. •

D. Evaluation Protocol We run SusVibes’s official functional and security tests inside isolated containers. FuncPass and SecPass require recognized terminal test summaries, including the benchmark’s expected-failure allowances. Hidden security tests and reference patches serve only for retrospective evaluation and are never given to an investigator as feedback. Investigators receive public functional feedback.

Each admitted patch is reviewed in four configurations by two investigators, for 3,096 repair runs. Table II compares the 345 patches whose original and candidate evaluations completed in all eight configurations, and DeepSeek additionally supports a comparison over 354 patches. An investigator error can leave a retained patch or a no-op that is evaluated later, in the same restored environment, with results reused only for byte-identical patches. We report conditional repair outcomes rather than benchmark-wide generator scores. Taskcluster intervals use 5,000 bootstrap replicates with seed 12, to account for repeated generator attempts on the same task [36]. A security conversion is an originally failing security outcome that becomes a verified pass after repair. Preservation of an original pass is recorded separately. An incomplete evaluation stays unresolved and is not counted as a failed security assertion. V. B ENCHMARK R EPAIR R ESULTS (RQ1–RQ3) This section answers the three benchmark questions on coverage, end-to-end verified repair, and the contribution of each provenance channel. A. RQ1. Symbol Localization Coverage Across the 387 admitted patches, CodeQL and Semgrep warnings fall in a reference security file in 228 cases (58.9%). The primary warning location falls on a reference symbol in 52 cases (13.4%), and 63 cases (16.3%) when flow endpoints are included. These figures measure how often the scanner directs attention to the relevant symbol and should not be read as vulnerability detection recall. Tracking modified symbols and their relations places a reference symbol in the review scope for 366 patches (94.6%), with the complete function body in 329. Among the 335 patches with no primary warning on the reference symbol, change tracking still supplies the full body in 282, and the task request names the reference target in 307 of them. A reference-symbol match determines which code to examine. It does not identify the missing condition. Answer to RQ1. T RACER places the reference symbol in the review scope for 94.6% of patches, against 13.4% for the primary SAST location.

and C3. The larger DeepSeek comparison over 354 patches keeps the same security ranking, with C2 reaching 77 passes from 69 originals. The two outcomes are reported side by side because an unchanged candidate preserves functionality without repairing the flaw. Answer to RQ2. C2 raises verified security passes from 68 to 76 with DeepSeek and to 75 with Gemini, with zero functional regressions for DeepSeek and one for Gemini.

C. RQ3. Impact of Provenance Channels Recorded rationale yields the highest aggregate SecPass count with both investigators. DeepSeek under C2 makes eight conversions across six distinct tasks, and Gemini under C2 makes seven across four tasks. These counts refer to repaired generator outputs, and several conversions concern the same task implemented by different generators. Combining both channels yields fewer verified repairs than rationale alone. DeepSeek falls from 76 under C2 to 74 under C3, and Gemini from 75 to 73. Context dilution is one possible explanation, but the comparison does not establish the mechanism. Rankings at the generator level are less uniform (Table II), and for C2 against C0 each investigator has six wins and one loss across the seven generators. The task-cluster 95% intervals for the C2 gain over C0 are 0.00 to 2.96 percentage points for DeepSeek and −0.32 to 3.86 for Gemini. The effect is small and both intervals include zero. The data support a consistent ordering across two investigators and do not support a claim of large improvement. Table II also splits the cohort into 235 development-exposed patches and 110 previously protected patches. On the previously protected subset, C2 adds no security passes over C0 for DeepSeek and two for Gemini. Answer to RQ3. Recorded rationale ranks highest with both investigators. Adding temporal AST history on top of it lowers the verified security count for both.

VI. R EPAIR F UNNEL AND FAILURE A NALYSIS (RQ4) This section follows every repair attempt through four nested steps, from the first hypothesis on a reference symbol to a retained candidate, and examines the three points at which most candidates are lost. A. The Four-Step Repair Funnel

B. RQ2. End-to-End Verified Security Repair Of the 345 shared patches, 342 initially pass the functional tests and 68 pass the security tests. With DeepSeek as investigator, C0 raises the security count to 71, C1 to 73, and C2 to 76 (22.03%). Gemini shows the same ordering, with 70 under C0, 74 under C1, and 75 under C2 (21.74%). C2 adds five net security passes over C0 with each investigator. The comparison is paired on the same original patches, so the difference is attributable to the investigation and repair rather than to a different task mix. Functional regressions are rare. DeepSeek retains all 342 functional passes under C1, C2, and C3, and loses one under C0. Gemini retains 341 under C0 and C2, and 340 under C1

High review coverage does not guarantee a correct diagnosis or a correct repair. Table III reports, for each investigator and configuration, how many of the 387 admitted patches reach each of four nested steps. 1) Hypothesis on Target Symbol. The investigator records a hypothesis on a symbol touched by the reference security patch (237 to 252 cases per configuration for DeepSeek, 213 to 235 for Gemini). 2) Supported Assessment. The investigator records an accepted supported assessment on a reference symbol (85 to 109 for DeepSeek, 15 to 32 for Gemini). This step records the investigator’s own assessment and does not independently confirm the flaw.

TABLE II V ERIFIED REPAIR OUTCOMES . PANEL A REPORTS F UNC PASS AND S EC PASS COUNTS . PANEL B REPORTS S EC PASS FOR D EEP S EEK AND G EMINI ON THE SAME 345 PATCHES . B OLD MARKS EACH INVESTIGATOR ’ S MAXIMUM ACROSS C0 TO C3, INCLUDING TIES . N

Original

C0

C1

C2

C3

A. End-to-end outcomes. FuncPass / SecPass counts DeepSeek-V3.2 (common cohort) 345 Gemini-3.8-Flash (common cohort) 345 DeepSeek-V3.2 (full completed cohort) 354

342 / 68 342 / 68 350 / 69

341 / 71 341 / 70 349 / 72

342 / 73 340 / 74 350 / 74

342 / 76 341 / 75 350 / 77

342 / 74 340 / 73 350 / 75

B. Common-cohort SecPass by coding generator and stratum. DeepSeek / Gemini Claude 4 Sonnet 83 15 16 / 16 Gemini 2.5 Pro 24 6 6/6 Gemini 3 Pro 55 16 16 / 16 GLM-4.7-Flash 30 5 5/6 GLM-5 93 16 16 / 16 Kimi K2 26 4 6/4 Qwen3-Coder-Next 34 6 6/6 SAST-seeded (Scoped warning) 176 36 37 / 38 Unseeded (No scoped warning) 169 32 34 / 32 Development exposed 235 44 45 / 45 Previously protected 110 24 26 / 25

16 / 16 6/6 17 / 16 6/6 16 / 16 6/8 6/6 37 / 40 36 / 34 48 / 47 25 / 27

18 / 18 6/6 17 / 17 5/5 18 / 17 6/6 6/6 40 / 40 36 / 35 50 / 48 26 / 27

16 / 16 6/6 17 / 18 5/6 17 / 16 6/5 7/6 38 / 40 36 / 33 50 / 48 24 / 25

Population or investigator

TABLE III F OUR - STEP INVESTIGATION AND REPAIR FUNNEL ACROSS 387 ADMITTED PATCHES PER CONFIGURATION . S EC PASS AND NEW FIXES COUNT ONLY CANDIDATES WITHIN S TEP 4. B OLD HIGHLIGHTS SUPPORTED ASSESSMENTS AND C2’ S NEW FIXES .

Config.

Step 1: Hypothesis on Target Symbol

Step 2: Supported Assessment

Step 3: Reference-File Edit

Step 4: Nonempty Retained Patch

Verified SecPass (New fixes)

DeepSeek-V3.2 DeepSeek-V3.2 DeepSeek-V3.2 DeepSeek-V3.2

C0 C1 C2 C3

245 251 237 252

109 109 85 97

89 98 74 89

58 65 54 60

13 (4 new) 13 (5 new) 13 (8 new) 10 (6 new)

Gemini-3.8-Flash Gemini-3.8-Flash Gemini-3.8-Flash Gemini-3.8-Flash

C0 C1 C2 C3

213 225 223 235

15 32 29 23

14 28 27 21

11 20 18 17

3 (2 new) 8 (6 new) 8 (7 new) 5 (4 new)

Investigator

3) Reference-File Edit. The case contains an edit in a reference file, which need not address the same symbol or hypothesis (74 to 98 for DeepSeek, 14 to 28 for Gemini). 4) Nonempty Retained Patch. The case ends with a nonempty candidate patch (54 to 65 for DeepSeek, 11 to 20 for Gemini). Retention is a protocol outcome and carries no functional or security verdict. The final column reports official security outcomes for this subset. Each step includes only the cases that reached the preceding steps, so the table shows at which step a candidate leaves the workflow. It answers a different question from Table II, which reports the final outcome of every evaluated patch, including those left unchanged. B. Drop-off 1. False Refutations During Diagnosis The largest loss occurs between forming a hypothesis and recording support for it. The ledgers show premature refutations, hypotheses on the wrong property, inconclusive assessments, and turn exhaustion. In several cases the investigator inspects a partial check or a helper call and concludes that the boundary is already protected. In one attribute-access inves-

tigation, for example, the existing restriction on underscoreprefixed names is judged sufficient, although inspection attributes remain unrestricted. In other cases the investigator finds the correct function and then examines an unrelated property, for instance dictionary copying while the actual contract concerns expiry. Because the funnel also includes outputs that were secure to begin with, an accepted refutation is not necessarily a false all-clear. These examples show that locating the relevant code and understanding its security obligation are distinct steps. A guard can be present and still enforce a property other than the one required at the boundary. C. Drop-off 2. Turn Exhaustion Before Validation A late edit can exhaust the turn budget before public validation completes, and validation slots and required probe reruns can also prevent retention. A separate recovery check, run during development, evaluates seventy saved candidates that were discarded for protocol completion reasons. Of these, 68 reached a turn or validation-slot limit and two lacked a probe rerun. Offline tests find 39 functional passes, eighteen failures, and thirteen unresolved executions. Nine candidates

TABLE IV M ANUAL INSPECTION OF FIFTEEN RETAINED PATCHES WITH REFERENCE - SYMBOL EDITS , VERIFIED FUNCTIONAL PASSES , AND FAILED SECURITY TESTS . C ATEGORIES DESCRIBE THIS SELECTED DIAGNOSTIC SAMPLE . Failure category

Count

Right bug class, incomplete fix

9

Wrong security property

4

Right condition, wrong failure behavior

2

and one connection error. The recorded labels distinguish provider responses, parsing failures, and run interruptions, and we do not attribute causes beyond what the labels state. Final candidate evaluations are reported separately from these investigation errors.

Repair attempt and remaining defect A plausible fix leaves a bypass or incomplete boundary. Path normalization misses symlink escape. Size checks follow decompression. Connection validation skips inherited settings. Request checks miss framing variants. URL checks leave a backslash bypass. Session checks concern keys rather than expiry. Request checks limit body size rather than media-type handling. Other candidates leave route anchoring or username normalization unresolved. An empty axis raises IndexError instead of ValueError. Malformed-byte decoding still raises instead of following the required failure behavior.

pass the security tests, eight of them preserved successes and one a new conversion. These recovery results are kept separate from the campaign scores. They indicate that a validation opportunity reserved after the final edit would retain usable candidates, without establishing that every loss was due to scheduling. D. Drop-off 3. Incomplete Fixes and Contract Mismatches A retained patch that passes the functional tests can still leave the security contract unsatisfied. We inspected fifteen such cases, outside the diagnostic pilot, by comparing the candidate diff, the reference patch, and the official failing assertions. Fourteen come from DeepSeek and one from Gemini. Table IV classifies nine of the fifteen as incomplete repairs of the correct vulnerability class, such as lexical path normalization that does not account for symbolic links, a size check applied after decompression, and validation that omits inherited connection settings. Four patches address a security property other than the one tested, such as session-key handling instead of expiry, or body-size limits instead of mediatype parsing. Two patches enforce the right condition with the wrong failure behavior. In all three categories, verifying the repair requires checking the full input and error contract rather than the presence of a new guard. A size check applied after decompression cannot prevent resource consumption that has already occurred, and the repair must enforce the intended boundary at the point where the unsafe operation can still be prevented. E. Operational and Provider Failures Gemini records 205 repair errors, comprising 153 blockReason: OTHER responses, 45 JSON decoding errors, six HTTP 500 errors, and one other value error. DeepSeek records 107 errors, comprising 63 explicit stop interruptions, 32 HTTP 429 responses, nine deadlines, two value errors,

Answer to RQ4. Candidates are lost during diagnosis, during validation scheduling, and in semantic repair. The funnel separates these losses from operational errors and from the final test outcome.

VII. I NDUSTRIAL C ASE S TUDY. S TRUCTURAL B IOLOGY P IPELINES (RQ5) This section evaluates the maintenance stage of T RACER on software outside the SusVibes benchmark. Following the silent-fix setting of CoLeFunDa [22], we repair five components of structural-biology software selected with our industrial co-author, then submit ten feature requests against the repaired code and measure, in a 2 × 2 factorial, whether repair memory and boundary feedback allow the feature to be delivered without undoing the repair. A. Repair Contracts and Independent Scope With our co-author at Nostrum Biodiscovery, we selected six open-source tools relevant to the company’s pipeline. RFdiffusion [24], ProteinMPNN [25], haddock3 [26], DockQ [27], AutoDock-Vina [28], and foldseek [29] cover protein structure generation, sequence design, docking, docking quality assessment, and structure search. Semgrep 1.101.0 reports 420 warnings across the six repositories. Review of source and history identifies five Python repair targets, none of which coincides with a warning location. Foldseek contributes eleven warnings but no suitable project-owned Python case and is therefore not part of the maintenance study. We verified the following five component contracts. 1. DockQ (load_PDB). A parser failure can leave a file handle open before a fallback parser opens another. The repair closes ordinary and compressed handles on both success and failure. 2. haddock3 (extract_files_flat). A flattened archive name can follow a pre-existing destination symlink to a location outside the extraction directory. The repair resolves paths, checks containment, and excludes non-regular archive members. 3. ProteinMPNN. StructureDataset.__init__ accepts unvalidated JSONL record names that later form part of output paths. The repair accepts only nonempty simple filenames and preserves ordinary identifiers and filtering. 4. RFdiffusion (Sampler.load_checkpoint). Unrestricted pickle loading under PyTorch 2.5.1 permits code execution from an untrusted checkpoint [30], [31]. The repair sets weights_only=True with a scoped safe_globals([slice]) allowlist for the accepted checkpoint forms. 5. AutoDock-Vina (Vina.write_poses). Numeric checks are skipped on entry to an elif not

TABLE V I NDEPENDENT BIOLOGY MAINTENANCE : TWO TASKS PER REPOSITORY. ROWS REPORT FINAL F UNC PASS /S EC PASS /J OINT PASS AFTER AT MOST ONE RETRY. F UNC PASS COMBINES ORDINARY AND FEATURE CHECKS . S EC PASS COMBINES SAVED AND EXTENDED CHECKS , INCLUDING D OCK Q AND AUTO D OCK -V INA RELIABILITY CONTRACTS . AGGREGATE COUNTS ARE OUT OF TEN . B OLD HIGHLIGHTS B3 (B OTH ). B0 Functional

B1 Memory

B2 Boundary

B3 Both

DockQ Lazy model iteration, parsed-model cache haddock3 Internal TAR aliases, selected nested extraction ProteinMPNN Hierarchical names and metadata, name mapping RFdiffusion Legacy config retry, checkpoint bytes cache AutoDock-Vina Dry-run preview, pose text in memory

1/1/1 1/2/1 2/0/0 1/1/0 2/2/2

2/2/2 1/2/1 2/1/1 2/2/2 2/2/2

1/2/1 0/2/0 2/0/0 1/1/0 2/2/2

2/2/2 2/2/2 2/1/1 2/2/2 2/2/2

First proposal, total (out of 10) Final proposal, total (out of 10) Final boundary assertion violations / unknowns Accepted proposals with observed boundary violation Fallbacks (not feature successes)

5/6/4 7/6/4 3/1 3 3

6/6/5 9/9/8 1/0 1 1

4/7/3 6/7/3 3/0 0 7

6/6/5 10/9/9 1/0 0 1

Repository

Two feature tasks

overwrite branch. The repair validates pose counts and energy ranges irrespective of the overwrite setting. The DockQ and AutoDock-Vina cases are reliability contracts and not exploitable vulnerabilities. Four of the five repairs were written by the model. For RFdiffusion, two model attempts failed and the remedy was specified by the developer. All five boundary assertions fail before repair and pass after it, and ordinary-input controls pass in both states. An external note for each repair records the invariant, its rationale, the source identity, and the check. Before the maintenance tasks, we strip comments and docstrings with tokenize and ast, keep the executable AST unchanged, and re-verify the cleaned starting code [34], [37], so that the model receives no textual hint of the repair. B. Equal-Budget Feature Evolution The ten tasks, two per repository, extend checkpoint compatibility, archive aliases, hierarchical names, in-memory pose output, and lazy parsing (Table V). They form a purposive stress sample, independent of SusVibes, across five repository clusters. The checks cover both the default mode and the newly requested mode, and each repaired starting point passes its existing checks. For every task, an evaluator-written reference implementation passes all checks and a deliberately weakened reference delivers the feature but violates a boundary. Both references are withheld from the model and serve only to validate the checks. DeepSeek-V3.2 at temperature zero receives the cleaned source and the feature request. The factorial varies repair memory and boundary feedback. Memory supplies the repair note together with a prospective clarification of the contract. Boundary feedback adds the saved and extended checks to the feedback the model receives. All four conditions receive functional feedback, the same token budget, and at most one retry. In B0 and B1, the boundary checks are run after generation and do not influence the retry. All 40 cells are retained, checks run in pinned offline containers, and the archive preserves the protocol and the correction history. FuncPass requires the ordinary and feature checks to pass,

SecPass requires the saved and extended boundary checks to pass, and JointPass requires both. Table V reports first and final proposals separately. An execution error is recorded as an unknown boundary outcome rather than as a violation. A fallback retains the repaired starting code and is counted as a failure to deliver the feature. Functional feedback alone (B0). B0 reaches 4/10 joint successes and accepts three proposals that violate a contract. The RFdiffusion proposal re-enables unrestricted checkpoint loading with weights_only=False, as described in Section II. Two ProteinMPNN proposals admit backslashes that the portable naming contract forbids, although neither demonstrates path traversal on Linux. The functional checks accept all three because the requested feature works. Boundary gate alone (B2). B2 accepts no observed violation but reaches only 3/10 joint successes, with seven fallbacks. Without the repair note, the model receives the failing check but has no record of why the boundary exists. The gate protects the boundary, and that protection is not sufficient to produce a proposal that satisfies both obligations. Memory alone (B1). With the note, the model implements the feature within the repaired contract in 8/10 tasks. One accepted proposal still relaxes ProteinMPNN’s default simplename restriction. Functional checks cannot reject this naming violation, even though the note states the intended contract. Memory and boundary feedback (B3). With both, 9/10 tasks succeed jointly and no contract violation is accepted. The gate rejects the remaining naming violation and retains the repaired starting code. The note tells the model how to preserve the boundary, and the executable check determines whether the proposal does so. The five joint successes that B3 adds over B0 consist of one restricted-loading case, one portable-naming case, and three feature or execution fixes. B3 leads in this sample, and five selected repository clusters do not establish a population effect.

Answer to RQ5. Under the same one-retry limit, joint success is 4/10 with functional feedback alone, 8/10 with memory, 3/10 with a boundary gate, and 9/10 with both. B3 rejects its one remaining contract-violating proposal. A fallback is counted as a failure to deliver the feature.

VIII. I NDUSTRIAL R ELEVANCE AND P RACTICAL TAKEAWAYS This section discusses the implications of the two studies for teams that integrate coding agents into their development process and summarizes the lessons we consider transferable to other pipelines. A. Relevance to Industrial Software Engineering In most industrial settings a change is accepted when the functional test suite passes, and our results indicate that this criterion is not sufficient for security. In the SusVibes cohort, 350 of the 354 patches with completed evaluations pass their functional tests, whereas 69 pass the associated security tests. In the structural-biology software, the five repaired components all read data from outside the program, and in each case the defect was located in code that passed the project’s existing tests. In both settings, the missing condition could be traced to an assumption about an upstream caller, a file path, or a serialization format that was never made explicit in the submitted code. Static analysis remains the first line of defense, and we do not propose to replace it. We observe, however, that in our cohort the primary warning location of CodeQL and Semgrep coincides with the symbol modified by the reference security fix in 13.4% of the admitted patches. T RACER is intended to complement static analysis at two points of a CI/CD workflow. During review, it uses the development history and the recorded reasoning of the coding agent to direct attention to the symbols that the pull request modified and to the assumptions that the agent stated about them. After a security fix has been accepted, it records the fix as a JSON repair note together with an executable boundary check, so that subsequent feature updates to the same component are tested against the earlier security work. B. Actionable Takeaways for Practitioner Pipelines 1) More historical context is not always better. In our benchmark study, the configuration that adds the agent’s recorded rationale to the current source and scanner findings (C2) produced more verified security passes than the configuration that adds both the rationale and the temporal AST edits (C3), with 76 against 74 passes for DeepSeek and 75 against 73 for Gemini. Our experiment does not identify the cause of this difference. Context dilution is a plausible explanation, since the investigator receives a larger and less focused prompt in C3, but it is not established by our data. The rationale provides a specific claim about the code, for example that an upstream caller already sanitizes its input, which the investigator can confirm or refute against the current implementation. For teams that log agent trajectories, this suggests storing reasoning excerpts linked to the symbols they

modified and retrieving at review time only the excerpts that concern the symbol under investigation. 2) Combine change tracking with static analysis. Symbollevel change tracking placed the symbol modified by the reference security fix in the review scope for 94.6% of the admitted patches, compared with 13.4% for the primary scanner location, and among the 335 patches for which the scanner did not point at that symbol it supplied the complete body in 282 cases. We therefore recommend that a review workflow for agent-generated changes define its scope from the modified symbols and their callers, and then use scanner warnings and recorded assumptions as evidence within that scope, so that a modified function without a warning can also be investigated. Coverage of the relevant symbol is a precondition for diagnosis and does not by itself indicate whether a condition is missing. 3) Reserve validation capacity after the final edit. During protocol development we evaluated seventy candidate patches that the campaign had discarded, 68 of them because the turn budget or the validation slot had been exhausted (Section VI). Of these candidates, 39 passed the functional tests and nine passed the security tests, including one new conversion. These candidates were lost to scheduling and not to the quality of the repair. We conclude that a validation step should be reserved after the last edit, independently of the budget consumed by exploratory probing. Recording candidate generation, retention, and completed evaluation as separate events also allowed us to recover from the 205 and 107 provider or protocol errors of the two investigators without repeating model requests. 4) Separate repair guidance from acceptance checks. The factorial design of the maintenance study allows the two interventions to be assessed separately. Repair memory alone led to joint success in 8 of 10 tasks but accepted one proposal that relaxed the naming contract repaired earlier. A boundary gate alone accepted no violating proposal but led to joint success in only 3 of 10 tasks. The combination led to joint success in 9 of 10 tasks with no accepted violation, whereas functional feedback alone led to 4 of 10 and accepted three violating proposals. The note informs the model of the invariant and the reason for it, which allows the feature to be implemented within the contract, and the check determines whether a given proposal respects it. We therefore recommend that teams store both with each accepted security fix. The check should cover the modes introduced by the new feature as well as the default mode, since in the ProteinMPNN tasks the proposals that violated the naming contract did so only on the new naming path. When a feature request widens the input domain of a repaired component, the contract should be revised explicitly, and a fallback that retains the repaired code should be reported as a feature that was not delivered rather than as a successful outcome. IX. T HREATS TO VALIDITY Construct validity. Overlap with the reference patch measures the opportunity to review the relevant code, and a broad symbol or an incidental edit can overstate the precision of a diagnosis. The SusVibes tests verify the specified contract and

do not certify the absence of other vulnerabilities [6], [7], [38], [39]. The recorded rationale is a stated development trace and is not guaranteed to reflect the model’s internal reasoning [20], [21]. Internal validity. The protocol was developed through repeated inspection of failures, so that individual engineering decisions were not evaluated as isolated interventions. The fifteen semantic reviews form a purposive sample inspected by a single reviewer and do not support a population estimate. The maintenance factorial holds the retry ceiling fixed but uses a single investigator and a single continuation per cell, and the memory condition includes prospective clarifications of the contract. The feature tasks were designed with knowledge of the repairs and constitute a stress sample. External validity. Selection favors Python outputs with publisher-reported functional success. The 42 admitted patches without complete shared outcomes may differ from the rest in environment difficulty or provider reliability. Repeated tasks across generators and the small number of security conversions limit precision. The scientific-software study covers ten tasks clustered within five selected components and was run by the researchers with input from the industrial co-author. Portable naming restrictions and reliability contracts do not by themselves constitute exploitable vulnerabilities. Our results do not establish the effectiveness of the approach for other languages or in production deployments. X. R ELATED W ORK Our work relates to four lines of research, reviewed in turn below. A. Benchmarks and Security of Agent-Generated Code SWE-bench evaluates the resolution of repository issues [40], and SusVibes separates functional from security outcomes for agent-generated changes [6], [7]. CVEfixes links historical vulnerabilities to their fixing commits, and Vul4J provides reproducible vulnerabilities with triggering tests [41], [42]. SWE-agent defines the interface through which an agent interacts with a repository [1], and ReAct interleaves explanations with actions [18]. We use the record these interfaces preserve as investigation evidence and evaluate existing generated patches against the security oracle of the benchmark, rather than generating new patches. B. Static Analysis and LLM-Assisted Vulnerability Detection Beyond CodeQL and Semgrep [8], [9], IRIS infers taint specifications with an LLM [13], LLMDFA combines model reasoning with dataflow checks [14], RepoAudit explores repository dataflows with memory and validation [15], PAGENT guides the generation of vulnerability-triggering inputs with program analysis [17], and KNighter synthesizes checkers from historical patches [43]. None of these approaches uses the development trajectory and recorded rationale of the coding agent that produced the code as evidence. T RACER compares these evidence channels within a single guarded repair protocol, and its conditional runtime checks follow PAGENT in

grounding a suspected flaw in observed program behavior before a repair is attempted. C. Code Evolution, Silent Fixes, and Automated Program Repair GumTree computes structural edit scripts between versions of a source file [19], [33], ViolationTracker tracks static-analysis warnings across revisions [44], CoLeFunDa represents function-level changes in order to identify silent vulnerability fixes [22], and FixMiner mines recurring repair patterns [45]. On the repair side, InferFix uses analyzer guidance [16], ChatRepair uses conversational test feedback [46], and RepairAgent inspects, edits, and tests code autonomously [2]. T RACER adopts the function-level change representation of CoLeFunDa and adds an analysis of where candidate repairs are lost between diagnosis, retention, and verified repair. D. Persistent Memory and Continuous Software Evolution Reflexion stores verbal feedback in an episodic memory [47], and Self-Refine iteratively revises its own outputs [48]. POSTURA maintains a graph of the security posture of a system [49], and Undoing the Fix studies deliberately elicited security regressions in an AI-control setting [23]. T RACER pairs each repair note with an independently owned executable check and evaluates benign feature evolution, rather than adversarial regression, against the saved contract. XI. C ONCLUSION We have presented T RACER, which uses the development history of an agent-generated change as evidence for security investigation and preserves the result of that investigation as a contract for subsequent maintenance. On SusVibes, symbollevel tracking placed the reference vulnerable symbol in the review scope for 94.6% of patches, compared with 13.4% for the primary SAST location, and the configuration that includes the recorded rationale produced the most verified security passes with both investigators, with at most one functional regression. In the structural-biology maintenance study, repair memory combined with boundary feedback delivered the requested feature while preserving the repair in 9 of 10 tasks, against 4 of 10 for functional feedback alone, and rejected the only remaining proposal that violated a contract. DATA AVAILABILITY The skill, analysis helpers, derived outcome records, supporting tables, and the RQ5 prompts, responses, candidates, and checks are archived on Zenodo [50]. Benchmark inputs originate from SusVibes [7]. Raw transcripts, evaluator logs, and container exports are excluded. ACKNOWLEDGEMENTS We thank Google for Gemini API credits and Nostrum Biodiscovery for verification assistance and feedback.

R EFERENCES [1] J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “SWE-agent: Agent-computer interfaces enable automated software engineering,” in Advances in Neural Information Processing Systems, vol. 37, 2024. [2] I. Bouzenia, P. Devanbu, and M. Pradel, “RepairAgent: An Autonomous, LLM-Based Agent for Program Repair,” in 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), 2025, pp. 2188–2200. [3] H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, “Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions,” in 2022 IEEE Symposium on Security and Privacy (SP), 2022, pp. 754–768. [4] N. Perry, M. Srivastava, D. Kumar, and D. Boneh, “Do Users Write More Insecure Code with AI Assistants?” in Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, 2023, pp. 2785–2799. [5] Y. Fu, P. Liang, A. Tahir, Z. Li, M. Shahin, J. Yu, and J. Chen, “Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Study,” ACM Transactions on Software Engineering and Methodology, vol. 34, no. 8, pp. 218:1–218:34, 2025. [6] S. Zhao, D. Wang, K. Zhang, J. Luo, Z. Li, and L. Li, “Is vibe coding safe? benchmarking vulnerability of agent-generated code in real-world tasks,” in Forty-third International Conference on Machine Learning, 2026. [Online]. Available: https://openreview.net/forum?id=qG8g00zR Za [7] LeiLiLab, “SusVibes: Benchmark and evaluation pipeline,” Software repository, 2026. [Online]. Available: https://github.com/LeiLiLab/susv ibes/tree/7e1b4b05240e56dc2b4e8253b2cc5c9481d016f3 [8] GitHub, “CodeQL CLI,” GitHub documentation. [Online]. Available: https://docs.github.com/en/code-security/concepts/code-scanning/codeql /codeql-cli [9] Semgrep, “Semgrep code overview,” Semgrep documentation. [Online]. Available: https://docs.semgrep.dev/semgrep-code/overview [10] B. Johnson, Y. Song, E. Murphy-Hill, and R. Bowdidge, “Why don’t software developers use static analysis tools to find bugs?” in 2013 35th International Conference on Software Engineering (ICSE), 2013, pp. 672–681. [11] C. Sadowski, E. Aftandilian, A. Eagle, L. Miller-Cushon, and C. Jaspan, “Lessons from building static analysis tools at Google,” Communications of the ACM, vol. 61, no. 4, pp. 58–66, 2018. [12] C. Calcagno, D. Distefano, J. Dubreil, D. Gabi, P. Hooimeijer, M. Luca, P. O’Hearn, I. Papakonstantinou, J. Purbrick, and D. Rodriguez, “Moving Fast with Software Verification,” in NASA Formal Methods, 2015, pp. 3–11. [13] Z. Li, S. Dutta, and M. Naik, “IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities,” in International Conference on Learning Representations (ICLR), 2025. [Online]. Available: https://openreview.net/forum?id=9LdJDU7E91 [14] C. Wang, W. Zhang, Z. Su, X. Xu, X. Xie, and X. Zhang, “LLMDFA: Analyzing Dataflow in Code with Large Language Models,” in Advances in Neural Information Processing Systems, vol. 37, 2024, pp. 131 545–131 574. [Online]. Available: https: //proceedings.neurips.cc/paper_files/paper/2024/hash/ed9dcde1eb9c597 f68c1d375bbecf3fc-Abstract-Conference.html [15] J. Guo, C. Wang, X. Xu, Z. Su, and X. Zhang, “RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditing,” in Proceedings of the 42nd International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 267, 2025, pp. 21 083–21 100. [Online]. Available: https://proceedings.mlr. press/v267/guo25n.html [16] M. Jin, S. Shahriar, M. Tufano, X. Shi, S. Lu, N. Sundaresan, and A. Svyatkovskiy, “InferFix: End-to-End Program Repair with LLMs,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2023, pp. 1646–1656. [17] A. Desai, M. Shafiuzzaman, W. Guo, and T. Bultan, “Program analysis guided LLM agent for proof-of-concept generation,” 2026. [Online]. Available: https://arxiv.org/abs/2604.07624 [18] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, “ReAct: Synergizing reasoning and acting in language models,” in International Conference on Learning Representations (ICLR), 2023. [Online]. Available: https://openreview.net/forum?id=WE_vluYUL-X

[19] J.-R. Falleri, F. Morandat, X. Blanc, M. Martinez, and M. Monperrus, “Fine-grained and accurate source code differencing,” in Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering (ASE), 2014, pp. 313–324. [20] M. Turpin, J. Michael, E. Perez, and S. R. Bowman, “Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting,” in Advances in Neural Information Processing Systems, vol. 36, 2023. [Online]. Available: https: //proceedings.neurips.cc/paper_files/paper/2023/hash/ed3fea9033a80fe a1376299fa7863f4a-Abstract-Conference.html [21] T. Lanham, A. Chen, A. Radhakrishnan, B. Steiner, C. Denison, D. Hernandez, D. Li, E. Durmus, E. Hubinger, J. Kernion, K. Lukošiūtė, K. Nguyen, N. Cheng, N. Joseph, N. Schiefer, O. Rausch, R. Larson, S. McCandlish, S. Kundu, S. Kadavath, S. Yang, T. Henighan, T. Maxwell, T. Telleen-Lawton, T. Hume, Z. Hatfield-Dodds, J. Kaplan, J. Brauner, S. R. Bowman, and E. Perez, “Measuring Faithfulness in Chain-of-Thought Reasoning,” 2023. [Online]. Available: https://arxiv.org/abs/2307.13702 [22] J. Zhou, M. Pacheco, J. Chen, X. Hu, X. Xia, D. Lo, and A. E. Hassan, “CoLeFunDa: Explainable silent vulnerability fix identification,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), 2023, pp. 2565–2577. [23] Ivan, “Undoing the fix: Evaluating AI control against security regression attacks,” Apart Research AI Control Hackathon report, Mar. 2026. [Online]. Available: https://github.com/strivn/undoing-the-fix/tree/6511 beccd75b63c93628ec8dc062ebb07a0bcc5d [24] J. L. Watson, D. Juergens, N. R. Bennett, B. L. Trippe, J. Yim, H. E. Eisenach, W. Ahern, A. J. Borst, R. J. Ragotte, L. F. Milles, B. I. M. Wicky, N. Hanikel, S. J. Pellock, A. Courbet, W. Sheffler, J. Wang, P. Venkatesh, I. Sappington, S. V. Torres, A. Lauko, V. De Bortoli, E. Mathieu, S. Ovchinnikov, R. Barzilay, T. S. Jaakkola, F. DiMaio, M. Baek, and D. Baker, “De novo design of protein structure and function with RFdiffusion,” Nature, vol. 620, pp. 1089–1100, 2023. [25] J. Dauparas, I. Anishchenko, N. Bennett, H. Bai, R. J. Ragotte, L. F. Milles, B. I. M. Wicky, A. Courbet, R. J. de Haas, N. Bethel, P. J. Y. Leung, T. F. Huddy, S. Pellock, D. Tischer, F. Chan, B. Koepnick, H. Nguyen, A. Kang, B. Sankaran, A. K. Bera, N. P. King, and D. Baker, “Robust deep learning–based protein sequence design using ProteinMPNN,” Science, vol. 378, no. 6615, pp. 49–56, 2022. [26] M. Giulini, V. Reys, J. M. C. Teixeira, B. Jiménez-García, R. V. Honorato, A. Kravchenko, X. Xu, R. Versini, A. Engel, S. Verhoeven, and A. M. J. J. Bonvin, “HADDOCK3: A Modular and Versatile Platform for Integrative Modeling of Biomolecular Complexes,” Journal of Chemical Information and Modeling, vol. 65, no. 13, pp. 7315–7324, 2025. [27] C. Mirabello and B. Wallner, “DockQ v2: improved automatic quality measure for protein multimers, nucleic acids, and small molecules,” Bioinformatics, vol. 40, no. 10, p. btae586, 2024. [28] J. Eberhardt, D. Santos-Martins, A. F. Tillack, and S. Forli, “AutoDock Vina 1.2.0: New Docking Methods, Expanded Force Field, and Python Bindings,” Journal of Chemical Information and Modeling, vol. 61, no. 8, pp. 3891–3898, 2021. [29] M. van Kempen, S. S. Kim, C. Tumescheit, M. Mirdita, J. Lee, C. L. M. Gilchrist, J. Söding, and M. Steinegger, “Fast and accurate protein structure search with Foldseek,” Nature Biotechnology, vol. 42, pp. 243– 246, 2024. [30] PyTorch Contributors, “Serialization API and Safe-Globals Context Manager,” PyTorch v2.5.1 source and API documentation, 2024. [Online]. Available: https://github.com/pytorch/pytorch/blob/v2.5.1/tor ch/serialization.py [31] Python Software Foundation, “pickle—Python Object Serialization,” Python 3.11 documentation. [Online]. Available: https://docs.python.or g/3.11/library/pickle.html [32] M. Souppaya, K. Scarfone, and D. Dodson, “Secure Software Development Framework (SSDF) Version 1.1: Recommendations for Mitigating the Risk of Software Vulnerabilities,” National Institute of Standards and Technology, Tech. Rep. SP 800-218, 2022. [33] GumTreeDiff, “GumTree, release 3.0.0,” Software release. [Online]. Available: https://github.com/GumTreeDiff/gumtree/releases/tag/v3.0.0 [34] Python Software Foundation, “ast—Abstract Syntax Trees,” Python 3.11 documentation. [Online]. Available: https://docs.python.org/3.11/l ibrary/ast.html

[35] LeiLiLab, “SusVibes leaderboard: Public SWE-agent trajectories and submissions,” Trajectory and submission repository, 2026. [Online]. Available: https://github.com/LeiLiLab/susvibes-leaderboard/tree/c74a 6a2ccaea9bae170ff04f4e7f741134ce03e3/public/submissions [36] B. Efron and R. J. Tibshirani, An Introduction to the Bootstrap. Chapman and Hall/CRC, 1994. [37] Python Software Foundation, “tokenize—Tokenizer for Python Source,” Python 3.11 documentation. [Online]. Available: https://docs.python.or g/3.11/library/tokenize.html [38] M. Monperrus, “Automatic Software Repair: A Bibliography,” ACM Computing Surveys, vol. 51, no. 1, pp. 17:1–17:24, 2018. [39] E. K. Smith, E. T. Barr, C. Le Goues, and Y. Brun, “Is the cure worse than the disease? overfitting in automated program repair,” in Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, 2015, pp. 532–543. [40] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” in International Conference on Learning Representations (ICLR), 2024. [Online]. Available: https://proceedings. iclr.cc/paper_files/paper/2024/hash/edac78c3e300629acfe6cbe9ca88fb8 4-Abstract-Conference.html [41] G. Bhandari, A. Naseer, and L. Moonen, “CVEfixes: automated collection of vulnerabilities and their fixes from open-source software,” in Proceedings of the 17th International Conference on Predictive Models and Data Analytics in Software Engineering, 2021, pp. 30–39. [42] Q.-C. Bui, R. Scandariato, and N. E. D. Ferreyra, “Vul4J: A Dataset of Reproducible Java Vulnerabilities Geared Towards the Study of Program Repair Techniques,” in Proceedings of the 19th International Conference on Mining Software Repositories, 2022, pp. 464–468. [43] C. Yang, Z. Zhao, Z. Xie, H. Li, and L. Zhang, “KNighter: Transforming static analysis with LLM-synthesized checkers,” in Proceedings of the 31st ACM Symposium on Operating Systems Principles (SOSP), 2025.

[44] P. Yu, Y. Wu, X. Peng, J. Peng, J. Zhang, P. Xie, and W. Zhao, “ViolationTracker: Building precise histories for static analysis violations,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), 2023. [Online]. Available: https: //conf.researchr.org/details/icse-2023/icse-2023-technical-track/12/Viol ationTracker-Building-Precise-Histories-for-Static-Analysis-Violations [45] A. Koyuncu, K. Liu, T. F. Bissyandé, D. Kim, J. Klein, M. Monperrus, and Y. Le Traon, “FixMiner: Mining relevant fix patterns for automated program repair,” Empirical Software Engineering, vol. 25, no. 3, pp. 1980–2024, 2020. [46] C. S. Xia and L. Zhang, “Automated Program Repair via Conversation: Fixing 162 out of 337 Bugs for $0.42 Each using ChatGPT,” in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024, pp. 819–831. [47] N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, “Reflexion: Language Agents with Verbal Reinforcement Learning,” in Advances in Neural Information Processing Systems, vol. 36, 2023. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2 023/hash/1b44b878bb782e6954cd888628510e90-Abstract-Conference. html [48] A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, and P. Clark, “Self-Refine: Iterative Refinement with Self-Feedback,” in Advances in Neural Information Processing Systems, vol. 36, 2023. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2023/hash/91edff07232 fb1b55a505a9e9f6c0ff3-Abstract-Conference.html [49] motornomad, “POSTURA: Agentic attack surface posture intelligence,” Software repository. [Online]. Available: https://github.com/motornoma d/postura [50] P. K. Rajput, S. Khan, E.-H. Diallo, Y. Song, J. Klein, and T. F. Bissyandé, “Securing Agent-Generated Code Through Development History,” Zenodo, 2026.

Related documents

• Securing Agent-Generated Code with Its Own Development History#1456921
• On the Footprints of Reviewer Bots Feedback on Agentic Pull Requests in OSS GitHub Repositories#138993
• RepoMirage: Probing Repository Context Reasoning in Code Agents with Perturbations#229581
• SafeClawBench: Separating Semantic, Audit-Evidence, and Sandbox Harm in Tool-Using LLM Agents#287076
• An Evaluation of Data Leakage Risks in Tool-Using LLM Agents in Realistic Scenarios#282741