arXiv CSopen access

EvoRepair: Enhancing Vulnerability Repair Agents Through Experience-Based Self-Evolution

arXiv:2605.30105v1 [cs.SE] 28 May 2026

E VO R EPAIR: Enhancing Vulnerability Repair Agents Through Experience-Based Self-Evolution

Haichuan Hu1

Guoqing Xie2

Shengcheng Yu3

Chunrong Fang2

1

Quanjun Zhang1 Zhenyu Chen2

Jiawei Liu2 Liang Xiao1

Nanjing University of Science and Technology 2 Nanjing University 3 Technical University of Munich

Abstract Large Language Models (LLMs) have shown promise for automated vulnerability repair (AVR), but they still face several limitations, including the lack of intravulnerability experience accumulation and the lack of cross-vulnerability experience reuse. As a result, LLMs may repeatedly make similar mistakes during iterative repair and underutilize valuable repair knowledge from historical vulnerabilities. To address these challenges, we propose E VO R EPAIR, the first experiencebased self-evolving AVR agent framework that enables LLMs to accumulate, refine, and leverage domain-specific knowledge across long-horizon vulnerability repairs. E VO R EPAIR follows a cyclic learn-and-repair process that retrieves relevant past experiences to guide repair, extracts new experiences from repair trajectories, and updates an experience bank using quality-aware scoring. We evaluate E VO R EPAIR against 12 representative vulnerability repair baselines on PATCHEVAL and SEC-bench using GPT-5-mini. Results show that E VO R EPAIR achieves the best overall performance, reaching 93.47% on PATCHEVAL, 87.00% on SEC-bench, and 90.46% overall. In particular, E VO R EPAIR outperforms latest LLM-based baseline LoopRepair by 39.56% and 33.50% on PATCHEVAL and SEC-bench, respectively, and surpasses IntentFix by 70.86% and 50.50%. Across both benchmarks, E VO R EPAIR also exceeds the recent self-evolving agent Live-SWE-Agent by 6.98% overall. Additional transfer experiments on VUL4J further demonstrate the robustness of E VO R EPAIR across models, programming languages, and datasets. These findings demonstrate that experience-based selfevolution substantially strengthens agentic AVR and goes beyond existing selfevolving techniques.

1

Introduction

As software systems continue to grow in size and complexity, security vulnerabilities have increased steadily, posing a significant threat to modern software systems [55, 7]. As shown in Figure 1, the number of reported CVEs accelerated after 2019; in 2024 alone, 38,942 new CVEs were reported, representing a 25% year-over-year increase and the highest annual total on record. Given the growing number of disclosed vulnerabilities, Automated Vulnerability Repair (AVR) [76, 54] has emerged as a promising approach for accelerating vulnerability fix and reducing security risks. Early work on AVR [24, 25, 28, 33], predominantly based on program analysis and searchbased methods, has shown effectiveness in synthesizing patches for certain types of vulnerabilities, albeit typically within constrained domains. Subsequent learning-based AVR techniques [22, 26, 72] fine-tune pre-trained models on large-scale vulnerability datasets to capture diverse repair patterns. However, learning-based methods face an important evaluation challenge. Prior work generally adopts CodeBLEU [48] and Exact Match as evaluation metrics, such execution-agnostic measures Preprint. Under review.

Number of CVE Entries

40000

30000

20000

10000

0 2015

2016

2017

2018

2019

2020

2021

Publication Year

2022

2023

2024

2025

Figure 1: Yearly growth in reported CVEs.

are insufficient for assessing real-world repair effectiveness. Specifically, they primarily capture surface-level similarity to reference patches and may therefore overestimate a model’s ability to generalize across heterogeneous datasets. Recently, the rise of Large Language Models (LLMs) has opened new opportunities for AVR [82]. Compared with earlier learning-based methods, LLM-based approaches [16, 4, 52, 53, 38, 3] can achieve strong repair performance under zero-shot or few-shot settings. Moreover, they make endto-end repair feasible and use test feedback to iteratively revise incorrect patches, reducing reliance on the matching-based evaluation schemes commonly used in prior learning-based methods. Although LLM-based methods represent the state of the art in AVR, they still face several technical challenges. (1) Lack of intra-vulnerability repair experience accumulation. Existing AVR methods make limited use of the experience generated during the repair of a single vulnerability. Although multiple repair trajectories may be explored, intermediate successes, failures, and diagnostic signals are rarely distilled into reusable experience for subsequent attempts. Here, we define experience as structured repair knowledge abstracted from repair trajectories, rather than raw interaction logs or one-off patches. Without such experience accumulation, LLMs may repeatedly follow unproductive repair paths, make similar mistakes across trajectories, and miss opportunities to iteratively refine its repair strategy. This stateless repair behavior reduces efficiency and weakens LLMs’ ability to improve within the current task. (2) Lack of cross-vulnerability repair experience reuse. Existing AVR methods [70, 20, 34] largely treat each vulnerability as an isolated repair task and do not systematically reuse repair experience across different vulnerabilities. This limitation becomes increasingly critical as the number of reported vulnerabilities continues to rise, as shown in Figure 1, making one-by-one isolated repair progressively less scalable. Because CWE and CVE provide structured taxonomies, vulnerabilities within the same or related categories often share similar causes, contexts, and repair patterns. However, although retrieval-augmented approaches [14] that retrieve reference patches or examples can provide useful guidance, they remain limited in AVR because the retrieved knowledge is typically static and instance-specific. In contrast, effective AVR requires repair experience that is reusable, generalizable, and continuously refined through a self-evolving repair process. Consequently, relying solely on static retrieval limits crosscase generalization and hinders the accumulation of transferable repair knowledge for large-scale vulnerability repair. Together these challenges indicate that effective AVR requires not only generating patches for the current vulnerability, but also accumulating, refining, and reusing repair experience both within and across vulnerabilities. This motivates the following research question: how can AVR systems effectively accumulate and reuse repair experience both across vulnerabilities and within a single vulnerability? To answer this question, we propose E VO R EPAIR, an experience-based selfevolving framework for AVR. The core idea of E VO R EPAIR is a cyclic two-stage learn-and-repair paradigm. In the learning stage, E VO R EPAIR extracts domain-specific repair knowledge from historical vulnerability repair trajectories and stores it as long-term repair experience. In the repair stage, E VO R EPAIR retrieves relevant experiences for similar vulnerabilities and uses them to guide the 2

repair process toward more promising repair directions. By alternating between these two stages, E VO R EPAIR continuously improves both repair effectiveness and the quality of the accumulated repair experience. We evaluate E VO R EPAIR against 12 representative AVR baselines on PATCHEVAL [60] and SECbench [41]. Overall, the results show that E VO R EPAIR achieves the best performance across both datasets, reaching 93.47% on PATCHEVAL, 87.00% on SEC-bench, and 90.46% overall. In particular, E VO R EPAIR outperforms the strongest LLM-based baseline, LoopRepair, by 39.56% and 33.50% on PATCHEVAL and SEC-bench, respectively, and surpasses IntentFix by 70.86% and 50.50%. E VO R EPAIR also exceeds the recent self-evolving SE agent Live-SWE-Agent by 6.98% overall. Furthermore, cross-dataset transfer experiments on VUL4J demonstrate that the experiences synthesized by E VO R EPAIR generalize across different datasets, programming languages, and model backbones. In summary, we make the following contributions: • New Dimension. We introduce experience-based self-evolution as a new research dimension for automated vulnerability repair. To the best of our knowledge, this is the first work to study how AVR systems can continuously accumulate, refine, and reuse repair experience to improve future AVR processes. • Novel Method. We propose E VO R EPAIR, an experience-based self-evolving framework for automated vulnerability repair. E VO R EPAIR follows a cyclic learn-and-repair paradigm: it learns reusable repair knowledge from historical repair trajectories and retrieves relevant experiences to guide future repairs. This design enables both cross-vulnerability experience reuse and intravulnerability experience accumulation. • Extensive Experiments. We evaluate E VO R EPAIR against 12 representative vulnerability repair baselines on PATCHEVAL and SEC-bench, and further assess the experience transfer ability of E VO R EPAIR on VUL4J. The results demonstrate the strong effectiveness of E VO R EPAIR, as well as its cross-dataset transferability, cross-language generality, and applicability across different backbone models.

2

Background and Motivation

2.1

Automated Vulnerability Repair

Automated Vulnerability Repair [46, 5] aims to fix security flaws with minimal human intervention. A typical AVR pipeline includes vulnerability detection, patch generation, verification, and deployment. Vulnerable locations can be identified using static-analysis tools such as Infer [1] and SpotBugs [2], or model-based methods [50, 21, 45]. Candidate patches are then synthesized [13] and validated through recompilation, functional testing, and security re-analysis before deployment [18, 75]. Existing AVR methods can be broadly grouped into three categories. Early NMTbased approaches formulate vulnerability repair as translation from vulnerable code to repaired code; representative examples include VRepair [13] and SeqTrans [15]. Pretraining-based methods further improve repair performance by fine-tuning pretrained code models on vulnerability datasets. For example, VulRepair [23] fine-tunes CodeT5 on CVEFixes [6] using byte-pair encoding [51], while VulMaster [79] extends CodeT5 with abstract syntax trees and CWE examples. More recently, LLM-based approaches have become dominant because they support end-to-end vulnerability localization, patch generation, and validation without task-specific training [57, 61, 81]. Representative examples include LLM4CVE [19] and VulnRepairEval [58]. Among them, agent-based methods currently represent the strongest line of work by augmenting LLMs with memory and external tools [44, 73, 37]. For example, VulnResolver [73] adopts a hybrid multi-agent design, while PatchIsland [37] integrates agent orchestration with continuous fuzzing pipelines. However, existing agent-based AVR systems still rely heavily on the parametric knowledge of the underlying LLM and do not explicitly accumulate and reuse repair experience across tasks. This limitation motivates our study of self-evolving vulnerability-repair agents. 3

2.2

Self-Evolving Agents

Self-evolving techniques aim to improve agents autonomously through interaction experience with minimal manual supervision. Expel [77] introduces experiential learning by extracting naturallanguage knowledge from prior interactions to support future decision-making. EvolveR [63] extends this idea by formalizing a full experience lifecycle that distills multi-turn interactions into reusable principles and closes the self-improvement loop through reinforcement learning. Absolute Zero Reasoner [78], R-Zero [31], and Multi-Agent Evolve [12] further advance this line of work through verifiable feedback, co-evolutionary learning, and structured multi-agent pipelines. Recently, self-evolving mechanisms have been applied to code agents in software engineering [42, 65, 30, 17, 62, 11, 27], yielding strong progress in program repair. However, most existing efforts focus on issue repair benchmarks such as SWE-bench [35], whereas vulnerability repair requires substantially more domain-specific expertise and security-aware reasoning. Accordingly, the application of self-evolving techniques to AVR remains underexplored. In this work, we study how self-evolving techniques can improve code agents for vulnerability repair. 2.3

Motivation

In complex problem-solving tasks, success often depends on whether a system can leverage prior experience rather than rely solely on stochastic trial-and-error. Without mechanisms to summarize past failures and reuse successful strategies, repeated attempts tend to yield diminishing returns. We observe the same pattern in AVR. When an LLM-based agent fails to identify a promising repair path early, simply increasing the number of repair steps or interaction turns usually leads to much higher cost with only limited gains. This blind scaling often causes the agent to repeat similar reasoning and repair mistakes. To address this limitation, we shift the repair process from exhaustive trial-and-error to experience-guided refinement. By maintaining a dynamic experience bank that records successful repair patterns and common failure modes, the agent can retrieve relevant prior knowledge, avoid redundant exploration, and improve both repair efficiency and repair quality. index.js

Experience.md

CVE-2020-8132 · index.js

exec(...) command injection

25 26 27 28 40

constructGetInfoCommand: function () { return util.format("pdfinfo \"%s\"", this.pdfFilePath); },

41

var getInfoCommand = this.constructGetInfoCommand(); Signal

42

exec(getInfoCommand, function (err, stdout, path flows stderr) into shell {

RETRIEVED EXPERIENCE

CVE-2020-7795 Similarity: 0.84 · CWE-94 / CWE-77 / CWE-78 Prefer execFile or spawn over exec / execSync. Pass user-controlled values as argument arrays instead of shell strings.

getInfo: function () {

43

execution if (err) return reject({ error: err, stdout, stderr });

44 45 46

return resolve(self.parseGetInfoCommandOutput(stdout)); });

Keep the patch minimal.

GUIDED REPAIR

Safer process invocation

}, execFile("pdfinfo", [this.pdfFilePath], function (err, stdout, stderr) { if (err) return reject({ error: err, stdout, stderr }); return resolve(self.parseGetInfoCommandOutput(stdout)); });

Figure 2: Motivation example of E VO R EPAIR. To illustrate this motivation, we conduct a case study on CVE-2020-8132 (Figure 2) using a vanilla GPT-5-mini agent, comparing its behavior with and without experience guidance. Without experience guidance, the agent exhibits the typical limitations of stochastic iteration: after encountering environment-related obstacles during patching, it repeatedly produces error-prone scripts and gradually shifts from principled repair to superficial modifications aimed only at satisfying test cases. In contrast, when equipped with an experience bank, the agent reuses repair knowledge from an analogous case (CVE-2020-7795) to perform more structured refinement. By retrieving secure repair patterns, such as replacing exec with execFile and adding input validation, it avoids repeated trialand-error. Moreover, when facing patch conflicts, it follows a more disciplined suspend-diagnose4

replan-repair workflow, suggesting that experience guidance improves not only repair correctness but also stability during complex troubleshooting.

3

Approach

3.1

Overview

Figure 3 presents the overall workflow of E VO R EPAIR. Built on top of a vanilla agent (Figure 3, upper-left), E VO R EPAIR continually accumulates and summarizes domain-specific experience across multi-turn vulnerability repair trajectories, enabling self-evolution in repair performance. In each repair turn, the agent operates on the vulnerabilities that remain unfixed from the previous turn and continues until it either submits a patch or exhausts the repair budget. Below, we briefly describe the main components of E VO R EPAIR. At the beginning of each turn, E VO R EPAIR retrieves historical experiences relevant to the current vulnerability from the experience bank and injects them into the repair context of the vanilla agent. The agent then interacts with external tools and the target vulnerability’s Docker environment to attempt repair within a bounded budget. After the repair process ends, E VO R EPAIR summarizes and reflects on the repair trajectory, compressing it into a reusable domain-specific experience and scoring it based on quality and generality. The resulting experience is stored in the experience bank for future reuse. At the end of each turn, E VO R EPAIR measures the number of newly fixed vulnerabilities and terminates once repair performance converges, at which point it submits all generated patches. E VO R EPAIR is framework-agnostic and can be adapted to mainstream code-agent frameworks. Moreover, experiences accumulated on one dataset or task can be transferred to other datasets or tasks, supporting cross-language and cross-model transfer. Section 3.3 provides detailed descriptions of all components. Vanilla Agent

Vulnerability-fixing Trajectories

Vulnerability Repair Bash

API

Desc: Stored XSS viva .svg file upload in Github repository ...

Backbone Task Memory

CVE-2025-27154

P Submit Patch

CVE-2020-26215

E Error Occur

CVE-2024-22199

C Cost Limit

Skill Framework-Agnostic

Other CVEs

Current CVE

Extract Experiences

Experience Design Vul Summary Experience Retriever

Fix Strategy

Domain-Specific

Multi-Dimension

Traj Analysis

Exp Learned

Reflection

Construct Memory 1 Description

Fetch By Similarity

Fix

Rerank By Score

Cost

Select Top-K

2 Type

Early Stop

2 Negative

3 Location 4 Symptom

1 Revisit

2 Goal

2 What to Keep

3 Guidance

3 What to Avoid

4 Examples

4 Future Plan

Failure Pattern

3 Effectiveness

Failure Step

LLM-as-a-Judge

Evaluated Experiences

1 Applicability

Instructive Thought Key Action

2 Actions to Take

Query

Experience Bank

1 Positive

1 Scenario

Generalizability

Quality

Turn 1

CVE-xxx

Turn 3

Hot-Start

CVE-xxx

Score: 5

Score: 4

Score: 3

Score: 2

Score: 1

👍 Very useful !

Good quality~

Maybe useful...

Not seem good

No use at all !

……

CVE-xxx

Turn K

Update Experience

Experience Cross-Lingual Transfer

Cross-Dataset

PATCHE VAL

VUL4J

Cross-Model

Figure 3: Overview of E VO R EPAIR. 3.2

Definition of Domain-Specific Experience

To make repair experience reusable, retrievable, and transferable across vulnerabilities, we define a standardized domain-specific experience schema. As shown in Table 1, we systematically model repair experiences from five complementary dimensions. Each dimension is designed to maximize reuse, retrieval precision, and actionable guidance while supporting continuous improvement. (1) Vulnerability introduction and analysis is designed to be a concise, factual description that locates and defines the vulnerability (type, location, reproduction steps, root cause, and affected components). (2) Repair rationale is designed to be a short statement of the chosen remediation approach 5

and its justification (method, expected effect, and considered alternatives). (3) Trajectory analysis is designed to be a distilled account of the repair process highlighting key successful actions, recurrent failure modes, and representative commands/tests or logs. (4) Experience summary is designed to be compact, prescriptive rules derived from the trajectory (preconditions, goal, concrete guidance, and a minimal example). (5) Reflection & Improvement is designed to be a brief evaluation of the outcome, remaining limitations, and suggested follow-ups (e.g., alternative strategies, different repair trajectories). This unified schema enables E VO R EPAIR to continuously accumulate, retrieve, and refine repair knowledge for experience-driven self-evolution. Table 1: Definition of domain-specific experience. Dimension

Purpose

Content

A. Vulnerability Introduction and Analysis

Clearly describe the vulnerability and its scope

Vulnerability type, location, observable symptoms/reproduction steps, root cause analysis, potential impact, affected components/systems

B. Repair Strategy

Summarize the remediation approach and why it was chosen

Chosen remediation method (e.g., input validation, least privilege, resource limits), rationale for selection, expected effectiveness, alternative options

C. Trajectory Analysis

Extract and analyze the most representative thoughts/actions during the repair

Positive Analysis: correct/instructive actions and why they worked; Negative Analysis: critical/common failure patterns and where they went wrong; representative interactions/commands/tests

D. Experience Summary

Synthesize analysis into actionable, reusable rules

For each rule provide: Applicability (preconditions, e.g., language or CWE), Goal (what to achieve), Guidance (concrete advice: “Avoid X, Suggest Y”) with short examples

E. Reflection and Improvement

Provide critical reflection and suggestions to improve future remediation

Evaluation of the fix, shortcomings discovered, next steps if fix failed (revisit root cause, consider alternatives), suggestions for continuous improvement

3.3 3.3.1

Components Experience Retrieval

Before each repair turn, E VO R EPAIR queries the experience bank through an experience retriever to identify relevant prior experiences. For the vulnerability currently under repair, its own historical experiences are always retrieved for direct reuse. For other vulnerabilities, the retriever matches experiences based on CVE and CWE information. Specifically, the retriever first selects the top-M (e.g., 10) candidate experiences from the experience bank according to similarity, as shown in Equation 1. This step filters out vulnerabilities whose similarity to the current vulnerability is too low, since even high-quality experiences may not be applicable when the underlying vulnerabilities differ substantially. The candidate experiences are then reranked in descending order by a consolidated score s′ that combines similarity and the experience score sexp , as shown in Equation 2. The scoring mechanism for sexp is described in Section 3.3.3. Finally, as shown in Equation 3, E VO R EPAIR selects the top-K (e.g., 3) experiences with the highest s′ . This procedure ensures that the retrieved experiences are not only highly similar to the target vulnerability but also sufficiently generalizable to provide useful guidance for the current repair. C = TopM {sim(q, ei )}N i=1 , M

s′ (e) = µ sim(q, e) + (1 − µ) sexp (e), S = TopK {s′ (e)}e∈C , K

(1) e∈C

(2) (3)

Warm-up strategy. E VO R EPAIR faces a cold-start problem in the first repair turn, when the experience bank is empty and no prior experiences can be retrieved. To analyze this issue, we consider two warm-up strategies. The first strategy (Standard Patch) is to collect historical vulnerabilities and their official patches from public vulnerability databases as warm-up samples, allowing the retriever to use patches from similar vulnerabilities as few-shot context before the first repair turn. The second strategy (Pre-repair) is to initialize the experience bank with pre-generated experiences from 6

other vulnerabilities. To avoid direct self-instance leakage, the target vulnerability itself is explicitly excluded. Moreover, to better simulate a realistic historical-memory scenario, only vulnerabilities disclosed earlier than the target vulnerability are eligible for initialization. 3.3.2

Vulnerability Repair

After retrieving relevant experiences, E VO R EPAIR injects them into the repair context to guide the agent in exploring repair trajectories. To minimize the confounding effect of agent-framework design on the performance of E VO R EPAIR, we construct a general-purpose baseline repair agent, referred to as the vanilla agent. The vanilla agent consists of the following components. Toolkit. To avoid performance variance introduced by different toolkits, we equip the vanilla agent with only a minimal Bash toolkit that can execute arbitrary shell commands and submit patches via the keyword SUBMIT. We also provide corresponding function-tool API interfaces and adapt them to different model families (e.g., MistralAI). Skills. We design task-specific skills to instruct the agent on how to perform actions at different stages of the vulnerability repair process. The detailed skill specifications are listed in Table 2. Table 2: Skills used by vanilla agent. Description

Goal

When to Use

Inputs

Outputs

Examples

Understand Vul- Quickly understand the CVE description and At the start of each repair cve_description, vulnerability_summary, hypothesis (likely root nerability CWE information, forming a repair plan and attempt cwe_info, experi- cause), checklist (items to verify) checklist. ence_context Reproduce PoC

Confirm the vulnerability is reproducible in the Before localization current repository by running the PoC. modification.

and source_dir, vul-run.sh

fix-run.sh, reproduction_result (PoC success/failure), fail- bash prepare.sh && bash ure_logs (if reproduction fails) vul-run.sh

Locate Vulnera- Locate the exact code lines, functions, or files After reproducing the vul- vul_location, codebase bility that trigger the vulnerability and mark the editable nerability and before editscope. ing. Verify Patch Reset ment

Verify the patch fixes the vulnerability and does After generating fix.patch prepare.sh, not break functionality. and before submitting. fix.patch

file_path, line_range, context_snippet

fix-run.sh, verification_result

Environ- Reset the repository to its initial state before each Always before running vul- prepare.sh verification attempt to ensure independent trials. run.sh or fix-run.sh.

clean repository state

nl -ba path/to/file | sed -n ’START,ENDp’ bash prepare.sh && bash fix-run.sh bash prepare.sh

Memory. To maintain cross-task knowledge, we equip the vanilla agent with a structured memory module. At the beginning of each repair task, this module is initialized with selected historical experiences retrieved from the experience bank (see Section 3.3.1). The task memory is organized as a 2×2 matrix along two dimensions: (1) Outcome, which distinguishes successful from failed repair attempts; and (2) Source, which distinguishes experiences of the current vulnerability (self ) from those of other vulnerabilities (other). This organization allows the agent to either reuse successful repair patterns or perform negative reasoning to avoid pitfalls observed in previous trajectories. Repair Context. In addition to memory, E VO R EPAIR constructs a repair context for the vanilla agent that includes the CVE and CWE information of the target vulnerability, its vulnerable location (if provided), task instructions, and a high-level description of the repair workflow. Repair Process. After the repair context is constructed, the vanilla agent initiates the repair process. Following the ReAct [69] paradigm, the agent interacts with the Docker environment by invoking the toolkit to autonomously explore repair trajectories. For both cost and efficiency reasons, we limit the number of repair steps and the maximum expenditure. If either limit is exceeded, E VO R EPAIR terminates the repair process early and marks the repair attempt as failed. In multi-turn repair, we further define the turn-level yield rate α to measure the cost-effectiveness of the current repair turn, as shown in Equation 4. α=

β γ

(4)

Here, β and γ denote the growth rate of repair performance and the growth rate of repair cost, respectively, defined as follows: Rn − Rn−1 Rn−1 Cn − Cn−1 γ= Cn−1

β=

7

(4.1) (4.2)

Here, Rn denotes the cumulative successful repair rate after the n-th turn, and Cn denotes the cumulative computational cost incurred up to the n-th turn. To maintain cost-efficiency under a limited repair budget, we adopt an early-stopping strategy based on a predefined threshold of α. When α falls below this threshold, the system terminates the repair process, thereby preventing E VO R EPAIR from continuing ineffective and resource-intensive repair attempts. Framework-agnostic design. Although E VO R EPAIR adopts the vanilla agent as the experimental baseline, it can be flexibly adapted to mainstream code-agent frameworks (e.g., SWE-agent [68] and OpenHands [59]) in practical deployment. 3.3.3

Experience Construction

After the vanilla agent completes a repair task, E VO R EPAIR constructs new experiences from the resulting repair trajectory, following the structural specification defined in Section 3.2. Beyond simple experience generation, E VO R EPAIR also refines historical knowledge to support self-evolution. Specifically, by reflecting on the retrieved experiences during analysis of the current trajectory, the agent can synthesize improved experience entries that preserve verified insights while correcting or discarding erroneous patterns. To prevent the context window from becoming a bottleneck over multiple repair turns, we apply a compression mechanism to keep each experience bounded in length while preserving information density. Finally, every synthesized experience is quantitatively evaluated through a scoring process, as described below. Experience Scoring. E VO R EPAIR employs an LLM-as-a-Judge strategy to score experiences. As shown in Equation 5, the scoring criteria consist of two dimensions: quality and generalizability, where λ denotes the weighting coefficient between the two. Quality measures whether an experience can effectively guide future repairs, for example by providing actionable steps or helping avoid incorrect repair paths. Generalizability measures whether the experience can transfer to other vulnerabilities beyond the current one. In our study, we set λ to 0.5 to balance quality and generalizability. The judge model assigns high scores only when both dimensions are well satisfied. sexp = λ · squality + (1 − λ) · sgeneral

(5)

To mitigate model bias, we experiment with multiple judge models and compare their scores against human ratings. Specifically, we compute the Pearson correlation coefficient (Equation 6) between model-assigned experience scores and human ratings. Among the evaluated judge models, Qwen3Max achieves the highest correlation with human judgment. In addition, to reduce variance from a single scoring pass, we score each experience three times and use the mean as the final score. ρ= 3.3.4

cov(sLLM , sHuman ) σsLLM σsHuman

(6)

Experience Updating

After an experience is evaluated, E VO R EPAIR persists it together with its score and stores it in the experience bank. For the same vulnerability, E VO R EPAIR compares the newly generated experience et from the current turn t with the experience et−1 stored from the previous turn t − 1, and updates the experience bank accordingly. Specifically, E VO R EPAIR adopts the following update strategies. Strategy 1: Discard. If the score of et is lower than that of et−1 , E VO R EPAIR discards et and retains et−1 as the formal experience entry for turn t. Strategy 2: Retain. If the score of et is higher than that of et−1 , E VO R EPAIR retains et . Strategy 3: Polish. If the scores of et and et−1 are identical, E VO R EPAIR feeds both experiences back into the model and triggers an automated fusion process, in which the model merges complementary details from et and et−1 to produce a polished and unified experience entry. During storage, the experiences are organized as a linked list. For a given vulnerability, experiences generated in different turns are linked sequentially as nodes, and pointers are used to record dependency relations among them. This design allows E VO R EPAIR to access the most recent experiences efficiently while also supporting subsequent offline analysis. 8

In terms of storage format, E VO R EPAIR maintains both textual and vector representations. It first stores experiences as Markdown files in a hierarchical directory structure, which makes them easy to inspect by researchers and reuse in other repair scenarios. The textual experiences are then vectorized and stored in a vector database to support efficient similarity-based retrieval and reasoning. After the experience bank is updated, E VO R EPAIR proceeds to the next repair turn and repeats the closed-loop process of experience retrieval, vulnerability repair, experience construction, and experience updating. 3.3.5

Experience Transfer

Through the cyclic interaction of the above four components, E VO R EPAIR enables self-evolution of agent-based repair capabilities on arbitrary vulnerability datasets. Moreover, the experiences distilled during this process can be transferred and reused in other vulnerability repair tasks and datasets. This transfer has three key characteristics. (1) Cross-language transfer. Learned experiences can generalize across programming languages (e.g., Python → Java) without manual adaptation, demonstrating the language-agnostic nature of our approach. (2) Cross-dataset / crosstask transfer. Experiences accumulated on one dataset or task can be reused effectively on unseen datasets and new repair scenarios, thereby avoiding repeated learning. (3) Cross-model transfer. The acquired experiences are independent of any specific model architecture and can be transferred across different backbone models, improving the flexibility and applicability of E VO R EPAIR.

4

Experimental Setup

4.1

Research Questions

We evaluate E VO R EPAIR on the following research questions: RQ1: How does E VO R EPAIR compare against existing vulnerability repair methods? RQ2: How does E VO R EPAIR improve repair performance through self-evolution over multiple repair turns? RQ3: How effective is each component of E VO R EPAIR? RQ4: Can the experience generated by E VO R EPAIR be transferred across languages and datasets? RQ5: How does E VO R EPAIR balance cost and performance? 4.2

Datasets

We evaluate E VO R EPAIR on the latest multilingual (JavaScript, Python, and Go) benchmark PATCHEVAL [60] and C-based benchmark SEC-bench [41]. PATCHEVAL comprises 1,000 vulnerabilities (230 w/ docker + 770 w/o docker), we use a subset of 230 instances with accompanying Docker images to evaluate E VO R EPAIR. SEC-bench comprises 200 C-language instances across 29 projects, including the corresponding vulnerability descriptions, sanitizer reports, and CVE identifiers. Following the original papers, we use oracle localization for PATCHEVAL and apply endto-end repair to SEC-bench (poc-desc mode [41]). In addition, we employ VUL4J [9] to further validate the experience transfer capability of E VO R EPAIR. VUL4J consists of 79 Java CVEs, which is orthogonal to both PATCHEVAL and SEC-bench in terms of programming language and CVE composition. Therefore, it serves as a suitable and independent benchmark for sufficiently verifying the transfer ability of E VO R EPAIR. 4.3

Baselines and Models

On PATCHEVAL and SEC-bench, we evaluate E VO R EPAIR against 12 baselines: 2 learningbased methods (VulRepair[23] and VulMaster[80]), 6 LLM-based methods (Zero-shot[39], Fewshot[8], ChatRepair[64], PailGen[53], LoopRepair[71], and IntentFix[36]), and 4 agent-based methods (SmolAgent[49], AgentMem[66], Vanilla Agent, and Live-SWE-Agent[65]). We directly adopt the official SEC-bench results for SmolAgent and AgentMem. We reproduce VulRepair and VulMaster with CodeT5 on both datasets. For all remaining baselines, we use GPT-5-mini as a unified base model for re-implementation, so as to ensure a fair comparison in the main experiments. All 9

hyperparameters and experimental settings follow the original papers as closely as possible. In addition, we study five representative backbone LLMs, including three closed-source models (GPT5-mini[56], DeepSeek-v3.1[43], and Qwen3.5-Plus[67]) and two open-source models (Devstral24B[47] and Devstral-123B[47]). On VUL4J, we compare E VO R EPAIR with 5 advanced vulnerability repair baselines [64, 10, 32, 40, 29]. 4.4

Evaluation Metrics

Following prior studies [41, 74], we use the following two metrics to evaluate the effectiveness of E VO R EPAIR. #Fix measures the number of vulnerabilities successfully fixed by E VO R EPAIR. %Fix measures the proportion of vulnerabilities successfully fixed by E VO R EPAIR. We determine whether a patch fixes a vulnerability based on the results of PoC tests and unit tests. 4.5

Implementation Details

To implement E VO R EPAIR, we first build a basic, general-purpose vulnerability-repair agent, which we refer to as the vanilla agent. The vanilla agent follows the widely used Mini-SWE-Agent paradigm [68]; we further adapt it for vulnerability repair and extend the existing bash tool with a function-tool interface to support various LLMs. Building on this foundation, we implement the components of E VO R EPAIR. Specifically, when constructing the experience bank we persist experiences as markdown using templates and vectorize them into Chroma. For experience retrieval, we use the LangChain framework with bge-large-en-v1.5 as the embedding model for similarity matching. For experience extraction and evaluation, we choose Qwen3-Max for its low cost and strong overall capabilities. When evaluating API models, we use OpenAI’s official API for the GPT family, Bailian’s DeepSeek API for the DeepSeek models, and deploy other open-source models with vLLM on an 8× RTX 5880 server using bf16 precision and a context length limit of 64,000 tokens. For reproducibility, all models are generated with temperature set to 0. E VO R EPAIR performs continuous multi-turn repairs on unresolved vulnerabilities, with a maximum of 15 turns in this work. If there are no newly fixed vulnerabilities in the current turn, we consider the agent to have reached its repair capacity and terminate the evaluation. We align the evaluation budget and stopping conditions across methods where applicable, while preserving each framework’s native interaction style and tooling assumptions. For each vulnerability, we cap E VO R EPAIR’s repair effort at 100 steps or $3, and the repair process terminates early if either condition is met. All experiments were run on a Linux server with 740 GB of RAM. We set the number of concurrent threads to 32 to speed up the experiments.

5

Evaluation and Results

5.1

RQ1: Overall Performance

Table 3 shows that E VO R EPAIR achieves the best overall performance on both PATCHEVAL and SEC-bench. Specifically, E VO R EPAIR repairs 215/230 vulnerabilities on PATCHEVAL (93.47%) and 174/200 on SEC-bench (87.00%), yielding an overall repair rate of 90.46% over 430 instances. Compared with existing state-of-the-art repair methods, E VO R EPAIR maintains a clear advantage. Among prior approaches, the strongest result is achieved by the recent self-evolving SE agent LiveSWE-Agent, which attains an overall repair rate of 83.48%. Notably, while both Live-SWE-Agent and E VO R EPAIR are built upon a self-evolving agent framework, E VO R EPAIR achieves a further 6.98% improvement overall, highlighting the advantage of experience-driven evolution over toolcentric self-evolution in vulnerability repair. In contrast, earlier LLM-based methods such as ChatRepair and LoopRepair reach only 50.69% and 53.72%, while learning-based methods perform substantially worse, likely because approaches such as VulRepair and VulMaster are constrained by fixed input formats and language-specific settings (e.g., C), and are primarily designed for relatively short repair contexts. These results show that our method substantially enhances the effectiveness of LLM-based and agent-based vulnerability repair, surpassing not only conventional repair paradigms but also prior self-evolving agent techniques. 10

Table 3: Performance comparison of different vulnerability repair methods on PATCHEVAL and SEC-bench. VulRepair and VulMaster use CodeT5, while all other methods use GPT-5-mini. Best results are in bold. Dataset

Metric

Learning-based

LLM-based

VulRepair VulMaster FSE 2022 ICSE 2024

Few-Shot 2022

Zero-Shot IntentFix 2020 ICSE 2026

Agent-based

Ours

PailGen TOSEM 2026

ChatRepair ISSTA 2024

LoopRepair ICSE 2026

SmolAgent 2025

AgentMem 2025

Vanilla 2026

Live-SWE 2026

E VO R EPAIR 2026

PATCHEVAL (230)

# Fix % Fix

0 0.00

0 0.00

39 16.95

34 14.78

52 22.61

52 22.61

115 50.00

124 53.91

-

-

194 84.34

199 86.52

215 93.47

SEC-bench (200)

# Fix % Fix

7 3.50

19 9.50

60 30.00

69 34.50

73 36.50

81 40.50

103 51.50

107 53.50

69 34.50

100 50.00

155 77.50

160 80.00

174 87.00

Total (430)

# Fix % Fix

7 1.62

19 4.41

99 23.02

103 23.95

125 29.06

133 30.93

218 50.69

231 53.72

-

-

349 81.16

359 83.48

389 90.46

5.2

RQ2: Turn-level Performance

Figure 4 and Figure 5 show the turn-level performance of E VO R EPAIR across 15 repair turns on PATCHEVAL and SEC-bench, respectively. Overall, the experimental results demonstrate the effectiveness of the self-evolution mechanism across different backbone models and repair turns. We summarize the key findings as follows: (1) Breaking the Performance Ceiling. Across both PATCHEVAL and SEC-bench, E VO R EPAIR consistently achieves more successful fixes than the vanilla agent. On PATCHEVAL, the final number of fixes improves from 194 to 215 on GPT-5mini, from 210 to 220 on Qwen3.5-Plus, and from 136 to 163 on DeepSeek-v3.1. On SEC-bench, the improvements remain clear, increasing from 155 to 174 on GPT-5-mini, from 94 to 170 on Qwen3.5-Plus, and from 127 to 131 on DeepSeek-v3.1. These results show that self-evolution effectively pushes repair performance beyond the ceiling of standard iterative prompting. (2) Consistent Gains Across Repair Turns. The advantage of E VO R EPAIR is not limited to the final turn. On both benchmarks, its performance curve stays above that of the vanilla agent for most repair turns, indicating that self-evolution provides stable benefits throughout the iterative repair process. (3) Faster Convergence and Higher Efficiency. E VO R EPAIR typically reaches its nearpeak performance within the first several turns, while the vanilla agent converges more slowly and often to a lower plateau. This trend is especially evident on GPT-5-mini and Qwen3.5-Plus across both PATCHEVAL and SEC-bench, showing that self-evolution reduces redundant trial-and-error and improves repair efficiency. (4) Robustness Across Model Families and Scales. The gains of E VO R EPAIR are consistent across both proprietary and open-source backbones, including GPT-5mini, Qwen3.5-Plus, DeepSeek-v3.1, Devstral-24B, and Devstral-123B. This shows that the effectiveness of self-evolution generalizes across different model families and scales. 215 194

200 150

#Fix

#Fix

150

100 0

100 50

Vanilla Agent EvoRepair (Ours) 1 2 3 4 5 6 7 8 9 101112131415

0

Turn

Vanilla Agent EvoRepair (Ours) 1 2 3 4 5 6 7 8 9 101112131415

Turn

(a) GPT-5-mini 200

136

150

#Fix

150

163

#Fix

100

(b) Qwen3.5-Plus 218 216

Vanilla Agent EvoRepair (Ours) 0 1 2 3 4 5 6 7 8 9 101112131415

Turn

(c) DeepSeek-v3.1

150

218

100

216 12

50

216 207

200

50

13

14

15

Vanilla Agent EvoRepair (Ours)

0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Turn

(d) Devstral-24B

#Fix

50

220 210

200

100 50 0

Vanilla Agent EvoRepair (Ours) 1 2 3 4 5 6 7 8 9 101112131415

Turn

(e) Devstral-123B

Figure 4: Turn-level performance on PATCHEVAL.

11

100

#Fix

100

#Fix

170 150

50

Vanilla Agent EvoRepair (Ours) 1 2 3 4 5 6 7 8 9 101112131415

0

50 0

Turn

(a) GPT-5-mini

94 Vanilla Agent EvoRepair (Ours) 1 2 3 4 5 6 7 8 9 101112131415

Turn

125 100 75 50 25 0

131 127

#Fix

174 155

150

Vanilla Agent EvoRepair (Ours) 1 2 3 4 5 6 7 8 9 101112131415

Turn

(b) Qwen3.5-Plus

(c) DeepSeek-v3.1

Figure 5: Turn-level performance on SEC-bench.

5.3

RQ3: Ablation Study

In RQ3, we conduct ablation studies on the core components and configurations of E VO R EPAIR, including the number of retrieved experiences, the mechanism for experience retrieval, the prompting strategies for experience construction, and the methodologies employed to address the cold-start problem of the experience bank. We choose PATCHEVAL for ablation study because it is multilingual and larger in scale compared to SEC-bench. Ablation 1: Experience Count. When constructing the repair context, E VO R EPAIR retrieves a specific number of experiences from the experience bank to serve as guidance. To investigate the relationship between the experience count and the resulting repair effectiveness, we conduct experiments by retrieving k experiences, where k ∈ {1, 3, 5, 7, 9}. As illustrated in Figure 6, we evaluate the repair efficacy of E VO R EPAIR across varying quantities of retrieved experiences. Overall, the repair performance exhibits a characteristic bell-shaped trajectory relative to the experience count, initially ascending to a peak before undergoing a marginal decline. Empirical evidence suggests that Exp=5 represents the optimal configuration for the majority of the evaluated models; specifically, it yields the highest repair count in three out of the five models and secures the second-best performance in the other two. Consequently, we adopt the Exp=5 configuration for all subsequent experiments.

220 200

Va ni Ex lla p= Ex 1 p= Ex 3 p= Ex 5 p= Ex 7 p= 9

180

210 200 190

DeepSeek-v3.1

Devstral-24B 230 220 210

Devstral-123B 230 220 210 200

200

190 Va ni Ex lla p= Ex 1 p= Ex 3 p= Ex 5 p= Ex 7 p= 9

230

180 170 160 150 140 130

Va ni Ex lla p= Ex 1 p= Ex 3 p= Ex 5 p= Ex 7 p= 9

Qwen3.5-Plus

Va ni Ex lla p= Ex 1 p= Ex 3 p= Ex 5 p= Ex 7 p= 9

#Fix

220

240

Va ni Ex lla p= Ex 1 p= Ex 3 p= Ex 5 p= Ex 7 p= 9

GPT-5-mini

Figure 6: Relationship between number of retrieved experiences and repair performance of E VO R EPAIR. Furthermore, the learning capacity for retrieved experiences varies across different model architectures. SOTA commercial models, such as Qwen3.5-Plus and GPT-5-mini, demonstrate a stable performance gain as the experience count increases. Even when additional experiences introduce potential noise, these models exhibit only marginal performance degradation, underscoring their superior robustness. In contrast, while the older DeepSeek-v3.1 benefits significantly from a larger context (peaking at Exp=9), indicating commendable In-Context Learning (ICL) capabilities, it remains constrained by lower performance bounds and higher sensitivity to the quantity of guidance. This suggests a persistent gap in reasoning maturity compared to the latest LLMs. For code-specialized models like Devstral, although they possess high intrinsic repair proficiency, they derive limited incremental utility from retrieved experiences. In fact, excessive experiences often lead to diminishing returns or negative performance impacts. This phenomenon suggests that while code-specific models excel in structural syntax and local logic, their ICL efficacy is heavily dictated by their underlying natural language understanding and knowledge transfer abilities. Consequently, the high specialization of code models may lead to a "knowledge saturation" point 12

where extra context acts more as a distractor than a catalyst, highlighting a critical trade-off between domain expertise and contextual adaptability.

Table 4: Effectiveness of the experience retrieval mechanism (w/ experience score vs. w/o experience score). Model w/o score w/ score

GPT-5-mini DeepSeek-v3.1 Qwen3.5-Plus Devstral-24B Devstral-123B

205 160 208 213 215

215 (+10) 163 (+3) 220 (+12) 218 (+5) 216 (+1)

Ablation 2: Experience Retrieval Mechanism. When retrieving experiences, E VO R EPAIR simultaneously considers the similarity between vulnerabilities and the score (quality+generalizability) of the experiences. To investigate the impact of experience score on retrieval effectiveness, in Ablation 2, we exclude the experience scoring component and retrieve experiences based solely on vulnerability similarity. As shown in Table 4, we find that the removal of the scoring component leads to a consistent performance degradation across all models to varying degrees. This observation suggests that vulnerability similarity alone is insufficient to guarantee the efficacy of retrieved guidance. The results underscore the critical role of the scoring module in filtering out low-quality or noisy experiences, thereby refining the retrieval results beyond raw similarity to provide more robust guidance for the repair process. Ablation 3: Experience Construction Strategy. To ensure that the experiences generated by E VO R EPAIR have good generalization while being easily practical for agents, we adopt an exampledriven strategy in constructing experiences. This involves requiring the model to provide minimal examples, including script commands and expected outcomes, while summarizing the experiences in natural language. To verify the effectiveness of the example-driven strategy, we remove this approach in Ablation 3, requiring the model only to summarize experiences in natural language for comparison of repair results. As shown in Table 5, all models achieve performance gains when employing the example-driven strategy, indicating that concrete examples effectively assist models in better internalizing and leveraging learned experiences.

Table 5: Effectiveness of example-driven experience construction. Model w/o example w/ example

GPT-5-mini DeepSeek-v3.1 Qwen3.5-Plus Devstral-24B Devstral-123B

209 160 207 214 207

215 (+6) 163 (+3) 220 (+13) 218 (+4) 216 (+9)

Ablation 4: Warm-up Strategy. To analyze the effect of initialization under cold-start, we implement two warm-up strategies: (1) Pre-repair, which initializes the experience bank with pregenerated experiences from other vulnerabilities while explicitly excluding the target vulnerability itself to avoid direct self-instance leakage. To better approximate a historical-memory scenario, we further restrict the initialization bank to vulnerabilities whose disclosure dates precede that of the target vulnerability, so that no future vulnerability information is used; and (2) Standard Patch, which uses standard patches from 770 vulnerabilities outside the PATCHEVAL evaluation set as the initial bank for few-shot guidance. As shown in Table 6, both strategies improve over the vanilla agent. However, directly retrieving similar vulnerability patches for few-shot learning leads to a substantial performance drop (-21) on GPT-5-mini, whereas the Pre-repair strategy yields more balanced gains across models. These results suggest that trajectory-derived experiences provide more effective guidance than raw patch examples in the cold-start setting. 13

Table 6: Comparison of different warm-up strategies.

Model

w/o warm-up

Standard Patch

Pre-repair

78 53 100 111 119

57 (-21) 46 (-7) 110 (+10) 120 (+9) 128 (+9)

105 (+27) 54 (+1) 131 (+31) 113 (+2) 124 (+5)

GPT-5-mini DeepSeek-v3.1 Qwen3.5-Plus Devstral-24B Devstral-123B

5.4

RQ4: Experience Transfer Performance

We select Qwen3.5-Plus, which achieves the best performance on PATCHEVAL, as the teacher model to transfer its synthesized experiences to VUL4J. For the student model, we employ two configurations: in configuration T1, the student model is also Qwen3.5-Plus, intended to validate intra-model experience transfer; in configuration T2, the student model is Qwen3-Max, utilized to evaluate cross-model transferability. As shown in Table 7, under both configurations, EvoRepair outperforms all baselines, achieving performance improvements of 9.67% and 8.69%, respectively, compared to repair without experience transfer. The transfer results suggest that some experiences learned on one benchmark can remain useful across datasets and programming languages. Table 7: The effectiveness of experience transfer (PATCHEVAL → VUL4J). Method

E VO R EPAIR-T1

E VO R EPAIR-T2

NTR

VRPILOT

APR4Vul

ChatRepair

TSAPR

# Fix

34/79 (+9.67%)

25/79 (+8.69%)

14/79

14/79

16/79

14/79

24/79

5.5

RQ5: Cost Analysis

As shown in Table 8, we compare E VO R EPAIR with the vanilla agent in terms of both per-CVE repair cost and total cost. Overall, the total cost remains nearly unchanged: the five vanilla models cost $1,666.9 in total, while E VO R EPAIR costs $1,663.7, a decrease of 0.19%. However, the cost impact varies across models. DeepSeek-v3.1 shows the largest increase (+41.86%), largely because its weaker repair ability causes many unrecoverable vulnerabilities to repeatedly hit the per-instance cost limit. In contrast, Qwen3.5-Plus reduces cost by 41.58%, suggesting that strong backbone models can benefit from experience guidance and converge more quickly. For the remaining models, the cost increase is roughly 10%–20%. We find that most of the additional cost comes from the prompt-token overhead introduced by injecting experiences into the repair context. Table 8: Cost analysis of E VO R EPAIR on PATCHEVAL. Per CVE Model DeepSeekv3.1 GPT-5-mini Devstral-24B Devstral-123B Qwen3.5-Plus

Total

Ref

Method

Step

Prompt (K)

Response (K)

Turn

Prompt (M)

Response (M)

Cost ($)

Charge ($/M)

Vanilla EvoRepair Vanilla EvoRepair Vanilla EvoRepair Vanilla EvoRepair Vanilla EvoRepair

64.15 76.18 20.02 25.64 68.99 66.22 73.04 65.25 82.01 70.28

823.8 1355.7 306.4 596.3 1321.7 1587.1 1203 1269.7 1881.6 1681.6

11.4 10.6 33.3 43.9 10.3 11.1 9.5 7.4 20.6 26

3.51 3.93 3.03 2.49 2.33 2.41 1.99 2.31 2.56 1.92

147.7 214 337.9 444.8 925.2 1101.4 715.8 883.6 1580.5 911.4

20.4 16.7 36.7 32.7 7.1 7.7 5.6 5.1 20.6 14

419.2 594.7 173.8 194.4 94.6 112.5 297.6 363.8 681.79 398.3

Prompt: 0.27 Response: 1 Prompt: 0.275 Response: 2.2 Prompt: 0.1 Response: 0.3 Prompt: 0.4 Response: 2 Prompt: 0.4 Response: 2

In terms of trajectory overhead, E VO R EPAIR achieves shorter average repair paths on three of the five models, with the largest reduction on Qwen3.5-Plus (-14.3%). Two models also require fewer repair turns overall. Notably, stronger models such as GPT-5-mini and Qwen3.5-Plus can achieve better final repair results with only modest additional cost (e.g., GPT-5-mini +11.8%) or even substantial cost reduction (e.g., Qwen3.5-Plus -41.58%), whereas weaker models such as DeepSeekv3.1 incur greater overhead from irreparable vulnerabilities. These results suggest that cost control, 14

such as early stopping, and experience compression are promising directions for future work on self-evolving AVR. Table 9: Early stop (ES) strategy under different α. Model DeepSeek-v3.1 Qwen3.5-Plus Devstral-123B Devstral-24B GPT-5-mini

α # Fix Cost ($) # Fix Cost ($) # Fix Cost ($) # Fix Cost ($) # Fix Cost ($)

0 (w/o ES)

0.1

0.15

0.2

0.25

0.3

163 594.74 220 398.35 216 363.81 218 112.45 215 194.4

163(-) 594.74(-) 220(-) 398.35(-) 208(-3.7%) 314.95(-13.43%) 218(-) 112.45(-) 204(-5.11%) 147.53(-24.11%)

163(-) 594.74(-) 207(-5.9%) 294.92(-25.96%) 195(-9.72%) 246.61(-32.21%) 214(-1.83%) 102.21(-9.1%) 204(-5.1%1) 147.53(-24.11%)

150(-7.97%) 474.63(-20.19%) 207(-5.9%) 294.92(-25.96%) 195(-9.72%) 246.61(-32.21%) 195(-10.5%) 75.86(-32.53%) 202(-6.04%) 132.92(-31.62%)

150(-7.97%) 474.63(-20.19%) 207(-5.9%) 294.92(-25.96%) 195(-9.72%) 246.61(-32.21%) 195(-10.55%) 75.86(-32.53%) 198(-7.9%) 120.49(-38.01%)

91(-44.17%) 200.18(-66.34%) 205(-6.8%) 274.43(-31.1%) 195(-9.72%) 246.61(-32.21%) 195(-10.55%) 75.86(-32.53%) 198(-7.9%) 120.49(-38.01%)

Early Stop. In Table 1, we discuss the efficacy of the threshold-based early-stopping strategy. The turn-level yield rate α, as defined in Equation 4, represents the ratio between the marginal gain in repair effectiveness and the marginal increase in repair cost. The results demonstrate that by setting the threshold between 0.15 and 0.25, all models can significantly reduce computational costs (↓ 20%-40%) with minimal degradation in performance (↓ 1%-10%), thereby validating the effectiveness of the early-stopping strategy.

6

Threats to Validity

Internal Validity. Internal validity concerns potential experimental biases that may affect the fairness or consistency of our evaluation. To reduce the impact of randomness during the generation process of LLMs on the experimental results, we repeat each experimental setting three times in the same hardware environment and take the median. Additionally, we save the trajectories generated during the agent’s repair process and conduct offline analysis and comparison of multiple trajectories for the same CVE to validate the consistency of the agent’s behavior. All experiments are conducted in isolated Docker environments to ensure that they are not affected by other processes. External Validity. External validity addresses the generalizability of our findings beyond the specific datasets and programming languages used in this study. Regarding the dataset issue, to validate the effectiveness of E VO R EPAIR across different datasets, we conduct extensive experiments on three high-quality vulnerability datasets (PATCHEVAL, SEC-bench, VUL4J) with five open-source and closed-source models. The three datasets cover vulnerabilities in five different programming languages (C, Java, Python, JavaScript, Go), demonstrating that the effectiveness of E VO R EPAIR is independent of the programming language.

7

Conclusion

In this paper, we address key limitations of existing agent-based AVR approaches, including the lack of intra-vulnerability experience accumulation and the lack of cross-vulnerability experience reuse. To overcome these challenges, we propose E VO R EPAIR, an experience-based self-evolving agent framework that enables AVR agents to accumulate, refine, and leverage domain-specific knowledge throughout iterative vulnerability repair. Evaluations on PATCHEVAL, SEC-bench, and VUL4J demonstrate that E VO R EPAIR consistently improves repair effectiveness over prior learning-based, LLM-based, and agent-based methods. Ablation studies further confirm the effectiveness of key components, including experience retrieval, quality-aware scoring, and cold-start strategies, supporting the framework’s transferability across datasets, languages, and model backbones.

References [1] Infer static analyzer, 2022. Last accessed: November 20, 2022. [2] Spotbugs: Find bugs in java programs, 2022. Last accessed: November 20, 2022. 15

[3] B. Ahmad, S. Thakur, B. Tan, R. Karri, and H. Pearce. On hardware security bug code fixes by prompting large language models. IEEE Transactions on Information Forensics and Security, 19:4043–4057, 2024. [4] K. Bao and S. Chen. A smart contract vulnerability detection method based on graph neural networks and zero-shot learning. In International Conference on Blockchain and Trustworthy Systems, pages 32–46. Springer, 2025. [5] B. Belleville, W. Shen, S. Volckaert, A. M. Azab, and M. Franz. KALD: detecting direct pointer disclosure vulnerabilities. IEEE Trans. Dependable Secur. Comput., 18(3):1369–1377, 2021. doi: 10.1109/TDSC.2019.2915829. URL https://doi.org/10.1109/TDSC.2019.2915829. [6] G. P. Bhandari, A. Naseer, and L. Moonen. Cvefixes: Automated collection of vulnerabilities and their fixes from open-source software. CoRR, abs/2107.08760, 2021. URL https://arxiv. org/abs/2107.08760. [7] L. Bilge and T. Dumitraş. Before we knew it: an empirical study of zero-day attacks in the real world. In Proceedings of the 2012 ACM conference on Computer and communications security, pages 833–844, 2012. [8] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. [9] Q.-C. Bui, R. Scandariato, and N. E. D. Ferreyra. Vul4j: A dataset of reproducible java vulnerabilities geared towards the study of program repair techniques. In Proceedings of the 19th International Conference on Mining Software Repositories, pages 464–468, 2022. [10] Q.-C. Bui, R. Paramitha, D.-L. Vu, F. Massacci, and R. Scandariato. Apr4vul: an empirical study of automatic program repair techniques on real-world java vulnerabilities. Empirical software engineering, 29(1):18, 2024. [11] S. Chen, S. Lin, X. Gu, Y. Shi, H. Lian, L. Yun, D. Chen, W. Sun, L. Cao, and Q. Wang. Sweexp: Experience-driven software issue resolution. arXiv preprint arXiv:2507.23361, 2025. [12] Y. Chen, Y. Wang, S. Zhu, H. Yu, T. Feng, M. Zhang, M. Patwary, and J. You. Multi-agent evolve: Llm self-improve through co-evolution. arXiv preprint arXiv:2510.23595, 2025. [13] Z. Chen, S. Kommrusch, and M. Monperrus. Neural transfer learning for repairing security vulnerabilities in C code. IEEE Trans. Software Eng., 49(1):147–165, 2023. doi: 10.1109/ TSE.2022.3147265. URL https://doi.org/10.1109/TSE.2022.3147265. [14] S. Cheng, Q. Yu, Y. Zhu, and Z. Huang. Automated vulnerability repair based on retrievalaugmented generation. In 2025 7th International Conference on Information Science, Electrical and Automation Engineering (ISEAE), pages 941–947. IEEE, 2025. [15] J. Chi, Y. Qu, T. Liu, Q. Zheng, and H. Yin. Seqtrans: Automatic vulnerability fix via sequence to sequence learning. IEEE Trans. Software Eng., 49(2):564–585, 2023. doi: 10.1109/TSE. 2022.3156637. URL https://doi.org/10.1109/TSE.2022.3156637. [16] A. Costin, H. Turtiainen, N. Yousefnezhad, V. Bogulean, and T. Hämäläinen. Evaluating zeroshot chatgpt performance on predicting cve data from vulnerability descriptions. In Proceedings of the European Conference on Cyber Warfare and Security, number 1. Academic Conferences International Ltd, 2024. [17] Y. Ding and L. Zhang. Swe-replay: Efficient test-time scaling for software engineering agents. arXiv preprint arXiv:2601.22129, 2026. [18] R. Duan, A. Bijlani, Y. Ji, O. Alrawi, Y. Xiong, M. Ike, B. Saltaformaggio, and W. Lee. Automating patching of vulnerable open-source software versions in application binaries. In 26th Annual Network and Distributed System Security Symposium, NDSS 2019, San Diego, California, USA, February 24-27, 2019. The Internet Society, 2019. URL https://www.ndss-symposium.org/ndss-paper/ automating-patching-of-vulnerable-open-source-software-versions-in-application-binaries/. 16

[19] M. Fakih, R. Dharmaji, H. Bouzidi, G. Q. Araya, O. Ogundare, M. Siddika, and M. A. A. Faruque. LLM4CVE: enabling iterative automated vulnerability repair with large language models. In 28th Euromicro Conference on Digital System Design, DSD 2025, Salerno, Italy, September 10-12, 2025, pages 592–599. IEEE, 2025. doi: 10.1109/DSD67783.2025.00087. URL https://doi.org/10.1109/DSD67783.2025.00087. [20] M. Fakih, R. Dharmaji, H. Bouzidi, G. Q. Araya, O. Ogundare, M. A. Siddika, and M. A. Al Faruque. Llm4cve: Enabling iterative automated vulnerability repair with large language models. In 2025 28th Euromicro Conference on Digital System Design (DSD), pages 592–599. IEEE, 2025. [21] M. A. Ferrag, A. Battah, N. Tihanyi, R. Jain, D. Maimut, F. Alwahedi, T. Lestable, N. S. Thandi, A. Mechri, M. Debbah, and L. C. Cordeiro. Securefalcon: Are we there yet in automated software vulnerability detection with llms? IEEE Trans. Software Eng., 51(4): 1248–1265, 2025. doi: 10.1109/TSE.2025.3548168. URL https://doi.org/10.1109/TSE.2025. 3548168. [22] M. Fu. Toward more effective deep learning-based automated software vulnerability prediction, classification, and repair. In 2023 IEEE/ACM 45th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), pages 208–212. IEEE, 2023. [23] M. Fu, C. Tantithamthavorn, T. Le, V. Nguyen, and D. Phung. Vulrepair: a t5-based automated software vulnerability repair. In Proceedings of the 30th ACM joint european software engineering conference and symposium on the foundations of software engineering, pages 935– 947, 2022. [24] X. Gao, S. Mechtaev, and A. Roychoudhury. Crash-avoiding program repair. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 8–18, 2019. [25] X. Gao, B. Wang, G. J. Duck, R. Ji, Y. Xiong, and A. Roychoudhury. Beyond tests: Program vulnerability repair via crash constraint extraction. ACM Transactions on Software Engineering and Methodology (TOSEM), 30(2):1–27, 2021. [26] W. Han, Y. Kwak, M. Yu, K. Kim, Y. Lee, H. Moon, and Y. Paek. Rethinking the capability of fine-tuned language models for automated vulnerability repair. arXiv preprint arXiv:2512.22633, 2025. [27] Z. Hao, H. Wang, J. Luo, J. Zhang, Y. Zhou, Q. Lin, C. Wang, H. Dong, and J. Chen. Recreate: Reasoning and creating domain agents driven by experience. arXiv preprint arXiv:2601.11100, 2026. [28] S. Hong, J. Lee, J. Lee, and H. Oh. Saver: scalable, precise, and safe memory-error repair. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, pages 271–283, 2020. [29] H. Hu, Y. Shang, W. Sun, and Q. Zhang. Tsapr: A tree search framework for automated program repair. arXiv preprint arXiv:2507.01827, 2025. [30] T. Hu, R. Chen, S. Zhang, J. Yin, M. X. Feng, J. Liu, S. Zhang, W. Jiang, Y. Fang, S. Hu, et al. Controlled self-evolution for algorithmic code optimization. arXiv preprint arXiv:2601.07348, 2026. [31] C. Huang, W. Yu, X. Wang, H. Zhang, Z. Li, R. Li, J. Huang, H. Mi, and D. Yu. R-zero: Self-evolving reasoning llm from zero data. arXiv preprint arXiv:2508.05004, 2025. [32] K. Huang, J. Zhang, X. Meng, and Y. Liu. Template-guided program repair in the era of large language models. In ICSE, pages 1895–1907, 2025. [33] Z. Huang, D. Lie, G. Tan, and T. Jaeger. Using safety properties to generate vulnerability patches. In 2019 IEEE symposium on security and privacy (SP), pages 539–554. IEEE, 2019. 17

[34] R. Jiao, Y. Zhang, J. Li, and J. Ma. Hit the bullseye on the first shot: Improving llms using multi-sample self-reward feedback for vulnerability repair. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 791–803. IEEE, 2025. [35] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023. [36] H. Jinseok, C. Dongwook, K. Jinyoung, K. Misoo, and L. Eunseok. Intentfix: Automated logic vulnerability repair via llm-driven intent modeling. In Proceedings of the IEEE/ACM 48th International Conference on Software Engineering, ICSE ’26. Association for Computing Machinery, 2026. [37] W. Kim, S. Min, M. Gwon, D. Baik, H. Lee, H. Heo, M. Lee, M. W. Baek, Y. Jin, Y. Park, Y. Choi, T. Kim, S. Park, and I. Yun. Patchisland: Orchestration of llm agents for continuous vulnerability repair. arXiv preprint arXiv:2601.17471, 2026. [38] Y. Kim, S. Shin, H. Kim, and J. Yoon. Logs in, patches out: Automated vulnerability repair via {Tree-of-Thought}{LLM} analysis. In 34th USENIX Security Symposium (USENIX Security 25), pages 4401–4419, 2025. [39] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213, 2022. [40] U. Kulsum, H. Zhu, B. Xu, and M. d’Amorim. A case study of llm for automated vulnerability repair: Assessing impact of reasoning and patch validation feedback. In Proceedings of the 1st ACM International Conference on AI-Powered Software, pages 103–111, 2024. [41] H. Lee, Z. Zhang, H. Lu, and L. Zhang. Sec-bench: Automated benchmarking of llm agents on real-world software security tasks. arXiv preprint arXiv:2506.11791, 2025. [42] J. Lin, Y. Guo, Y. Han, S. Hu, Z. Ni, L. Wang, M. Chen, H. Liu, R. Chen, Y. He, et al. Seagent: Self-evolution trajectory optimization in multi-step reasoning with llm-based agents. arXiv preprint arXiv:2508.02085, 2025. [43] A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. [44] Z. Liu, Y. Ma, J. Xu, J. Ai, X. Gao, H. Sun, and A. Roychoudhury. Agent that debugs: Dynamic state-guided vulnerability repair. CoRR, abs/2504.07634, 2025. doi: 10.48550/ARXIV.2504. 07634. URL https://doi.org/10.48550/arXiv.2504.07634. [45] Q. Mao, Z. Li, X. Hu, K. Liu, X. Xia, and J. Sun. Towards explainable vulnerability detection with large language models. IEEE Trans. Software Eng., 51(10):2957–2971, 2025. doi: 10. 1109/TSE.2025.3605442. URL https://doi.org/10.1109/TSE.2025.3605442. [46] Y. Noller, R. Shariffdeen, X. Gao, and A. Roychoudhury. Trust enhancement issues in program repair. In 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022, pages 2228–2240. ACM, 2022. doi: 10.1145/3510003. 3510040. URL https://doi.org/10.1145/3510003.3510040. [47] A. Rastogi, A. Yang, A. Q. Jiang, A. H. Liu, A. Sablayrolles, A. Héliou, A. Martin, A. Agarwal, A. Ehrenberg, A. Lo, et al. Devstral: Fine-tuning language models for coding agent applications. arXiv preprint arXiv:2509.25193, 2025. [48] S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, and S. Ma. Codebleu: a method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009.10297, 2020. [49] A. Roucher, A. V. del Moral, T. Wolf, L. von Werra, and E. Kaunismäki. smolagents: A smol library to build great agentic systems. Hugging Face, 2025. 18

[50] C. Seas, G. Fitzpatrick, J. A. H. Jr., and M. C. Carlisle. Automated vulnerability detection in source code using deep representation learning. In R. Paul and A. Kundu, editors, 14th IEEE Annual Computing and Communication Workshop and Conference, CCWC 2024, Las Vegas, NV, USA, January 8-10, 2024, pages 484–490. IEEE, 2024. doi: 10.1109/CCWC60891.2024. 10427574. URL https://doi.org/10.1109/CCWC60891.2024.10427574. [51] R. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/V1/P16-1162. URL https://doi.org/10.18653/v1/p16-1162. [52] A. Shahriar, S. J. Hisham, K. A. Rahman, M. R. Islam, M. S. Hossain, R.-H. Hwang, and Y.-D. Lin. 5gpt: 5g vulnerability detection by combining zero-shot capabilities of gpt-4 with domain aware strategies through prompt engineering. IEEE Transactions on Information Forensics and Security, 2025. [53] M. Shao, Y. Ding, C. Gao, J. Wang, and G. Zhu. Fix pattern-aware vulnerability patch generation via in-context learning. ACM Transactions on Software Engineering and Methodology, 2026. [54] Z. Shen and S. Chen. A survey of automatic software vulnerability detection, program repair, and defect prediction techniques. Security and Communication Networks, 2020(1):8858010, 2020. [55] Y. Shin, A. Meneely, L. Williams, and J. A. Osborne. Evaluating complexity, code churn, and developer activity metrics as indicators of software vulnerabilities. IEEE transactions on software engineering, 37(6):772–787, 2010. [56] A. Singh, A. Fry, A. Perelman, A. Tart, A. Ganesh, A. El-Kishky, A. McLaughlin, A. Low, A. Ostrow, A. Ananthram, et al. Openai gpt-5 system card. arXiv preprint arXiv:2601.03267, 2025. [57] P. Wang, X. Liu, and C. Xiao. Cve-bench: Benchmarking llm-based software engineering agent’s ability to repair real-world CVE vulnerabilities. In L. Chiruzzo, A. Ritter, and L. Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2025 - Volume 1: Long Papers, Albuquerque, New Mexico, USA, April 29 - May 4, 2025, pages 4207–4224. Association for Computational Linguistics, 2025. doi: 10.18653/V1/2025. NAACL-LONG.212. URL https://doi.org/10.18653/v1/2025.naacl-long.212. [58] W. Wang, W. Ma, Q. Hu, Y. Zhang, J. Sun, B. Wu, Y. Liu, G. Xu, and L. Jiang. Vulnrepaireval: An exploit-based evaluation framework for assessing large language model vulnerability repair capabilities. CoRR, abs/2509.03331, 2025. doi: 10.48550/ARXIV.2509.03331. URL https: //doi.org/10.48550/arXiv.2509.03331. [59] X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song, B. Li, J. Singh, et al. Openhands: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741, 2024. [60] Z. Wei, J. Zeng, M. Wen, Z. Yu, K. Cheng, Y. Zhu, J. Guo, S. Zhou, L. Yin, X. Su, et al. Patcheval: A new benchmark for evaluating llms on patching real-world vulnerabilities. arXiv preprint arXiv:2511.11019, 2025. [61] X. Wen, Z. Lin, Y. Yang, C. Gao, and D. Ye. Vul-r2: A reasoning LLM for automated vulnerability repair. In 40th IEEE/ACM International Conference on Automated Software Engineering, ASE 2025, Seoul, Korea, Republic of, November 16-20, 2025, pages 26–38. IEEE, 2025. doi: 10.1109/ASE63991.2025.00011. URL https://doi.org/10.1109/ASE63991.2025.00011. [62] Z. Weng, A. Antoniades, D. Nathani, Z. Zhang, X. Pu, and X. E. Wang. Group-evolving agents: Open-ended self-improvement via experience sharing. arXiv preprint arXiv:2602.04837, 2026. 19

[63] R. Wu, X. Wang, J. Mei, P. Cai, D. Fu, C. Yang, L. Wen, X. Yang, Y. Shen, Y. Wang, et al. Evolver: Self-evolving llm agents through an experience-driven lifecycle. arXiv preprint arXiv:2510.16079, 2025. [64] C. S. Xia and L. Zhang. Automated program repair via conversation: Fixing 162 out of 337 bugs for $0.42 each using chatgpt. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 819–831, 2024. [65] C. S. Xia, Z. Wang, Y. Yang, Y. Wei, and L. Zhang. Live-swe-agent: Can software engineering agents self-evolve on the fly? arXiv preprint arXiv:2511.13646, 2025. [66] W. Xu, Z. Liang, K. Mei, H. Gao, J. Tan, and Y. Zhang. A-mem: Agentic memory for llm agents. In Advances in Neural Information Processing Systems, 2025. [67] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. [68] J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press. Sweagent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528–50652, 2024. [69] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations, 2022. [70] Z. Ye, X. Sun, S. Cao, L. Bo, and B. Li. Well begun is half done: Location-aware and traceguided iterative automated vulnerability repair. arXiv preprint arXiv:2512.20203, 2025. [71] Z. Ye, X. Sun, S. Cao, L. Bo, and B. Li. Well begun is half done: Location-aware and traceguided iterative automated vulnerability repair. In Proceedings of the IEEE/ACM 48th International Conference on Software Engineering, ICSE ’26. Association for Computing Machinery, 2026. [72] J. Zhang, C. Wang, A. Li, W. Wang, T. Li, and Y. Liu. Vuladvisor: Natural language suggestion generation for software vulnerability repair. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, pages 1932–1944, 2024. [73] M. Zhang, X. Wang, J. Zhang, X. Meng, J. Zhang, and C. Hu. Vulnresolver: A hybrid agent framework for llm-based automated vulnerability issue resolution. CoRR, abs/2601.13933, 2026. doi: 10.48550/ARXIV.2601.13933. URL https://doi.org/10.48550/arXiv.2601.13933. [74] M. Zhang, X. Wang, J. Zhang, X. Meng, J. Zhang, and C. Hu. Vulnresolver: A hybrid agent framework for llm-based automated vulnerability issue resolution. arXiv preprint arXiv:2601.13933, 2026. [75] Q. Zhang, Y. Zhao, W. Sun, C. Fang, Z. Wang, and L. Zhang. Program repair: Automated vs. manual. CoRR, abs/2203.05166, 2022. doi: 10.48550/ARXIV.2203.05166. URL https: //doi.org/10.48550/arXiv.2203.05166. [76] Q. Zhang, C. Fang, B. Yu, W. Sun, T. Zhang, and Z. Chen. Pre-trained model-based automated software vulnerability repair: How far are we? IEEE Transactions on Dependable and Secure Computing, 21(4):2507–2525, 2023. [77] A. Zhao, D. Huang, Q. Xu, M. Lin, Y.-J. Liu, and G. Huang. Expel: Llm agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19632–19642, 2024. [78] A. Zhao, Y. Wu, Y. Yue, T. Wu, Q. Xu, M. Lin, S. Wang, Q. Wu, Z. Zheng, and G. Huang. Absolute zero: Reinforced self-play reasoning with zero data. arXiv preprint arXiv:2505.03335, 2025. [79] X. Zhou, K. Kim, B. Xu, D. Han, and D. Lo. Out of sight, out of mind: Better automatic vulnerability repair by broadening input ranges and sources. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ICSE ’24. Association for Computing Machinery, 2024. 20

[80] X. Zhou, K. Kim, B. Xu, D. Han, and D. Lo. Out of sight, out of mind: Better automatic vulnerability repair by broadening input ranges and sources. In Proceedings of the IEEE/ACM 46th international conference on software engineering, pages 1–13, 2024. [81] X. Zhou, S. Cao, X. Sun, and D. Lo. Large language model for vulnerability detection and repair: Literature review and the road ahead. ACM Trans. Softw. Eng. Methodol., 34(5):145:1– 145:31, 2025. doi: 10.1145/3708522. URL https://doi.org/10.1145/3708522. [82] X. Zhou, S. Cao, X. Sun, and D. Lo. Large language model for vulnerability detection and repair: Literature review and the road ahead. ACM Transactions on Software Engineering and Methodology, 34(5):1–31, 2025.

21

Appendix Contents A Algorithm

22

B Full Result on SEC-bench

22

C Overlap Analysis

23

D Combined With Live-SWE-Agent

24

E Experience Transfer

26

F Early Stop

26

A

Algorithm

As shown in Figure 1, we provide a complete pseudocode representation of the E VO R EPAIR workflow, which delineates the step-by-step logic and decision-making mechanisms integrated into each remediation cycle. Algorithm 1: Self-Evolving Framework for Automated Vulnerability Repair Input: Current vulnerability q, initial experience bank E Output: Final patched code begin // Initialize variables T ←0; // Repair turns V ← Unfixed vulnerabilities while V is not empty do T ←T +1 // Step 1: Retrieve relevant experiences S ← Experience Retrieval(q, E) // Step 2: Prioritize retrieval based on quality and similarity R ← Prioritize(S) // Step 3: Inject retrieved experiences into the repair context Inject(R) // Step 4: Perform repair using the vanilla agent Patch ← VanillaAgentRepair(q) // Step 5: Validate patch and update experience bank if Validation(Patch, q) then E ← UpdateExperience(P atch, T ) V ← V \{q} ; // Remove fixed vulnerability return Final patched code

B

Full Result on SEC-bench

We present in Table 10 the complete results of 15 turns of repairs on SEC-bench for the Vanilla agent, Live-SWE-Agent, and E VO R EPAIR. The results show that E VO R EPAIR generally outperforms both Live-SWE-Agent and the Vanilla agent. When using Qwen3.5-Plus as the base model, E VO R EPAIR achieves the most pronounced improvement, fixing 26 more vulnerabilities than Live-SWE-Agent and 76 more than the Vanilla agent. With DeepSeek-v3.2 and GPT-5-mini, although E VO R EPAIR’s final repair ceiling is roughly on par with Live-SWE-Agent, E VO R EPAIR typically converges 3–5 turns faster. Additionally, in Tables 11 and 12 we report each method’s file-level and function-level localization accuracy at every turn. 22

Table 10: Comparison of test-time scaling performance (E VO R EPAIR, Live-SWE-Agent, Vanilla Agent) on SEC-bench. Model DeepSeek-v3.1 DeepSeek-v3.2 Qwen3.5-Plus GPT-5-mini

Method

Turn 1

Turn 2

Turn 3

Turn 4

Turn 5

Turn 6

Turn 7

Turn 8

Turn 9

Turn 10

Turn 11

Turn 12

Turn 13

Turn 14

Turn 15

# Fix

Vanilla Live-SWE-Agent E VO R EPAIR Vanilla Live-SWE-Agent E VO R EPAIR Vanilla Live-SWE-Agent E VO R EPAIR Vanilla Live-SWE-Agent E VO R EPAIR

22 28 44 94 86 101 36 67 64 51 35 66

50 59 73 122 122 134 60 84 98 81 74 119

65 73 90 140 135 147 71 101 120 98 102 135

75 83 102 149 146 156 77 111 135 110 117 142

83 95 110 149 150 163 78 116 141 122 125 149

90 102 112 154 168 82 121 146 128 132 151

95 108 116 159 169 87 125 149 131 141 155

98 112 120 163 171 89 126 152 138 145 158

99 120 125 166 173 94 128 156 142 148 161

101 121 126 167 173 94 131 159 145 151 164

111 125 131 171 134 165 150 152 166

114 127 131 173 137 168 152 156 170

117 127 173 138 170 153 157 171

118 141 170 155 160 174

118 144 155 160 174

118 127 131 149 173 173 94 144 170 155 160 174

Table 11: File-level localization accuracy on SEC-bench. Model DeepSeek-v3.1 DeepSeek-v3.2 Qwen3.5-Plus GPT-5-mini

C

Method

Turn 1

Turn 2

Turn 3

Turn 4

Turn 5

Turn 6

Turn 7

Turn 8

Turn 9

Turn 10

Turn 11

Turn 12

Turn 13

Turn 14

Turn 15

Vanilla Live-SWE-Agent E VO R EPAIR Vanilla Live-SWE-Agent E VO R EPAIR Vanilla Live-SWE-Agent E VO R EPAIR Vanilla Live-SWE-Agent E VO R EPAIR

49.07% 41.28% 49.69% 46.67% 47.08% 47.26% 41.18% 62.16% 53.23% 30.94% 33.07% 33.20%

41.76% 39.60% 47.75% 52.88% 52.73% 45.76% 61.90% 38.33% 54.79% 32.37% 30.97% 42.13%

43.55% 33.77% 41.84% 56.06% 48.15% 35.29% 42.42% 53.33% 55.81% 26.90% 27.63% 37.89%

38.60% 42.86% 45.45% 48.00% 52.24% 38.71% 41.38% 57.14% 50.00% 25.62% 27.42% 35.00%

41.46% 27.50% 37.65% 46.81% 50.00% 40.74% 38.46% 57.89% 65.22% 27.64% 19.81% 42.86%

40.82% 39.47% 39.39% 47.37% 42.50% 47.37% 47.37% 61.11% 23.64% 23.85% 33.80%

45.45% 48.15% 41.43% 50.00% 63.33% 57.89% 66.67% 27.78% 25.00% 21.21% 32.81%

35.42% 25.93% 48.28% 41.18% 46.43% 28.57% 46.15% 33.33% 30.85% 23.26% 40.00%

30.43% 33.96% 42.86% 59.26% 50.00% 54.17% 62.50% 72.73% 22.58% 16.25% 33.33%

51.52% 44.00% 36.54% 41.18% 47.83% 63.64% 50.00% 53.33% 20.43% 36.36% 38.30%

52.17% 35.71% 40.35% 44.83% 63.64% 53.33% 25.88% 18.75% 34.88%

25.45% 25.00% 36.67% 57.69% 53.85% 37.50% 22.64% 20.00% 32.61%

54.17% 35.14% 52.63% 54.55% 60.00% 25.42% 18.97% 45.71%

50.00% 87.50% 40.00% 36.73% 15.09% 43.59%

32.14% 60.00% 21.57% 20.00% 42.11%

Overlap Analysis

Overlap analysis between vanilla agent and E VO R EPAIR. on PATCHEVAL, we conduct an overlap analysis of the vulnerabilities remediated by E VO R EPAIR and the vanilla agent to evaluate their respective repair capabilities. As illustrated in Figures 7(a) to 7(e), we compare the CVEs remediated by E VO R EPAIR and the vanilla agent across various base models. We observe a high degree of overlap between the vulnerabilities resolved by the two methods. Notably, E VO R EPAIR effectively encompasses the remediation coverage of the vanilla agent; specifically, when using Qwen3.5-Plus as the base model, there is only one vulnerability that E VO R EPAIR fails to remediate while the vanilla agent succeeds. This indicates that E VO R EPAIR successfully preserves the baseline agent’s strengths while simultaneously augmenting its repair capabilities. These results demonstrate the robustness of E VO R EPAIR, confirming that the integration of the experience library does not introduce excessive noise or degrade the original repair performance of the base model. Overlap analysis between different models. Beyond analyzing the overlap in remediated vulnerabilities across different methods, we also investigate the overlap among various base models. As illustrated in Figure 7(f), we observe that even prior to the integration of E VO R EPAIR, the vanilla agents across different base models already exhibit a significant degree of overlap in their remediation coverage. Specifically, the 129 jointly repaired vulnerabilities account for 60% or more of the successful repairs for any individual model, indicating a substantial level of inherent consistency in how different LLMs handle standard security flaws. However, upon the introduction of E VO R EPAIR, as shown in Figure 7(g), the number of shared successful repairs across the five models increases to a striking 157, representing over 70% of any single model’s repair set. This significant expansion in the overlap set demonstrates that E VO R EPAIR successfully preserves the original strengths of the vanilla agents while consistently augmenting their capabilities. More importantly, this trend suggests that the experience library acts as a powerful ’knowledge anchor,’ harmonizing the repair trajectories of diverse base models toward a convergent, Table 12: Method-level localization accuracy on SEC-bench. Model DeepSeek-v3.1 DeepSeek-v3.2 Qwen3.5-Plus GPT-5-mini

Method

Turn 1

Turn 2

Turn 3

Turn 4

Turn 5

Turn 6

Turn 7

Turn 8

Turn 9

Turn 10

Turn 11

Turn 12

Turn 13

Turn 14

Turn 15

Vanilla Live-SWE-Agent E VO R EPAIR Vanilla Live-SWE-Agent E VO R EPAIR Vanilla Live-SWE-Agent E VO R EPAIR Vanilla Live-SWE-Agent E VO R EPAIR

38.20% 31.63% 36.69% 40.20% 39.38% 41.84% 41.94% 52.08% 47.47% 22.62% 28.97% 23.41%

34.62% 29.03% 29.25% 34.26% 33.88% 36.52% 50.00% 36.17% 48.28% 23.18% 27.42% 33.55%

26.92% 35.29% 26.74% 39.44% 40.28% 30.99% 42.31% 50.00% 35.71% 14.18% 22.63% 24.18%

21.43% 31.48% 28.92% 29.41% 38.81% 38.71% 25.00% 42.11% 36.67% 18.81% 20.39% 34.21%

24.32% 18.52% 29.63% 31.91% 28.30% 27.12% 33.33% 40.00% 46.43% 17.31% 12.24% 33.33%

20.45% 41.94% 25.76% 28.57% 31.71% 19.05% 31.58% 34.78% 15.38% 17.20% 20.63%

22.45% 27.59% 25.68% 29.79% 47.22% 38.10% 53.85% 20.00% 13.95% 17.07% 24.07%

21.43% 28.21% 28.57% 30.77% 21.21% 16.67% 35.71% 13.33% 13.25% 16.44% 25.00%

20.51% 25.00% 30.00% 33.33% 31.25% 40.00% 27.27% 33.33% 12.99% 13.04% 21.15%

43.75% 26.92% 23.53% 22.22% 23.08% 40.00% 20.00% 25.00% 8.86% 20.45% 16.67%

34.09% 24.39% 30.00% 21.21% 42.86% 23.53% 12.99% 6.67% 11.36%

20.45% 17.86% 20.37% 23.33% 37.50% 20.00% 8.93% 11.48% 19.05%

29.63% 33.33% 26.32% 38.46% 40.00% 14.04% 9.80% 24.24%

21.88% 62.50% 25.00% 16.67% 7.32% 24.32%

13.79% 37.50% 12.00% 11.36% 21.62%

23

28

187

11

7

EvoRepair Vanilla Agent

209

EvoRepair Vanilla Agent

(a) GPT-5-mini

36

127

(b) Qwen3.5-Plus

7

9

EvoRepair Vanilla Agent

1

211

5

14

5

EvoRepair Vanilla Agent

EvoRepair Vanilla Agent

(c) DeepSeek-v3.1

202

(d) Devstral-24B

(e) Devstral-123B

optimal solution space. It further proves that E VO R EPAIR possesses excellent robustness, as the high-quality diagnostic priors effectively guide different models to resolve complex vulnerabilities without introducing excessive noise that might otherwise degrade their baseline performance. Devstral-24B

2 Devstral-123B

3 6 0

2

0 2

DeepSeek-v3.1

0

0 0

0 0

129

0 4 2

Qwen3.5-Plus

0

0

2

15 3

45 5 1

Devstral-123B 0

0

GPT-5-mini

Qwen3.5-Plus

0

0 0

DeepSeek-v3.1

0

0 0

1 157

0 5 0

1 0

0 0

0

(f) Vanilla Agent

D

0 4

1 00 1

Devstral-24B

0

2

3 0

43 5 2

0

0 00 0

3 1

GPT-5-mini

(g) E VO R EPAIR

Combined With Live-SWE-Agent

During the reproduction of Live-SWE-Agent, we observe that while synthesizing bash tools does not achieve full self-evolution, with multi-turn convergence results falling between the vanilla agent and E VO R EPAIR, combining fine-grained bash commands into fixed-function scripts effectively reduces the length of the agent’s repair trajectories and accelerates convergence. Driven by these observations, we conduct further exploratory experiments by integrating Live-SWEAgent with E VO R EPAIR, aiming for a breakthrough in both performance and efficiency. 24

Specifically, we incorporate the prompt (as shown below) from Live-SWE-Agent, which instructs the model to synthesize new tools during the repair process, into the E VO R EPAIR prompt while maintaining all other configurations. We evaluate this hybrid approach across three models (Qwen3.5-Plus, Devstral-24B, and DeepSeek-v3.1) on PATCHEVAL. The experimental results indicate the following: (1) Performance ceiling. The integration of Live-SWE-Agent does not significantly raise the upper bound of repair capabilities. Performance remains on par with the original results for Devstral-24B and DeepSeek-v3.1, while Qwen3.5-Plus successfully fixes three additional vulnerabilities. (2) Model efficiency. The integration shows a measurable impact on efficiency. Both Qwen3.5-Plus and Devstral-24B reach convergence five turns earlier than their respective baselines. Combined with the experimental results above, we believe that tool evolving and experience evolving are two critical pathways to achieving self-evolving agents. Among them, experience evolving focuses more on raising the performance ceiling, while tool evolving focuses more on enhancing efficiency. The organic integration of both can achieve a superior outcome that is both fast and effective. Tool evolving prompt in Live-SWE-Agent ## Creating your own tools - You can also create your own tools in Python to help with your workflow - Compared to basic bash commands, the tools you create should be able to better aid your workflow in solving the task - Ensure each tool you create is in Python, contains informative outputs or error messages, and can be ran from the command line - You should at least create a simple edit tool that can help you effectively edit arbitrary files instead of using bash commands - The tools you create can be for any purpose, it does not need to be general, instead think about how it can help you specifically with the current task at hand ### Example of creating a custom tool: <example_response> THOUGHT: I noticed that in order to solve the issue I need to ... therefore I should create a custom tool to help me ... ‘‘‘bash cat <<’EOF’ > /path/to/tool_name.py #!/usr/bin/env python3 import sys # Import other packages if needed def main(): # Your tool logic here ... if __name__ == "__main__": main() EOF ‘‘‘‘ </example_response>

25

### Example of using the tool you created: <example_response> THOUGHT: Let me use the custom tool I created to help me with ... ‘‘‘bash python /path/to/tool_name.py <<EOF your_input_here EOF ‘‘‘ </example_response>

E

Experience Transfer

As shown in Table 13, we present the full results of the experience transfer experiment. Specifically, we conduct experiments under three configurations, where the teacher model is consistently Qwen3.5-Plus, and the student models are Qwen3.5-Plus, Qwen3-Max, and Qwen3-Coder, respectively. We transfer the experience generated by E VO R EPAIR using Qwen3.5-Plus on PATCHEVAL to VUL4J, and then compare it with the repair results without experience transfer under the same model. Overall, experience transfer demonstrates effectiveness across all models. Specifically, Qwen3.5-Plus fixes 3 additional vulnerabilities (31 → 34), Qwen3-Max fixes 2 additional vulnerabilities (23 → 25), and Qwen3-Coder-Plus fixes 1 additional vulnerability (22 → 23). By examining the repair details of each vulnerability in Table 13, we observe that the model’s repair behavior changes after applying experience transfer. Specifically, vulnerabilities that the model consistently fails to repair without prior knowledge can now be successfully fixed, as the introduced experience provides valuable guidance for the model to discover correct patches. Taking Qwen3.5-Plus as an example, the three vulnerabilities VUL4J-56, VUL4J-58, and VUL4J-71 cannot be fixed consistently by the model across multiple independent runs without experience guidance. However, after introducing experience transfer, the model achieves stable and reliable repair for these vulnerabilities. On the other hand, the introduced experience may also bring noise. For instance, Qwen3-Max can successfully repair VUL4J-25 without experience guidance but fails after applying experience transfer. This phenomenon is more pronounced in cross-model transfer (e.g., Qwen3.5-Plus → Qwen3-Max, Qwen3.5-Plus → Qwen3-Coder-Plus), while the effectiveness of experience is relatively robust under identical-model transfer (e.g., Qwen3.5-Plus → Qwen3-Plus). Besides the model choice, experience transfer is also affected by many other factors. For instance, the vulnerabilities in PATCHEVAL are written in JavaScript, Go, and Python, whereas VUL4J contains Java programs. The differences among programming languages impose a non-negligible impact on the effectiveness of experience transfer. As a result, the performance of experience transfer across datasets, languages, and models degrades compared with experience evolution under the single dataset, same model, and identical language setting. In future work, we will further investigate how to improve the effectiveness of experience transfer. We plan to enhance its performance by extracting and transforming knowledge across programming languages, as well as by reducing the amount of experience to alleviate noise interference.

F

Early Stop

To balance the performance and cost of E VO R EPAIR, we have integrated an early-stopping strategy into our original methodology. The core rationale behind this strategy is that when the remediation cost of a single turn for E VO R EPAIR outweighs the marginal gain from new fixes beyond a certain threshold, we deem the repair process no longer cost-effective and thus terminate the entire workflow prematurely. Specifically, we define the turn-level yield rate α (detailed in Equation 4) as the ratio between the marginal increase in repair rate and the marginal increase in cost. When α falls below a certain threshold t, E VO R EPAIR triggers an early-stopping strategy. To facilitate the determination of the optimal threshold for α, Table 14 provides a detailed breakdown of E VO R EPAIR’s 26

Table 13: Experience transfer result on the VUL4J dataset. VUL VUL4J-1 VUL4J-2 VUL4J-3 VUL4J-4 VUL4J-5 VUL4J-6 VUL4J-7 VUL4J-8 VUL4J-9 VUL4J-10 VUL4J-11 VUL4J-12 VUL4J-13 VUL4J-14 VUL4J-15 VUL4J-16 VUL4J-17 VUL4J-18 VUL4J-19 VUL4J-20 VUL4J-21 VUL4J-22 VUL4J-23 VUL4J-24 VUL4J-25 VUL4J-26 VUL4J-27 VUL4J-28 VUL4J-29 VUL4J-30 VUL4J-31 VUL4J-32 VUL4J-33 VUL4J-34 VUL4J-35 VUL4J-36 VUL4J-37 VUL4J-38 VUL4J-39 VUL4J-40 VUL4J-41 VUL4J-42 VUL4J-43 VUL4J-44 VUL4J-45 VUL4J-46 VUL4J-47 VUL4J-48 VUL4J-49 VUL4J-50 VUL4J-51 VUL4J-52 VUL4J-53 VUL4J-54 VUL4J-55 VUL4J-56 VUL4J-57 VUL4J-58 VUL4J-59 VUL4J-60 VUL4J-61 VUL4J-62 VUL4J-63 VUL4J-64 VUL4J-65 VUL4J-66 VUL4J-67 VUL4J-68 VUL4J-69 VUL4J-70 VUL4J-71 VUL4J-72 VUL4J-73 VUL4J-74 VUL4J-75 VUL4J-76 VUL4J-77 VUL4J-78 VUL4J-79

Qwen3.5-Plus Vanilla

Qwen3.5-Plus Transfer

Qwen3-Max Vanilla

Qwen3-Max Transfer

Qwen3-Coder-Plus Vanilla

Qwen3-Coder-Plus Transfer

% ! % % % % % % ! ! ! % % % % % % ! ! ! % ! % % ! % % % % % % % % % % ! % ! % % ! % ! % ! % % ! ! % % ! % % % % ! % ! ! ! ! % ! ! ! % ! ! ! % % % % % ! ! % !

% ! % % % % % % ! ! ! % % % % % % ! ! ! % ! % % ! % % % % % % % % % % ! % ! % % ! % ! % ! % % ! ! % % ! % % % ! ! ! ! ! ! ! % ! ! ! % ! ! ! ! % % % % ! ! % !

% ! % % % % % % ! % ! % % % % % % ! % % % % % % ! % % % % % % % % % % % % ! % % ! % ! % ! % % ! ! % % ! % % % % % % ! % ! ! % ! ! ! % ! ! ! ! % % % % % ! % %

% ! % % % % % % ! ! ! % % % % % % ! % ! % % % % % % % % % % % % % % % % % ! % % ! % ! % ! % % ! ! % % % % % % % ! % ! % ! ! % ! ! ! % ! ! ! ! % % % % % ! % !

% ! % % % % % % ! % ! % % % % % % % % ! % ! % % % % % % % % % % % % % % % % % % % % ! % ! % % ! ! % % % % % % ! ! % % ! ! ! % ! ! % % ! ! ! ! % % % % % ! % !

% ! % % % % % % ! ! ! % % % % % % ! ! % % % % % % % % % % % % % % % % ! % % % % % % ! % ! % % ! ! % % ! % % % ! % % % ! ! ! % ! ! % % ! ! % ! % % % % % ! % !

27

turn-level computational costs and incremental fixes, based on which the corresponding values of α are calculated. As shown, α exhibits a generally downward trend as the number of turns increases. By adjusting the threshold t, E VO R EPAIR can achieve an optimal balance between performance and cost-effectiveness across different models. Based on the empirical data in the table, we recommend setting the threshold t within the range of 0.15 to 0.25.

28

Table 14: Breakdown of new fixes and additional costs per turn on PATCHEVAL, used as a criterion for the early-stopping strategy. Turn

CVEs

New Fixes

Cumulative Fixes

New Cost ($)

Cumulative Cost ($)

β

γ

α

DeepSeek-v3.1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

230 176 148 139 126 108 97 89 84 81 80 76 72 70 -

54 28 9 13 18 11 8 5 3 1 4 4 2 3 -

54 82 91 104 122 133 141 146 149 150 154 158 160 163 -

75.8867 64.7944 59.5065 51.6457 47.594 41.7326 37.4431 36.1995 30.5494 29.284 32.6795 30.8755 26.253 30.2974 -

75.8867 140.6811 200.1876 251.8333 299.4273 341.1599 378.603 414.8025 445.352 474.6359 507.3154 538.191 564.444 594.7414 -

∞ 0.5185 0.1098 0.1429 0.1731 0.0902 0.0602 0.0355 0.0205 0.0067 0.0267 0.026 0.0127 0.0187 -

∞ 0.8538 0.423 0.258 0.189 0.1394 0.1098 0.0956 0.0736 0.0658 0.0689 0.0609 0.0488 0.0537 -

∞ 0.6073 0.2595 0.5537 0.9158 0.6469 0.5481 0.3709 0.279 0.1021 0.3873 0.4268 0.2595 0.3493 -

Qwen3.5-Plus

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

230 99 46 31 25 23 21 18 16 12 11 10 -

131 53 15 6 2 2 3 2 4 1 1 0 -

131 184 199 205 207 209 212 214 218 219 220 220 -

137.3843 68.6887 40.2869 28.0739 20.4882 22.8841 17.7669 18.2184 11.7496 12.4055 10.1384 10.2706 -

137.3843 206.073 246.36 274.4339 294.9221 317.8062 335.5731 353.7915 365.5411 377.9466 388.0851 398.3557 -

∞ 0.4046 0.0815 0.0302 0.0098 0.0097 0.0144 0.0094 0.0187 0.0046 0.0046 0 -

∞ 0.5 0.1955 0.114 0.0747 0.0776 0.0559 0.0543 0.0332 0.0339 0.0268 0.0265 -

∞ 0.8092 0.417 0.2646 0.1307 0.1245 0.2568 0.1738 0.5628 0.1352 0.1702 0 -

Devstral-123B

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

230 106 66 51 38 35 29 25 23 22 20 19 17 15 -

124 40 15 13 3 6 4 2 1 2 1 2 2 1 -

124 164 179 192 195 201 205 207 208 210 211 213 215 216 -

104.8407 53.7326 36.4763 26.5779 24.9846 19.3673 18.6482 15.5734 14.7502 11.872 8.8812 8.6617 10.5507 8.9027 -

104.8407 158.5732 195.0495 221.6274 246.612 265.9793 284.6275 300.2009 314.9511 326.8231 335.7044 344.3661 354.9168 363.8195 -

∞ 0.3226 0.0915 0.0726 0.0156 0.0308 0.0199 0.0098 0.0048 0.0096 0.0048 0.0095 0.0094 0.0047 -

∞ 0.5125 0.23 0.1363 0.1127 0.0785 0.0701 0.0547 0.0491 0.0377 0.0272 0.0258 0.0306 0.0251 -

∞ 0.6294 0.3976 0.533 0.1386 0.3918 0.2838 0.1783 0.0983 0.2551 0.1752 0.3674 0.3065 0.1854 -

Devstral-24B

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

230 117 68 51 39 35 31 28 23 17 16 14 13 12 -

113 49 17 12 4 4 3 5 6 1 2 1 1 0 -

113 162 179 191 195 199 202 207 213 214 216 217 218 218 -

28.4529 18.1357 11.6543 9.8368 7.7828 6.9982 6.1207 5.8086 3.9927 3.4361 2.7336 2.6001 2.4921 2.4103 -

28.4529 46.5886 58.243 68.0797 75.8625 82.8607 88.9814 94.79 98.7828 102.2188 104.9524 107.5525 110.0446 112.4548 -

∞ 0.4336 0.1049 0.067 0.0209 0.0205 0.0151 0.0248 0.029 0.0047 0.0093 0.0046 0.0046 0 -

∞ 0.6374 0.2502 0.1689 0.1143 0.0922 0.0739 0.0653 0.0421 0.0348 0.0267 0.0248 0.0232 0.0219 -

∞ 0.6803 0.4195 0.3969 0.1832 0.2224 0.2041 0.3792 0.6881 0.135 0.3495 0.1869 0.1989 0 -

GPT-5-mini

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

230 125 76 54 42 35 32 28 26 24 20 19 18 16 -

105 49 22 12 7 3 4 2 2 4 1 1 2 1 -

105 154 176 188 195 198 202 204 206 210 211 212 214 215 -

41.1859 28.9068 17.4825 12.1541 12.2516 8.5106 12.4333 14.608 9.4343 9.369 7.0953 9.1991 4.9421 6.8364 -

41.1859 70.0927 87.5752 99.7293 111.9809 120.4915 132.9248 147.5327 156.967 166.336 173.4313 182.6304 187.5725 194.4089 -

∞ 0.4666 0.1429 0.0682 0.0372 0.0154 0.0202 0.0099 0.0098 0.0194 0.0048 0.0047 0.0094 0.0047 -

∞ 0.7019 0.2494 0.1388 0.1228 0.076 0.1032 0.1099 0.0639 0.0597 0.0427 0.053 0.0271 0.0364 -

∞ 0.6648 0.5728 0.4913 0.3031 0.2024 0.1958 0.0901 0.1533 0.3253 0.1116 0.0894 0.3486 0.1282 -

Model

29

Related documents

• The Complexity Kink: A Prompt-Side Structural Complexity Index for Code-Generation Reliability#978496
• Semantic Layer Induction from Raw Telemetry via Hierarchical LLM and RAG Abstraction#978497
• The Output-Space Hypothesis: Enumerative Equivalence Checking for Tensor Programs#978498
• DeltaSelect: Affordable A/B Testing for Coding Agents#978499
• Code-as-Auditor: Executable Compliance Reasoning via Regulation-to-Code#978500