V ERI P ORT: Automated and Verified Patch Backporting at Scale Jonah Ghebremichael† * , Wenxin Jiang‡ * , Mikola Lysenko‡ , Benjamin Barslev Nielsen‡ , William Enck† , Alexandros Kapravelos†‡ † North Carolina State University — {jghebre, whenck, akaprav}@ncsu.edu
arXiv:2606.22704v1 [cs.CR] 21 Jun 2026
‡ Socket Inc. — {wenxin, mik, barslev, alexandros}@socket.dev
Abstract—One of the key challenges for securing the software supply chain is addressing known vulnerabilities in third-party open-source dependencies. Security patches are frequently only available for the latest version of a dependency, leaving developers with the choice of either upgrading to the latest version (risking breaking changes) or manually backporting the security fix. Prior work backports to a single version that must be specified in advance and does not produce sufficient evidence to demonstrate that their patches block exploitation and preserve functionality. In this paper, we present V ERI P ORT, an end-toend agentic system that scalably backports a patch for a given vulnerability advisory to every affected version of the package. For each backport, V ERI P ORT builds a chain of evidence to confirm that the patch blocks exploitation and preserves intended behavior. V ERI P ORT reliably resolves 95.3% of 128 backporting tasks in BackportBench, outperforming the best existing solution (Claude Code) by 22.7 percentage points. We further deployed V ERI P ORT on 169 high- and critical-severity CVEs and have generated over 5,000 verified backported patches. Moreover, V ERI P ORT’s value extends beyond simply backporting patches. It uncovered 2,100 versions incorrectly reported as affected and 127 previously unidentified vulnerable versions across 92 advisories, and 23 advisories have since been corrected upstream by removing 387 versions and adding 81.
1. Introduction Software vulnerabilities are being disclosed at an unprecedented rate [1], [2], [3]. This pace is accelerating as frontier Large Language Models (LLMs) become capable of autonomously discovering zero-day vulnerabilities in widely deployed software [4]. Most vulnerabilities affect more than one major version of a package, but a fix is usually only published for the latest release [5]. Developers using an older version cannot always adopt that fix by upgrading, since a newer release often introduces breaking changes [6], [7]. The alternative is backporting, which adapts the upstream fix to apply to an older version. Backporting remains a primarily human-driven task that requires an engineer to understand the vulnerability, reconstruct the upstream fix against code that has diverged from *
These authors contributed equally to this work.
the patched release, and confirm that the result still blocks the exploit without breaking existing behavior. The effort is significant enough that projects like the Linux kernel maintain dedicated teams to backport fixes to specific older releases [8]. Most projects cannot sustain this investment, making existing backports scarce. To close this gap, a line of research has emerged to automate the backporting process. Early automated backporting tools adapt a fix by matching it against a predefined model, whether as syntactic patterns or as program analysis that locates the corresponding code in the target [9], [10], [11], [12]. These tools work well when the divergence between versions follows welldefined patterns, but they cannot generalize to structural changes that fall outside their predefined rules. Each of these tools requires as input a minimal security patch (MSP): a patch containing only the changes that fix the vulnerability. An MSP rarely exists in published advisories, and even when a vulnerability-fixing commit (VFC) is linked, it can tangle the fix with unrelated changes; therefore extracting one is a hard problem in its own right [13], [14]. Moreover, these automated backporting tools judge a backport by its resemblance to a reference patch or by static checks, rather than confirming it blocks Proof-of-Concept (PoC) exploits and preserves existing functionality [15]. Together these limitations confine existing tools to a narrow slice of advisories, and require significant human effort for verification. Insights. Through our experience backporting packages, we observe that an automated backporting tool must perform four core steps. It must 1 gather rich context about the affected package, the vulnerability, and the upstream fix; 2 generate PoCs; 3 generate regression suites; and 4 isolate the MSP from non-security changes, adapt it to affected versions, and validate the proposed patch via generated tests. Each of these steps is hard because it requires reasoning about code semantics rather than matching against fixed patterns. A system must separate the security fix from the unrelated changes around it, follow that fix through versions where code has been restructured, and craft inputs that genuinely trigger the vulnerability. Modern LLMs have grown capable of exactly this kind of code reasoning, which makes them a natural foundation for this workflow [16]. Recent systems have begun to apply LLMs to this task, adapting the fix directly and reducing much of the rigidity of earlier rule-based methods [17], [18]. They generalize
backporting across the npm ecosystem, consisting of 393 backporting tasks. Running each tool three times per task, V ERI P ORT reliably solves 91.3% of tasks on all three runs against 71.1% for Claude Code. Its worst case exceeds Claude Code’s best case, by 10.7 percentage points, and its pass rate varies by only 2.3 points across runs against Claude Code’s 9.5. We perform an ablation study to isolate the contributions of AUTO P OC and AUTOT ESTER. Removing both verification stages lowers V ERI P ORT performance by 9.8 points, showing their importance. We make the following contributions: • End-to-end verified backporting. We present V ERI P ORT, the first system to backport an advisory’s fix across its entire affected range, with each backport verified by a PoC the patch blocks and a regression suite it passes. • A new benchmark. We present CVEPatchBench [23], 393 npm backporting tasks, each with a containerized environment, a working PoC, a regression suite, and a reference backport that blocks the PoC and passes the suite. • Real-world impact. We deployed V ERI P ORT in production at Socket [24], a software supply chain security company, generating over 5,000 verified backported patches [25] for 169 high- and critical-severity CVEs, each reviewed by a security engineer before release. Since V ERI P ORT runs an exploit against every release in an advisory’s affected range, it observes when a listed version is not exploitable. V ERI P ORT also probes outward from the edges of the affected version range to find unlisted versions that are exploitable. In deployment, V ERI P ORT has flagged 2,100 versions incorrectly listed as affected and 127 missed vulnerable versions across 92 advisories. We reviewed each flagged error and reported confirmed corrections upstream. At the time of writing, corrections to 23 of these advisories have been merged into the GitHub Advisory Database, removing 387 of the incorrectly listed versions and adding 81 of the missing versions.
further than prior approaches but still require an MSP and leave the same verification gaps. Using an LLM also introduces failure modes of its own. A model can hallucinate or fabricate information, producing a convincing but wrong result [19]. A model also has a fixed context limit, and approaching that limit degrades performance [20]. A single advisory spans dozens of affected versions, so regenerating every artifact per version multiplies cost and gives each failure a fresh chance to recur. Our Approach. We present V ERI P ORT, an end-to-end agentic system that takes a vulnerability advisory as input and backports the corresponding fix into a minimal, verified patch for each affected version. Our key insight is that each of the three LLM limitations above yields a corresponding design principle. First, V ERI P ORT never accepts a claim on a single model’s output alone, pairing freshcontext critic agents with deterministic checks. Second, V ERI P ORT decomposes backporting into stages, and each stage into narrowly scoped agents, so that every task fits into a bounded context. Third, V ERI P ORT generates and verifies each artifact once and then adapts it to lower affected versions, regenerating from scratch only when the code drift between versions is too large to adapt across. V ERI P ORT realizes these principles in four stages. 1 AUTO R ESEARCH reads the advisory and produces a grounded analysis of the package, the vulnerability, the upstream fix, and how the code varies across affected versions. 2 AUTO P OC builds a vulnerability oracle for each affected version: an exploit that succeeds on a vulnerable version and is blocked once the patch is applied. 3 AUTOT ESTER builds a functionality oracle for each affected version: a regression suite that exercises intended functionality on both the vulnerable and patched versions. 4 AUTO PATCH isolates the MSP, adapts it for downstream versions, and accepts a backport only when its vulnerability and functionality oracles pass. Together these stages form a chain of evidence, each piece built independently of the patch. A defective backport is accepted only by evading every check at once. Results. We compare V ERI P ORT against four baselines on the npm and PyPI tasks of BackportBench [15], a multilingual benchmark of backporting tasks. The baselines pair two generic agents, Claude Code [21] and mini-sweagent [16], with two patching tools, MagentLess [22] and the state-of-the-art agentic backporting system PortGPT [18]. We run each tool ten times per task and evaluate two settings, one that provides the upstream MSP and one that withholds it. With the MSP provided, the generic agents solve nearly every task on a single run, and V ERI P ORT leads on every metric. Withholding the MSP costs Claude Code 13.2 percentage points of single-run success and mini-sweagent 17.3, while V ERI P ORT falls only 0.8. Without the MSP, V ERI P ORT reliably solves 95.3% of tasks on all ten runs, against 72.6% for Claude Code, the strongest baseline. V ERI P ORT’s worst case across the ten runs exceeds every baseline’s best case, and it is the only tool in either setting to solve every task within ten runs. We further compare V ERI P ORT to Claude Code on CVEPatchBench, a benchmark we introduce to evaluate
2. Problem and Related Work By treating backporting primarily as a code transformation problem, prior work overlooks a key property required in practice: reliability. A reliable backport of a patch (a) fixes the vulnerability and (b) preserves existing behavior. However, in order to test reliability, the backporting tool requires an understanding of the package, vulnerability, and fix. Reliably backporting patches requires overcoming four challenges presented in the remainder of this section.
2.1. Challenge 1: Data Quality Security advisories are a practical source for context about the package, vulnerability, and fix. They often contain version ranges, fix references, PoCs, and external write-ups. However, they are also known to be noisy and therefore require validation before use in backporting.
2
TABLE 1: V ERI P ORT compared to prior automated backporting tools. ✓ denotes full support, and ✗ denotes no support. Capability Vulnerability and fix modeling Minimal-Security-Patch isolation Patch adaptation Exploit oracle Regression oracle Deterministic
V ERI P ORT (This work) ✓ ✓ ✓ ✓ ✓ ✗
denotes partial support,
PortGPT [18] (S&P 26)
Mystique [17] (FSE 25)
TSBPORT [11] (CCS 23)
SKYPORT [12] (USENIX 22)
✗ ✓
✗ ✓ ✗ ✗ ✗
✗ ✓ ✗ ✗ ✓
✗
✗
Core Difficulty. Vulnerability-fixing commits (VFCs) are missing from 63% of vulnerability database reports [13] and from roughly 93% of CVEs more broadly [26]. Even when present, 70% of VFCs span multiple correlated hunks and are often bundled with unrelated code modifications [14], [27]. Affected-version metadata is similarly unreliable. Only 59.8% of vulnerability reports match NVD’s standardized version entries [28], and over half of examined CVEs contain spurious affected-version information [29]. References are not authoritative either; 91.8% of GitHub Security Advisories pass through the review pipeline without editorial review [30]. Existing approaches and gaps. Existing automated backporting tools such as FixMorph [9], TSBPORT [11], SKYPORT [12], and PortGPT [18] assume that a minimal security patch (MSP) or known upstream security fix has already been identified. Mystique [17] further requires the vulnerable function as input. VFC-linking and silent-fix detection tools [13], [14], [31] narrow this gap by ranking candidate fixing commits or classifying commits as vulnerability fixes, but their outputs remain imperfect and commit-level. Hence they are unable to separate the MSP from the VFC. More recent work [32] has begun to test affected version ranges directly using adapted exploits. This information confirms vulnerability presence but not fix correctness, and so is insufficient for backporting on its own.
✗ ✗ ✓
FixMorph [9] (ISSTA 21) ✗ ✗ ✓ ✗ ✓
by similarity to developer-created backports, static properties, or supplied tests. A more robust approach is to generate and execute a PoC for the specific version of the code that the patch is created for. Recent work has considered LLMdriven PoC generation [33], [35], but exploit construction remains unreliable in practice.
2.3. Challenge 3: Regression Validation A backported patch requires a functionality oracle to ensure it does not break intended functionality. Project regression tests can help create such an oracle; however, they are not ubiquitous. Automatically generating regression tests is particularly subtle, because the tests need to differentiate intended behavior from unintended (vulnerable) behavior. Core Difficulty. More than 90% of analyzed npm releases ship without test code, and reconstructed suites often provide limited coverage [36]. Further, we found naive regression tests can include vulnerable behavior. Therefore, a backporting tool must identify or generate a relevant set of regression tests for a given vulnerability. Traditional automated generators [37] infer assertions from current behavior, which can encode buggy or incomplete semantics. Modern LLM-based generators [38], [39], [40] use code and naturallanguage context to infer intent, but still create invalid tests, hallucinated APIs, and flaky assertions. Existing approaches and gaps. Existing backporting tools [9], [11], [12], [17], [18] rely on developer-created backports, static checks, or supplied test suites rather than generating regression tests for the target version. Generalpurpose test generation can exercise code near the fix, but no existing approach determines which pre-patch behavior should survive and which is related to the vulnerability.
2.2. Challenge 2: Exploit Validation Since vulnerable code can shift between versions, it is insufficient to simply check if a backported patch applies cleanly. A reliable backporting tool requires a vulnerability oracle to determine if the vulnerability is actually fixed. PoCs commonly serve as such oracles [18], [33]. Core Difficulty. A working PoC must identify an attackercontrolled entry point, construct input that reaches the vulnerable behavior, rebuild the environment where the bug manifests, and define a success condition that cannot be satisfied by an unrelated crash or fabricated output. These requirements vary by vulnerability class and runtime context. Reproducing the bug may require non-default configurations or deployment settings [33], [34]. Security advisories often omit executable PoCs and describe the bug informally [35]. Existing approaches and gaps. Prior automated backporting tools [9], [11], [12], [17], [18] evaluate candidate patches
2.4. Challenge 4: Code Divergence Across versions, files move, functions are renamed or split, APIs change, and the same behavior can be implemented through different control flow. As divergence accumulates, applying the upstream diff directly can fail, patch the wrong location, or miss supporting changes that are required only in the older version. Core Difficulty. The pervasiveness of breaking changes highlights the core difficulty of backporting patches to older versions. Nearly one-third of Maven releases introduce at least one breaking change [41]. In npm, 12% of dependent
3
<3.0.3
<4.0.1
3.x.x 4.0.1
Github Security Advisory
3.1.3 4.0.0
1 Review Ad..... 1 Review Advisory 2
2
Exploit....... PoC Exploits
3 Regression Test 3 Regression Tests
3.0.8 3.0.5 3.0.0 3.0.4
<3.0.3
3.1. LLM Limitations
<4.0.1
3.x.x 4.0.1
Section 2 identified four challenges that stand between an advisory and that evidence: data quality, exploit validation, regression validation, and code divergence. The difficulty in each challenge is largely semantic, not mechanical. Resolving them requires verifying and enriching advisory data, constructing working exploits, inferring intended behavior, and restructuring a fix across code drift. LLMs can supply this semantic reasoning, but an LLM’s output is a stochastic claim, not trusted evidence. Three limitations stand between that reasoning and a trustworthy backport. Unsupported claims. An LLM can produce a convincing but wrong result. It may cite a reference that does not support its claim, mistake an unrelated crash for exploitation, or generate a test that preserves buggy behavior. Each of these mistakes can lead to a defective patch. Bounded context. A single agent that researches, exploits, tests, and patches accumulates stale assumptions, failed attempts, and irrelevant details in one long context. The context window imposes a hard limit on how much of this a model can see at once. Well before that limit, the noise competes with the facts needed for the next decision. Compounding at scale. A single advisory often spans dozens of affected versions, and reviewed npm GitHub advisories average 77 affected versions. Every artifact costs time and compute to generate. Generating each artifact independently for every version multiplies that cost by the size of the range and gives the failures above a fresh chance to surface each time. The rule- and graph-based tools sidestep all three by being deterministic, but suffer in generality (Table 1). The LLM-based tools improve in generality, but inherit the same limitations: both accept patches on checks a defective patch can pass, both adapt at a narrow granularity, PortGPT [18] hunk by hunk and Mystique [17] at the function level, and both repeat the full effort for every affected version. V ERI P ORT mitigates this tradeoff, retaining the generality of stochastic reasoning while recovering trust through evidence.
3.1.3 4.0.0 3.0.8 3.0.5
4
Backporting
3.0.0 3.0.4
Figure 1: V ERI P ORT system overview.
packages and 14% of their releases are broken by non-major dependency updates, and 44% of observed breaking changes occur in minor or patch releases [42]. These measurements understate the pervasiveness of breaking changes: semantic changes can also alter behavior but without changing API signatures [43]. Therefore, each affected version may require a different adaptation of the same underlying fix. Existing approaches and gaps. Existing backporting tools make important progress, but they each have key limitations. FixMorph [9] synthesizes transformation rules from the upstream patch and target-version alignment. This approach works when the required change fits the learned syntactic transformation, but it struggles with larger semantic or structural edits. SKYPORT [12] improves localization with graph-based program analysis to backport injectionrelated patches. However, this approach requires predefined sinks, which does not generalize to all vulnerability classes. TSBPORT [11] uses program-dependence-graph matching and patch-type-specific migration, but it is limited to predefined patterns. Mystique [17] replaces fixed rules with a fine-tuned LLM, but it operates at function granularity and assumes the corresponding vulnerable function in the target version is available as input. PortGPT [18] generalizes further with an agentic workflow and repository tools, but it fails to reason across hunks and its effectiveness drops on complex cases that require prerequisite changes, missing functions, or large structural edits. These tools show that patch adaptation is feasible, but they leave open the harder setting where one advisory must be adapted across many affected versions whose files, APIs, and code paths have drifted in different ways.
3.2. Our Approach V ERI P ORT turns unreliable semantic reasoning into trustworthy patches through three design principles, one per limitation. Evidence before trust. V ERI P ORT never accepts a claim on the model’s output alone. No agent grades its own work, and every claim must pass an independent check before anything downstream depends on it. Claims that can be executed require execution before they can be trusted. An exploit must trigger the vulnerability on a vulnerable version and fail on a patched one. A regression test must pass on both the vulnerable and patched versions. Claims that cannot be executed are audited by critic agents with fresh context that search for missing evidence, inconsistent reasoning, and invalid assumptions. When both forms of evidence exist, execution outranks judgment.
3. Overview V ERI P ORT’s goal is to produce a non-breaking security patch for every affected version of a given advisory. The obstacle is not generating these patches but trusting them. Modern LLMs can draft a plausible patch for any affected version, but a plausible patch is not a trustworthy one. A backport that breaks functionality incentivizes users to delay or skip future security updates. A backport that fails to block the exploit leaves users vulnerable while appearing safe. We call a patch with either flaw defective. A patch is therefore only useful when paired with evidence that it is not.
4
Decompose for independent verification. V ERI P ORT decomposes backporting into stages whose tasks each fit a bounded context. Each stage hands downstream a structured artifact with a short rationale rather than a conversation transcript. The same separation is what gives the first principle its force. The exploit, the tests, and the critics are constructed before the patch exists and independently of it. A defective patch ships only if it evades all of them at once. Solve once, then adapt. A key insight is that although a security advisory may have many affected versions, these versions share the same vulnerability and differ only in how the surrounding code has drifted. To take advantage of this sharing, V ERI P ORT resolves an advisory’s affected version ranges into backport chains (Definition 1), one per fixed version. A chain orders the affected versions that one fix must reach, beginning at the highest affected version, the affected version closest to that fix. V ERI P ORT generates and verifies every artifact once per chain, at the highest affected version, where divergence from the fix is smallest. It then adapts the exploit, the tests, and the patch down the chain, re-verifying on every version, so verified work is reused rather than regenerated. V ERI P ORT also maps how the vulnerability-relevant code changes across the versions of each chain, so each adaptation works from the actual differences it must bridge. Together, these structures reduce each additional version to a small, verified adaptation step and let backporting scale reliably across an advisory’s full affected range.
Definition 2 (Minimal Security Patch). Let the upstream fix be the diff between the fixed-vulnerable pair (vf , vn ) (Def. 1), partitioned into hunks H = {h1 , . . . , hm }. Each hunk is either a security hunk, one required to block the vulnerability, or a non-security hunk. The minimal security patch M ⊆ H is the set of security hunks. Section 4 details how each stage within V ERI P ORT assembles this evidence.
4. V ERI P ORT V ERI P ORT takes a vulnerability advisory and produces a verified backport for every affected version, decomposing the task into the four stages of Figure 2. AUTO R ESEARCH grounds the advisory, AUTO P OC and AUTOT ESTER build a vulnerability and a functionality oracle, and AUTO PATCH isolates and adapts the fix. We call a backport verified when an executable exploit can no longer trigger the vulnerability on it and an executable regression suite still passes; this is evidence from two oracles, not a proof, since each oracle witnesses specific behavior rather than the absence of every vulnerable path. Each stage targets one of the backporting challenges of Section 2 while addressing the LLM limitations discussed in Section 3, and deterministic scaffolding (git-apply ladder, fingerprinting, scope bounding, and flag checking) bounds where stochastic agents may act. We instantiate V ERI P ORT for the npm ecosystem and use the advisory GHSA-4jqc-8m5r-9rpr for set-value as a running example throughout this section.
Definition 1 (Backport Chain). Let vf be a fixed version and v1 . . . vn be the affected version range for vf . The backport chain for vf is the sequence (vf , vn , . . . , v1 ). We refer to vn as the highest affected version and to the pair (vf , vn ) as the fixed-vulnerable pair.
4.1. AUTO R ESEARCH Advisory data includes raw information about the package, the vulnerability, and the fix that later stages depend on. This information is often unreliable, with incorrect version ranges, imprecise summaries, and references that point to external sources whose own PoCs and fix descriptions may be inaccurate. The goal of AUTO R ESEARCH is to produce a vulnerability manifest that verifies and extends this information and places the advisory in a normalized form the rest of the pipeline consumes.
These principles shape the four stages of Figure 1, each addressing one challenge from Section 2. 1 AUTO R E SEARCH addresses data quality. It verifies and enriches the advisory’s data, identifies the security-relevant portions of the upstream fix, and resolves the affected ranges into backport chains. It emits two artifacts that every later stage consumes. The vulnerability manifest holds the verified and enriched advisory data. The cross-version map records how the vulnerable code and its surroundings change across the versions of each chain. 2 AUTO P OC addresses exploit validation. It generates executable PoCs from the manifest and verifies each against the fixed-vulnerable pair, requiring success on the vulnerable build and failure on the patched build. The verified PoCs establish which versions are actually exploitable and serve as the vulnerability oracle for every candidate backport. 3 AUTOT ESTER addresses regression validation. It generates regression tests that pin the intended behavior around the vulnerable code path without preserving the vulnerability itself, and verifies that they pass on both builds. These tests form the functionality oracle. 4 AUTO PATCH addresses code divergence. AUTO PATCH first isolates the minimal security patch (MSP). It then adapts the MSP down each chain and accepts a backport only when it clears both oracles.
4.1.1. Advisory Refinement. AUTO R ESEARCH compares the vulnerability the advisory describes against the vulnerability as it appears in source. Because the vulnerability can manifest differently across versions, and showing an agent every version would overload its context (Section 3), AUTO R ESEARCH provides the vuln-analyzer agent 1 with the fixed-vulnerable pair (Def. 1) for a given backport chain. The agent also reads the advisory description, CWE, severity, and references. The references often carry the richest evidence, since they can link issues containing PoCs, vulnerability-fixing commits (VFCs), and pull requests in which the maintainers explain the fix. A listed reference can link onward to richer evidence than it contains, and valuable sources such as a security write-up’s proof of concept may go unlisted in the advisory. A deep-research
5
A
AutoResearch
START BACKPORTING
GENERATE ONCE per backporting chain => [email protected] -> 3.0.4, [email protected] -> 3.0.0, [email protected] -> 0.0.0
B
AutoPoc
GHSA-4jqc-8m5r-9rpr npm set-value · prototype pollution
5 poc-builder
10 scope restricter
1 vuln-analyzer
a build vuln env [email protected]
a fetch refs · read CWE-1321
b write canonical run.sh
b write vuln_manifest
critic ⊳ 9-item build checklist
critic ⊳ Grounding Cert. (G1–G5)
critic ⊳ Classification Cert. (H1–H4)
3 fingerprint
DETERMINISTIC
SHA-256 ( package.json · sorted paths )
4 version-analyzer
b fresh {} polluted ⇒ ^FLAG=
b grade majors · build 3 chains critic ⊳ Consistency Cert. (C1–C4)
produced once by AutoResearch · read by every stage
JSON
package summary
MD
vuln summary
entry points
MD
fix summary
JSON
code paths
Cross-version Map JSON
fix applicability
JSON
bundle analysis
Backport Chains
fix → highest-affected (=representative) ⋯ lowest
fix 2.0.1
2.0.0
⋯ 0.0.0
fix 3.0.3
3.0.2
⋯ 3.0.0
fix 4.0.1
4.0.0
⋯ 3.0.4
3 git apply --3way 4 structural acorn apply
16 patch-adapter a adapt MSP to drifted code b synthesize guard where absent c retry · re-clear two-sided gate
DETERMINISTIC
# STDOUT (vuln @4.0.0) ^FLAG=CTF_FLAG_23c8… # patched → blocked
▸ functionality oracle (4.0.0)
MULTI-VERSION ADAPTATION
Applicable?
9 poc-adapter
direct
13 test-adapter
adapt PoC → re-test on vuln build
vulnerability oracle (vX.Y)
E
inputs
adapted
candidate vuln oracle (vX.Y)
▸ minimal security patch (4.0.0)
incompatible
Vulnerability Manifest MD
2 git apply --fuzz
DETERMINISTIC
critic ⊳ Eval Cert. (V1–V6 · flag)
▸ PoC oracle (4.0.0)
Shared Artifacts
DETERMINISTIC
1 git apply --exact
# run suite on vuln build cover the patch-touched region 92 / 92 pass
a verifier (read-only)
8 Flag Checker
a trace set() → unsafe-key sink
15 apply ladder
critic ⊳ Functionality Cert. (D1–D5)
12 coverage gate
14 patch-isolator
critic ⊳ Minimality Cert. (P1–P4)
c drop tests that flip w/ patch
7 poc-verifier
D
c write minimal_hunks/ · syntax gate
b exercise a range of inputs
critic ⊳ Eval Cert. (V1–V4 · E1–E4)
b 3 hunks → 1 security hunk
func_O
b keep String(key) hunk
a tests on set() · fix-region fns
b set(o,[['__proto__']],v) → flag
vul_O
a git 4.0.0 + 4.0.1 tags
11 test-generator
a write exploit.js · set-value
a review vuln → patched source
AutoPatch
DETERMINISTIC
# entry points + code paths classified hunks → fix region target functions to test
6 poc-exploiter
2 diff-classifier
C
AutoTester
17 patch-adapter
adapt suite → re-test on vuln build
PASS
FAIL
candidate func oracle (vX.Y)
vul_O
functionality oracle (vX.Y)
PASS
backport MSP → rewrite hunk sites
FAIL
func_O
candidate backport patch (vX.Y)
▸ backport patch (vX.Y)
PASS
FAIL
END
Figure 2: V ERI P ORT system design. GHSA-4jqc-8m5r-9rpr is provided as a running example. fingerprints each version’s shape 3 as a SHA-256 over its package.json entry fields and sorted file paths, and groups versions whose fingerprints match. The fingerprint is deterministic and content-blind: a shared fingerprint means two versions have the same file layout and package entry structure, not necessarily identical contents. Reusing a representative’s structure report across its group is therefore sound, whereas reusing its content-dependent code-path and fix-applicability reports is a heuristic that assumes versions of identical shape expose the vulnerability alike. V ERI P ORT accepts this gap because the fingerprint captures exactly the structural divergence (Section 2.4) that defeats mechanical reuse, and because every reused artifact is later rechecked by execution on its own version, so an unsound reuse surfaces as a failed oracle rather than a silent error. A cosmetic layout change instead over-fragments a group, wasting analysis effort but never merging versions that truly diverge in shape. V ERI P ORT analyzes one representative per group rather than once per version.
step within the vuln-analyzer workflow follows links deeper within a reference and out to related sources to gather this evidence, bounded to a fixed link depth and a curated allowlist of authoritative domains. For example, the advisory for CVE-2025-55182, a deserialization flaw in React Server Components, listed a third-party repository as its proof-ofconcept reference; by reading that repository’s issues alongside the upstream fix, deep-research found that the listed exploit did not trigger the vulnerability. Left unchecked, that reference would have reached AUTO P OC as ground truth and produced an exploit for the wrong code path. The end result is the vulnerability manifest, which records the package, vulnerability, and fix summaries together with the package’s entry points and a structured report of every reference reviewed and its relevance. 4.1.2. Version Analysis. Adapting the patch, PoC, and regression suite to every affected version requires knowing what each version contains and how far it has drifted from the fix. Analyzing every version individually would be wasteful, since affected versions can be structurally identical and differ in only a few lines. V ERI P ORT therefore
A diff-classifier agent 2 makes a pass over the hunks H of the upstream fix (Def. 2) and gives each a preliminary security or non-security label. When the vulnerability
6
nerable build, but not that it is blocked once the patch is applied, which is the gate AUTO P OC adds next. Verification. Automated exploit generation is prone to false positives, exploits that satisfy a validator without ever triggering the vulnerability [33], [35]. Like CVEGenie [33], AUTO P OC has an LLM author a flag-based verifier whose embedded flag is read by a deterministic, nonLLM check 8 . AUTO P OC adds a second gate against this, requiring the same exploit that fires on the vulnerable build also be blocked once the patch is applied. A check is only as honest as the build it runs on, so the harness rather than the agent installs the exact version, applies the patch, and rejects any run where the exploit reinstalled the vulnerable code or the patched code never loaded. Because the patched build passes when the exploit does not fire, differential guards ensure a pre-existing crash, an infrastructure failure, or a verdict with no executed checks is never taken as a pass. The result is the vulnerability oracle for the highest affected version of the chain, which establishes this exploit is blocked on the patched build rather than that no exploit remains. When AUTO P OC cannot build a working exploit for a version, that version carries no vulnerability oracle, and AUTO PATCH reports its backport as unverified rather than certifying it. The exploit for GHSA-4jqc-8m5r-9rpr wraps an unsafe key in an array to slip past a blocklist an earlier fix added. On the vulnerable build it pollutes a fresh object and AUTO P OC captures its flag, while the patched build coerces the key to a string the blocklist rejects, so the object stays clean. The flag is gated on the observed pollution, so a build that never pollutes cannot pass verification.
manifest recovered a fixing commit, this classification is scoped to the files that commit touched; when no fixing commit was recovered, the common case for advisories (Section 2.1), the classifier instead ranges over the full fix diff. This triage is fast rather than exhaustive, producing only a rough estimate of M; the authoritative isolation happens later in AUTO PATCH (Section 4.4). Analyzing Each Group. A version-analyzer agent 4 then examines one group representative, comparing it against the fixed version using the security hunks and the manifest’s entry points, and emits three reports. A structure report records how the package is laid out and built, such as its entry fields and whether it ships built artifacts a patch must also reach. A code-path report traces each entry point to the vulnerable sink and confirms the vulnerability is reachable in that group. A fix-applicability report grades how the fix lands as direct, adapted, or incompatible, together with the concrete differences behind the grade. For GHSA-4jqc-8m5r-9rpr, the analyzer returns adapted for 0.x and direct for the other three groups. The 3.x group is graded direct because its representative already defines the guard the fix hardens, isValidKey, so the coercion applies in place. The 0.x group is graded adapted for a sharper reason, since its representative lacks isValidKey entirely, so the fix cannot be transplanted and must be synthesized against a different key-splitting path. V ERI P ORT stitches these reports into the cross-version map, which tells each later stage which files and code paths the fix must touch in a given version and what must change for it to land.
4.2. AUTO P OC
4.2.2. Multi-Version Adaptation. AUTO P OC carries each vulnerability oracle across the rest of the affected range for its backport chain rather than re-running the build→exploit→verify chain for each version. It reuses AU TO R ESEARCH ’s fingerprint groups (Section 4.1.2), validating the oracle once per group representative and adapting it to that group’s structural siblings. As the versions share one vulnerability and differ only in structure, the work left for each is to fit the exploit to its own code, and AUTO R ESEARCH’s cross-version map has already graded how far each has drifted. Where the surface is unchanged (direct), the validated exploit is reused verbatim and a single agent-free harness run certifies the version if it still passes, falling back to adaptation if it does not. Where references have drifted (adapted), a pocadapter agent 9 rewrites only what moved, such as import paths, function signatures, and return shapes, and leaves the verified exploit logic intact. Only when the package has been reorganized enough that the fix no longer maps (incompatible) does AUTO P OC regenerate a fresh oracle for the group representative and adapt inward to its siblings. For GHSA-4jqc-8m5r-9rpr, the exploit drives only the package’s public entry point, which is stable across the range, so AUTO P OC reuses it on every version and re-checks the two-sided result rather than regenerating an oracle.
A backported patch that applies and even builds correctly can still fail to truly block the vulnerability. A Proof-ofConcept exploit (PoC) that reproduces the vulnerable behavior provides executable evidence that a patch correctly mitigates the stated vulnerability. V ERI P ORT originally had a single agent write this exploit from the advisory, which worked for simple cases but proved brittle across projects and could not handle incorrect affected ranges. Reliable PoC generation instead has to build a working environment, craft an exploit that genuinely triggers the vulnerability, and verify that the exploit fired while rejecting false positives. CVE-Genie [33] covers these three steps with a build→exploit→verify chain of agents that reproduces a vulnerability for a particular CVE. AUTO P OC extends this chain for backporting, producing a vulnerability oracle for each backport: a PoC that fires on a vulnerable version’s build and is blocked on the patched version. 4.2.1. PoC Generation. AUTO P OC uses the vulnerability manifest, cross-version map, and a fixed-vulnerable pair per backport chain, rather than the raw advisory. A poc-builder agent 5 stands up a working environment for the highest affected version. Then a poc-exploiter agent 6 writes an attack against it, anchored by the vulnerable path and sink extracted in the vulnerability manifest. CVE-Genie’s pocverifier agent 7 confirms that the exploit fires on the vul-
7
Advisory Range Correction. Published advisories routinely misstate the affected version range in both directions, listing false-positive releases that are not actually affected and omitting false-negative releases that are. Pinning down either error requires running a real exploit against the release and observing whether it fires. False positives surface naturally while AUTO P OC builds its per-version oracles across the affected range. When an oracle verified on a neighboring affected version fails to trigger on a listed version, and AUTO R ESEARCH also grades that version structurally incompatible with the vulnerable path, V ERI P ORT flags the release as a candidate false positive. For example, the advisory GHSA-vg7j-7cwx-8wgw for the widely used mongoose package marks much of its release history as affected, yet V ERI P ORT’s oracle does not fire on 422 of these releases, the kind of over-approximation that motivates AUTO R ESEARCH (Section 2.1). False negatives are harder, since confirming them requires oracles for versions outside the affected range, and testing every such version naively would be prohibitively expensive. AUTO P OC instead runs a differential boundary search that probes the immediate neighbors of each edge of the affected range. At the lower edge it walks outward one version at a time while the oracle keeps firing, stopping at the first release it cannot exploit, and that last exploitable release becomes the corrected boundary. At the upper edge it instead probes only the single release just above the fix: a vulnerability surviving the stated fix usually signals a broken patch rather than a wider range, so V ERI P ORT deliberately favors precision over recall here and does not walk further, accepting the risk of missing a true upperedge false negative. A clean result confirms the boundary, while an exploit there is flagged as a likely incomplete upstream fix for review rather than widening the range automatically. For example, the advisory for the vm2 sandbox escape (GHSA-ffh4-j6h5-pg66) initially named a single affected release, but V ERI P ORT’s oracle fired on earlier releases as well, revealing that the escape was present well before the named version; the upstream advisory now reflects this wider range.
and esm bundles. The agent refines the suite over several rounds, guided by the critic’s feedback and a coverage gate 12 that each run reports. Focused Generation. A patch is most likely to perturb behavior near the code it changes (Section 2.3), so for large packages a scope restricter 10 confines generation to that region rather than the full public API, where exercising every entry point would be prohibitively expensive. This focusing is gated on package size, taking effect once a package exceeds a configurable file-count threshold, 50 source files in our deployment. We set the cutoff well above the size of a typical npm package, which carries on the order of 20 source files, so the common case is still tested comprehensively and only the large frameworks above it, where comprehensive generation runs to millions of tokens, are scoped down. Package sizes are heavily skewed into a cheap majority and a costly tail, so the precise cutoff matters little provided it separates the two regimes, which is why we leave it configurable rather than tuned. For a focused run, it reads AUTO R ESEARCH’s entry points and traced code paths to find which functions to target and the classified hunks to see what the fix changes, then exercises those functions with a range of inputs to cover the behavior the fix can affect. Because these are exactly the paths a patch can perturb, a backport that over-restricts the fixed code is caught here; behavior far from the fix is out of scope by design, on the assumption that a minimal security patch leaves it untouched. A Patch-Invariant Oracle. A regression test must stay independent of the fix, so V ERI P ORT keeps a test only when it passes on both the vulnerable and the patched build, making the suite patch-invariant. A test that fails on the vulnerable build reflects a pre-existing failure rather than a regression. A test that asserts on the behavior the patch changes is the opposite hazard, flipping from pass to fail once the fix lands and mistaking the fix for a regression, so the generator and its critic drop it, leaving the suite invariant by construction. V ERI P ORT then verifies this rather than assuming it. After AUTO PATCH applies the backport, it re-runs the suite on the patched build. A failure is now unambiguous. One that appears only after the patch is a genuine regression, while one present on both builds is a preexisting failure, set aside rather than blamed on the patch. The functionality oracle for GHSA-4jqc-8m5r-9rpr exercises the package’s ordinary key-setting behavior, such as nested writes like set(obj, ’a.b.c’, v). Its security-adjacent tests use string keys like __proto__, which the blocklist already rejects on both the vulnerable and the patched build. Each such test therefore yields the same verdict on either side of the fix. A test built on the array-wrapped key the exploit relies on would instead flip between builds, since that key pollutes before the fix and is rejected after it. AUTOT ESTER’s critic re-derives the fix and confirms that no test depends on this path, leaving the suite patch-invariant by construction.
4.3. AUTOT ESTER A backported patch that blocks the vulnerability can still break intended functionality within the package. A regression suite guards against this by checking that a patch preserves existing safe behavior. However, crafting these tests is subtle because a security regression test must distinguish between the behavior the patch removes and the intended behavior it preserves. AUTOT ESTER produces a functionality oracle for each backport, a regression suite that passes on both the vulnerable and patched builds. 4.3.1. Test Generation. A test-generator agent 11 grounded in AUTO R ESEARCH’s artifacts writes tests that exercise the package’s public API, and a separate critic agent reviews them. These tests run against both the package source and the build artifacts it ships, such as npm’s dist
4.3.2. Multi-Version Adaptation. AUTOT ESTER carries the functionality oracle across the affected range with the
8
holds as an immutable reference to the original fix. The nonsecurity hunks cover the refactoring, formatting, and unrelated edits the version bump happens to include. The agent grounds this classification in AUTO R ESEARCH’s compiled evidence (package, vulnerability, and fix summaries, code paths) and confirms it by exploring the two versions, after which a critic re-examines it against the same evidence and either accepts it or returns it for another round. Because the classification is not executable, M is a textual estimate rather than a proven minimum, and execution during backporting can still reveal it to be incomplete. Of GHSA-4jqc-8m5r-9rpr’s three chains, we follow the chain for 3.0.3 (affected versions 3.0.0–3.0.2). The upstream diff bundles README and package.json churn with the fix. The isolator keeps a single four-line hunk, the String(key) coercion, as M.
same hierarchical strategy as AUTO P OC (Section 4.2.2), driven by the cross-version map’s direct, adapted, and incompatible grades. Where the surface is unchanged (direct), the suite is reused as is and a single harness run certifies the version if it still passes. Where references have drifted (adapted), a test-adapter 13 subagent rewrites the moved function signatures, import paths, and return shapes that the cross-version map records, and leaves the assertions intact. Holding the assertions fixed is what carries patchinvariance to the adapted version, since the adapter changes only how the suite reaches the API and not the behavior it checks. Only when the package has been reorganized enough that the suite no longer maps (incompatible) does AUTOT ESTER regenerate a fresh suite for the group representative and adapt inward to its siblings. Each adapted suite must pass on its own vulnerable build to become that version’s functionality oracle, and its patched-build check completes once AUTO PATCH applies that version’s patch, as with the representative version. For GHSA-4jqc-8m5r-9rpr, set-value exposes one stable set(obj, path, value) entry point across all four groups. The functionality oracle is therefore reused without adaptation: each version is certified by a single harness run, and only its patched-build re-check waits until that version’s backport lands. Had the public surface drifted in some group, the test-adapter would rewrite only how the tests call the API, leaving the set(obj, ’a.b.c’, v) assertions untouched, so the suite checks the same behavior and returns the same verdict on both builds.
4.4.2. Backporting the Highest Affected Version. The patch adapter agent 16 reshapes M to fit the highest affected version vn (Def. 1), whose code might have drifted from the upstream fix. It works in an editable checkout of vn and keeps the immutable fixed-vulnerable checkouts available for comparison. For each security hunk in M, the agent produces an adapted hunk that fits vn ’s code. The agent never edits M itself, so the original fix stays intact for the rest of the chain. Unlike the hunk classification in isolation, a candidate backport can be executed, so AUTO PATCH verifies it with the two oracles instead of a critic. The vulnerability oracle must block the exploit on the patched build and the functionality oracle must pass, and the agent revises the candidate and retries whenever either fails. A failure can reveal that M does not fit this version, either because one of its security hunks does not apply or because a hunk that isolation labeled non-security is in fact needed here. The agent responds by adding or removing hunks from the candidate, never by editing M itself, so execution refines the security-relevant set that isolation could only estimate textually. The accepted adapted hunks form the backport for vn . One release below the fix, the highest affected version 3.0.2 already defines isValidKey. The MSP applies as written and clears both oracles.
4.4. AUTO PATCH AUTO PATCH produces a verified backport for each affected version of the package. A natural design has a single agent adapt the fix directly from the vulnerable and patched versions, but we found that such an agent overlooks changes that appear unrelated to the vulnerability and loses track of the original fix on a long chain, leaving the backport incomplete. AUTO PATCH instead separates patch isolation from patch adaptation and uses three steps for each chain. (1) It isolates the minimal security patch M (Def. 2) from the upstream fix. (2) It adapts M into a candidate backport for the highest affected version vn (Def. 1) and accepts it only when the vulnerability oracle blocks the exploit and the functionality oracle still passes. (3) It carries the verified backport across the remaining versions in the chain.
4.4.3. Multi-Version Backporting. AUTO PATCH carries the verified backport from vn toward v1 along the chain (Def. 1), using the same hierarchical strategy as AU TO P OC and AUTOT ESTER , driven by the cross-version map’s direct, adapted, and incompatible grades. To land the backport on each version, AUTO PATCH climbs an escalating git apply ladder 15 and takes the first rung that produces a syntactically valid, oracleverified patch. The lower, agent-free rungs run git apply with progressively looser matching (exact, fuzzed context, then three-way merge), followed by a structural pass that relocates each hunk by its parsed context and rewrites renamed identifiers; only when all of these fail does the top rung invoke the patch adapter agent 17 . Versions whose surface is unchanged (direct) typically clear a mechanical
4.4.1. Patch Isolation. AUTO PATCH reuses the vulnerability manifest and cross-version map that AUTO R ESEARCH produced, which describe the vulnerability, the package, and the upstream fix. It works in a dedicated Git workspace that holds two immutable checkouts, the fixed-vulnerable pair (vf , vn ) (Def. 1), which the agents read but never modify. AUTO PATCH deterministically diffs the pair into the hunks H of the upstream fix (Def. 2). The patch isolator agent 14 labels each hunk in H as a security hunk or a non-security hunk. The accepted security hunks form M, the MSP for the chain, which AUTO PATCH
9
rung and apply without an agent, whereas versions whose references have drifted (adapted) fall through to the patch adapter, which rewrites how each hunk lands in the version, and leaves M intact. Grounding each adaptation in M’s immutable hunks rather than the backport it starts from is what keeps the fix faithful across a long chain, since an agent that adapts from an already-backported version would otherwise inherit its drift and lose track of the original fix. Only when the package has been reorganized enough that the backport no longer maps (incompatible) does AUTO PATCH adapt M afresh for the group representative and adapt inward to its siblings. A version’s backport is accepted only once that version’s vulnerability oracle blocks the exploit and its functionality oracle passes on the patched build, as at vn ; a version that never clears both oracles is reported as an unverified backport rather than shipped as a fix, so a failure is surfaced rather than hidden. Once every version holds a verified backport the chain is complete, and repeating this across all chains yields a verified fix for every affected version. On 3.0.1 the hunk applies unchanged. Version 3.0.0 inherits the group’s direct grade through its content-blind fingerprint, but isValidKey does not exist there. The mechanical rungs fail, so the adapter synthesizes the guard and filters the parsed keys through it. Both oracles accept the result even though it no longer resembles the upstream fix. The grade routes the attempt; the oracles decide.
sons discussed in the corresponding subsections, RQ1 uses BackportBench, while RQ2 and RQ3 use CVEPatchBench, a benchmark we created for this paper. Additionally, RQ1 compares V ERI P ORT to two patching tools and two generic agents. RQ2 and RQ3 only compare to Claude Code, the best-performing baseline from RQ1. 5.1.1. Benchmarks. Our evaluation uses two benchmarks: BackportBench [15] is a repository-level, executionvalidated benchmark for backporting. It is constructed from existing vulnerability-fixing commits and their maintainer backports. Each task evaluates whether a tool can adapt an existing upstream fix to a historical target version for which a maintainer-written backport is already available. The released benchmark contains 202 tasks from 12 repositories across three ecosystems: PyPI, Maven, and npm. Each task consists of an upstream (golden) patch, an unpatched target version, a containerized execution environment, and two test lists: Fail-to-Pass (F2P) tests that encode the expected fix behavior and Pass-to-Pass (P2P) tests that guard against regressions. A task passes when all F2P and P2P tests pass. We excluded two types of tasks from BackportBench: (1) 67 Maven tasks, as V ERI P ORT does not currently support Java, (2) 7 tasks (2 PyPI and 5 npm) for which we could not reproduce a passing result from the provided inputs. Appendix B lists each excluded task and its reason. The resulting BackportBench evaluation set contains 110 PyPI tasks and 18 npm tasks. CVEPatchBench is a benchmark we created to evaluate npm backporting more broadly than BackportBench supports. BackportBench constructs tasks from maintainerwritten backports, and npm maintainers rarely backport [44], so its released set contains only 23 npm tasks from four repositories covering 9 CVEs and 11 CWEs. CVEPatchBench instead was created by combining output from a preliminary version of V ERI P ORT and significant human manual curation. For each included task, a human security expert (1) reviewed the PoC and regression test suite, (2) edited or replaced incorrect artifacts, (3) confirmed the final PoC triggers the vulnerable release, and (4) confirmed the PoC is blocked by the patched reference release. The expert also ensured the regression suite exercises package behavior that should be preserved. This lets CVEPatchBench cover 393 npm tasks from 44 repositories, spanning 60 CVEs and 33 CWEs. Each CVEPatchBench task contains: (1) a vulnerable package release, (2) a GitHub security advisory (GHSA), (3) a containerized execution environment, (4) a proofof-concept exploit, (5) a regression test suite, and (6) a reference backport that blocks the PoC and passes the regression suite. CVEPatchBench resolves each GHSA affected range to stable npm releases, groups them by affected major version, and selects the lowest, median, and highest affected release in each major line when those releases exist. This design gives CVEPatchBench greater affected-range coverage than BackportBench while keeping the benchmark small enough for human review. CVEPatchBench evaluates 6.5 versions per advisory on average versus 1.8 for Back-
4.4.4. Patch Refinement. A backport is accepted once it clears both oracles, but that verdict can rest on incomplete evidence, for instance when a stronger PoC later exposes a residual bypass or a functionality failure surfaces only after the patch lands. When such a failure appears, AUTO PATCH enters a refinement loop rather than discarding the backport. It forks a new revision from the failing one, never mutating the accepted record, and runs a bounded fix-and-verify loop guided by the failure. A surviving exploit directs it to harden the gate, whereas a regression with the exploit still blocked directs it to repair functionality without weakening the security fix. The loop is fail-closed, so a revision is promoted only when it re-clears both oracles, and an iteration that reproduces the prior patch unchanged is rejected rather than accepted as progress.
5. Evaluation This section evaluates V ERI P ORT by answering the following research questions. RQ1 How effective is V ERI P ORT relative to existing backporting techniques? RQ2 How effective is V ERI P ORT at backporting across the npm ecosystem? RQ3 How much does each component of V ERI P ORT contribute to backporting effectiveness?
5.1. Experimental Setup We now define the benchmarks, task inputs, comparison tools, and metrics used throughout the evaluation. For rea-
10
TABLE 2: BackportBench results pooled across npm and PyPI. Time is mean wall-clock seconds per task, weighted by task count (110 PyPI, 18 npm).
portBench. It covers 19.6% of affected ranges on average versus 7.5%. 5.1.2. Comparison Tools. We compare V ERI P ORT against four baselines. All tools run in isolated containers with identical CPU, memory, and wall-clock limits. RQ1 experiments use Claude Opus 4.7 [45] and RQ2 and RQ3 experiments use Claude Opus 4.6 [46], the strongest coding models available at the time of each experiment. Claude Code [21] is Anthropic’s production agentic coding tool. Since V ERI P ORT is implemented on top of Claude Code, this baseline isolates the contribution of V ERI P ORT’s task decomposition, artifact contracts, critics, and verification workflow beyond the underlying coding agent. mini-swe-agent [16] is the maintained successor to SWEagent, an agent-computer-interface system designed for repository-level automated repair, where an LLM navigates a codebase, edits files, and runs tests to produce patches for GitHub/SWE-bench-style issues. mini-swe-agent preserves this repair-oriented workflow while reducing the interface to a lightweight bash-only tool. We use the official swebench configuration distributed with mini-swe-agent. MagentLess [22] is a multilingual adaptation of Agentless [47], an llm-based procedural repair tool that localizes relevant files and functions from a repository and generates edit operations. We adapt MagentLess for backporting by providing the upstream patch as the problem description and converting its edit output into a unified diff for evaluation. PortGPT [18] is the state-of-the-art academic agentic backporting tool that uses a two-stage hunk-adaptation and patch-validation workflow with tools for code viewing, Git-history tracing, hunk application, and compile feedback. We extended PortGPT’s CompileTest stage with three language-native parse-level validators: node --check (JavaScript), tsc --noEmit (TypeScript), and py_compile (Python). Excluded Systems. We considered direct comparisons with prior automated backporting systems, including FixMorph [9], TSBPORT [11], SKYPORT [12], and Mystique [17]. These systems are not straightforward to adapt to Python and JavaScript, because their core algorithms are coupled to language-specific analyses. Porting them would therefore require reimplementing front-end parsers, symbol analysis, patch-type rules, and validation harnesses for Python and JavaScript, which would constitute a new system rather than a faithful baseline adaptation.
pass10
pass@10
Time (s)
Patch Adaptation (with MSP) V ERI P ORT 99.6% 98.4% Claude Code 97.4% 95.3% mini-swe-agent 97.3% 89.8% PortGPT 89.0% 80.5% MagentLess 63.6% 50.8% git apply 40.6% –
100.0% 99.2% 99.2% 93.0% 75.0% –
271 174 60 71 18 0.02
Patch Isolation and Adaptation (without MSP) V ERI P ORT 98.8% 95.3% 100.0% Claude Code 84.2% 72.6% 91.4% mini-swe-agent 80.0% 67.2% 90.6%
323 179 77
Tool
pass1
we report pass1 = Ei [ci /n] as expected single-run success. We report passk = Ei cki / nk , which estimates the probability that all k sampled nruns solve the task. We i report pass@k = 1 − Ei n−c / k k , which estimates the probability that at least one of k sampled runs solves the task. We use pass10 to represent reliability and pass@10 as best-of-ten recall. Benchmark Execution Settings. We evaluated BackportBench in two settings. (1) In the Patch Adaptation setting, each tool receives the benchmark-provided upstream MSP to adapt to its target version. This setting implicitly isolates the patch requiring adaptation, because the tool does not need to infer the MSP from the patched release. We also report a git apply baseline, which applies the provided MSP directly without any adaptation. (2) In the Patch Isolation and Adaptation setting, each tool receives directories with the vulnerable and patched versions along with the advisory metadata. For this setting, we remove any fixing-commit URLs to ensure the evaluated tool recovers the security fix from the difference between the two versions. This setting requires each tool to isolate the MSP itself before adapting it. We only evaluate V ERI P ORT, Claude Code, and mini-swe-agent in this setting, as PortGPT and MagentLess require the MSP as input and cannot infer it. 5.2.1. Patch Adaptation (with MSP). V ERI P ORT exceeds every baseline on every metric (Table 2). The margins are small because this setting is nearly saturated. git apply alone resolves 40.6% of tasks, and both general agents exceed 97% pass1 . The repeated-run metrics expose differences that pass1 hides. mini-swe-agent reaches 97.3% pass1 but drops to 89.8% pass10 , so its single-run success does not survive repeated trials. V ERI P ORT holds the highest pass10 at 98.4%, 3.1 points above Claude Code, and is the only tool to solve every task within ten runs (100% pass@10).
5.2. RQ1: Comparison to Existing Techniques We compared V ERI P ORT against the baseline tools using BackportBench, as it provides outside validation of our tool. We ran each tool ten times per task and measured both single-run success and reliability across runs rather than judging it on a single outcome. Multiple runs are necessary to measure reliability rather than best-case success due to the stochastic nature of LLM agents [33], [48]. Repeated-run metrics. Let ci be the number of successful runs for task i out of n = 10 runs. Following τ -bench [48],
Takeaway. With the MSP provided, single-run success of BackportBench is saturated and reliability is what separates tools.
11
TABLE 3: CVEPatchBench results across 393 tasks with three runs per task. Each ablation removes the named oracle from V ERI P ORT. Time is mean wall-clock seconds per advisory.
5.2.2. Patch Isolation and Adaptation (without MSP). The Patch Adaptation setting hands each tool the MSP, which advisories rarely provide. This setting withholds it. A tool must first recover the fix from the full difference between the vulnerable and patched versions before adapting it. V ERI P ORT is nearly unchanged between the two settings, falling 0.8 points under pass1 pooled across both ecosystems, while Claude Code falls 13.2 points and miniswe-agent falls 17.3 (Table 2). The drops are consistent across npm and PyPI (Table 4). V ERI P ORT’s worst case across ten runs exceeds every baseline’s best case on both ecosystems, with a pass10 of 94.4% on npm and 95.5% on PyPI against a best baseline pass@10 of 88.9% and 91.8%. Observations. The drop in performance does not come from running out of resources. Every mini-swe-agent and Claude Code failure we examined stopped on its own, several costing less than the corresponding run with the MSP provided. Neither agent builds a check of its own, so nothing in the loop contradicts a confident but incomplete fix. Two failure shapes account for the lost tasks, both from reading the fix off the patched version. In the first, the agent ports too little. On django_261, both agents remove the AJAX bypass named in the advisory but miss the compensating change that lets legitimate callers satisfy the check through a request header, which is the behavior the test verifies. In the second, the agent ports too much. On django_193, both agents transplant host-allowlist machinery from a later major version onto the older target and break existing behavior. On six tasks, both agents fail every run while V ERI P ORT succeeds on all ten, and each fix requires a hunk in a file the advisory never names. The agents scope their search to the symbols the advisory mentions, so an incomplete advisory leaves them blind to a needed hunk. V ERI P ORT classifies every hunk in the full version diff and gates each patch on both oracles, recovering the hunks an advisory omits and rejecting the over-wide ports that break behavior.
Condition
pass1
pass3
pass@3
Time (s)
V ERI P ORT
92.9%
91.3%
93.6%
627
88.9% 84.4% 83.1%
87.5% 81.7% 78.9%
90.1% 86.8% 86.8%
498 524 398
76.7%
71.1%
80.6%
313
w/o AUTOT ESTER w/o AUTO P OC w/o both Claude Code
10.7 percentage points better than Claude Code’s best case (pass@3). V ERI P ORT also has much smaller variance across the three runs: 2.3 percentage points vs Claude Code’s 9.5. Observations. We examined nine tasks that Claude Code failed on. While the patches appear plausible and apply cleanly, they failed due to reasoning gaps. Eight of the nine patches pass the CVEPatchBench regression suite, but the exploit still works. Without a vulnerability oracle, Claude Code left an exploitable path open and the PoC still succeeded. The ninth patch ports the upstream fix that blocks the exploit but changes call behavior in a way that fails CVEPatchBench’s regression suite. Without a functionality oracle, Claude Code over-reasoned about the exploit and broke functionality. Timing. V ERI P ORT is slower than Claude Code on CVEPatchBench, averaging 627 seconds per advisory against 313 (Table 3). This is the expected cost of V ERI P ORT’s verification-oriented design. It not only drafts a patch but also isolates the security-relevant change and accepts a candidate only after the exploit and regression oracles validate it. Relative to Claude Code, V ERI P ORT spends about five additional minutes per advisory and improves pass3 by 20.2 points and pass@3 by 13.0 points. The added time therefore buys reliability rather than raw throughput. For security backporting, where a plausible patch can still leave the vulnerability exploitable or break existing behavior, this tradeoff is deliberate.
Takeaway. Withholding the MSP costs Claude Code three times more reliability than recall, 22.7 points of pass10 against 7.8 of pass@10. V ERI P ORT trades minutes of extra runtime for substantially fewer unreliable or incomplete backports.
Takeaway. On CVEPatchBench’s 393 npm backporting tasks, the reliability gap between V ERI P ORT and generic agents persists. V ERI P ORT is slower, but the additional verification work preserves both reliability and recall. Compared with Claude Code, it improves pass3 by 20.2 points and pass@3 by 13.0 points.
5.3. RQ2: Backporting Across npm As noted in Section 5.1, BackportBench has relatively few npm backporting tasks, while CVEPatchBench has 393. The fact that npm maintainers rarely backport [44] makes it a valuable target for novel backporting tools. We now compare V ERI P ORT to Claude Code (the best approach in RQ1) using CVEPatchBench. Results. As shown in Table 3, V ERI P ORT scores higher than Claude Code on every metric: by 16.2 percentage points for pass1 , 20.2 for pass3 , and 13.0 for pass@3. Across the three runs, V ERI P ORT’s worst case (pass3 ) is
5.4. RQ3: Ablation Study We now evaluate the contribution of the AUTOT ESTER and AUTO P OC components to V ERI P ORT’s design. These components are designed to provide reliability through the use of functionality and vulnerability oracles, respectively. Results. As shown in Table 3, removing either oracle degrades every metric. The vulnerability oracle contributes the most. Removing it costs 8.5 percentage points of pass1
12
and 9.6 of pass3 , against 4.0 and 3.8 for the functionality oracle. Removing both costs 9.8 percentage points of pass1 , less than the 12.5 point sum of the individual losses. This demonstrates that the oracles overlap on some failures while each catches defects the other misses. The oracle-free variant still exceeds Claude Code by 6.4 percentage points of pass1 and 7.8 of pass3 , isolating the contribution of decomposition and AUTO R ESEARCH’s artifacts. Observations. Claude Code also fails on the tasks the ablation variants leave unsolved. On these tasks the first candidate patch, from AUTO PATCH and Claude Code alike, is plausible, often a textbook mitigation, yet carries a defect subtle enough to survive review. Two tasks, one per oracle, illustrate the pattern. In deep-object-diff (CVE-2022-41713), the first patch creates a diff with Object.create(null), a standard prototype-pollution defense that blocks five of the six exploit vectors. On a null-prototype object the __proto__ write lands as an own property, the remaining vector reads the planted value back, and the exploit still fires on the patched build. In global-modules-path (CVE-2022-21191), the first patch adopts the upstream rewrite of execSync into spawnSync, a stronger fix that fully blocks the PoC. The regression suite drives the package through a mocked execSync, the rewrite bypasses the mock, and the suite fails on the patched build. Neither error is a localization failure. Such defects are hard to catch by review alone.
in principle, since the pipeline depends on a normalized manifest rather than the GHSA shape.
6. Production Deployment A backport shipped to users must also be reviewable, composable with other fixes for the same package version, and delivered in a form downstream users can adopt without changing their dependency graph. In production at Socket [24], a software supply chain security company, V ERI P ORT has generated over 5,000 verified backported patches [25] for 169 high- and critical-severity CVEs. Human-in-the-Loop Certification. The vulnerability and functionality oracles provide strong evidence, but the agents producing them are inherently stochastic (Section 3). Therefore, every patch passes a human certification step before release. A security engineer reviews the upstream fix analysis, isolated security hunks, per-version PoC verdicts, regression-test results, and the agent rationale behind each artifact. Review thus checks executable evidence and backtraces decisions rather than reconstructing each backport manually, allowing engineers to efficiently process V ERI P ORT’s output. When review finds a defect, the patch is withheld, the failing case re-enters the refinement loop (Section 4.4.4) with structured feedback, and the concrete error is distilled into reusable guidance so that later runs avoid it. Patch Delivery. V ERI P ORT publishes each verified patch with a signed manifest identifying the package PURL, target version, advisory, severity, changed files, and the associated PoC, test, and review artifacts. The manifest gives scanners, package-manager plugins, and CI systems an integration boundary while preserving provenance back to the certification evidence. Multi-CVE Merging. A single package version can be affected by multiple CVEs, yet patches are typically only generated for a single vulnerability. Simply composing multiple patches for a given version can cause issues. One patch may edit the same code as another or invalidate its assumptions. V ERI P ORT therefore composes verified patches targeting the same package version into a single unified patch, preserves the provenance of each constituent advisory, and re-certifies the merged artifact against the accumulated PoCs and regression suites. The result is one drop-in patch per version, addressing all relevant CVEs. Advisory Correction. AUTO P OC verifies an exploit against every version in an advisory’s range, discovering per version whether the vulnerability is present (Section 4.2.2). A security engineer reviews each flagged version against this evidence and files each confirmed case as a pull request to the GitHub Advisory Database to correct the affected range. The advisory for the vm2 sandbox escape (GHSA-ffh4-j6h5-pg66) had been pinned to the single version its reporter tested, while V ERI P ORT’s per-version verdicts marked many earlier versions equally exploitable, and the reviewed correction widened the range to cover them. A single mongoose advisory
Takeaway. Both verification components improve V ERI P ORT’s reliability, with AUTO P OC contributing the largest single gain. Relative to the oracle-free variant, the full system improves pass1 by 9.8 points and pass3 by 12.4 points, at a cost of 229 additional seconds per advisory.
5.5. Limitations Testing-data contamination. Since BackportBench uses known fixes for its backporting tasks, it is possible that the model used (Opus 4.7) was trained on both the vulnerable packages and the published CVE descriptions in our test set, so the model may recall fix locations rather than derive them, inflating apparent performance on advisories disclosed before its training cutoff. We provide another benchmark for evaluation, CVEPatchBench, which is based on new backports that are not listed in an advisory and were created after the model’s training cutoff. Ecosystem coverage. Our quantitative evaluation mostly focuses on npm as V ERI P ORT has the strongest support for this ecosystem. We have support for PyPI and early support for other ecosystems but we treat cross-ecosystem generalization as an empirical claim requiring its own study. GHSA as the main advisory source. V ERI P ORT primarily consumes GHSA as its advisory source (Section 4.1). V ERI P ORT has not been evaluated on its ability to ingest vulnerabilities indexed in OSV, ecosystem feeds, or vendor bulletins. However, extending ingestion is straightforward
13
(GHSA-m7xq-9374-9rvx) was corrected in both directions at once. Versions below 3.6.0-rc0, where the vulnerable match option of populate did not yet exist, were not exploitable and were dropped, and one later version on the 5.x line that had backported the fix was excluded as well. Excluding that release would have orphaned the stillvulnerable 6.x line, so the same review added a new range to re-cover it.
7. Conclusion
[13] T. Dunlap, E. Lin, W. Enck, and B. Reaves, “Vfcfinder: Pairing security advisories and patches,” in Proceedings of the 19th ACM Asia Conference on Computer and Communications Security, ser. ASIA CCS ’24. New York, NY, USA: Association for Computing Machinery, 2024, p. 1128–1142. [Online]. Available: https://doi.org/10.1145/3634737.3657007
References
C. Hicks, C. Attridge, A. Janjeva, and C. Ashurst, “Claude Mythos: What does Anthropic’s new model mean for the future of cybersecurity?” CETaS Expert Analysis, Centre for Emerging Technology and Security, The Alan Turing Institute, April 2026. [Online]. Available: https: //cetas.turing.ac.uk/publications/claude-mythos-future-cybersecurity
[6]
[7]
[15] Z. Zhong, J. Huang, and P. He, “BackportBench: A multilingual benchmark for automated backporting of patches,” arXiv preprint arXiv:2512.01396, 2025, under review. [Online]. Available: https: //arxiv.org/abs/2512.01396
Jerry Gamblin, “2024 CVE data review,” Blog post, Jan. 2025, analysis of NVD data. Reproducible code available at https://github.com/jgamblin/2024CVEBlog. Accessed: 202603-19. [Online]. Available: https://jerrygamblin.com/2025/01/05/ 2024-cve-data-review/
[4]
[5]
[14] Y. Cheng et al., “Fixseeker: An empirical driven graph-based approach for detecting silent vulnerability fixes in open source software,” 2025. [Online]. Available: https://arxiv.org/abs/2503.20265
J. Gamblin, “2025 CVE data review,” Blog post, Jan. 2026, accessed: 2026-03-19. [Online]. Available: https://jerrygamblin.com/2026/01/ 01/2025-cve-data-review/
National Institute of Standards and Technology, “National vulnerability database,” https://nvd.nist.gov/, accessed: 2026-03-19.
R. Shariffdeen, X. Gao, G. J. Duck, S. H. Tan, J. Lawall, and A. Roychoudhury, “Automated patch backporting in Linux (experience paper),” in Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, ser. ISSTA 2021. New York, NY, USA: Association for Computing Machinery, 2021, pp. 633–645. [Online]. Available: https://doi.org/ 10.1145/3460319.3464821
[12] Y. Shi, Y. Zhang, T. Luo, X. Mao, Y. Cao, Z. Wang, Y. Zhao, Z. Huang, and M. Yang, “Backporting security patches of web applications: A prototype design and implementation on injection vulnerability patches,” in 31st USENIX Security Symposium (USENIX Security 22). Boston, MA: USENIX Association, 2022, pp. 1993–2010. [Online]. Available: https: //www.usenix.org/conference/usenixsecurity22/presentation/shi
We thank Jordan Harband and Marvin Fleischer for their expert security review of V ERI P ORT’s published patches and for feedback on its PoCs, regression tests, and backports that helped refine the system. We also thank Husain Sharaf for help crafting the architectural figures for V ERI P ORT.
[3]
[9]
[11] S. Yang, Y. Xiao, Z. Xu, C. Sun, C. Ji, and Y. Zhang, “Enhancing oss patch backporting with semantics,” in Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, ser. CCS ’23. New York, NY, USA: Association for Computing Machinery, 2023, p. 2366–2380. [Online]. Available: https://doi.org/10.1145/3576915.3623188
Acknowledgements
[2]
The kernel development community, “Everything you ever wanted to know about Linux -stable releases,” The Linux Kernel documentation, version 7.1.0-rc6. [Online]. Available: https://docs.kernel.org/process/ stable-kernel-rules.html, accessed: Jun. 2, 2026.
[10] Y. Padioleau, J. Lawall, R. R. Hansen, and G. Muller, “Documenting and automating collateral evolutions in Linux device drivers,” in Proceedings of the 3rd ACM SIGOPS/EuroSys European Conference on Computer Systems (EuroSys ’08). New York, NY, USA: ACM, 2008, pp. 247–260.
We presented V ERI P ORT, which backports an advisory’s fix to every affected version and ships each backport with executable evidence: a PoC the patch blocks and a regression suite it passes. By decomposing backporting into independently verified stages and letting execution, not the model, accept each patch, V ERI P ORT resolves 95.3% of BackportBench tasks across all ten runs, 22.7 points above the strongest baseline, and 91.3% of CVEPatchBench’s 393 tasks across all three runs. In production it has generated over 5,000 verified patches for 169 CVEs and corrected 23 upstream advisories.
[1]
[8]
[16] J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. R. Narasimhan, and O. Press, “SWE-agent: Agent-computer interfaces enable automated software engineering,” in The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. [Online]. Available: https://arxiv.org/abs/2405.15793 [17] S. Wu, R. Wang, Y. Cao, B. Chen, Z. Zhou, Y. Huang, J. Zhao, and X. Peng, “Mystique: Automated vulnerability patch porting with semantic and syntactic-enhanced LLM,” Proceedings of the ACM on Software Engineering, vol. 2, no. FSE, pp. 130–152, 2025. [18] Z. Li, Z. Yu, J. Song, M. Xu, Y. Luo, and D. Mu, “Portgpt: Towards automated backporting using large language models,” in Proceedings of the 47th IEEE Symposium on Security and Privacy, 2026.
A. Decan, T. Mens, A. Zerouali, and C. D. Roover, “Back to the past – analysing backporting practices in package dependency networks,” IEEE Transactions on Software Engineering, vol. 48, no. 10, pp. 4087–4099, 2022.
[19] Y. Zhang, Y. Li, L. Cui, D. Cai, L. Liu, T. Fu, X. Huang, E. Zhao, Y. Zhang, C. Xu, Y. Chen, L. Wang, A. T. Luu, W. Bi, F. Shi, and S. Shi, “Siren’s song in the ai ocean: A survey on hallucination in large language models,” https://arxiv.org/abs/2309.01219, 2025.
C. Bogart, C. Kästner, J. D. Herbsleb, and F. Thung, “When and how to make breaking changes: Policies and practices in 18 open source software ecosystems,” ACM Transactions on Software Engineering and Methodology, vol. 30, no. 4, pp. 42:1–42:56, 2021.
[20] C.-P. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y. Zhang, and B. Ginsburg, “Ruler: What’s the real context size of your long-context language models?” 2024. [Online]. Available: https://arxiv.org/abs/2404.06654
L. Zhang, C. Liu, Z. Xu, S. Chen, L. Fan, L. Zhao, J. Wu, and Y. Liu, “Compatible remediation on vulnerabilities from third-party libraries for java projects,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), 2023, pp. 2540–2552.
[21] Anthropic, “Claude code,” https://code.claude.com/docs/en/overview, accessed: 2026-04-17.
14
[22] D. Zan, Z. Huang, W. Liu, H. Chen, S. Xin, L. Zhang, Q. Liu, A. Li, L. Chen, X. Zhong, S. Liu, Y. Xiao, L. Chen, Y. Zhang, J. Su, T. Liu, R. LONG, M. Ding, and liang xiang, “MultiSWE-bench: A multilingual benchmark for issue resolving,” in The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2026. [Online]. Available: https://openreview.net/forum?id=MhBZzkz4h9 [23] “CVEPatchBench-Public,” CVEPatchBench-Public.
[36] H. Sun, A. Rosà, D. Bonetta, and W. Binder, “Automatically assessing and extending code coverage for NPM packages,” in Proceedings of the 2nd IEEE/ACM International Conference on Automation of Software Test, ser. AST ’21. IEEE, 2021, pp. 40–49. [37] G. Fraser and A. Arcuri, “EvoSuite: Automatic test suite generation for object-oriented software,” in Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering, ser. ESEC/FSE ’11. New York, NY, USA: ACM, 2011, pp. 416–419.
https://github.com/SocketDev/
[24] Socket, Inc., “Socket: Software supply chain security,” https://socket. dev, 2026, accessed: 2026-06-19. [25] “Socket’s Patches Page,” https://socket.dev/features/patches.
[38] M. Schäfer, S. Nadi, A. Eghbali, and F. Tip, “An empirical evaluation of using large language models for automated unit test generation,” 2023. [Online]. Available: https://arxiv.org/abs/2302.06527
[26] H. H. Nguyen, D. M. Tran, Y. Cheng, T. Le-Cong, H. J. Kang, R. Widyasari, S. L. Khin, O. E. Lieh, T. Zhang, and D. Lo, “Mapping NVD records to their VFCs: How hard is it?” arXiv preprint arXiv:2506.09702, 2025.
[39] A. Berndt, T. Bach, R. Gemulla, M. Kessel, and S. Baltes, “On the flakiness of LLM-generated tests for industrial and opensource database management systems,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice, ser. ICSE-SEIP ’26. New York, NY, USA: ACM, 2026.
[27] S. Herbold, A. Trautsch, B. Ledel, A. Aghamohammadi, T. A. Ghaleb, K. K. Chahal, T. Bossenmaier, B. Nagaria, P. Makedonski, M. N. Ahmadabadi, K. Szabados, H. Spieker, M. Madeja, N. Hoy, V. Lenarduzzi, S. Wang, G. Rodrı́guez-Pérez, R. Colomo-Palacios, R. Verdecchia, P. Singh, Y. Qin, D. Chakroborti, W. Davis, V. Walunj, H. Wu, D. Marcilio, O. Alam, A. Aldaeej, I. Amit, B. Turhan, S. Eismann, A.-K. Wickert, I. Malavolta, M. Sulı́r, F. Fard, A. Z. Henley, S. Kourtzanidis, E. Tuzun, C. Treude, S. M. Shamasbi, I. Pashchenko, M. Wyrich, J. Davis, A. Serebrenik, E. Albrecht, E. U. Aktas, D. Strüber, and J. Erbel, “A fine-grained data set and analysis of tangling in bug fixing commits,” Empirical Softw. Engg., vol. 27, no. 6, Nov. 2022. [Online]. Available: https://doi.org/10.1007/s10664-021-10083-5
[40] N. Alshahwan, J. Chheda, A. Finogenova, B. Gokkaya, M. Harman, I. Harper, A. Marginean, S. Sengupta, and E. Wang, “Automated unit test improvement using large language models at Meta,” in Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, ser. FSE Companion ’24. New York, NY, USA: ACM, 2024, pp. 185–196.
[28] Y. Dong, W. Guo, Y. Chen, X. Xing, Y. Zhang, and G. Wang, “Towards the detection of inconsistencies in public security vulnerability reports,” in Proceedings of the 28th USENIX Security Symposium (USENIX Security 19). Santa Clara, CA: USENIX Association, 2019, pp. 869–885.
[41] S. Raemaekers, A. van Deursen, and J. Visser, “Semantic versioning and impact of breaking changes in the Maven repository,” Journal of Systems and Software, vol. 129, pp. 140–158, 2017.
[29] L. Bao, X. Xia, A. E. Hassan, and X. Yang, “V-SZZ: Automatic identification of version ranges affected by CVE vulnerabilities,” in Proceedings of the 44th IEEE/ACM International Conference on Software Engineering (ICSE ’22). New York, NY, USA: ACM, 2022, pp. 2352–2364.
[42] D. Venturini, F. R. Cogo, I. Polato, M. A. Gerosa, and I. S. Wiese, “I depended on you and you broke me: An empirical study of manifesting breaking changes in client packages,” ACM Transactions on Software Engineering and Methodology, vol. 32, no. 4, pp. 1–26, 2023.
[30] C. Segal, P. Segal, C. E. de Schuller Banjar, F. P. ao, H. S. Borges, P. S. Neto, E. S. de Almeida, J. C. S. Santos, A. Kocheturov, G. K. Srivastava, and D. S. Menasché, “Characterizing and modeling the GitHub security advisories review pipeline,” in Proceedings of the 23rd IEEE/ACM International Conference on Mining Software Repositories (MSR ’26), 2026.
[43] L. Zhang, C. Liu, Z. Xu, S. Chen, L. Fan, B. Chen, and Y. Liu, “Has my release disobeyed semantic versioning? Static detection based on semantic differencing,” in Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering (ASE ’22). New York, NY, USA: ACM, 2022.
[31] H. H. Nguyen, A. T. Nguyen, T. Le-Cong, Y. Li, H. W. Ang, Y. Yin, F. Liauw, S. L. Khin, O. E. Lieh, T. Zhang, and D. Lo, “Patchseeker: Mapping nvd records to their vulnerability-fixing commits with llm generated commits and embeddings,” 2025. [Online]. Available: https://arxiv.org/abs/2509.07540
[44] Y. Wang, P. Sun, L. Pei, Y. Yu, C. Xu, S.-C. Cheung, H. Yu, and Z. Zhu, “ Plumber: Boosting the Propagation of Vulnerability Fixes in the npm Ecosystem ,” IEEE Transactions on Software Engineering, vol. 49, no. 05, pp. 3155–3181, May 2023. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/TSE.2023.3243262
[32] Z. Chen, Z. Xue, J. Zhou, X. Hu, X. Xia, and X. Yang, “Diffploit: Facilitating cross-version exploit migration for open source library vulnerabilities,” 2025. [Online]. Available: https: //arxiv.org/abs/2511.12950
[45] Anthropic, “Introducing Claude Opus 4.7,” https://www.anthropic. com/news/claude-opus-4-7, Apr. 2026, accessed: 2026-06-10.
[33] S. Ullah, P. Balasubramanian, W. Guo, A. Burnett, H. Pearce, C. Kruegel, G. Vigna, and G. Stringhini, “From cve entries to verifiable exploits: An automated multi-agent framework for reproducing cves,” 2026. [Online]. Available: https://arxiv.org/abs/ 2509.01835
[46] ——, “Introducing Claude Opus 4.6,” https://www.anthropic.com/ news/claude-opus-4-6, Feb. 2026, accessed: 2026-06-10. [47] C. S. Xia, Y. Deng, S. Dunn, and L. Zhang, “Agentless: Demystifying llm-based software engineering agents,” 2024. [Online]. Available: https://arxiv.org/abs/2407.01489
[34] B. Ruan, J. Liu, C. Zhang, and Z. Liang, “Kernjc: Automated vulnerable environment generation for linux kernel vulnerabilities,” in The 27th International Symposium on Research in Attacks, Intrusions and Defenses, ser. RAID ’24. ACM, Sep. 2024, p. 384–402. [Online]. Available: http://dx.doi.org/10.1145/3678890.3678891
[48] S. Yao, N. Shinn, P. Razavi, and K. Narasimhan, “τ -bench: A benchmark for tool-agent-user interaction in real-world domains,” 2024. [Online]. Available: https://arxiv.org/abs/2406.12045
[35] D. Simsek, A. Eghbali, and M. Pradel, “Pocgen: Generating proof-of-concept exploits for vulnerabilities in npm packages,” 2025. [Online]. Available: https://arxiv.org/abs/2506.04962
15
Appendix A. Additional Figures and Tables
provided changes therefore cannot satisfy the scored tests on these five tasks. Formatting-sensitive verdict. On django_650 (CVE2022-28347), the scored test asserts the exact error-message text of the maintainer’s 2.2.28 backport. The provided upstream patch uses a different string format, so functionally identical patches fail on formatting alone, while the test that exercises the vulnerability passes. Harness instability. On django_211 (CVE-2022-28346), the baseline test run crashes inside the container, truncating its log and recording seven passing regression tests as failures.
Table 2 reports BackportBench pooled across npm and PyPI. We pool because npm contributes only 18 of the 128 tasks, where a single task shifts its rate by 5.6 points, leaving the per-ecosystem npm numbers coarse. Table 5 and Table 4 isolate the with-MSP and without-MSP settings by ecosystem. TABLE 4: BackportBench without MSP Ecosystem
Tool
pass1
pass10
pass@10
PyPI
V ERI P ORT Claude Code mini-swe-agent
98.7% 84.5% 79.8%
95.5% 71.8% 69.1%
100.0% 91.8% 90.9%
npm
V ERI P ORT Claude Code mini-swe-agent
98.9% 82.8% 81.1%
94.4% 77.8% 55.6%
100.0% 88.9% 88.9%
TABLE 5: BackportBench with MSP provided Ecosystem
Tool
pass1
pass10
pass@10
PyPI
V ERI P ORT Claude Code mini-swe-agent PortGPT MagentLess git apply
99.6% 97.5% 97.3% 89.9% 67.5% 40.0%
99.1% 95.5% 90.9% 80.0% 55.5% –
100.0% 99.1% 99.1% 94.5% 79.1% –
npm
V ERI P ORT Claude Code mini-swe-agent PortGPT MagentLess git apply
99.4% 97.2% 97.8% 83.3% 39.4% 44.4%
94.4% 94.4% 83.3% 83.3% 22.2% –
100.0% 100.0% 100.0% 83.3% 50.0% –
Appendix B. Excluded BackportBench Tasks This appendix details the 7 BackportBench tasks we exclude from our evaluation (Section 5.1), we omit the 67 Maven tasks as V ERI P ORT does not have full support of that ecosystem yet. We confirmed each case by execution inside the task’s released container. On six tasks, the benchmark’s gold patch passes while a patch built faithfully from the task’s provided inputs cannot, so the outcome is independent of the evaluated tool. We will share reproduction details for each task with the BackportBench authors. Provided patch omits required changes. On vite_31 and vite_37 (CVE-2024-31207), the scored fail-to-pass tests require three fixture files that the provided upstream patch omits. On vite_50 (CVE-2024-23331), the scored test probes an HTML fixture element that the provided patch omits. On socket.io-parser_71 and socket.io-parser_72 (CVE-2023-32695), the provided patch reflects the 4.x source commit and omits the error-to-throw change that the maintainer’s 3.4.3 and 3.3.4 backports and the scored test include. A patch limited to the
16