On the Reliability of LLM-Based Vulnerability Patching Benchmarks

On the Reliability of LLM-Based Vulnerability Patching Benchmarks

arXiv:2610.10150v1 [cs.CR] 7 Oct 2026

Dang K Le Northwestern University dang.le@northwestern.edu

Wenxuan Shi Northwestern University wenxuan.shi@northwestern.edu

Abstract

tasks [34] to end-to-end agents that localize, patch, and validate fixes against real-world software projects [42,49]. Benchmarks such as SWE-bench [20] and its multi-language successors [22,31,48] have become the de facto yardstick [30]: a new model is judged by the fraction of issues it “resolves,” [25] and leaderboards are updated weekly. This enthusiasm has spawned a vibrant ecosystem of agent frameworks, finetuned models, and training datasets derived from benchmark traces [29, 45]. Unfortunately, the same ecosystem is remarkably opaque about how those numbers are produced. A typical benchmark run places an agent in a buggy repository with a bug report, exposes some set of tools and auxiliary artifacts, and scores the resulting patch against one or more automated oracles. Seemingly small choices, such as whether the harness tells the agent how to build the project, or whether the target’s git history is included in the workspace, can significantly affect the reported pass rate. Worse, an agent can “pass” a benchmark case by silencing the symptom used to score it while leaving the underlying defect, or even the whole codebase, demonstrably worse off [39]. If these results are then used to train the next generation of code models, the pitfalls are baked into model weights [43]. In this paper we argue that the benchmarking pipeline itself, not just the model or the dataset, is a first-class research artifact that deserves principled design and critical scrutiny: benchmark scores are highly sensitive to evaluation design and can materially misrepresent model capability. Drawing on our experience developing and stress-testing LLM patching agent harnesses across three languages, we identify a taxonomy of pitfalls along three dimensions: Agent-level factors (§3.1): prompting choices, tool permissions, feedback oracles, and auxiliary capabilities such as Internet access; framework-level factors (§3.2): permission misconfigurations, timeouts, hidden agent mechanics, and post-agent test interference; and dataset-level factors (§3.3): how vulnerabilities are described, whether PoCs are provided, how the source is packaged, and how “success” is operationalized. To study these factors empirically, we curate 112 historical

Large language models (LLMs) have shown remarkable potential for automated vulnerability patching, with benchmarks such as SWE-bench driving rapid progress. However, through extensive experience developing, running, and stress-testing such evaluation frameworks, we identify critical but underexamined pitfalls in current benchmarking practices that can substantially distort reported results. These pitfalls span three dimensions: (1) agent-level factors, where prompting strategies, tool availability, and detailed instructions inflate success rates noticeably without improving developer-aligned patch quality; (2) framework-level factors, where permission misconfigurations, infrastructural bugs, and timeout issues silently degrade or artificially boost agent performance; and (3) dataset-level factors, where reliance on bug reports and single proof-of-concept (PoC) tests fails to capture whether patches address root causes or adhere to developer design intent. To demonstrate the effects of these shortcomings, we curate a dataset of 112 real-world historical bugs across 84 open-source C/C++, Go, and Rust projects, each equipped with PoC tests, regression tests, and added developer’s tests that evaluate whether generated patches align with the original developers’ design principles. Through three controlled experiments and multiple case studies, we demonstrate that while LLMs can achieve high PoC passing rates in ideal scenarios, many factors in executing a benchmark can noticeably lower the success rate. Moreover, their developer’s test passing rates remain low and improve only marginally with newer models, revealing that models are increasingly adept at suppressing symptoms but not at producing patches that satisfy upstream developer-authored tests. Our results show that current benchmark scores are highly sensitive to evaluation design choices, and we provide practical guidelines for making future evaluations more rigorous, reliable, and reproducible.

1

Xinyu Xing Northwestern University xinyu.xing@northwestern.edu

Introduction

Automated program repair (APR) with large language models (LLMs) has progressed from synthetic code-completion 1

bugs across 84 C/C++, Go, and Rust projects, each packaged in a reproducible Docker harness (§4) with PoCs, regression tests, and developer’s tests (§4.2)—tests that encode the developer’s design intent, not just the absence of a crash. We then run three controlled experiments and five case studies to probe how each factor shifts apparent patch quality (§5, §6). Our experiments are designed to isolate principles rather than individual knobs. For example, across a modern frontier model, we observe that detailed build and test instructions lift PoC-pass rates by a large margin in C/C++ and much less so in Go/Rust—because in Go/Rust the build system is uniform and discoverable from the repository, whereas C/C++ projects’ build systems are heterogeneous. This illustrates one of the principles we demonstrate in this paper: the reported success rate of a benchmark is only meaningful relative to the information its harness makes available to the agent. We generalize this style of observation into a set of guidelines (§8) for how benchmarks should be built and reported. Contributions. We make the following contributions: (1) A three-level taxonomy of pitfalls in LLM vulnerability patching benchmarks, grounded in concrete failures we observed in practice. (2) A controlled evaluation dataset and harness designed to isolate benchmarking factors: 112 real-world security-relevant bugs across three languages, each packaged with reproducible builds, PoCs, regression tests, and developer’s tests derived from upstream commits. (3) Three controlled experiments quantifying how prompting, tool availability, PoC access, and model choice move both PoC-pass and developer-test-pass rates; and five case studies that show how “incidental” framework issues can silently dominate headline numbers. (4) A survey of 13 existing real-world patching benchmarks and vulnerability datasets along the same factor axes, showing that most exhibit at least one of the pitfalls we characterize. (5) A set of concrete guidelines and a reporting checklist for future benchmark designers. Scope. This work is not intended to propose another benchmark for model ranking. Instead, the dataset is deliberately designed to highlight evaluation pitfalls under controlled conditions rather than to support competitive comparison. Our contribution is methodological: to strengthen the validity, interpretability, and reproducibility of reported APR results and of the training signals subsequently derived from them.

2 2.1

cess to the target repository at a buggy commit, and (c) optionally a PoC input and/or instructions related to building and testing the target [47, 50]. The agent is graded by an automated oracle, usually whether the PoC no longer reproduces the crash and whether a reference test suite still passes.

2.2 “Just Another Benchmark” Is Not Enough Existing benchmarks occupy a wide design space, but their evaluation protocols have converged on a surprisingly narrow notion of correctness: did the supplied test(s) pass? [47, 51] This definition is seductive because it is cheap and objective. But it conflates very different outcomes: (1) the agent produced a patch the original maintainer would have accepted; (2) the agent produced a patch that makes the oracle happy, e.g., by short-circuiting the crashing code path, disabling the feature, or special-casing the PoC input; or (3) the agent produced a patch that looks perfect, by taking advantage of a pitfall in the benchmark, such as git history, Internet access, permission misconfiguration, etc. All of the outcomes above appear identical on a leaderboard. We argue that the difference matters for two downstream uses: Deployment. If an APR system is proposed as a developer tool or as part of an autonomous pipeline, a fix that silences the PoC but breaks a corner of the feature is strictly worse than no fix at all, because it provides false assurance and masks the underlying vulnerability. Training. Increasingly, agent trajectories that “pass” the benchmark are used as supervised or reinforcement-learning signal [28, 38, 53]. Every false-positive pass is a data point that teaches the model to produce similar surface-level patch.

3

A Taxonomy of Benchmarking Pitfalls

We organize the pitfalls into three levels, corresponding to the three artifacts a benchmark designer controls: the agent (what the model is allowed to do), the framework (how its behavior is measured), and the dataset (what it is measured on). We define a factor as any variable in the evaluation pipeline that can change the measured success rate without changing the underlying model. We refer to factors studied via controlled ablations as systematic, and those revealed through individual failures as incidental; both can substantially affect reported performance. These factors are not independent. In practice, their interactions can amplify effects. Throughout this section and in §5, §6, we analyze factors in isolation for clarity, but we note that real-world benchmarks might exhibit multiple interacting factors simultaneously (§7).

Background and Motivation LLM-Based Vulnerability Patching

Modern LLM patching systems typically wrap a frontier model (e.g., Claude, GPT, Gemini) in an agent harness: a scaffold that exposes shell, file-editing, and sometimes webbrowsing tools, loops over model turns, and terminates when the agent emits a “done” signal or hits a step/turn budget [46, 50]. A typical patching task gives the agent: (a) a textual description of the bug (often a GitHub issue), (b) ac-

3.1

Agent-Level Factors

A1. Build and test instructions in the prompt (systematic). An impactful prompt-level variable we found is whether the agent is told, in-context, how to build and test the project. 2

Without such guidance, a capable agent must infer the build system from files scattered across the repository; with it, the agent can validate its own changes after every edit. We show in §5.1 that supplying these instructions raises PoC-pass rates by ≈ 20% in C/C++ cases and by a far smaller margin in Go/Rust cases, where the build system is more uniform. A2. Internet and web-search access (incidental). LLM agents are increasingly given live web-search and URL-fetch tools [37]. For a benchmark drawn from public GitHub issues, these tools leak ground truth in several ways. We observed several case traces where the agent directly searched for the issue number and retrieved the fix verbatim (§6.1).

3.2

a Go test function) reveals which API is exercised and with what expectations. We show in §5.2 that PoCs presented as source-level harnesses yield markedly higher PoC-pass rates than binary PoCs, independent of patch quality. D3. Source provenance and contamination (incidental). The method used to retrieve the source code determines what incidental signals are exposed in the workspace. Shallow clones, tarballs, and git worktrees each reveal different amounts of history. In particular, the presence of an unsanitized Git history is risky: a future commit in the history may contain the upstream fix for the buggy version (§6.2). D4. Language balance (systematic). Most benchmarks are dominated by a single language (typically Python or Java for SWE-bench variants). We find that the sensitivity of patch quality to specific factors varies sharply by language: C/C++ benchmarks are more sensitive to harness quality, Go/Rust benchmarks are more sensitive to PoC framing.

Framework-Level Factors

F1. Permission misconfiguration (incidental). A harness typically mounts the target source tree together with sensitive metadata related to the issue. If the agent is granted overly broad permissions, it may access these files and extract information about the ground-truth fix. Conversely, if permissions are too restrictive (e.g., preventing installation of dependencies required for a legitimate build), the agent may underperform for reasons unrelated to its capabilities. F2. Timeouts and hidden mechanics (incidental). Closedsource agent systems contain behaviors that are not obvious from the documentation alone. Evidently, we observed one such behavior in Claude Code, leading to unintended timeouts (§6.4). Analogously, slow-building projects (e.g., large C/C++ code bases) can exhaust wall-clock budgets before the agent has had a chance to test. These effects produce false negatives that are indistinguishable from real model failures. F3. Agent-written tests interfering with the evaluation pipeline (incidental). Some agents routinely create their own tests to validate a fix. If the evaluation framework subsequently applies additional tests to a file that the agent has already modified to add its own tests, the test application step may fail due to conflicts. As a result, the case is marked as a failure even though the patch itself is correct (§6.5). This is particularly problematic: the agent follows the intended workflow and produces a valid fix, yet the failure arises entirely from the evaluation pipeline rather than the agent’s behavior.

3.3

4

Methodology and Dataset

We built a reproducible evaluation infrastructure in which each of the above factors can be toggled in isolation. This section describes the dataset, the harness, and the agent/model choices used throughout §5–§6.

4.1

Dataset

Our dataset contains 112 real-world historical bugs spanning 84 open-source projects across three languages, drawn from public GitHub issues, GitHub Security Advisories (GHSA), RUSTSEC advisories, and curated commit histories. The perlanguage breakdown is summarized in Table 1; the full dataset instances and their metadata are listed in Appendix B. Table 1: Per-language dataset breakdown. “Bug-desc. source” is the primary artifact provided to the agent as the bug description; “PoC form” is the format of the proof-of-concept input bundled with each case.

Dataset-Level Factors

D1. Oracle granularity: PoC versus developer’s test (systematic). The single most common oracle in APR benchmarks is “does the PoC still crash?” [18, 44, 51] This is a necessary but insufficient condition for correctness. We propose developer’s tests (§4.2) as a practical middle ground between PoC-only and full regression testing: a targeted, authorauthored oracle that encodes the fix’s intent. D2. PoC format and framing (systematic). When a PoC is included, how it is presented matters. A raw binary input tells the agent nothing about intent, while a test harness (e.g.,

Language

Proj.

Cases

Bug-desc. source

PoC form

C/C++

10

25

GitHub/GitLab issue + ASan trace

Go

51

62

GH issue, GHSA

Rust

23

25

GH issue, RUSTSEC, GHSA

binary/text blob & C source harness Go source harness Rust source harness

Total

84

112

—

—

The C/C++ slice spans 10 projects: interpreters, image/dataprocessing libraries, command-line tools, and binary tooling. The Go slice spans 51 projects, including web frameworks, infrastructure tooling, and security-relevant libraries. The Rust slice spans 23 projects with memory-safety and logic CVEs, 3

including RUSTSEC advisories affecting data structure & cryptographic libraries, and HTML sanitizers. Each case is pinned to a specific trigger commit and patch commit, and is packaged as a self-contained Docker build. The per-case directory includes: (1) target-specific scripts for building, reproducing the PoC, and running regression tests. The PoC reproduction script is carefully designed to return a non-zero exit code only when a crash associated with the vulnerability is triggered, rather than when the program itself exits with a non-zero status (e.g., due to invalid inputs). (2) PoC inputs, which are binary artifacts (most C/C++ cases) or source-level harnesses (all Go and Rust cases). (3) The upstream developer’s patch, used exclusively by the groundtruth validator (§4.3) and never exposed to the agent. (4) The developer’s test derived from the upstream commit (§4.2). (5) The bug description provided to the agent: a textual issue/advisory write-up, a sanitizer crash trace, or both, depending on what is available upstream. (6) A YAML configuration file that captures metadata and pins all of the above components. Data collection and curation. Cases were collected by scraping historical security issues and fix commits, restricted to projects whose build-and-test cycle completes in under half an hour in our containerized harness. For C/C++ cases, PoCs are the binary inputs attached to the upstream issue; the bug description is the GitHub or GitLab issue text combined with a sanitizer crash trace we reproduce in the harness. For Go cases, the bug description is taken from the upstream issue or GHSA advisory; PoCs are source-level Go programs, of which 25 were imported from the upstream issue or advisory and 37 were minimal harnesses we hand-wrote when no upstream PoC was available. For Rust cases, the bug description is taken from RUSTSEC advisories, GHSA advisories, or the upstream issue; PoCs are source-level Rust programs sourced from the RUSTSEC advisory’s reference exploit when available, with the remaining cases using harnesses adapted from the upstream report or hand-written by us. Each case is validated by our ground-truth validator (§4.3): with the upstream patch applied, the target must build successfully, avoid crashing on the PoC, and pass all regression and developer’s tests. A case is admitted into the dataset only if it passes all four phases; cases that failed any phase were either repaired or discarded, yielding the final 112 curated instances.

4.2

tions, coding conventions, and intended design. In our dataset, we extract these tests from the upstream fix commit and reconstruct them as a patch named test.diff, which can be applied to the pre-fix source tree. This allows us to execute developer’s tests in the same manner as the project’s regression suite. Conceptually, a developer’s test answers a different question than a PoC: Oracle

Question answered

Build

Is the codebase still syntactically/linkable valid? Does the known crashing input still crash? Do pre-existing regression tests still pass? Does the patch behave as the developer intended on inputs adjacent to the bug?

PoC Regression Developer’s test

Example: PHP range() type confusion (GH-13094). To illustrate the difference between PoC-based evaluation and developer-intent evaluation, consider PHP issue GH13094 [5], a type-confusion bug in range(). The bug arises because mixed-type inputs can be misclassified as string inputs, leading the function to dereference a non-string operand and crash on the PoC range(9.9, []). At a high level, the vulnerable code uses an arithmetic predicate over internal type tags to decide whether to treat the arguments as strings: The relevant code, simplified for presentation, is the following: if (start_type >= IS_STRING || end_type >= IS_STRING) { if (start_type + end_type < 2 * IS_STRING) { goto handle_numeric_inputs; } /* unsafe string access */ }

Two patches can plausibly “fix” this bug. The upstream developer patch replaces the brittle arithmetic condition with a direct semantic check for mixed non-string inputs, preserving PHP’s intended coercion behavior. A second, more defensive patch instead adds type checks immediately before the unsafe string dereference and throws an error when either operand is not a string. Both patches suppress the original crash and pass the pre-existing PHP regression suite. However, the two patches are not behaviorally equivalent. PHP’s range() is intended to permit mixed numeric/string inputs [7] such as range(9.9, '0'), coercing the string operand and producing a numeric range. The upstream fix preserves this behavior; the defensive patch rejects it with ValueError. The developer’s test added in the upstream commit encodes exactly this expected behavior by checking the output of range(9.9, '0') and asserting it to an exact array value. As a result, the upstream patch passes the developer’s test, while the defensive patch fails. What the example shows. PHP GH-13094 illustrates that a benchmark which scores patches with the PoC and the pre-patch regression suite alone treats the developer’s and

Developer’s Tests

A developer’s test is a unit or integration test authored by the developers and extracted from the upstream fix commit. In mature projects, when developers fix an issue, they often introduce additional tests to validate one or more of the following [32]: (1) the patch addresses the root cause rather than simply its symptoms; (2) the patch preserves the intended functionality of the modified code, especially in cases where existing regression tests do not adequately cover the affected area; or (3) the patch conforms to the software’s specifica4

5

defensive patches as identical successes, while a benchmark that adds the developer’s test treats only the developer’s patch as a success. This is the kind of distinction we want LLM patching benchmarks to be capable of drawing. Authoring fidelity. Because developer’s tests are derived directly from upstream commits, they reflect the maintainer’s notion of correctness rather than ours. We only normalize test infrastructure (e.g., expected-output formatting, harness wiring) when necessary to run in our sandbox. Cases whose upstream commit does not add or meaningfully modify a test are excluded from the dataset rather than instrumented post-hoc by us; we discuss the resulting threats in §9.

4.3

We run three controlled experiments that hold the dataset constant and vary one agent- or dataset-level factor at a time. All experiments are performed under restrained container environments, hosted in typical Linux environment with 16 CPU cores, 32GB RAM, under normal workloads. The artifact includes per-case phase-level results for all conditions.

5.1

Experiment 1: Build/Test Instructions

Setup. We run the full 112-case dataset twice with OpenCode/Claude Sonnet 4.0. In one condition, the prompt includes the verbatim contents of the scripts the evaluator uses to build the project and run regression test; in the other, both are omitted, forcing the agent to infer how to build and test the project from the repository itself. All other conditions are fixed. Expected effect. Providing build and test instructions should primarily improve PoC-pass rates by making self-validation easier, with the largest effect in C/C++ projects whose build systems are heterogeneous and less discoverable than those of Go and Rust. Results. Table 2 reports per-language pass rates for both conditions across three test oracle phases.

Oracle Pipeline

After the agent produces a patch, we evaluate it in four phases: (1) Build: re-build the target project, with the LLM-generated patch applied, to completion. (2) PoC: execute each PoC ten times and require that no run result in a vulnerability-related crash. Repeating executions helps capture vulnerabilities with non-deterministic trigger rates (e.g., race conditions), ensuring they are reliably detected. (3) Regression: run the target project’s built-in regression test (before applying the developer’s test). (4) Developer’s test: apply the developer’s test patch file to the patched tree and re-run the regression test, which now includes the added developer’s test. A case is considered fully resolved only if all four phases pass. We separately report the PoC pass rate, the regression pass rate, and the developer’s test pass rate throughout §5.

4.4

Experiments of Systematic Factors

Table 2: Experiment 1: PoC, regression, and developer-test pass rates with and without build/test instructions in the prompt, broken down by language. Values are percentages computed over pass + fail outcomes (denominator = total cases per row).

Agents and Models

We use two distinct agent/model combinations, matched to the scale of each experiment: Controlled experiments (§5). We use OpenCode [12] in combination with Claude Sonnet 4.0 [10]. OpenCode is opensource, which allows us to audit its behavior end-to-end, and Sonnet 4.0 offers a cost/quality tradeoff that makes full-factor sweeps across hundreds of runs tractable. Case studies (§6). We use Claude Code [8] in combination with Claude Sonnet 4.5. Claude Code is a production closedsource agent with non-trivial internal mechanics, making it useful for exposing framework-level pitfalls that are difficult to observe in simpler harnesses. Across both settings, the evaluation harness is held fixed: the agent operates on the checked-out buggy repository inside a containerized environment, while prompt components such as build/test instructions and PoC exposure are toggled through configuration so that individual factors can be isolated cleanly. Model access is routed through a LiteLLM [11] proxy for uniform invocation and logging. Appendix A summarizes the container layout, isolation model, runtime controls, and evaluation pipeline; full implementation details are included in the artifact.

With instructions (Exp. 1)

Without (Exp. 2)

Lang.

PoC

Reg.

Dev.

PoC

Reg.

Dev.

C/C++ Go Rust

92.0 69.4 84.0

88.0 100.0 100.0

36.0 41.9 56.0

72.0 72.6 80.0

92.0 100.0 96.0

28.0 40.3 56.0

Overall

77.7

97.3

43.8

74.1

97.3

41.1

Interpretation. Including build/test instructions raises overall PoC-pass from 74.1% to 77.7%, but the effect is highly uneven across languages. The C/C++ slice improves by 20 percentage points (72.0% to 92.0%), whereas Rust changes only modestly (80.0% to 84.0%) and Go is nearly unchanged (72.6% to 69.4%, a slight negative within the noise of a single run). This matches the structure of the ecosystems in our dataset: Go and Rust projects usually expose recoverable defaults for building and testing such as go build & go test and cargo build & cargo test, while C/C++ projects rely on far more heterogeneous build and test procedures (such as CMake, Autotools, hand-written Makefiles, project-specific shell wrappers, etc.). By contrast, regressionpass rates remain constant overall (97.3% in both conditions), and developer-test-pass changes only slightly (41.1% 5

to 43.8%). Together, these results suggest that build/test instructions mainly alter the benchmark’s information budget— especially in C/C++—rather than patch quality as measured by our stronger developer-test oracle. Takeaway. The headline PoC-pass rate of an otherwise identical benchmark can shift by 20 percentage points in C/C++ purely as a function of whether the harness discloses build/test instructions in the prompt. Because current benchmark papers disclose this detail inconsistently, reported scores across benchmarks are not directly comparable, and the magnitude of the discrepancy is itself language-dependent.

5.2

even more because the bug description already includes an ASan stack trace that localizes the fault. Takeaway. The benefit of PoC access depends much more on its representation than on the mere fact that a PoC is provided. Two benchmarks that nominally “provide a PoC” are therefore not comparable unless they also disclose the PoC’s representation. Crucially, the gains accrue far more to oracle alignment than to developer-aligned patch quality.

5.3

Experiment 3: Model Generations

Setup. We run the full 112-case dataset three times with the same OpenCode harness used in Experiment 1 (build/test instructions included, no PoC body), varying only the underlying model, from older baseline to current frontier reasoning model: Claude Sonnet 4.0, Claude Sonnet 4.5, and Claude Opus 4.7. We record overall PoC-, regression-, and developertest-pass rates across all three languages. Expected effect. As models improve, PoC-pass should rise faster than developer-test-pass. If so, PoC-only evaluation will increasingly overstate how close the benchmark is to being solved. Results. Figure 1 shows the three pass rates as a function of model generation; the underlying overall numbers are summarized in Table 4.

Experiment 2: PoC Framing

Pass rate (%)

Setup. We study two subsets of the dataset according to the natural form of their PoC artifacts. The Go subset (62 cases) uses source-harness PoCs written in Go that exercise specific APIs in the target package and trigger the vulnerability; the C/C++ subset (25 cases) uses binary or text-blob PoCs fed directly to an executable. For each subset, we compare a noPoC condition against one in which the PoC is exposed to the agent: inline source code for Go and a file path for C/C++. Build and test instructions are included in both conditions. Expected effect. Exposing a PoC should help most when the artifact is readable and reveals API-level intent, as with source-level harnesses. Opaque binary PoCs should offer much less benefit because they indicate only that some input crashes the program, not why or at what semantic level. Results. Table 3 shows a strong contrast between these two PoC forms.

100 80 60 40 Sonnet 4.0 PoC pass

Table 3: Experiment 2: effect of PoC form on PoC, regression, and developer-test pass rates. Each row is restricted to the language subset in which the corresponding PoC form is natural; the no-PoC rows are the language-matched baselines from Experiment 1. PoC form

PoC

Reg.

Dev.

No PoC (Go, n=62) Source-harness PoC (Go, n=62)

69.4 95.2

100.0 98.4

41.9 48.4

No PoC (C/C++, n=25) Binary blob PoC (C/C++, n=25)

92.0 92.0

88.0 92.0

36.0 36.0

Sonnet 4.5 Regression pass Model generation

Opus 4.7 Developer-test pass

Figure 1: Experiment 3: PoC-pass, regression-pass, and developer-test-pass rates across three Claude generations (overall, n=112). PoC-pass climbs steadily toward saturation; developer-test-pass improves but stays well below; regressionpass is flat.

Table 4: Experiment 3: overall pass rates (%) across Claude model generations and the persistent PoC−Dev gap. Within parentheses we report the share of PoC-passing cases that nonetheless fail the developer’s test.

Interpretation. In Go, adding the source-harness PoC raises PoC-pass from 69.4% to 95.2% and developer-test-pass from 41.9% to 48.4%. In C/C++, exposing the binary blob yields no measurable change in PoC-pass (92.0% in both conditions) and no change in developer-test-pass (36.0% in both), with only a small increase in regression-pass (88.0% to 92.0%). The asymmetry is consistent with the information each PoC carries. A source-level Go harness exposes both the crashing call site and the API-level expectation, whereas a binary input is largely opaque. In our C/C++ cases, that opacity matters

Model

PoC

Reg.

Dev.

PoC−Dev (PoC-only %)

Sonnet 4.0 Sonnet 4.5 Opus 4.7

77.7 84.8 90.2

97.3 94.6 94.6

43.8 48.2 60.7

33.9 (43.7%) 36.6 (43.2%) 29.5 (32.7%)

Interpretation. PoC-pass rises steadily across model generations, from 77.7% to 84.8% to 90.2%, approaching saturation by Opus 4.7. Developer-test-pass also improves, but much 6

more slowly: 43.8% to 48.2% to 60.7%. The resulting PoC– Dev gap remains large throughout, shrinking only from 33.9 to 29.5 percentage points, and even with Opus 4.7, 32.7% of PoC-passing cases still fail the developer’s test. In other words, roughly one in three patches that appear successful under a PoC-only oracle would still be rejected by an oracle aligned with maintainer intent. Regression-pass remains nearly flat (97.3%, 94.6%, 94.6%), suggesting that pre-existing regression suites do little to distinguish symptom suppression from developer-acceptable repair. Takeaway. These results show that PoC-pass is a misleading progress signal near the top of the capability curve: a PoC-only benchmark would suggest that the task is nearly solved, whereas developer’s tests reveal substantial remaining headroom. Every benchmark currently using only PoC/test oracles is therefore reporting a scaling curve that overstates how much of the remaining headroom matters to practitioners.

5.4

consume unbounded input, under a default Claude Code configuration with WebSearch and WebFetch enabled. This is the configuration that a benchmark operator gets “out of the box” if they do not edit .claude/settings.json; many of the other coding agents share this similar default configuration. Trace. The agent’s first four tool calls after reading the bug description were all web fetches surrounding ReadUvarint, with the final call fetching the upstream patch commit directly. Eleven turns later, the agent committed a fix and reported success with the line “The fix follows the same approach used in the Go standard library fix,” citing the very URL it had fetched. Both the PoC and the developer’s test passed. Mechanism. Public GitHub issues and security advisories are indexed by search engines within hours of being filed. For any benchmark derived from historical bugs, the ground-truth commit is at most one web query away, and the case identifier or the bug description usually contains enough hints (CVE id, GitHub issue number, function name) to land the agent on it on the first try. Implication. For benchmarks targeting historical bugs, web tools must be disabled by default and explicitly re-enabled only when the benchmark’s claim is about tool-use rather than about patching. Our harness disables WebSearch and WebFetch through the Claude Code permissions.deny list; frameworks without an equivalent knob must isolate the container network. Note that disabling these two tools is not in itself sufficient: an agent that retains Bash can run curl or git fetch against the public origin remote. We address that channel in Case 2.

Cross-Cutting Observations

Across all three experiments, three patterns recur. First, developer-test-pass is tightly constrained by PoC-pass: among the 734 case-level outcomes in our condition-by-language analysis (aggregating all controlled conditions in §5), only a single case has PoC=fail and Dev=pass simultaneously. Second, the ratio of developer-test-pass to PoC-pass varies substantially by condition and model, indicating that some benchmark configurations improve oracle alignment much more than they improve true patch quality. Third, the pool of “PoC-only” patches, i.e. cases where the agent suppresses the PoC but fails the developer’s test, remains large across model generations: 38 cases for Sonnet 4.0, 41 for Sonnet 4.5, and 33 for Opus 4.7. This persistent slab is exactly the region that PoC-only benchmarks miscount as progress.

6

6.2

Setup. This case study is observed in commit 524f142 of the Go package bluemonday [3], which is an XSS sanitizer regression where strings.ToLower() is used to compare HTML tag names, against a variant of our harness in which the project’s full upstream .git history was preserved. This is meant to simulate the most common “benchmark from a tarball or unsanitized clone” scenario. The bug description shipped with the case mentions, as a hint, that the issue is a regression of an earlier issue #56. Trace. The agent’s reasoning before any code edit acknowledged the regression hint provided by the bug description and decided to look for the original fix from git history. It then issued a single git log query, filtering by issue #56, and inspected the resulting commit. By doing so, it found commit 5793ebc, which contains a unified diff that has the exact API change as the upstream fix for the current case. The agent’s subsequent edits track that diff almost line-for-line; it concludes by quoting “the fix pattern used before” in its final message. The case passes the PoC and the developer’s test. It is worth noting that the agent’s invocation of git log here is not malicious: it is a perfectly reasonable strategy for understanding a regression, and a developer asked to do

Case Studies of Incidental Factors

The experiments in §5 show how systematic factors manifest at scale. In this section, we walk through five concrete incidents drawn from real runs with Claude Code paired with Claude Sonnet 4.5, in which a single framework- or datasetlevel decision either leaked the answer to the agent or caused the pipeline to mis-score a correct patch. We call these factors incidental because they affect only a subset of cases rather than the dataset as a whole. Our goal is to show that for model training, leakage detection, or leaderboard reporting, even rare incidents matter. Every excerpt below is taken from raw agent traces. These traces are archived in the artifact.

6.1

Case 2: Git History Leakage

Case 1: WebSearch Leakage

Setup. This case study is observed in commit 69c6093 of the Go package xz [2], where ReadUvarint can be made to 7

Implication. The agent and the evaluator must run in disjoint mounts, and the agent container should run as a non-root user. The launch script materializes only the inputs the agent is allowed to see (config without patch references, scripts, PoCs) into a separate path before starting the agent; the metadata directory in its full form is only mounted in the evaluator container. We recommend a discipline of one write role, one read role: the agent can only read what the harness has explicitly staged for it, and the evaluator’s view is invisible to the agent.

the same task would do exactly the same thing. The leakage is structural: any benchmark that ships the upstream history alongside the bug effectively answers “how was this fixed?” in the workspace itself. Mechanism. The trigger commit is almost always followed, within a small number of commits, by the upstream fix; for regression-style bugs, an even older commit on a different file may already encode the answer. A shallow clone, a tarball, or an explicit git log sanitization step closes this channel. Note that WebSearch (Case 1) and git log are two routes to the same destination, and disabling either one alone is insufficient. Implication. Every patching benchmark should document, and preferably automate, its source-tree sanitization step. In our default harness, the per-case Dockerfile clones the upstream repository, checks out the trigger commit, then removes the git history and initializes a new one containing only the buggy commit, leaving the agent with a single-commit working tree. The variant we used to produce this trace exists exclusively as a control for the case study and is not part of the headline dataset.

6.3

6.4

Case 4: Hidden Sub-Model Dependency

Setup. This case is an outlier in the section: the failure mode does not surface in the agent’s transcript at all, and we therefore cannot point to a log excerpt. We discovered it indirectly, by cross-referencing per-case wall-clock with model-side traffic in the LiteLLM admin panel. Symptom. Across many cases we observed unexpectedly long wall-clock times that did not match any obvious build or test latency. A given case might issue tens of Bash tool calls and accumulate tens of minutes of overhead even when each individual command, run outside the harness, completed in well under a second. Re-running the same commands from a shell inside the same container gave the expected sub-second timings, so the latency was not in the project under test. Mechanism. Inspecting LiteLLM’s request log revealed two distinct streams of model traffic from a single agent run. The first stream targets the model alias we had configured (anthropic/sonnet-4-5) and corresponds to the main agent loop; this stream succeeds normally. The second stream targets the bare model name claude-haiku-4-5-20251001, which our LiteLLM configuration does not route, and corresponds to a per-Bash-command “safety” classification call that Claude Code issues internally. Each such call hangs for ≈ 120 seconds—the underlying HTTP timeout— before Claude Code falls back to allowing the command anyway. The bash command then runs in a few milliseconds. Schematically, every bash invocation looks like: (1) Main model proposes Bash(...); (2) Claude Code calls claude-haiku-4-5-20251001 for safety classification; (3) no route; the call times out after ≈ 120 s; (4) Claude Code defaults to “allow”; and (5) the Bash command is finally executed. The behavior is not visible in the agent’s stream-JSON output, in the harness logs, or in the LiteLLM logs of the configured alias—only in the LiteLLM logs of the unrouted exact model name, which an operator would have no reason to look at. The behavior also appears to be hard-coded into the most recent versions of Claude Code at the time we conducted our experiments: it cannot be disabled through agent settings or prompt instructions. Workaround. We mitigate the issue by setting the lesser-known environment variable ANTHROPIC_DEFAULT_HAIKU_MODEL [9] in the agent

Case 3: Permission Misconfiguration

Setup. In an earlier version of our harness, the agent ran as root inside the case container, as a consequence, the entire metadata directory was both readable and writable to it. The motivation was operational: we wanted to avoid permission errors during builds and dependency installation. Rather than inlining build/test/PoC content into the prompt, we instructed the agent to consult the metadata directory as needed. The directory, however, also contains sensitive files used by the post-agent evaluator, such as the upstream fix and the developer’s test. We re-created this configuration to drive the case in GitHub issue GH-13931 [6] of the PHP project, which contains an ASan-detected null-pointer dereference in phpdbg’s compiler shutdown path. Trace. The agent’s first six tool calls all touched the metadata directory. Listing this directory revealed the existence of the YAML config file. By inspecting this file, the agent could see the path to the ground-truth patch file used by the validator, which contains the upstream patch verbatim. Worse, the config file itself disclosed the upstream patch commit hash (c3acfb1b...) and the issue URL, enough metadata to pivot to Case 1’s WebFetch leak if the tool is enabled. The agent’s subsequent edits to the code base are textually identical to the upstream patch. Mechanism. The harness conflated two distinct sets of files: agent-visible task inputs (config, bug report, build/test scripts, PoCs) and evaluator-visible ground-truth metadata (upstream patch, developer’s tests, patch-commit references). Bundling both into a single mount made permission misconfiguration trivial: the agent did not need root, only read access to the wrong directory. 8

container, which redirects the safety-check slot to our routed alias. With this set, per-Bash overhead drops back to near-negligible latency and overall case wall-clock becomes consistent with the underlying project’s build and test latency. Implication. The effect of this issue is not random; it disproportionately penalizes large projects in which the agent must navigate through many directories or run many tests, exactly the cases where the per-Bash 120 s tax compounds. A benchmark operator who is not aware of raw model traffic will read these as model-capability failures. More broadly, every closed-source agent harness may call sub-models the operator did not configure, or generally any hidden mechanics; benchmark designs must be aware of the possibility that such mechanics exist, monitor unexpected behaviors, and treat them as a run-invalidation event rather than a scored failure.

6.5

from test run failure, reporting the former as a pipeline error rather than as a patch failure, otherwise correct patches are penalized whenever they happen to touch the same test file the maintainer did, which for security fixes is common.

6.6

Why “Incidental” Factors Matter

We refer to these factors as “incidental” because they do not affect the majority of cases in full-dataset experiments and do not consistently manifest in the same instances across repeated runs. However, their impact is far from negligible. In our experiments, each individual factor can appear in as low as ≈ 5% (Permission Misconfiguration) up to as high as ≈ 32% (Test Collision) of cases across one complete run on our full dataset, and crucially, their effects are systematically biased rather than random. As a result, even a relatively small incidence rate can distort aggregate metrics and downstream uses of benchmark data. Despite their limited prevalence, incidental factors have a disproportionate impact in two key contexts: Leaderboard reporting. Modern LLM-based patching systems are often compared using aggregate success rates that differ by only a few percentage points. In this regime, even low-frequency distortions can alter the relative ranking of systems. For example, a framework issue affecting as little as 5% of cases can shift the reported success rate enough to reverse leaderboard positions [1]. Because these distortions are not independent of system behavior—for instance, certain agents/models may be more likely to trigger timeouts, exploit leakage channels, or generate conflicting tests—they introduce systematic bias rather than uniform noise. This phenomenon mirrors concerns raised in prior work on benchmark reliability and evaluation sensitivity, where small perturbations in evaluation setup can lead to significant changes in reported performance [30, 39]. In the context of LLM agents, the problem is worsened by the complexity of the evaluation pipeline, where infrastructure-level artifacts can dominate model-level differences. Training data generation. The impact of incidental factors is even more pronounced when benchmark traces are reused as training data for supervised fine-tuning (SFT) or reinforcement learning (RL). In such pipelines, each evaluated instance is treated as a labeled example, and mislabeling directly affects the learned policy. A case where a correct fix is incorrectly marked as failure— due to timeout, infrastructure error, or test collision—becomes a false negative, discouraging behaviors that are in fact desirable. Conversely, a case solved via unintended shortcuts, such as WebSearch leakage or access to git history, becomes a false positive, reinforcing spurious strategies such as copying ground-truth patches rather than reasoning about the bug. This asymmetry is particularly problematic because learning algorithms amplify consistent biases in the training signal. Recently, studies of LLM benchmarking have highlighted how

Case 5: Agent-Written Test Collision

Setup. This case study is observed in commit b8c9ede of the Fiber web framework [4] written in Go. The upstream fix adds a test to ctx_test.go, in addition to fixing the vulnerability. Trace. The agent produced a correct root-cause fix, matching the intention of the upstream change. Then it proactively rewrote the existing Test_Ctx_IsFromLocal test in ctx_test.go to exercise the new code path. When doing so, it also added "net" to the import block of ctx_test.go so the new code compiles. The collision. The upstream developer’s test also modifies ctx_test.go: it renames Test_Ctx_IsFromLocal to Test_Ctx_IsFromLocal_X_Forwarded and rewrites the body to assert the same property the agent’s added cases assert, and it adds the same "net" import. When the evaluator applies the developer’s test using git apply on top of the agent’s commit: the import-block hunk fails because the agent already added "net", while the function-body hunk fails because the original Test_Ctx_IsFromLocal body that the diff expects to find has been replaced by the agent’s expanded version. git apply aborts the entire diff, the developer’s-test phase is recorded as a failure, and the case is scored fail despite the agent’s patch being functionally equivalent to (and arguably better tested than) the upstream one. Mechanism. Developer-test application is a textual diff, not a semantic merge; any benign collision aborts the application. The collision in this case is not adversarial—the agent did exactly what its system prompt requested, namely add regression tests covering the security property—and the failure lives entirely in the scoring pipeline, not in the patch. Implication. Any added test files to the evaluation pipeline must be robust to such benign collisions, or scoped to files the agent is unlikely to touch. In our harness, before applying the added developer’s test, we revert all files that are modified in test.diff back to their original states. At minimum, the evaluator should distinguish test addition/application failure 9

contamination and evaluation artifacts can be absorbed into model behavior during training [39, 43]. Incidental factors act as an additional source of such bias: although they occur less frequently than systematic factors, their per-instance impact on the training signal is disproportionately large. Taken together, these observations suggest that incidental factors should not be treated as edge cases or experimental noise. Instead, they represent a distinct class of evaluation failure modes that can meaningfully affect both reported performance and the quality of downstream models. A reliable benchmark must therefore detect, report, and where possible eliminate these factors, rather than averaging over them.

7

the upstream fixing commit; VJBench relies entirely on the project’s existing test suite without any PoC execution. The SWE-bench family uses test-suite oracles. D2, PoC format. No benchmark documents or controls the format in which PoCs are presented, despite its measurable effect on reported quality (§5.2). Among the five security benchmarks that supply PoCs (SEC-bench, PatchAgent, ARVO, ExtractFix, Vul4C), all use binary inputs exclusively. Vul4J and VJBench side-step the issue by relying on source-level test cases rather than PoC inputs, but this is an oracle-design choice (D1), not a deliberate PoC-format decision. D3, source provenance and contamination. How the target source code is provisioned determines what incidental signals are exposed to the agent. SEC-bench, MultiSWE-bench, ARVO, ExtractFix, Vul4C, and VJBench all retain full, unsanitized git history, making the upstream fix commit discoverable via git log (§6.2). Only SWE-bench’s Python instances perform multi-step sanitization (remote removal, tag pruning, reflog expiry, aggressive GC); its non-Python instances only remove the remote. PatchAgent removes .git from the patch-application copy but not from the source-reading copy that the agent’s viewcode tool accesses, leaving git history partially exposed. Vulnerability collections (Big-Vul, CVEfixes, VADER) distribute source code as standalone snippets, database entries, or file archives rather than as repository checkouts, so git-history-based contamination does not arise from their design. However, researchers who repurpose these datasets for LLM-agent patch evaluation must supply their own source-retrieval step; naively cloning the upstream repository introduces the same contamination risk described above. D4, language balance. Every security benchmark with execution infrastructure covers a single language family: C/C++ (SEC-bench, PatchAgent, ARVO, ExtractFix, Vul4C) or Java (Vul4J, VJBench). Vulnerability collections such as CVEfixes (27 languages) and VADER (15+ languages) offer broader language coverage but lack infrastructure. Among all surveyed benchmarks, only MultiSWE-bench combines multi-language support with execution infrastructure.

Survey of Real-World Benchmarks

The controlled experiments in §5 and case studies in §6 demonstrate that agent-level, framework-level, and datasetlevel factors can materially affect reported patch quality. We now ask: how prevalent are these pitfalls in existing benchmarks? To answer that, we surveyed 13 benchmarks and datasets that are commonly used in prior work for evaluating LLM-based vulnerability patching. Because these benchmarks differ in whether they provide an agent interface or execution infrastructure, we organize them into three categories (Table 5) that determine which factors from §3 apply: LLM-agent patching benchmarks provide an agent harness or are used in prior work with LLM agents. All factors (A1– A2, F1–F3, D1–D4) apply. We also include the SWE-bench family in this category because it is widely used with LLM agents despite being agent-agnostic. Although its tasks focus on general software engineering rather than security vulnerabilities, it involves bug repair workflows where many of the pitfalls we analyze still arise. Automated vulnerability repair (AVR) benchmarks are designed for traditional program-repair tools, with build and test infrastructure but no agent harness. Dataset-level factors (D1– D4) apply directly; framework-level factors (F1–F3) become relevant if the benchmark is repurposed for agent evaluation. Agent-level factors (A1–A2) do not apply, as these benchmarks do not define an agent interface. Vulnerability collections are curated datasets of vulnerability metadata without execution infrastructure. Only dataset-level factors (D1–D4) are relevant.

7.1

7.2

Framework-Level Observations

F1, permission model and harness isolation. No surveyed benchmark enforces a complete permission model that limits agent access at both the application and network layers. PatchAgent restricts agent actions to a defined tool set (viewcode, locate, validate) with no shell or web access. SWE-bench and MultiSWE-bench are agent-agnostic: they delegate isolation entirely to the agent framework (e.g., SWEagent provides its own container, but the benchmarks themselves do not enforce boundaries). SEC-bench provides a partial harness with tag-based information separation but no documented permission model. AVR benchmarks provide no agent isolation, which is appropriate for their original scope but insufficient if repurposed for agent evaluation. When per-

Dataset-Level Observations

D1, oracle granularity. Many security benchmarks rely on PoC-only oracles, which overestimate correctness (§5.3). SEC-bench, ARVO, and ExtractFix determine correctness solely by whether a single exploit input triggers a sanitizer report, which may allow superficial fixes that suppress crashes without addressing root cause. PatchAgent and Vul4C supplement the PoC oracle with the project’s existing regression tests; Vul4J provides source-level PoV tests extracted from 10

Table 5: Survey of 13 benchmarks and datasets used for evaluating LLM-based vulnerability patching Dataset Benchmark

D1

Framework

D2

D3

D4

issue + hints issue issue multi-source sanitizer rpt

shallow; part. san. full; unsan. full; part. san. full; unsan. full; part. san.

Agent

F1

F2

F3

A1

Py 6 lang. Py C/C++ C/C++

delegateda delegated N/Ab partial tool-restr.d

✓ ✗ ✓ ✗ ✓

✗c agent-dep. ✗c not doc. – fixed – not doc. ✗e ✓

A2

LLM-agent patching benchmarks SWE-bench [20] MultiSWE-bench [48] Agentless [41] SEC-bench [22] PatchAgent [47]

test suite test suite test suite PoC PoC + regr.

not contr.a not contr. N/A not contr. restrictedd

Automated vulnerability repair (AVR) benchmarks ARVO [31] Vul4C [23] Vul4J [14] VJBench [40] ExtractFix [18]

PoC PoC + regr. PoV tests regr. tests PoC

issue + PoC NVD + crash trace metadata only CVE + CWE ID PoC only

full; unsan. full / tarball; unsan. full; san. full; unsan. full; unsan.

C/C++ C/C++ Java Java C/C++

✗ ✗ ✗ ✗ ✗

✗ ✗ ✗ ✗ ✗

– ✗e ✗e ✗e –

– – – – –

– – – – –

none none none

CVE summary CVE desc + msg expert report

– – –

C/C++ 27 lang. 15+ lang.

– – –

– – –

– – –

– – –

– – –

Vulnerability collections Big-Vul [17] CVEfixes [13] VADER [24]

D1 =oracle granularity; D2 =bug description; D3 =source provenance (clone depth; sanitization status); D4 =languages; F1 =permission model; F2 =timeout documented; F3 =test integrity; A1 =build/test instructions; A2 =internet control. ✓ =present/addressed; ✗ =absent/unaddressed; – =not applicable. a SWE-bench is agent-agnostic; permission model and internet control depend on which agent framework is used (e.g., SWE-agent provides its own container, but the benchmark itself does not enforce isolation). b Agentless delegates build/test to the SWE-bench Docker harness; it does not provide standalone scripts or an agent shell. c SWE-bench and MultiSWE-bench apply test_patch via git apply, which can conflict with agent-modified files (§6.5). d PatchAgent restricts the agent to three tools with no shell or web access, but does not enforce this via containerized network isolation. e Benchmark runs the project’s existing test suite on the agent-modified tree; the agent may modify or delete test files to influence the oracle outcome.

mission models are weak or absent, the source-provenance issues described in D3 become exploitable: agents with shell access can discover ground-truth fixes via git log or by reading mounted metadata. F2, timeouts and hidden mechanics. Only PatchAgent and SWE-bench document timeout thresholds; no benchmark monitors hidden sub-model calls. Without documented timeouts, capability failures are indistinguishable from budget exhaustion (§6.4). F3, test-patch collision. SWE-bench applies its test_patch via git apply without handling merge conflicts against agent-modified files; MultiSWE-bench follows the same pattern. Benchmarks that run the project’s pre-existing test suite on the agent-modified tree (PatchAgent, Vul4C, Vul4J, VJBench) face a complementary risk: the agent may modify or delete test files, inadvertently or deliberately, to influence the oracle outcome. Any evaluation that runs tests on agentmodified code should address this collision risk (§6.5).

build scripts inside the container without documenting what the agent is told. No benchmark ablates the impact of build and test instructions on reported scores. Because different harness configurations can shift PoC-pass rates by up to 20 % (§5.1), two studies on the same dataset can report substantially different numbers without either being wrong. A2, internet and web-search access. PatchAgent and Agentless both prevent the agent from reaching the internet with restricted agent workflows, but neither enforces this at the network level. SWE-bench and MultiSWE-bench leave network policy entirely to the agent framework. SEC-bench containers use default bridge networking, which permits outbound connections. No surveyed benchmark enforces network-level isolation (e.g., --network=none or firewall rules). For benchmarks derived from historical bugs, web search provides a direct path to the upstream fix (§6.1).

7.3

Among the five LLM-agent benchmarks, none addresses all nine factors. Every pitfall identified in §5–§6 is present in at least one surveyed benchmark. PatchAgent covers the most ground—restricted tool set (F1), documented timeouts and iterative budget (F2), and regression tests supplementing the PoC oracle (D1). But it still lacks network-level isolation (A2) and full git-history sanitization (D3). SWE-bench provides strong source sanitization for Python instances (D3)

7.4

Agent-Level Observations

A1, build and test instructions. Agent benchmarks generally provide build and test infrastructure, but whether and how these instructions reach the agent varies: SWE-bench and MultiSWE-bench are agent-agnostic and leave prompt content to the agent framework; PatchAgent abstracts building and testing behind its validate tool; SEC-bench places 11

Summary

and documented timeouts (F2), but delegates all agent-level and permission concerns to external frameworks (F1, A1, A2). SEC-bench and MultiSWE-bench leave the majority of factors unaddressed or undocumented. AVR benchmarks and vulnerability collections are increasingly repurposed for LLM-agent evaluation despite not being designed for that setting. Repurposing either category requires supplying agent harnesses, oracles, and isolation— engineering choices that introduce the very confounds characterized in §5–§6. In particular, unsanitized git histories (D3), absent permission models (F1), and unrestricted network access (A2) become immediate risks once an autonomous agent operates inside these environments.

8

axes used in this paper: wrong build, regression-test failure, developer-test failure, pipeline error, timeout, etc. Without this, cross-benchmark comparison is impossible.

8.2

We propose a minimal reporting checklist that LLM vulnerability patching benchmarking reports should include: (1) The exact agent prompt(s) and any case-specific instructions included. (2) List of tools exposed to the agent, including network scope. (3) Source tree provenance and sanitization steps. (4) Oracle definition: PoC-only, regression, developer-test, or combination. (5) Inspection and analysis of the failure case logs with the failure taxonomy of G7.

Guidelines for Reliable Benchmark Design 9

We translate the principles established across §5–§6 into a checklist that benchmark designers and model benchmark operators can apply directly.

8.1

A Reporting Checklist

Discussion and Threats to Validity

Scope and external validity. Our dataset is intentionally small and highly curated: 112 cases across C/C++, Go, and Rust, restricted to projects with reproducible Dockerized builds, executable PoCs, and developer-added tests. This design improves experimental control, but it also introduces selection bias [25]. In particular, our cases likely overrepresent better-maintained projects and bugs with cleaner reproduction and validation artifacts. We therefore do not claim that the absolute pass rates reported here generalize to the full population of real-world bugs or to benchmark suites in other language ecosystems. Our goal is narrower: to identify and measure evaluation factors whose effects are large enough to remain visible even in a controlled dataset. For that purpose, controlled comparisons within a fixed harness are more important than scale alone. Our dataset versus real-world benchmarks. Our approach to uncovering benchmark pitfalls is to systematically inject factors that may influence outcomes into our own dataset via configuration flags, and then survey real-world benchmarks to assess whether these factors are present. We argue that this methodology is preferable to directly running individual benchmarks: it allows us to keep both the agent harness and dataset fixed while isolating and controlling each factor in separate experiments, rather than being constrained by the design and limitations of existing benchmarks. Oracle validity. Developer’s tests are a stronger oracle than PoC-only evaluation, but they are not a perfect proxy for patch acceptability [33, 36]. They capture the maintainer’s notion of correctness only to the extent that the upstream fix commit adds or meaningfully revises tests, and some upstream tests may themselves be incomplete or overly narrow. To reduce this threat, we exclude cases whose fix commits do not add meaningful test coverage, and we derive developer’s tests directly from upstream artifacts rather than authoring them ourselves. Even so, passing a developer’s test should be interpreted as evidence of closer alignment with maintainer intent, not as a proof of absolute correctness.

Guidelines Derived from the Principles

G1 (from §5.1). Disclose the harness’s information budget. Publish the exact prompt including any case-specific instructions provided, the list of tools, and the default values used in headline numbers. G2 (from §5.2). Separate capability from knowledge. If the benchmark supplies PoCs, also report the number without PoCs; if it supplies build scripts, also report the number without them. Diff-style ablations are cheap and disambiguate which part of the headline reflects which axis. G3 (from §5.3). Report developer-test results alongside PoC results. At minimum, a benchmark based on historical bugs should report both a crash-suppression rate and a developer-intent rate, understood as a lower bound on real patch quality. We advocate for developer’s tests derived from upstream commits as a cheap, high-signal default. G4 (from F1/D3). Quarantine the ground truth. Never mount ground-truth artifacts in the agent’s container. Automate source-tree sanitization (.git removal, build-cache cleanup) as part of the setup of the harness environment. G5 (from F2). Be aware of hidden mechanics in closedsource agents. Monitor agent logs and raw LLM request logs to detect hidden behaviors. For closed-source agents with hidden sub-models, benchmark operators should monitor perslot model availability accordingly. G6 (from F3). Use collision-tolerant added test application or classify collisions as pipeline errors. Additional tests (whether it is PoC-related, regression, or developer’s test) application should tolerate benign conflicts with agent-added files, and should distinguish pipeline errors from patch errors in the scoring output. G7 (cross-cutting). Publish failure taxonomies, not just pass rates. Each failed case should be labeled along the 12

Agent, model, and run dependence. Absolute performance depends on the chosen agent/model pair and on run-level stochasticity [27, 52]. Our controlled experiments use OpenCode with Claude Sonnet 4.0 for tractability, while our case studies use Claude Code with Claude Sonnet 4.5 to expose behaviors of a production closed-source agent. As a result, the exact percentages in §5 should not be read as universal estimates of model capability. Our main claims instead concern differences across conditions with the dataset and harness held fixed—for example, the effect of build/test instructions, PoC representation, and oracle choice. These comparative patterns are less sensitive to the specific model choice than the absolute rates themselves. We likewise place more weight on large and consistent shifts (e.g., the 20-point C/C++ swing in §5.1) than on small percentage-point differences that may fall within single-run variance. Memorization, contamination, and leakage. Because our cases are historical bugs drawn from public repositories, some fixes may have appeared in model training data [15, 43]. This threat mainly affects absolute success rates. However, it does not explain the core findings of the paper, which are driven by changes in benchmark configuration while holding the underlying model fixed. Our experiments show that build/test instructions (§5.1), PoC framing (§5.2), and framework-level leakage or timeout behaviors (§6) substantially change measured outcomes without changing the target bug or the model. A pure memorization account also does not explain the persistent gap between PoC-pass and developer-test-pass rates (§5.3): if success were dominated by exact recall of upstream fixes, one would expect substantial alignment with developerintent oracles. We therefore view memorization as a possible contributor to headline pass rates, but not as a sufficient explanation for the benchmark-sensitivity effects we report.

10

questioning whether benchmark scores faithfully measure the intended capability. In APR, Liu et al. [25] critically review the evaluation practices used to assess repair systems, and prior work has long shown that test-suite-passing patches can be plausible but incorrect or overfit to weak oracles [26,33,36]. In the LLM era, recent audits of SWE-bench-style evaluation similarly show that apparent issue resolution can diverge from true correctness [30, 39]. More broadly, benchmark contamination and evaluation artifacts have become a central concern in LLM assessment [15, 43], and the machine-learning community has repeatedly warned that rapid benchmark progress can outpace empirical rigor [35]. Our contribution specializes these concerns to agentic vulnerability patching, where reliability depends not only on the dataset and oracle, but also on harness permissions, source sanitization, hidden agent mechanics, and post-patch evaluation pipelines. Classical program repair. Automated program repair predates LLMs [21], and many of the evaluation pathologies we study have clear precedents in that literature, especially the tension between test-suite adequacy and semantic correctness [26, 33, 36]. Our developer’s-test proposal inherits this lesson, but adapts it to modern agentic patching settings in which the model can localize, edit, execute tools, and exploit benchmark infrastructure. LLMs for SE more broadly. Broad surveys of LLMs for software engineering [16, 19] cover tasks such as code generation, testing, vulnerability detection, and repair. Our focus is narrower: the integrity of the benchmark pipelines used to evaluate LLM-based patch generation.

11

Conclusion

Benchmarks for LLM-based automated vulnerability patching have become a de facto measure of progress, but we show that their headline numbers can be misleading. Reported success depends not only on the model, but also on prompt design, framework behavior, source packaging, and oracle choice. In particular, passing a PoC often overstates true repair quality: across our experiments, models improved much faster at suppressing benchmark symptoms than at producing patches that satisfy developer-intent tests. Our contribution is methodological. We introduced a taxonomy of benchmarking pitfalls, a controlled multi-language dataset and harness, and a set of experiments and case studies that make these pitfalls visible and measurable. The resulting guidelines and reporting checklist are meant to help future work distinguish model capability from the highly sensitive evaluation pipeline. If benchmark scores are to demonstrate the capability of a new model, support credible scientific claims, or provide a reliable training signal for future patching systems, then the evaluation pipeline itself must be treated as part of the contribution. For this task, what a benchmark measures is at least as important as how high the score is.

Related Work

LLM-based patching benchmarks and agents. Recent work has popularized real-repository issue resolution as a benchmark for code agents, most notably SWE-bench and its multilingual extensions [20], together with agent frameworks such as SWE-agent and Agentless [41, 46]. Our goal is not to provide another benchmark catalog; instead, §7 surveys existing benchmarks specifically through the lens of evaluation pitfalls such as oracle choice, source leakage, and configurations. Vulnerability-focused repair. A parallel line of work studies security-oriented repair using CVEs, advisories, exploit inputs, or sanitizer traces [14, 22, 31, 40, 47]. These efforts are closer to our setting than general issue-resolution benchmarks, but most still evaluate fixes primarily through PoC suppression or existing regression tests. Our work is complementary: rather than proposing a new leaderboard, we focus on how benchmark design choices change the measured reliability of LLM-generated security patches. Evaluation methodology. Our closest antecedents are studies 13

References [1] SWE-bench Leaderboard. swebench.com/.

[15] Jialun Cao, Wuqi Zhang, and Shing-Chi Cheung. Concerned with Data Contamination? Assessing Countermeasures in Code Language Model, March 2024. arXiv:2403.16898 [cs]. URL: http://arxiv.org/abs/2403.16898, doi:10.48550/arXiv.2403.16898.

URL: https://www.

[2] ulikunitz/xz commit 69c6093. https://github.com/ ulikunitz/xz/commit/69c6093, 2020.

[16] Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M. Zhang. Large Language Models for Software Engineering: Survey and Open Problems, November 2023. arXiv:2310.03533 [cs]. URL: http://arxiv.org/abs/2310.03533, doi:10.48550/arXiv.2310.03533.

commit 524f142. [3] microcosm-cc/bluemonday https://github.com/microcosm-cc/bluemonday/ commit/524f142, 2021. [4] gofiber/fiber commit b8c9ede. https://github.com/ gofiber/fiber/commit/b8c9ede, 2023. [5] PHP issue GH-13094: range(9.9, ’0’) causes segmentation fault. https://github.com/php/ php-src/issues/13094, 2024.

[17] Jiahao Fan, Yi Li, Shaohua Wang, and Tien N. Nguyen. A C/C++ code vulnerability dataset with code changes and CVE summaries. In Sunghun Kim, Georgios Gousios, Sarah Nadi, and Joseph Hejderup, editors, MSR ’20: 17th International Conference on Mining Software Repositories, Seoul, Republic of Korea, 29-30 June, 2020, pages 508–512. ACM, 2020. doi:10.1145/ 3379597.3387501.

[6] PHP issue GH-13931: applying zero offset to null pointer in Zend/zend_opcode.c. https://github. com/php/php-src/issues/13931, 2024. [7] PHP manual: range(). https://www.php.net/ manual/en/function.range.php, 2024.

[18] Xiang Gao, Bo Wang, Gregory J. Duck, Ruyi Ji, Yingfei Xiong, and Abhik Roychoudhury. Beyond Tests: Program Vulnerability Repair via Crash Constraint Extraction. ACM Trans. Softw. Eng. Methodol., 30(2):14:1– 14:27, February 2021. URL: https://dl.acm.org/ doi/10.1145/3418461, doi:10.1145/3418461.

[8] Claude Code: agentic coding tool by Anthropic. https: //www.anthropic.com/product/claude-code, 2025. [9] Claude Code documentation: model configuration. https://code.claude.com/docs/en/ model-config, 2025.

[19] Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. Large Language Models for Software Engineering: A Systematic Literature Review, April 2024. arXiv:2308.10620 [cs]. URL: http://arxiv.org/abs/2308.10620, doi:10. 48550/arXiv.2308.10620.

[10] Claude models documentation. https://platform. claude.com/docs/en/about-claude/models, 2025. [11] LiteLLM: a Python library for calling LLM apis. https: //github.com/BerriAI/litellm, 2025. [12] OpenCode: an AI coding agent. https://github. com/anomalyco/opencode, 2025.

[20] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, November 2024. arXiv:2310.06770 [cs]. URL: http://arxiv.org/abs/2310.06770, doi:10.48550/arXiv.2310.06770.

[13] Guru Prasad Bhandari, Amara Naseer, and Leon Moonen. Cvefixes: automated collection of vulnerabilities and their fixes from open-source software. In Shane McIntosh, Xin Xia, and Sousuke Amasaki, editors, PROMISE ’21: 17th International Conference on Predictive Models and Data Analytics in Software Engineering, Athens Greece, August 19-20, 2021, pages 30– 39. ACM, 2021. doi:10.1145/3475960.3475985.

[21] Claire Le Goues, ThanhVu Nguyen, Stephanie Forrest, and Westley Weimer. GenProg: A Generic Method for Automatic Software Repair. IEEE Transactions on Software Engineering, 38(1):54–72, January 2012. URL: https://ieeexplore.ieee.org/abstract/ document/6035728, doi:10.1109/TSE.2011.104.

[14] Quang-Cuong Bui, Riccardo Scandariato, and Nicolás E. Díaz Ferreyra. Vul4j: A dataset of reproducible java vulnerabilities geared towards the study of program repair techniques. In 19th IEEE/ACM International Conference on Mining Software Repositories, MSR 2022, Pittsburgh, PA, USA, May 23-24, 2022, pages 464–468. ACM, 2022. doi:10.1145/3524842.3528482.

[22] Hwiwon Lee, Ziqi Zhang, Hanxiao Lu, and Lingming Zhang. Sec-bench: Automated benchmarking of LLM agents on real-world software security tasks. CoRR, 14

abs/2506.11791, 2025. URL: https://doi.org/ 10.48550/arXiv.2506.11791, arXiv:2506.11791, doi:10.48550/ARXIV.2506.11791.

ieeexplore.ieee.org/document/11186078/, doi: 10.1109/ICSME64153.2025.00042. [30] Matias Martinez and Xavier Franch. What’s in a Benchmark? The Case of SWE-Bench in Automated Program Repair, February 2026. arXiv:2602.04449 [cs]. URL: http://arxiv.org/abs/2602.04449, doi:10. 1145/3786583.3786904.

[23] Ying Li, Faysal hossain shezan, Bomin wei, Gang Wang, and Yuan Tian. Sok: Towards effective automated vulnerability repair, 2025. URL: https://arxiv.org/ abs/2501.18820, arXiv:2501.18820. [24] Ethan TS. Liu, Austin Wang, Spencer Mateega, Carlos Georgescu, and Danny Tang. Vader: A human-evaluated benchmark for vulnerability assessment, detection, explanation, and remediation, 2025. URL: https:// arxiv.org/abs/2505.19395, arXiv:2505.19395.

[31] Xiang Mei, Pulkit Singh Singaria, Jordi Del Castillo, Haoran Xi, Abdelouahab, Benchikh, Tiffany Bao, Ruoyu Wang, Yan Shoshitaishvili, Adam Doupé, Hammond Pearce, and Brendan Dolan-Gavitt. Arvo: Atlas of reproducible vulnerabilities for open source software, 2024. URL: https://arxiv.org/abs/2408.02153, arXiv:2408.02153.

[25] Kui Liu, Li Li, Anil Koyuncu, Dongsun Kim, Zhe Liu, Jacques Klein, and Tegawendé F. Bissyandé. A critical review on the evaluation of automated program repair systems. Journal of Systems and Software, 171:110817, January 2021. URL: https://www.sciencedirect. com/science/article/pii/S0164121220302156, doi:10.1016/j.jss.2020.110817.

[32] Qing Mi and Jacky Keung. An empirical analysis of reopened bugs based on open source projects. In Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering, EASE ’16, pages 1–10, New York, NY, USA, June 2016. Association for Computing Machinery. URL: https:// dl.acm.org/doi/10.1145/2915970.2915986, doi: 10.1145/2915970.2915986.

[26] Fan Long and Martin Rinard. An analysis of the search spaces for generate and validate patch generation systems. In Proceedings of the 38th International Conference on Software Engineering, ICSE ’16, pages 702–713, New York, NY, USA, May 2016. Association for Computing Machinery. URL: https:// dl.acm.org/doi/10.1145/2884781.2884872, doi: 10.1145/2884781.2884872.

[33] Yuhua Qi, Xiaoguang Mao, Yan Lei, Ziying Dai, and Chengsong Wang. The strength of random search on automated program repair. In Proceedings of the 36th International Conference on Software Engineering, ICSE 2014, pages 254–265, New York, NY, USA, May 2014. Association for Computing Machinery. URL: https:// dl.acm.org/doi/10.1145/2568225.2568254, doi: 10.1145/2568225.2568254.

[27] Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8086–8098, Dublin, Ireland, May 2022. Association for Computational Linguistics. URL: https:// aclanthology.org/2022.acl-long.556/, doi:10. 18653/v1/2022.acl-long.556.

[34] Francisco Ribeiro. Large Language Models for Automated Program Repair. In Companion Proceedings of the 2023 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity, SPLASH 2023, pages 7–9, New York, NY, USA, October 2023. Association for Computing Machinery. URL: https:// dl.acm.org/doi/10.1145/3618305.3623587, doi: 10.1145/3618305.3623587.

[28] Xufang Luo, Yuge Zhang, Zhiyuan He, Zilong Wang, Siyun Zhao, Dongsheng Li, Luna K. Qiu, and Yuqing Yang. Agent Lightning: Train ANY AI Agents with Reinforcement Learning, August 2025. arXiv:2508.03680 [cs]. URL: http://arxiv.org/abs/2508.03680, doi:10.48550/arXiv.2508.03680.

[35] D. Sculley, Jasper Snoek, Alex Wiltschko, and Ali Rahimi. Winner’s Curse? On Pace, Progress, and Empirical Rigor. June 2018. [36] Edward K. Smith, Earl T. Barr, Claire Le Goues, and Yuriy Brun. Is the cure worse than the disease? overfitting in automated program repair. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, pages 532–543, New York, NY, USA, August 2015. Association for Computing Machinery. URL: https://

[29] Roman Macháček, Anastasiia Grishina, Max Hort, and Leon Moonen. The Impact of Fine-Tuning Large Language Models on Automated Program Repair. In 2025 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 380–392, September 2025. ISSN: 2576-3148. URL: https:// 15

[45] Boyang Yang, Zijian Cai, Fengling Liu, Bach Le, Lingming Zhang, Tegawendé F. Bissyandé, Yang Liu, and Haoye Tian. A Survey of LLM-based Automated Program Repair: Taxonomies, Design Paradigms, and Applications, December 2025. arXiv:2506.23749 [cs]. URL: http://arxiv.org/abs/2506.23749, doi:10. 48550/arXiv.2506.23749.

dl.acm.org/doi/10.1145/2786805.2786825, doi: 10.1145/2786805.2786825. [37] Aditya Bharat Soni, Boxuan Li, Xingyao Wang, Valerie Chen, and Graham Neubig. Coding Agents with Multimodal Browsing are Generalist Problem Solvers, June 2025. arXiv:2506.03011 [cs]. URL: http://arxiv.org/abs/2506.03011, doi:10. 48550/arXiv.2506.03011.

[46] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, November 2024. arXiv:2405.15793 [cs]. URL: http://arxiv.org/abs/2405.15793, doi:10. 48550/arXiv.2405.15793.

[38] Qi Tian, Kun Kuang, Furui Liu, and Baoxiang Wang. Learning from Good Trajectories in Offline Multi-Agent Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 37(10):11672–11680, June 2023. URL: https://ojs. aaai.org/index.php/AAAI/article/view/26379, doi:10.1609/aaai.v37i10.26379.

[47] Zheng Yu, Ziyi Guo, Yuhang Wu, Jiahao Yu, Meng Xu, Dongliang Mu, Yan Chen, and Xinyu Xing. PATCHAGENT: A Practical Program Repair Agent Mimicking Human Expertise. pages 4381–4400, 2025. URL: https://www.usenix.org/conference/ usenixsecurity25/presentation/yu-zheng.

[39] You Wang, Michael Pradel, and Zhongxin Liu. Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study, September 2025. arXiv:2503.15223 [cs]. URL: http://arxiv.org/ abs/2503.15223, doi:10.1145/3744916.3764576.

[48] Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, Rui Long, Kai Shen, and Liang Xiang. MultiSWE-bench: A Multilingual Benchmark for Issue Resolving, April 2025. arXiv:2504.02605 [cs]. URL: http://arxiv.org/abs/2504.02605, doi:10. 48550/arXiv.2504.02605.

[40] Fangzhou Wu and Tan Lin. How well does LLM generate security patches? a large empirical study with VJBench. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), 2023. [41] Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying llm-based software engineering agents. CoRR, abs/2407.01489, 2024. URL: https://doi.org/ 10.48550/arXiv.2407.01489, arXiv:2407.01489, doi:10.48550/ARXIV.2407.01489.

[49] Quanjun Zhang, Chunrong Fang, Yang Xie, Yuxiang Ma, Weisong Sun, Yun Yang, and Zhenyu Chen. A Systematic Literature Review on Large Language Models for Automated Program Repair. ACM Trans. Softw. Eng. Methodol., March 2026. Just Accepted. URL: https://dl.acm.org/doi/10.1145/3799693, doi:10.1145/3799693.

[42] Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. Automated Program Repair in the Era of Large Pretrained Language Models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 1482–1494, May 2023. ISSN: 1558-1225. URL: https://ieeexplore.ieee.org/document/ 10172803/, doi:10.1109/ICSE48619.2023.00129.

[50] Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. AutoCodeRover: Autonomous Program Improvement, July 2024. arXiv:2404.05427 [cs]. URL: http://arxiv.org/abs/2404.05427, doi:10. 48550/arXiv.2404.05427.

[43] Cheng Xu, Shuhao Guan, Derek Greene, and M.-Tahar Kechadi. Benchmark Data Contamination of Large Language Models: A Survey, June 2024. arXiv:2406.04244 [cs]. URL: http://arxiv.org/abs/2406.04244, doi:10.48550/arXiv.2406.04244.

[51] Yuntong Zhang, Jiawei Wang, Dominic Berzin, Martin Mirchev, Dongge Liu, Abhishek Arya, Oliver Chang, and Abhik Roychoudhury. Fixing Security Vulnerabilities with AI in OSSFuzz, November 2024. arXiv:2411.03346 [cs]. URL: http://arxiv.org/abs/2411.03346, doi:10.48550/arXiv.2411.03346.

[44] Xuezheng Xu, Yulei Sui, Hua Yan, and Jingling Xue. VFix: Value-Flow-Guided Precise Program Repair for Null Pointer Dereferences. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pages 512–523, May 2019. ISSN: 1558-1225. URL: https://ieeexplore.ieee.org/document/ 8812101, doi:10.1109/ICSE.2019.00063.

[52] Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate Before Use: Improving 16

Few-shot Performance of Language Models. In Proceedings of the 38th International Conference on Machine Learning, pages 12697–12706. PMLR, July 2021. URL: https://proceedings.mlr.press/ v139/zhao21c.html.

the default configuration, and redirects internal sub-model slots to the configured LiteLLM endpoint. For OpenCode, the harness constructs an equivalent configuration against the same endpoint. Unless otherwise stated, all runs use a 1800-second wall-clock budget and a 500-turn budget.

[53] Yuqi Zhou, Sunhao Dai, Changle Qu, Liang Pang, Jun Xu, and Ji-Rong Wen. Learning to Retrieve from Agent Trajectories, March 2026. arXiv:2604.04949 [cs]. URL: http://arxiv.org/abs/2604.04949, doi:10.48550/arXiv.2604.04949.

A

A.3 LiteLLM, networking, and resource limits All model traffic is routed through a LiteLLM proxy, which provides a uniform API surface across providers and preserves raw request logs for post hoc analysis. This indirection is important both for controlled model substitution and for diagnosing hidden agent behavior, such as the internal safetymodel traffic discussed in §6.4. Each case runs in a resource-limited Docker container attached only to an internal bridge network that exposes the LiteLLM proxy but not the public Internet. This enforces the default no-network policy for agents: even if application-level web tools are re-enabled, shell-based network access (e.g., curl or git fetch) remains blocked unless the harness is intentionally reconfigured.

Infrastructure Details

This appendix summarizes the evaluation harness used in §4.4. The full artifact contains per-language base images, per-case container recipes, agent images, a LiteLLM proxy, and scripts for launching cases and validating ground truth.

A.1

Container structure and isolation

Each run is built in three layers: (1) a language-specific base image with the required toolchain, (2) a case image that checks out the upstream repository at the trigger commit and installs case-specific dependencies, and (3) an agent image that installs the agent runtime and launches the patching workflow under a non-root model user. For case-study controls involving source leakage, the harness can deliberately relax these defaults, but the headline configuration uses the isolated setup described below. Inside the container, the project source tree is exposed at /src/<project> and is the only location the agent is intended to modify. Per-case metadata is staged separately under /ssebench. In the default configuration, the agent runs as an unprivileged user and cannot read ground-truth artifacts such as the upstream patch or developer’s test. The evaluator, which runs after the agent terminates, retains full access to /ssebench. This separation is what allows us to study permission and leakage failures as controlled deviations rather than as properties of the default harness.

A.2

A.4

Evaluation and ground-truth validation

After the agent terminates, the evaluator executes the fourphase pipeline described in §4.3: rebuild the project, run the PoC(s), run the pre-existing regression suite, and then apply the developer’s test and re-run the tests. Each phase is timeoutbounded, and failures in applying additional test patches are recorded as evaluation-phase failures rather than evaluator crashes. Dataset curation uses a separate ground-truth validator built from the same case image but without any agent. It applies the upstream developer patch to a fresh checkout and runs the same four-phase pipeline. A case is included in the dataset only if the upstream patch passes all phases, ensuring that the benchmark instances remain internally consistent as scripts and metadata evolve.

Agent configuration

A small Python driver constructs the user prompt from a fixed template and launches the agent with its working directory set to the checked-out source tree. The prompt is parameterized so that we can independently toggle the factors studied in §5.1 and §5.2, including whether build/test scripts are inlined and whether the PoC is exposed to the agent. Model selection is likewise parameterized through a LiteLLM endpoint, allowing the same harness to be reused across providers and model families. For Claude Code, the harness writes a per-case settings file that restricts tool permissions, disables web-search tools in 17

B

Dataset Inventory

Table 6 lists every case in our dataset: 25 C/C++, 62 Go, and 25 Rust cases (112 total). The Case ID columns contain the target project’s name along with the case identifier. The Bug Source column reports the primary upstream identifier (GitHub issue or pull request, CVE, GHSA advisory, RUSTSEC advisory, or commit hash when no tracker entry exists). The PoC column abbreviates the PoC representation used by the harness: src is a source-level harness compiled and executed against the project, text is a textual input file (e.g., XML, Vim script, WebAssembly text), and bin is a binary blob fed to a project binary. Trigger and patch commits are git commit IDs in short form. Table 6: Dataset inventory Case ID

Lang

Bug Source

PoC

Trigger

Patch

cpython-py_pr_124555

C/C++

issue #120378

bin

7bd9dbf

c2ba931

exiv2-exiv2_gh_1901

C/C++

issue #1901

text

4d55cb6

395389a

exiv2-exiv2_gh_1913

C/C++

issue #1913

bin

04f4624

be29681

exiv2-exiv2_gh_2007

C/C++

issue #2007

bin

aebb3fb

65fad9d

exiv2-exiv2_gh_789

C/C++

issue #789

bin

f05d100

8cd95e2

hdf5-hdf5_gh_3842

C/C++

issue #3842

src

3ea21cc

af4c6c4

hdf5-hdf5_gh_4656

C/C++

issue #4656

src

6cf3389

9fd4fd0

hdf5-hdf5_gh_4705

C/C++

issue #4705

bin

b11f2d0

aea4c53

jasper-jas_gh_383

C/C++

issue #383

bin

29154a5

2d6b6e8

jq-jq_gh_2825

C/C++

issue #2825

text

f86566b

3fa10e8

libxml2-xml_commit_0bcd05c

C/C++

commit 0090675

text

0090675

0bcd05c

libxml2-xml_commit_a820dbe

C/C++

commit db07dd6

text

db07dd6

a820dbe

libxml2-xml_gh_931

C/C++

commit 24d7e15

src

24d7e15

71e1e8a

libxml2-xml_gh_932

C/C++

commit 069bcda

src

069bcda

499bcb7

libxml2-xml_gh_933

C/C++

commit 24d7e15

src

24d7e15

71e1e8a

php-php_gh_13094

C/C++

issue #13094

bin

71a1def

1d6f344

php-php_gh_13931

C/C++

issue #13931

bin

5f9b9c4

c3acfb1

vim-vim_gh_14738

C/C++

issue #14738

text

8c35c26

6779719

vim-vim_gh_16606

C/C++

issue #16606

text

97baa1c

c5654b8

vim-vim_gh_17005

C/C++

issue #17005

text

c6336ac

f13c856

wabt-wabt_gh_1524

C/C++

issue #1524

bin

40f6a8a

713bece

wabt-wabt_gh_2398

C/C++

issue #2398

bin

c6853eb

874caa4

wabt-wabt_gh_989

C/C++

issue #989

text

21a9de4

943164d

wabt-wabt_gh_991

C/C++

issue #991

text

943164d

fb7e866

wireshark-wireshark_commit_129565f

C/C++

commit 10e9024

text

10e9024

129565f

altcha-lib-go-4a56107

Go

GHSA-6GVQ-JCMP-8959

src

361dbe3

4a56107

apiserver-4df268e

Go

GHSA-833M-37F7-JQ55

src

c17b7df

4df268e

archiver-09b5706

Go

CVE-2018-25046

src

7291196

09b5706

avro-b4a402f

Go

GHSA-9X44-9PGQ-CF45

src

3abfe1e

b4a402f

aws-iam-authenticator-029d1dc

Go

issue #472

src

0a72c12

029d1dc

beats-dd3af18

Go

PR #47839

src

a042afd

dd3af18

beego-64cf44d

Go

issue #4946

src

daf779f

64cf44d

beego-939bb18

Go

GHSA-2J42-H78H-Q4FG

src

1f40a88

939bb18

beego-d5df5e4

Go

GHSA-28R6-JM5H-MRGG

src

d7a918f

d5df5e4

bluemonday-524f142

Go

issue #111

src

3cce251

524f142

chi-6eb3588

Go

commit 6eb3588

src

de0d16e

6eb3588

consul-template-d6a6f4a

Go

commit d6a6f4a

src

a8022bc

d6a6f4a

docconv-b19021a

Go

commit b19021a

src

44d6739

b19021a

echo-0ac4d74

Go

issue #2259

src

d77e8c0

0ac4d74

Continued on next page

18

Table 6 (continued) Case ID

Lang

Bug Source

PoC

Trigger

Patch

etcd-7d1cf64

Go

GHSA-M332-53R6-2W93

src

91b1a91

7d1cf64

exporter-toolkit-2528877

Go

GHSA-7RG2-CXVP-9P7P

src

48868f5

2528877

expr-0d19441

Go

GHSA-93MQ-9FFX-83M2

src

77e876b

0d19441

fiber-b8c9ede

Go

GHSA-3Q5P-3558-364F

src

b8c9ede

b8c9ede

gin-81ac7d5

Go

PR #3500

src

d07db17

81ac7d5

gin-bfc8ca2

Go

issue #2473

src

f3de813

bfc8ca2

git-lfs-0cffe93

Go

GHSA-6PVW-G552-53C5

src

e735de5

0cffe93

gjson-192-f0ee9eb

Go

issue #192

src

9f58baa

f0ee9eb

gjson-196-bf4efcb

Go

issue #196

src

9f58baa

bf4efcb

gnark-crypto-5660088

Go

issue #1483

src

1873045

5660088

go-rpmutils-a64058c

Go

commit a64058c

src

e314ffa

a64058c

go-slug-764785b

Go

commit 28cafc5

src

dbc66eb

764785b

go.uuid-75cca53

Go

issue #73

src

36e9d2e

75cca53

goutils-869801f

Go

GHSA-XG2H-WX96-XGXR

src

f192353

869801f

goyave-5836bff

Go

issue #3293

src

fc57b4c

5836bff

http-swagger-b7d83e8

Go

commit b7d83e8

src

25e73d2

b7d83e8

interceptor-fa5b35e

Go

issue #3148

src

7914551

fa5b35e

jose2go-a4584e9

Go

issue #31

src

c5d6a57

a4584e9

jsonparser-188-91ac968

Go

issue #188

src

1a29609

91ac968

jwt-7b1c1c0

Go

GHSA-29WX-VH33-7X7R

src

9358574

7b1c1c0

jwt-go-ec0a89a

Go

issue #422

src

32acca3

ec0a89a

jwx-0e8802c

Go

GHSA-PVCR-V8J8-J5Q3

src

ea70886

0e8802c

jwx-64f2a22

Go

GHSA-7F9X-GW85-8GRF

src

55000b3

64f2a22

jwx-d43f2ce

Go

GHSA-HJ3V-M684-V259

src

778ed27

d43f2ce

kubernetes-dad0e93

Go

issue #101695

src

90f5048

dad0e93

mapstructure-742921c

Go

GHSA-2464-8J7C-4CJM

src

5d1b937

742921c

minio-c1a4949

Go

issue #21647

src

334c313

c1a4949

nuclei-0da993a

Go

GHSA-7H5P-MMPP-HGMM

src

4d5eb9c

0da993a

oauth2-proxy-780ae4f

Go

GHSA-4MF2-F3WH-GVF2

src

c377466

780ae4f

opa-064f616

Go

issue #4762

src

243f526

064f616

phonenumbers-152-0479e35

Go

issue #148

src

ccd8c4b

0479e35

protobuf-go-f01a588

Go

GHSA-8R3F-844C-MC37

src

235ef28

f01a588

proxyprotocol-5c4a101

Go

issue #1

src

18c1c89

5c4a101

rekor-39bae3d

Go

GHSA-273P-M2CW-6833

src

812e699

39bae3d

rekor-cf42ace

Go

GHSA-2H5H-59F5-C5X9

src

46ac0b2

cf42ace

revel-d160ecb7

Go

issue #1424

src

60c3d7a

d160ecb

saml-8e92368

Go

GHSA-5MQJ-XC49-246P

src

2aeb2ef

8e92368

saml-b07b16c

Go

GHSA-267V-3V32-G6Q5

src

5f6e5ab

b07b16c

schema-cd59f2f

Go

GHSA-3669-72X9-R9P3

src

180f71e

cd59f2f

shoutrrr-6a27056

Go

issue #240

src

2c9378b

6a27056

submariner-operator-b27a04c

Go

issue #3041

src

7c94448

b27a04c

text-23ae387

Go

issue #39491

src

3a82255

23ae387

woodpecker-8aa3e5e

Go

issue #3929

src

904d8da

8aa3e5e

xz-CVE-2020-16845

Go

issue #35

src

067145b

69c6093

yaml-53403b5

Go

issue #89535

src

1f64d61

53403b5

yaml-8f96da9

Go

issue #666

src

8f96da9

539c8e7

yaml-bb4e33b

Go

PR #375

src

7b8349a

bb4e33b

yaml-f221b84

Go

PR #515

src

bb4e33b

f221b84

Continued on next page

19

Table 6 (continued) Case ID

Lang

Bug Source

PoC

afire-da7904c

Rust

commit da7904c

src

5644d94

da7904c

ammonia-142-4b8426b

Rust

GHSA-5325-XW5M-PHM3

src

ae3fb56

4b8426b

branca-7da3274

Rust

issue #22

src

aeadd4d

7da3274

clatter-b65ae6e

Rust

GHSA-253Q-9Q78-63X4

src

0fb7a00

b65ae6e

cocoon-1b63921

Rust

issue #22

src

680b68c

1b63921

cors-4c32059

Rust

issue #170

src

8d33ca4

4c32059

fuel-vm-9c97c2b

Rust

PR #941

src

d8fa4ae

9c97c2b

nano-arena-rustsec20210031

Rust

RUSTSEC-2021-0031

src

f5306c7

6b83f9d

rust-smallvec-rustsec20190009

Rust

RUSTSEC-2019-0009

src

dfd78d1

4ba0d0f

rust-smallvec-rustsec20210003

Rust

RUSTSEC-2021-0003

src

7852204

9998ba0

stack_dst-rs-rustsec20210033

Rust

RUSTSEC-2021-0033

src

807e9d4

2a4d538

string-interner-rustsec20190023

Rust

RUSTSEC-2019-0023

src

6a9246d

04731e1

hpack-d669282

Rust

issue #11

src

e833eca

d669282

mongo-rust-driver-b918cd6

Rust

PR #1453

src

007894c

b918cd6

nanoid-10-a902277

Rust

GHSA-9HC7-6W9R-WJ94

src

d8d2113

a902277

reffers-rs-6dd7ca0

Rust

issue #7

src

34d9878

6dd7ca0

rs-stellar-strkey-83adad0

Rust

issue #58

src

9d308ac

83adad0

rust-ordered-float-c55cda3

Rust

RUSTSEC-2020-0082

src

e8fe06a

c55cda3

rust-phonenumber-b792151

Rust

RUSTSEC-2024-0369

src

c0fdd69

b792151

rust-protobuf-f06992f

Rust

RUSTSEC-2024-0437

src

8d045d2

f06992f

rustix-31fd98c

Rust

GHSA-C827-HFW6-QWVM

src

4bb2f4b

31fd98c

serde-yaml-b93aff6

Rust

RUSTSEC-2018-0005

src

49e1e6f

b93aff6

stack_dst-2a4d538

Rust

issue #5

src

807e9d4

2a4d538

string-interner-d91dac0

Rust

issue #9

src

f5ffa9d

d91dac0

webp-62b4706

Rust

issue #40

src

d523893

62b4706

20

Trigger

Patch

Related documents

• On the Reliability of LLM-Based Vulnerability Patching Benchmarks#1234517
• SWE-Test: Benchmarking LLM Vulnerability Discovery via Input Prediction#973824
• Repair or Resample? Rethinking Failure Debugging in LLM Multi-Agent Systems#793925
• Repair or Resample? Rethinking Failure Debugging in LLM Multi-Agent Systems#789906
• ProtoArena: Can AI Agents Discover and Exploit Real-World Cross-Component Protocol Vulnerabilities?#1343041