VulnGym: Benchmarking Coding Agents for Repository-Level Vulnerability Detection

VulnGym: Benchmarking Coding Agents for Repository-Level Vulnerability Detection Kexing Ji1, Jiachen Liu1, Enze Hu1, Cuiyun Gao2, Keke Lian1, Yongheng Liu3, Lei Zhang3, Tian Dong4, Hao Chen4, Wang Bin5 1Tencent Wukong 2The Chinese University of Hong Kong 3Fudan University 4The University of Hong Kong 5Peking University kexing1208@gmail.com, jiachenliu1024@gmail.com, slamhu@tencent.com, cuiyungao@outlook.com, kekelian@tencent.com, yhliu24@m.fudan.edu.cn, zxl@fudan.edu.cn, nsectian@gmail.com, chenho@hku.hk, 2201111747@stu.pku.edu.cn

arXiv:2608.02001v1 [cs.SE] 3 Aug 2026

Abstract Recent advances in LLM-based vulnerability detection have shown promising results, while coding agents further extend this capability from isolated code snippets to complete repositories. This shift requires agents to autonomously explore repositories and locate vulnerability-relevant code, instead of performing detection on preselected functions. However, existing benchmarks primarily focus on vulnerability classification over preselected code snippets, limiting their ability to evaluate coding agents in repository-level vulnerability detection. Moreover, without fine-grained vulnerability trace annotations, the capability limitations underlying the detection process remain difficult to explore. To address these limitations, we present VulnGym, a real-world repository-level benchmark for evaluating vulnerability detection by coding agents. VulnGym aligns reviewed GitHub advisories with their corresponding vulnerable version repositories. It contains 184 advisories and 408 vulnerability entries across 23 repositories, with each entry annotated with line-level entry points, critical operations, and vulnerability traces. Using this fine-grained ground truth, VulnGym defines an end-to-end detection task and three oracle-based subtasks to jointly evaluate vulnerability detection and diagnose limitations in code localization and evidence construction. Our evaluation indicates that current coding agents remain limited in both end-to-end repositorylevel vulnerability detection and the construction of accurate supporting traces.

Introduction Large language models (LLMs) have shown promising performance in function-level vulnerability detection (Zhou et al. 2025a; Yildiz et al. 2025; Zhou, Zhang, and Lo 2024). Building on these advances, coding agents further combine iterative reasoning and planning with tool interaction (Hong et al. 2024; Yang et al. 2024; Shinn et al. 2023), enabling them to search files, trace code dependencies, and perform repository-level vulnerability detection (Wang et al. 2024; Yildiz et al. 2025). This paradigm better aligns vulnerability Copyright © 2027, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.

detection with security practice through complete-repository analysis rather than preselected code snippets (Li, Dutta, and Naik 2025; Wen et al. 2026). However, this shift has exposed a substantial gap between existing vulnerability benchmarks and the requirements for evaluating code agents (Islam, Ali, and Parvez 2024; Zhang et al. 2024; Qian et al. 2024; Anysphere, Inc. 2026; Anthropic 2025; Yang et al. 2024; All Hands AI 2024) at the repository level. These benchmarks generally focus on preselected functions, without extending the evaluation scope from located code to the broader repository (Lian et al. 2026; Yildiz et al. 2025). In contrast, real-world vulnerabilities often span multiple functions, files, and modules. For example, a stored XSS vulnerability (GitHub Advisory Database 2025) in Open WebUI (Baek 2026) propagates user-controlled prompt content through multiple components before reaching an unsafe innerHTML assignment. Repository-level vulnerability detection requires coding agents to navigate the codebase and trace paths from entry points to unsafe operations to identify a vulnerability (Wang et al. 2025). Recent work has broadened the evaluation scope from isolated functions to repository-level detection scenarios. VulEval (Wen et al. 2026) augments target functions with repository-level dependency context and evaluates vulnerability classification under broader code contexts. Similarly, JITVul (Yildiz et al. 2025) evaluates LLMs through binary classification of preselected functions extracted from paired vulnerable and patched repository versions. Despite offering valuable insights into LLM-based vulnerability detection, these works share several key limitations for comprehensive benchmarking. First, their evaluation remains centered on predefined target functions, so agents are not required to explore the broader repository and locate vulnerability-relevant code autonomously. Second, their classification-oriented ground truth lacks fine-grained vulnerability-relevant traces, making in-depth analysis of detection behavior difficult. Consequently, an agent may predict the correct label on given codes yet locate the wrong defect or rely on irrelevant evidence, while the evaluation cannot reveal where in the de-

tection process the failure occurs. Together, these limitations raise a critical question: how can we evaluate and diagnose where the vulnerability detection process of coding agents fails in real-world repositories? To answer this question, we introduce VulnGym, a benchmark for evidence-grounded vulnerability detection by coding agents in real-world repositories. VulnGym contains 184 reviewed advisories and 408 vulnerability entries across 23 open-source repositories, with each entry aligned to its vulnerable repository version. Rather than providing only vulnerability labels or predefined targets, VulnGym annotates each entry with line-level entry points, critical operations, and vulnerability traces. These annotations enable both evaluating repository-level vulnerability detection and diagnosing where in the detection process an agent fails. Based on this ground truth, VulnGym defines an end-to-end detection task and three oracle-based subtasks. The end-to-end task jointly evaluates vulnerability findings and their supporting evidence, while the oracle-based subtasks diagnose limitations in locating relevant code and constructing traces. Our main contributions are as follows: • We introduce VulnGym, a real-world repository-level benchmark with fine-grained code evidence annotations. It contains 184 reviewed advisories and 408 vulnerability entries across 23 open-source repositories. • We design an evaluation framework comprising an endto-end detection task and three oracle-based subtasks that jointly assess vulnerability detection and diagnose capability limitations in code localization and evidence construction. • We conduct a systematic empirical study of current coding agents across model scales and LLM–agent combinations. Our findings reveal capability limitations underlying vulnerability detection.

Related Work Early vulnerability benchmarks formulate detection as function-level binary classification (Lin et al. 2019). BigVul (Fan et al. 2020) and Devign (Zhou et al. 2019) provide large labeled function collections but suffer from label noise and duplication (Croft et al. 2021; Tamberg and Bahsi 2025; Liu et al. 2024). PrimeVul (Ding et al. 2024) improves labeling accuracy through automated strategies and removes exact duplicates using MD5 hashing, while SecVulEval (Ahmed et al. 2026) refines annotations to the statement level and extracts multiple forms of code context. These benchmarks still lack inter-procedural context and complete execution chains from entry points to critical operations. VulEval (Wen et al. 2026) and CWE-Bench-Java (Li, Dutta, and Naik 2025) extend evaluation to repositories. VulEval localizes vulnerable files and lines from patches and provides cross-function call dependencies; CWE-BenchJava contains 120 manually verified vulnerabilities in compilable Java projects, each linked to its fix commit. Both primarily evaluate classification accuracy, rather than autonomous repository exploration, and neither provides vulnerability execution chains.

Benchmark

Level

Location Cause Chain Val.

PrimeVul Function SecVulEval Function VulEval Repository CWE-Bench-Java Repository JITVul Repository

% ! ! % %

% % % % %

% % % % %

% % % ! %

VulnGym

!

!

!

!

Repository

Table 1: Comparison of vulnerability benchmarks. Level denotes the evaluation scope; Location denotes vulnerable file and line localization; Cause denotes a natural-language defect cause; Chain denotes a line-level execution trace across files; Val. indicates whether annotations are human-verified. JITVul (Yildiz et al. 2025) is the first benchmark of ReAct (Yao et al. 2022) agents for repository-level vulnerability detection. It frames detection as pairwise classification of vulnerability-introducing and fixing commits, lets agents retrieve cross-procedural context through tool calls, and proposes pAcc to avoid inflated F1 from over-predicting the vulnerable class. It nevertheless targets preselected functions and lacks chain-level annotations for assessing reasoning completeness. Table 1 summarizes existing benchmarks across five dimensions. No benchmark covers all five, particularly Chain and Cause, which are needed to evaluate autonomous repository exploration and complete evidence-chain construction. VulnGym fills these gaps with repository-level line annotations, natural-language defect causes, and human-verified cross-module Traces.

VulnGym Benchmark Construction As illustrated in Figure 1, VulnGym is constructed in three phases: dataset generation, manual verification, and entry refinement. Phase I aligns advisories with their vulnerable repository snapshots and generates structured vulnerability records. Phase II verifies these records through source-code alignment and human review. Records that do not pass verification enter Phase III and return to Phase II after refinement, forming an iterative verification loop. Phase I: Dataset Construction. We begin with 3,632 reviewed high risk severity GitHub Advisories(GitHub 2026) published between January 2025 and April 2026, covering 1,579 open-source repositories. To reduce potential data leakage when benchmarking coding agents(Deng et al. 2023; Zhou et al. 2025b), we select advisories published after the training cutoff of LLM, from November 2025 to April 2026, and retain only repositories with more than 10,000 GitHub stars. For each remaining advisory, we identify the corresponding vulnerable repository and establish its vulnerable snapshot using the detailed description and vulnerability databases. Advisories whose repositories cannot be located or whose vulnerable snapshots cannot be reliably established are excluded, resulting in 184 advisories.

Phase I: Dataset Construction

Phase II: Manual Verification

Phase III: Entry Refinement

1 Vulnerability Report Collection

1 Code-Aligned Verification

1 Repair Target Selection

Report Retrieval Description Selection Vulnerability Deduplication

2 Entry Construction

Vuln. Trace

Entry Point

Critical Operation

Exact Match

Line Offset

Mismatch

Missing File

Annotation Filter

Target

2 Entry Reconstruction Code Relocation Entry Update

Vulnerability Repository Verified Data-flow

Control-flow

Trace Construction

2 Manual Review Correct

Repair Incorrect Unsolved

Update

Final Verified Dataset

Figure 1: Overview of the VulnGym benchmark construction pipeline. For each vulnerable repository snapshot, we construct reachable-entry records by identifying externally reachable Entry Points (EPs), security Critical Operations (COs), and the Traces connecting them through data-flow and controlflow analysis. The EP is an externally reachable code location that can trigger the vulnerability, such as an API handler or event callback. The CO is a security-critical location where the vulnerability takes effect, e.g., an unsafe operation or a missing security check. The Trace is an ordered execution path connecting an EP to a CO, with each step corresponding to a specific file location and code snippet. When multiple independent EP–CO paths exist, each is stored as a separate reachable-entry record. Phase II: Manual Verification. This phase first compares the code locations associated with each EP, CO, and Trace step line by line against the target source code and classifies the result as an exact match, line offset, content mismatch, or missing file. Two reviewers then independently verify the correctness of each entry based on the vulnerability description, including the external reachability of the EP, security relevance of the CO, and ordering of the Trace. Each entry is labeled as Correct, Repair, Incorrect, or Unsolved. Correct denotes a fully valid entry. Repair indicates an otherwise valid entry with localized errors in line numbers or the Trace, while Incorrect indicates that the entry misidentifies the vulnerability entirely. Unsolved marks cases that reviewers cannot determine confidently and that require third-party adjudication. When the two reviewers disagree on a label or an entry is labeled as Unsolved, a third senior reviewer adjudicates the case and makes the final decision. We further conduct inter-rater agreement measurement using Cohen’s Kappa (Ma et al. 2024), with a score of κ = 0.87, indicating high agreement. The review labels, together with annotations of erroneous line numbers and manual corrections for entries requiring changes, are then passed to Phase III for entry refinement.

Other(20.93%) SSRF(9.30%) Code Injection(16.28%) Business Logic(51.16%) Business Logic(100.00%) Business Logic(62.50%) Business Logic(55.56%) Path Traversal(33.33%) Business Logic(66.67%) Business Logic(94.12%) Other(13.64%) Deserialization(86.36%) SSRF(9.30%) Business Logic(90.70%) Other(15.38%) Sandbox Escape(9.62%) Command Injection(9.62%) Business Logic(15.38%) XSS(17.31%) Code Injection(30.77%) Business Logic(86.22%) Other(5.61%) Command Injection(2.04%) Sandbox Escape(3.06%) Path Traversal(3.06%)

Figure 2: Repository-level composition of VulnGym. The inner ring shows the proportion of reachable-entry records contributed by each repository; the outer ring shows the distribution of vulnerability categories.

Phase III: Entry Refinement. This phase selects repair targets based on the annotation labels and line-level correction feedback produced in Phase II. For each selected entry, the refinement process first identifies the corresponding source files and code lines in the target repository. It then updates the relevant entry fields based on the relocated code. Entries requiring no revision after source-code validation are marked as Verified, whereas entries with sourcegrounded corrections are marked as Update and returned to Phase II for another round of verification. Entries whose corresponding source files or line numbers cannot be reliably identified are excluded from the final verified dataset. Following the pipeline above, we construct VulnGym with 408 reachable-entry records derived from 184 advisories

Evaluation Metrics

T1 End-to-End Vulnerability Detection

Detection Output

Input s Advisory

Entry Point (EP)

(Title & Desc.)

Vulnerable Repository

Critical Operation (CO) Coding Agent

T2 EP Identification

T3 CO Localization

Input

Input

Oracle CO

Oracle EP

Vulnerability Trace

T4 Trace Construction

Input Oracle CO Oracle EP

Detect

Detect

Outputs

Outputs

EP

CO

Detect Outputs Trace

Figure 3: Overview of the four tasks in VulnGym. T1 performs end-to-end vulnerability detection from an advisory and a vulnerable repository, whereas T2–T4 provide oracle components to isolate EP identification, CO localization, and Trace construction.

across 23 open-source repositories. Figure 2 shows the number of entries from each repository and the corresponding distribution of vulnerability categories.

Evaluation Framework As shown in Figure 3, VulnGym defines four repository-level tasks comprising one end-to-end vulnerability detection task (T1) and three oracle-conditioned subtasks (T2–T4). The upper panel illustrates the end-to-end setting, while the lower panels show how oracle components isolate the capabilities required to identify EPs, localize COs, and construct vulnerability Traces. End-to-End Vulnerability Detection. In T1, the coding agent receives a vulnerable repository together with the advisory title and description, and identifies one or more vulnerability entries comprising an EP, a CO, and their connecting Trace. The task jointly evaluates repository exploration, vulnerability localization, reachability analysis, and evidence construction. Oracle-Conditioned Tasks. For fine-grained capability evaluation in end-to-end vulnerability detection, we further construct three oracle-conditioned tasks. • T2: Entry-Point Identification provides an oracle CO and requires the coding agent to identify the externally reachable EPs that can lead to it. • T3: Critical-Operation Localization provides an oracle EP and requires the coding agent to locate all COs reachable from it. • T4: Trace Construction provides an oracle-bound EP–CO pair and requires the coding agent to construct its ordered evidence Trace between them.

Across all four tasks, evaluation is based on vulnerability entries represented as EP → Trace → CO. An advisory comprises one or more entries, each corresponding to an independently reachable EP–CO path. We compare predicted and reference entry components using a unified line-level location-matching criterion: two code locations match when they share the same normalized repositoryrelative path and are within k = 5 lines of each other. Accordingly, we next introduce the metrics for each task. End-to-End Vulnerability Detection. For T1, we report Entry Recall (ER), Advisory Recall (AR), and Edit Similarity (ES). A reference entry is hit when a prediction matches both its EP and CO locations. • ER measures the proportion of reference entries hit: ER =

hit Nentry , Nentry

(1)

hit where Nentry is the number of reference entries and Nentry the number hit. • AR measures the proportion of reference advisories hit, where an advisory is hit if at least one of its entries is hit:

AR =

hit Nadv , Nadv

(2)

hit where Nadv is the number of reference advisories and Nadv the number hit. • ES measures code-location and invocation-order similarity between predicted and reference Traces for hit entries:

ES(P, G) = 1 −

DLev (P, G) , max(n, m)

(3)

where P = (p1 , . . . , pn ) and G = (g1 , . . . , gm ) are the predicted and reference Traces, and DLev is their Levenshtein edit distance. EP Identification and CO Localization. The EPidentification and CO-localization tasks share three metrics: File Recall (FR), Line Recall (LR), and Finding Precision (FP). Given oracle COs, T2 evaluates predicted EPs, while T3 evaluates predicted COs under oracle EPs. • FR measures the proportion of reference targets localized to the correct file: Mfile FR = , (4) NG where Mfile is the number of file-level matches and NG the number of reference targets. • LR measures the percentage of reference targets matched by a prediction within k lines: LR =

Mk , NG

where Mk is the number of line-level matches.

(5)

Easy 30% 20% 10% 0%

2B

4B 9B 27B Model Size (a) Advisory Recall

Medium

Hard

20%

20%

15%

15%

10%

10%

5%

5%

0%

2B

0%

4B 9B 27B Model Size (b) Entry Recall

2B

4B 9B 27B Model Size (c) Edit Similarity

Figure 4: Scaling performance of Qwen3.5 models under Claude Code across the Easy, Medium, and Hard subsets. • FP measures the percentage of predictions that correctly match a reference target within k lines: FP =

Mk , NP

(6)

where NP is the number of predictions. Trace Construction. T4 reuses ES and additionally reports Trace Node Recall (TNR) and Trace Node Precision (TNP). • TNR measures the percentage of reference intermediate nodes matched by a prediction: TNR =

MI , NIG

(7)

where MI is the number of one-to-one node matches and NIG the number of reference intermediate nodes. • TNP measures the percentage of predicted intermediate nodes that match a reference node: MI TNP = P , (8) NI where NIP is the number of predicted intermediate nodes.

Experiment Experimental Setup LLM Selection To support repository-level code exploration, we select models with context windows of at least 256K tokens Specifically, we use Qwen3.5-2B, 4B, 9B, and 27B (Qwen Team 2026) for scaling analysis within a single model family. We also include three frontier LLMs for model comparison: DeepSeek-V4-Flash (Xu et al. 2026), MiniMaxM3 (MiniMax 2026), and GLM-5.2 (Z.ai 2026). Agentic Scaffolds We evaluate three widely used codingagent scaffolds: Claude Code (Anthropic 2025), OpenHands (All Hands AI 2024), and MiniSWE (Yang et al. 2024). For each task, the agent is restricted to invoking only read-only commands, including cat, find, grep, head, ls, nl, pwd, rg, sed, tail, and wc.

Scaffold

LLM

End-to-End Detection AR (%)↑ ER (%)↑ ES (%)↑

DeepSeek-V4-Flash Claude Code GLM-5.2 MiniMax-M3

9.68 12.90 12.90

6.52 8.70 8.70

3.37 6.94 6.25

OpenHands

DeepSeek-V4-Flash GLM-5.2 MiniMax-M3

22.58 6.45 16.13

15.22 4.35 10.87

11.63 2.15 7.93

MiniSWE

DeepSeek-V4-Flash GLM-5.2 MiniMax-M3

16.13 16.13 16.13

10.87 10.87 10.87

7.67 8.65 6.95

Table 2: End-to-end vulnerability performance of frontier coding agent on Hard cases.

Model Scaling across Difficulty Levels We stratify VulnGym’s verified entries by Trace length into Easy (≤ 7), Medium (= 8), and Hard (≥ 9), keeping Medium and Hard comparable, comprising 110/31/31 advisories and 278/56/46 entries, respectively. To investigate model scaling, we evaluate Qwen3.5-2B, 4B, 9B, and 27B using Claude Code as the agentic scaffold and report their T1 performance at each difficulty level. With model scaling, coding agents generally achieve stronger end-to-end vulnerability detection. As shown in Figure 4, the 2B and 4B models obtain non-zero scores only on Easy, while 9B extends non-zero performance to Medium. Qwen3.5-27B leads on every metric at every difficulty level and is the only model to achieve non-zero performance on Hard. Notably, scaling up model size does not consistently lead to better performance. 4B outperforms 9B on all three metrics at the Easy level, suggesting that the gains from model scaling may vary with case difficulty. Across model sizes, end-to-end vulnerability detection performance declines consistently as case difficulty increases. For Qwen3.5-9B, AR, ER, and ES decrease from 6.36%, 4.32%, and 2.52% on Easy to 3.23%, 1.79%, and 1.67% on Medium, respectively, and all reach zero on Hard.

No Usable Finding EP Miss (CO-only) CC / DS

5

CC / GLM 1

No Endpoint File CO Miss (EP-only)

5 11

15 12

3 MS / DS 2 MS / GLM 1 OH / M3

8 16 6

2 20%

8 5

12 9

8 16 8 15 40%

60%

3 3

7

15 18 12

OH / GLM

5 5

9

9

OH / DS 1

0%

10 10

18

CC / M3 2

MS / M3

Endpoint File Only

9 10 12 80%

5 6

1 3 3 3 100%

Figure 5: Composition of relation-level T1 failures on Hard. CC: Claude Code; OH: OpenHands; MS: MiniSWE; DS: DeepSeek-V4-Flash; GLM: GLM-5.2; M3: MiniMax-M3. The 2B and 4B variants reach zero on all three metrics at Medium and remain on Hard. Even Qwen3.5-27B achieves merely 12.90% AR, 8.70% ER, and 4.55% ES on Hard.

Frontier Coding Agents on Hard Cases Although Qwen3.5-27B performs best among the scaling models, its end-to-end performance on Hard remains limited. To assess whether stronger coding agents can improve upon these limited results, we pair DeepSeek-V4-Flash, GLM-5.2, and MiniMax-M3 with all three scaffolds and evaluate the resulting nine frontier coding agents on T1 over Hard level. Frontier coding agents improve over Qwen3.5-27B, but Hard entries remain largely unresolved. As shown in Table 2, DeepSeek-V4-Flash with OpenHands performs best, achieving 22.58% AR, 15.22% ER, and 11.63% ES, compared with 12.90%, 8.70%, and 4.55% for Qwen3.5-27B with Claude Code. Nevertheless, even this best-performing coding agent detects fewer than one quarter of hard-level entries, leaving substantial room for improvement. Coding-agent performance varies across LLM– scaffold combinations, with scaffolds differing in their sensitivity to LLM selection. DeepSeek-V4-Flash performs best with OpenHands, but its performance drops to 9.68% AR, 6.52% ER, and 3.37% ES with Claude Code. By contrast, GLM-5.2 exhibits the opposite pattern, with performance increasing from 6.45% AR, 4.35% ER, and 2.15% ES with OpenHands to 12.90%, 8.70%, and 6.94% with Claude Code. Unlike these model-specific reversals, MiniSWE yields the same AR of 16.13% and ER of 10.87% across all three LLMs, while its ES ranges narrowly from 6.95% to 8.65%. These results indicate coding agent compatibility is critical to end-to-end performance, while showing that MiniSWE remains comparatively stable across the evaluated LLMs. T1 failures mainly involve missing the vulnerable files entirely or failing at line-level localization. Figure 5 summarizes the composition of 347 relation-level T1 failures. No Endpoint File indicates that neither predicted EP or CO matches its corresponding reference file (93; 26.8%), whereas Endpoint File Only means that at least one endpoint

reaches the correct file but neither is localized to the correct line (103; 29.7%). Together, these two categories account for 56.5% of all failures. When one endpoint is localized, EP Miss (CO-only) is substantially more common than CO Miss (EP-only), with 80 (23.1%) versus 32 (9.2%) cases, suggesting that EP identification is more frequently unresolved in T1 outputs. No Usable Finding accounts for the remaining 39 failures (11.2%), including 18 from OpenHands with GLM-5.2 alone. Overall, these results reveal where end-toend predictions fall short, but provide only a coarse-grained view of the underlying capability limitations.

Oracle-Conditioned Capability Analysis To better understand these limitations, we next perform oracle-conditioned evaluation to analyze the capabilities underlying end-to-end vulnerability detection. Under the respective oracle conditions, EP identification is generally more challenging than CO localization. As shown in Table 3, CO localization yields higher FR across all LLM–scaffold combinations except OpenHands with GLM-5.2. For LR, only OpenHands with GLM-5.2 yields higher performance on EP identification than on CO localization among the nine coding agents, reaching 9.30% and 2.33%, respectively. Across all coding agents, the highest CO-localization FR and LR are 51.16% and 30.23%, respectively, compared with 30.23% and 18.60% for EP identification. However, FP reveals a precision–recall trade-off, as EP identification generally achieves higher scores than CO localization, indicating more selective but lower-coverage predictions. Nevertheless, FP peaks at only 11.11% for EP identification and 5.22% for CO localization, indicating that most predicted findings remain unmatched at the line level. Even with the oracle EP–CO pair provided, the vulnerable evidence Traces remain incomplete and noisy. Across coding agents, TNR ranges from 30.60% to 53.96%, whereas TNP remains substantially lower, ranging from 14.61% to 26.71%. This recall–precision imbalance indicates that coding agents recover some reference intermediate nodes but include many unmatched nodes in their predictions. After endpoint uncertainty is removed, the highest ES is still only 36.37%, indicating persistent difficulty in constructing accurate evidence Traces.

Discussion Practical Efficiency Analysis Beyond detection effectiveness, we further evaluate the practical efficiency of coding agents. We examine how per-run resource consumption relates to end-to-end vulnerability detection. As shown in Figure 7(a), the token–AR Pareto frontier extends from Claude Code with GLM-5.2 to OpenHands with DeepSeek-V4-Flash. Since API pricing varies across models, Figure 7(b) further compares per-run cost with AR. OpenHands with DeepSeek-V4-Flash achieves the highest AR while remaining among the least expensive configurations, whereas OpenHands with GLM-5.2 incurs the highest cost but obtains the lowest AR. Overall, higher resource consumption does not necessarily yield more effective vulnerability discovery.

Scaffold

EP Identification

LLM

CO Localization

Trace Construction

FR (%)↑ LR (%)↑ FP (%)↑ FR (%)↑ LR (%)↑ FP (%)↑ ES (%)↑ TNR (%)↑ TNP (%)↑ DeepSeek-V4-Flash Claude Code GLM-5.2 MiniMax-M3

13.95 25.58 13.95

4.65 9.30 2.33

1.69 5.97 0.79

34.88 48.84 39.53

16.28 30.23 16.28

0.91 5.22 1.23

34.68 32.07 22.01

53.96 39.86 53.60

20.22 19.79 14.61

OpenHands

DeepSeek-V4-Flash GLM-5.2 MiniMax-M3

18.60 18.60 23.26

2.33 9.30 16.28

1.01 8.16 8.05

37.21 6.98 46.51

20.93 2.33 23.26

2.23 1.33 3.25

26.14 16.47 26.37

41.64 30.60 52.67

19.63 26.71 19.53

MiniSWE

DeepSeek-V4-Flash GLM-5.2 MiniMax-M3

16.28 30.23 13.95

6.98 18.60 6.98

3.49 11.11 4.69

51.16 39.53 46.51

16.28 18.60 18.60

1.64 2.54 3.11

27.77 36.37 30.20

37.72 49.47 39.86

16.13 23.28 20.07

Table 3: Oracle-conditioned tasks performance on Hard cases. Within each scaffold, bold values denote the best performance in each column.

Number of Actions

OpenHands

MiniSWE

Claude Code 60

70

60

50

60

50

50

40

40

40

30

30

30

20

20

20

10

10

10

0

0

0 0

20

40

60

80

100

0

20

Turns

40

60

80

100

0

Explore

Search

Read

20

40

60

80 100 120 140

Turns

Turns Other

Failed

Figure 6: Turn-wise action composition of DeepSeek-V4-Flash on Hard under T1 across the three agentic scaffolds.

DS-v4-Flash

GLM-5.2

MiniMax-M3

25% 20%

OpenHands

MiniSWE

20%

15%

AR

AR

15%

10%

10%

5% 0%

Claude Code 25%

5%

2

3

4

5

6

0%

0

0.2

0.4

0.6

Tokens per Run (M)

Cost per Run (USD)

(a) Tokens vs. AR

(b) Cost vs. AR

0.8

Figure 7: Efficiency–effectiveness trade-offs among frontier coding agents on Hard under T1. The two panels compare AR with tokens and cost per run, respectively.

Agent Trajectory Analysis To characterize agent behavior across interaction turns, we analyze action distributions for DeepSeek-V4-Flash. For this analysis, we categorize the provided commands into three 1 Explore commands (find, ls), O 2 types of actions: O 3 Search commands (rg and grep), and O Read commands 4 Other denotes actions beyond (cat, head, sed, tail). O those explicitly provided to the agent in the prompt.

As shown in Figure 6, Read accounts for the largest share of actions across all configurations, while Explore is concentrated in the initial turns and Search persists throughout the trajectories. The three scaffolds also exhibit distinct action patterns. Claude Code produces comparatively few Failed actions and also executes few Other actions. MiniSWE exhibits a larger share of Other actions, corresponding to commands beyond those specified in the prompt. By contrast, OpenHands produces substantially more Failed actions, particularly during the early turns.

Conclusion We presented VulnGym, a repository-level benchmark for evaluating vulnerability detection by coding agents. It contains 184 advisories and 408 vulnerability entries , with finegrained annotations of entry points, critical operations, and vulnerability traces. Based on these annotations, VulnGym combines an end-to-end task with three oracle-based subtasks to assess overall performance and diagnose capability limitations. Our evaluation shows that challenging vulnerabilities remain largely unresolved and performance varies across LLM–agent combinations. With this benchmark, VulnGym provides a foundation for advancing coding agents toward reliable repository-level vulnerability detection.

References Ahmed, M. B. U.; Harzevili, N. S.; Shin, J.; Pham, H. V.; and Wang, S. 2026. SecVulEval: Context-Aware Benchmarking of LLMs for Vulnerability Detection. In Proceedings of the 3rd ACM International Conference on AI-Powered Software, 388–396. All Hands AI. 2024. OpenHands: An Open Platform for AI Software Developers as Generalist Agents. https://github. com/All-Hands-AI/OpenHands. Anthropic. 2025. Claude Code. https://docs.anthropic.com/ en/docs/claude-code. Agentic command-line tool for software development. Anysphere, Inc. 2026. Cursor: The AI Code Editor. https: //www.cursor.com/. Accessed: 2026-07-29. Baek, T. J. 2026. Open WebUI: User-Friendly AI Interface. https://github.com/open-webui/open-webui. GitHub repository. Accessed: 2026-07-28. Croft, R.; Newlands, D.; Chen, Z.; and Babar, M. A. 2021. An empirical study of rule-based and learning-based approaches for static application security testing. In Proceedings of the 15th ACM/IEEE international symposium on empirical software engineering and measurement (ESEM), 1–12. Deng, C.; Zhao, Y.; Tang, X.; Gerstein, M.; and Cohan, A. 2023. Benchmark probing: Investigating data leakage in large language models. In NeurIPS 2023 workshop on backdoors in deep learning-The good, the bad, and the ugly. Ding, Y.; Fu, Y.; Ibrahim, O.; Sitawarin, C.; Chen, X.; Alomair, B.; Wagner, D.; Ray, B.; and Chen, Y. 2024. Vulnerability detection with code language models: How far are we? arXiv preprint arXiv:2403.18624. Fan, J.; Li, Y.; Wang, S.; and Nguyen, T. N. 2020. AC/C++ code vulnerability dataset with code changes and CVE summaries. In Proceedings of the 17th international conference on mining software repositories, 508–512. GitHub. 2026. GitHub Advisory Database. https://github. com/advisories. Accessed: 2026-07-29. GitHub Advisory Database. 2025. Open WebUI Vulnerable to Stored DOM XSS via Prompts When “Insert Prompt as Rich Text” Is Enabled, Resulting in ATO/RCE. https: //github.com/advisories/GHSA-w7xj-8fx7-wfch. GHSAw7xj-8fx7-wfch; CVE-2025-64495. Accessed: 2026-07-28. Hong, S.; Zhuge, M.; Chen, J.; Zheng, X.; Cheng, Y.; Wang, J.; Zhang, C.; Yau, S.; Lin, Z.; Zhou, L.; et al. 2024. MetaGPT: Meta programming for a multi-agent collaborative framework. In International Conference on Learning Representations, volume 2024, 23247–23275. Islam, M. A.; Ali, M. E.; and Parvez, M. R. 2024. Mapcoder: Multi-agent code generation for competitive problem solving. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 4912–4944. Li, Z.; Dutta, S.; and Naik, M. 2025. IRIS: LLM-assisted static analysis for detecting security vulnerabilities. In International conference on learning representations, volume 2025, 35735–35758.

Lian, K.; Bin, W.; Zhang, L.; Chen, L.; Wang, J.; Zhao, Z.; Yang, Y.; Lin, M.; Duan, H.; Zhao, H.; et al. 2026. Ase: A repository-level benchmark for evaluating security in aigenerated code. In Findings of the Association for Computational Linguistics: ACL 2026, 31390–31405. Lin, G.; Xiao, W.; Zhang, J.; and Xiang, Y. 2019. Deep learning-based vulnerable function detection: A benchmark. In International Conference on Information and Communications Security, 219–232. Springer. Liu, Y.; Gao, L.; Yang, M.; Xie, Y.; Chen, P.; Zhang, X.; and Chen, W. 2024. Vuldetectbench: Evaluating the deep capability of vulnerability detection with large language models. arXiv preprint arXiv:2406.07595. Ma, Y.; Tian, W.; Gao, X.; Sun, H.; and Li, L. 2024. API Misuse Detection via Probabilistic Graphical Model. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 88–99. MiniMax. 2026. MiniMax M3: Frontier Coding, 1M Context, Native Multimodality—All in One Model. https://www. minimax.io/blog/minimax-m3. Published June 1, 2026; accessed July 29, 2026. Qian, C.; Liu, W.; Liu, H.; Chen, N.; Dang, Y.; Li, J.; Yang, C.; Chen, W.; Su, Y.; Cong, X.; et al. 2024. Chatdev: Communicative agents for software development. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), 15174–15186. Qwen Team. 2026. Qwen3.5: Towards Native Multimodal Agents. https://qwen.ai/blog?id=qwen3.5. Published February 15, 2026; accessed July 29, 2026. Shinn, N.; Cassano, F.; Gopinath, A.; Narasimhan, K.; and Yao, S. 2023. Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems, 36: 8634–8652. Tamberg, K.; and Bahsi, H. 2025. Harnessing large language models for software vulnerability detection: A comprehensive benchmarking study. IEEE Access. Wang, X.; Hu, R.; Gao, C.; Wen, X.-C.; Chen, Y.; and Liao, Q. 2024. Reposvul: A repository-level high-quality vulnerability dataset. In Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings, 472–483. Wang, Z.; Shi, T.; He, J.; Cai, M.; Zhang, J.; and Song, D. 2025. CyberGym: Evaluating AI Agents’ RealWorld Cybersecurity Capabilities at Scale. arXiv preprint arXiv:2506.02548. Wen, X.-C.; Wang, X.; Chen, Y.; Hu, R.; Lo, D.; and Gao, C. 2026. From Function to Repository: Towards RepositoryLevel Evaluation of Software Vulnerability Detection. IEEE Transactions on Software Engineering. Xu, A.; Lin, B.; Xue, B.; Wang, B.; Xu, B.; Wu, B.; Zhang, B.; Lin, C.; Dong, C.; Ling, C.; et al. 2026. Deepseek-v4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348. Yang, J.; Jimenez, C. E.; Wettig, A.; Lieret, K.; Yao, S.; Narasimhan, K. R.; and Press, O. 2024. SWE-agent: AgentComputer Interfaces Enable Automated Software Engineer-

ing. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. Yao, S.; Zhao, J.; Yu, D.; Shafran, I.; Narasimhan, K. R.; and Cao, Y. 2022. React: Synergizing reasoning and acting in language models. In NeurIPS 2022 Foundation Models for Decision Making Workshop. Yildiz, A.; Teo, S. G.; Lou, Y.; Feng, Y.; Wang, C.; and Divakaran, D. M. 2025. Benchmarking llms and llm-based agents in practical vulnerability detection for code repositories. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 30848–30865. Z.ai. 2026. GLM-5.2: Built for Long-Horizon Tasks. https: //z.ai/blog/glm-5.2. Published June 16, 2026; accessed July 29, 2026. Zhang, K.; Li, J.; Li, G.; Shi, X.; and Jin, Z. 2024. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 13643– 13658. Zhou, X.; Cao, S.; Sun, X.; and Lo, D. 2025a. Large language model for vulnerability detection and repair: Literature review and the road ahead. ACM Transactions on Software Engineering and Methodology, 34(5): 1–31. Zhou, X.; Weyssow, M.; Widyasari, R.; Zhang, T.; He, J.; Lyu, Y.; Chang, J.; Zhang, B.; Huang, D.; and Lo, D. 2025b. Lessleak-bench: A first investigation of data leakage in llms across 83 software engineering benchmarks. arXiv preprint arXiv:2502.06215. Zhou, X.; Zhang, T.; and Lo, D. 2024. Large language model for vulnerability detection: Emerging results and future directions. In Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results, 47–51. Zhou, Y.; Liu, S.; Siow, J.; Du, X.; and Liu, Y. 2019. Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks. Advances in neural information processing systems, 32.

Related documents

• OpenAnt: LLM-Powered Vulnerability Discovery Through Code Decomposition, Adversarial Verification, and Dynamic Testing#287048
• Deep Agentic Search for Repository-Level Code Question Answering: An Empirical Study#424048
• AutoPenBench: A Vulnerability Testing Benchmark for Generative Agents#1175260
• ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues#820515
• SWE-Cycle: Benchmarking Code Agents across the Complete Issue Resolution Cycle#180726