VEX-B ENCH: Benchmarking LLM Agents for Assessing Exploitability of Software Supply Chain Vulnerabilities Jiahao Shi1 , Edward Tsien2 , Yifeng Di1 , Hongjiao Zhang2 , Yuan Tang2 , Ronit Dey2 , Ilona Shishov2 , Gal Netanel 2 , Zvi Grinberg2 , Vladimir Belousov2 , Bat-Zion Rotman2 , Ilan Pinto2 , Tianyi Zhang1 1 Department of Computer Science, Purdue University 2 Red Hat {shi768, tianyi}@purdue.edu {etsien}@redhat.com Abstract
supply chain attacks have grown 156% year-overyear (Sonatype, 2024), and the 2025 OWASP Top 10 ranks Software Supply Chain Failures third overall, with 50% of surveyed practitioners ranking it their top concern (OWASP Foundation, 2025). A single vulnerability, Log4Shell (NVD, 2021), affected hundreds of millions of devices and 93% of enterprise cloud environments, causing billions of dollars in industry-wide remediation costs. This growing severity stems largely from modern software’s reliance on expansive package ecosystems (e.g., npm, PyPI, Maven) to reduce development costs and accelerate delivery. This creates complex dependencies among packages, allowing a single vulnerable package to propagate risk into a vast number of downstream projects. To defend against such risks, existing tools such as GitHub Dependabot (Dependabot, 2026) scan software components against vulnerability databases to identify known vulnerabilities. However, such coarse-grained matching produces a large number of false alerts, as it relies primarily on package names and versions, without considering contextual information or fine-grained reachability analysis (Pu et al., 2026; Pashchenko et al., 2022). Security analysts are often overwhelmed by these alerts and spend substantial effort triaging them, leading to alert fatigue and causing critical vulnerabilities to be buried among spurious ones (He et al., 2023; Rombaut et al., 2023; Zhou et al., 2026). These limitations motivate leveraging LLM agents, whose strong coding and cybersecurity capabilities (Anthropic, 2026a; OpenAI, 2026b) make them well-suited for assessing exploitability of software supply chain vulnerabilities. However, existing cybersecurity benchmarks for evaluating LLM agents (Zhang et al., 2026; Wang et al., 2026; Lee et al., 2026; Zhu et al., 2025; Zhang et al., 2025) focus on zero-day settings, where agents discover or exploit previously unknown vulnerabilities within a single codebase.
arXiv:2609.08040v1 [cs.CR] 7 Sep 2026
The software supply chain has become an increasingly exposed attack surface because of its reliance on intricate yet fragile dependencies. Existing defenses such as GitHub Dependabot often raise many false alerts because their coarse-grained matching cannot determine whether a vulnerable dependency is actually exploitable. Security analysts typically spend substantial time assessing vulnerability exploitability case by case. Recent LLM agents have emerged as promising candidates for this task given their advanced capabilities in coding and cybersecurity, yet no existing benchmark evaluates them on it. Prior benchmarks target zero-day settings, where agents detect and exploit previously unknown vulnerabilities. In contrast, software supply chain security focuses on how known vulnerabilities in upstream dependencies affect downstream projects. This requires agents to reason across repositories and determine whether an upstream vulnerability is exploitable in the downstream project. To address this gap, we introduce VEX-B ENCH, the first benchmark for evaluating LLM agents’ ability to assess the exploitability of software supply chain vulnerabilities. It contains 75 real-world cases mined from GitHub and labeled by security experts, covering Python, Java, and Go. We evaluate nine models across three agent harnesses. While GPT-5.5 and Claude Opus 4.6 reach approximately 80% F1 on binary vulnerabilitystatus classification, only GPT-5.5 surpasses 70% macro-F1 on fine-grained justification classification. This gap highlights the challenge of moving beyond binary exploitability assessment to identifying fine-grained exploitability reasons. Code and data: https: //github.com/steven1518/vex-bench
1
Introduction
The software supply chain faces increasingly severe security threats. Sonatype reports that software 1
In contrast, software supply chain security poses a completely different setting—the vulnerability is already known and is introduced into the local project through a third-party dependency. This task therefore requires agents to integrate external vulnerability evidence with cross-repository reasoning to assess downstream exposure. To address this gap, we construct VEX-B ENCH, the first benchmark for evaluating LLM agents’ ability to assess the exploitability of software supply chain vulnerabilities. VEX-B ENCH contains 75 real-world cases mined from GitHub across Go, Python, and Java projects, with ground-truth labels manually annotated by security experts. Given a codebase and a known vulnerability in one of its dependencies, an agent must determine whether the codebase is genuinely affected. Beyond the binary affected/not-affected decision, we define fine-grained justification labels that explain why a downstream project is not affected, mirroring how vulnerability status is communicated in industry. These choices together enable a more realistic evaluation of LLM agents on real-world software supply chain tasks. We evaluate nine models across three agent harnesses on VEX-B ENCH. For binary vulnerabilitystatus classification, Claude Opus 4.6 and GPT5.5 achieve the highest status F1 scores, reaching 81.6% and 79.9%, respectively. Performance drops at the finer justification granularity: only GPT-5.5 exceeds 70% macro-F1. This indicates that identifying the precise exploitability reason is more challenging than making the coarse affected/notaffected decision. We further analyze performance across inference cost, programming language, repository size, and harness choice, providing a more detailed picture of where fine-grained exploitability reasoning remains difficult.
2
Related Work
2.1
Software Supply Chain Security
false alerts (Pashchenko et al., 2022; Pu et al., 2026). Another line of work uses reachability analysis to filter alerts whose vulnerable functions are never invoked (Ponta et al., 2020; Jia et al., 2025). However, reachability analysis still faces several limitations. Identifying the exact vulnerable functions is itself challenging (Dunlap et al., 2024). Call graphs are difficult to make both sound and precise. Most importantly, reachability does not imply exploitability, which may additionally depend on configuration and environmental conditions. In this work, we propose VEX-B ENCH to evaluate whether LLM agents can perform end-to-end exploitability assessments of vulnerable dependencies in downstream projects. 2.2
Cybersecurity Benchmarks for Agents
Recent benchmarks evaluate LLM agents on cybersecurity tasks from two main angles. CyBench (Zhang et al., 2025) uses Capture the Flag (CTF) challenges, where agents operate in compact, self-contained environments and recover hidden flags by exploiting intentionally planted vulnerabilities. Other benchmarks use real-world software projects, drawing from historical CVEs, OSS-Fuzz reports, or bug-bounty disclosures to ask agents to detect, exploit, or patch vulnerabilities in the repositories that contain the flawed code (Wang et al., 2026; Lee et al., 2026; Zhu et al., 2025; Zhang et al., 2026). These benchmarks primarily focus on firstparty vulnerabilities within a single codebase, overlooking software supply chain vulnerabilities. VEX-B ENCH instead studies dependency-level exploitability assessment: given a downstream project and a known CVE in one of its third-party dependencies, the agent must determine whether the project is actually affected by searching for upstream vulnerability information and inspecting downstream dependency usage.
3
Many known vulnerabilities propagate risk to downstream projects through software supply chains (Wu et al., 2023; Shen et al., 2025). Software composition analysis (SCA) tools automatically identify third-party components in a project and flag vulnerable dependencies by matching package names and versions against vulnerability databases (Ponta et al., 2020; Imtiaz et al., 2021; Zhao et al., 2023). However, such metadata-based matching is often imprecise and generates many
VEX-B ENCH
We construct VEX-B ENCH to evaluate whether LLM-based agents can determine if a project is genuinely affected by a known vulnerability in its dependencies. The task requires agents to analyze the local codebase, retrieve information from external sources, inspect upstream projects and vulnerability reports, and synthesize these signals into an end-to-end exploitability analysis. This assessment is routinely performed by security analysts, and its 2
Output
Task: Analyze whether this codebase is affected by the third-party vulnerability. Codebase python-app/ main.py app/ utils/ requirement.txt ...
requirement.txt FastAPI==0.110.0 uvicorn==0.29.0 Jinja2==3.1.3 ...
1) Status Not Affected
LLM Agents
3) Reasoning Capabilities:
depends on Third-Party Dependency (External Source)
Web search Code Analysis
CVE-2024-22195
Jinja2 v3.1.3
2) Justification code_not_reachable
Vulnerability in Jinja2 Server-Side Injection Affected versions: < 3.1.4
The project uses Jinja2 v3.1.3, a version affected by CVE-202422195. However, exploitation requires rendering usercontrolled templates with autoescape disabled, which the codebase never does. Therefore, although the vulnerable dependency is present, the vulnerability is not exploitable in this project.
Figure 1: VEX-B ENCH sources task instances from real-world projects by pairing codebases with third-party dependency vulnerabilities (CVEs) to determine real-world exploitability. Provided with the project’s source code and CVE identifier, LLM agents search external information and analyze code to produce a vulnerability status (Affected / Not Affected), a justification label (e.g., code_not_reachable), and reasoning grounded in the codebase.
results are communicated through the Vulnerability Exploitability eXchange (VEX) format (NTIA, 2021), a machine-readable specification for reporting whether a known vulnerability actually affects a given product. 3.1
sources and locate the relevant code paths in the codebase without assistance. This mirrors the situation a human expert faces when a scanner flags a production repository, and ensures that our evaluation reflects the agent’s own cross-source reasoning rather than the leverage of human-curated hints.
Task Formulation
Output Formulation The agent produces a structured output with three fields. The vulnerability status is a binary label indicating whether the codebase is affected. When the status is not-affected, the justification further specifies the reason, drawn from four fine-grained not-affected categories. The reasoning trace is a free-form explanation recording which call sites the agent inspected, which input paths it followed, and which mitigations it identified — making the prediction auditable rather than a black-box verdict. Together, these fields support both quick triage on the binary status and deeper review through the justification and trace.
We formalize the task as cross-repository vulnerability exploitability assessment. As shown in Figure 1, the agent operates within a target codebase as its working environment and is prompted with a CVE identifier for a known vulnerability in dependencies. The agent must analyze the codebase against the vulnerability and produce a structured prediction together with supporting evidence. Input Formulation Each task consists of a target codebase and a vulnerability identifier. The codebase is a complete real-world project that serves as the agent’s working environment, in which it reads files, runs commands, and analyzes code. The vulnerability identifier is a CVE referring to a known issue in one of the codebase’s third-party dependencies, and is delivered to the agent as part of its instruction prompt. The full prompt template is provided in Appendix B.1. By design, the agent receives no curated advisory text, no preselected vulnerable files, and no pre-computed call graph slice. It must retrieve information about the vulnerability from external
Justification Labels Although the exploitability assessment is binary at the top level, a single not_affected status can arise from different reasons. In one case the vulnerable function may never be invoked along any reachable call path; in another, the exploit may require preconditions that the project’s default configuration prevents. Distinguishing these cases serves two purposes. First, it has direct practical value, since different reasons imply different follow-up actions for the 3
downstream maintainer, and the specific reason an agent identifies reveals which kind of reasoning it is actually performing. Second, it is necessary for reliable evaluation. not_affected cases dominate real-world triage outcomes, so the dataset is heavily imbalanced and a binary status prediction is easy to inflate by simply predicting the majority class. Our schema is adapted from the VEX standard (CISA, 2022). VEX-B ENCH defines four justification categories that capture the most common reasons a known vulnerability fails to be exploitable in a downstream project. Detailed definitions and examples for each category appear in Appendix A. 3.2
Section 2.1. Moreover, PR statuses on GitHub (e.g., merged or closed) cannot be directly interpreted as exploitability labels, since such decisions are often influenced by compatibility concerns or maintenance trade-offs. We therefore opt for manual annotation throughout. Five annotators with prior experience in software engineering and security manually analyze and label the candidate PRs, with each case taking roughly 30 minutes to annotate. To ensure inter-annotator consistency, we adopt a calibrationthen-scaling protocol. In the calibration phase, all five annotators independently label the same 15 cases drawn across the three languages, achieving an initial Fleiss’ Kappa of 0.667. They then meet to reconcile disagreements and align on labeling criteria. In the scaling phase, annotators label distinct sets of cases under the calibrated guidelines, and each annotation is reviewed by at least one additional annotator. During review, 88.3% of the initial labels are confirmed without revision, while the remaining disagreements are resolved through discussion until consensus is reached. This process yields a final benchmark of 75 cases spanning Go, Python, and Java.
Benchmark Construction
We construct VEX-B ENCH through three stages: (1) mining candidate cases from GitHub, (2) rulebased filtering, and (3) manual annotation. Mining GitHub Data We begin by targeting popular and mature open-source projects on GitHub, as such projects better reflect real-world software complexity, active maintenance, and practical dependency management scenarios. Specifically, we select the top 100 starred repositories with permissive open-source licenses (MIT, Apache 2.0, BSD 2-Clause, and BSD 3-Clause). From these repositories, we collect all pull requests (PRs) created after 2021 whose titles or descriptions reference a CVE identifier, yielding a coarse-grained pool of potential dependency-level vulnerabilities.
3.3
Benchmark Statistics
Table 1: This table reports the composition and codebase size of VEX-B ENCH, covering 75 cases across 67 CVEs and 35 real-world projects in Go, Python, and Java, where LOC denotes lines of code and Files denotes the number of source files in the target codebase.
Rule-based Filtering Because the initial candidate pool contains irrelevant cases, such as release notes and internal project vulnerabilities, we apply heuristic filters to remove low-quality candidates. Specifically, we first exclude cases in which all modified files are under test or example directories. Second, we exclude PRs that show more than five distinct CVEs. Finally, the most decisive rule requires the PR to modify a dependency manifest or lockfile (e.g., go.mod, requirements.txt, setup.py, pom.xml), which provides direct evidence that the host project actually consumes and reacts to the referenced vulnerable dependency. The PRs surviving this stage form the candidate pool for manual annotation.
Language
Cases
CVEs
Projects
Median LOC
Median Files
Java Go Python
25 30 20
23 24 20
10 15 10
545,806 90,013 165,122
6,494 924 2,124
All
75
67
35
272,791
2,671
VEX-B ENCH contains 75 cases covering 67 CVEs drawn from 35 real-world repositories across Go, Python, and Java, as shown in Table 1. The target codebases are full projects, not curated snippets. The median case spans roughly 273K lines of code across more than 2,600 files, and the Java subset is the largest, with a median of 546K LOC. An agent therefore cannot solve the task by scanning the entire project; it must locate the few dependency call sites relevant to the vulnerability within a large codebase. The cases also vary widely in scale, ranging from 9,902 LOC to about 4.0M, which prevents strong performance from being explained by overfitting to a single repository size.
Manual Annotation The filtered candidates still require manual labeling, as no reliable automatic signal can serve as ground truth. Existing automated approaches are either inaccurate or require substantial manual effort, as discussed in 4
Label
VEX-Bench
75 tasks
n
%
Affected
22
29.3%
vulnerable
22
29.3%
Not Affected
53
70.7%
code_not_reachable
38
50.7%
code_not_present
6
8.0%
requires_configuration
7
9.3%
requires_environment
2
2.7%
preserving reproducibility. Another potential source of leakage is web access. Since agents can search the web, they could in principle attempt to retrieve task-specific information online. However, the benchmark ground truth is not publicly available: all vulnerability-status labels and justifications are manually annotated by our security experts rather than derived from existing online sources. An agent may therefore gather background information about a vulnerability, but it cannot directly recover the benchmark answers. We further examine potential model-side leakage from the source PRs and find no meaningful evidence that they expose the benchmark answers or that the evaluated models memorized their contents. The full analysis is provided in Appendix C.
Figure 2: Label distribution of VEX-B ENCH across 75 cases, where the inner ring shows the binary vulnerability status (affected vs. not affected) and the outer ring breaks down each status into its fine-grained justification category.
Most scanner-reported dependency vulnerabilities in VEX-B ENCH are not exploitable in the downstream project. Figure 2 shows that only 29.3% of cases are genuinely affected, while 70.7% are not. Within the not_affected group, code_not_reachable alone accounts for 38 cases, or 50.7% of the entire benchmark and 71.7% of all not_affected cases. The remaining not_affected cases are split across configuration, missing-code, and environment reasons. This longtailed structure motivates fine-grained labels: a single not_affected status would conflate reachability analysis with configuration, environment, and dependency reasoning, all of which require distinct agent capabilities. 3.4
4
Evaluation
4.1
Experimental Setup
Models and Harness Modern agents usually have two parts: a model that performs reasoning and a harness that manages execution, memory, tool use, and interaction with the environment. We therefore evaluate agents on VEX-B ENCH from both perspectives. We select nine models, covering both closedsource and open-weight models. The closedsource group includes Claude Opus 4.6 (Anthropic, 2026a), Claude Sonnet 4.6 (Anthropic, 2026b), GPT-5.5 (OpenAI, 2026b), and GPT-5.4 mini (OpenAI, 2026a). The open-weight group includes five recent models reported to be competitive on agentic and coding benchmarks: Kimi K2.6 (Moonshot AI, 2026), DeepSeek-V4-Pro and DeepSeek-V4-Flash (DeepSeek-AI, 2026), MiniMax-M2.7 (MiniMax, 2026), and GLM-5.1 (GLM-5-Team et al., 2026). For harnesses, we evaluate both vendor-specific and open-source frameworks. Vendor-specific harnesses are paired with their corresponding models: Claude Code (Anthropic, 2025) with Claude Opus 4.6 and Claude Sonnet 4.6, and Codex (OpenAI, 2025) with GPT-5.5 and GPT5.4 mini. We evaluate the open-weight models with OpenCode (Anomaly Innovations (SST), 2025), a widely used open-source agent harness. Details of these models and frameworks are in Appendix D. Unless otherwise stated, model names below refer to the model under its corresponding default harness.
Defense Against Benchmark Hacking
As LLM agents become more capable and are granted broader execution privileges, recent work has shown that they can exploit weaknesses in benchmark environments, such as accessing hidden labels or tampering with evaluation logic to inflate reported scores (Thaman, 2026; Atinafu and Cohen, 2026). To mitigate these risks, we execute all experiments inside isolated Docker environments. Each container is separated from the benchmark labels and evaluation artifacts, ensuring that agents can only solve tasks through reasoning rather than by accessing ground-truth answers. In addition, our evaluation does not rely on executable test cases, eliminating the possibility of test-harness manipulation. To further prevent leakage through version history, we remove Git history and provide agents only with a standalone snapshot of the target commit. The result is a sealed environment with no path to the answer, and it is identical across runs,
Metrics We use classification metrics to assess predictions of vulnerability status and justification. 5
Table 2: Performance of agents on VEX-B ENCH, reported as mean±standard deviation over three runs. Vulnerability status metrics treat the task as binary classification; justification metrics evaluate fine-grained reasoning. Token and cost columns report per-case averages. Best and second-best mean values are bolded and underlined, respectively; lower is better for Tokens/Cost. Status
Model
Justification F1
Tokens
Cost
Accuracy Precision
Recall
Accuracy Macro-F1 Weighted-F1
w/ Claude Code Claude Opus 4.6 Claude Sonnet 4.6
88.4±2.0 81.8±5.4
76.3±1.4 68.4±7.1
87.9±6.9 81.6±3.8 75.8±10.5 71.8±8.5
76.4±2.8 67.1±5.4
61.1±3.4 56.1±2.6
78.2±3.0 70.5±4.8
297.47K $0.595 276.24K $0.281
w/ Codex GPT-5.5 GPT-5.4 mini
89.3±4.0 82.2±3.9
88.7±6.2 79.1±3.7
72.7±9.1 79.9±8.0 78.2±3.9 53.0±13.1 63.1±10.3 66.7±4.0
73.5±3.3 60.4±3.3
81.5±3.3 71.9±3.7
700.28K $1.082 947.25K $0.274
w/ OpenCode Kimi K2.6 DeepSeek-V4-Pro DeepSeek-V4-Flash MiniMax-M2.7 GLM-5.1
81.8±2.0 84.0±6.1 72.9±5.1 71.1±10.1 84.4±4.1
74.5±2.0 74.6±9.9 62.7±11.3 75.1±1.8 77.8±6.8
66.7±6.9 68.2±13.6 40.9±0.0 54.6±12.0 68.2±7.9
64.6±7.9 60.3±5.1 47.2±3.3 44.8±12.0 61.7±3.2
75.8±4.1 77.3±6.2 62.8±3.8 64.5±11.1 77.8±3.6
579.01K 170.60K 189.72K 245.28K 201.45K
70.3±4.8 71.2±11.9 49.3±3.4 62.7±7.7 72.5±6.2
72.9±4.3 76.0±7.4 58.7±5.8 59.1±13.5 74.7±4.8
4.2
The vulnerability status is a binary label indicating whether the codebase is affected by the vulnerability. We report standard binary classification metrics, including accuracy, precision, recall, and F1 score. The justification label is a multi-class label over the affected class and the four not-affected reasons, evaluating the agent’s fine-grained reasoning beyond the binary status prediction. We report accuracy, macro-F1, and weighted-F1 for this task.
$0.222 $0.075 $0.027 $0.037 $0.095
Main Results
Table 2 reports the performance of nine model and harness configurations. While current agents perform well on the coarse vulnerability-status task, they remain far from saturated on fine-grained justification. GPT-5.5 obtains the best vulnerabilitystatus accuracy at 89.3%, followed by Claude Opus 4.6 at 88.4%. Claude Opus 4.6 achieves the best status recall and F1 score, reaching 87.9% recall and 81.6% F1, while GPT-5.5 obtains the best status precision at 88.7%. GPT-5.5 also leads all three justification metrics, reaching 78.2% accuracy, 73.5% macro-F1, and 81.5% weighted-F1. At the lower end, MiniMax-M2.7 has the lowest status accuracy at 71.1% and the lowest justification macro-F1 at 44.8%. DeepSeek-V4-Flash records the lowest status precision, recall, and F1 score, as well as the lowest justification accuracy and weighted-F1, reaching 58.7% justification accuracy and 62.8% weighted-F1.
Other Settings To ensure reproducibility and prevent benchmark leakage, all experiments are conducted inside isolated Docker environments. We build separate Linux-based Docker images for different programming languages, each equipped with the corresponding build toolchains. The environment details are in Appendix D. The task prompt contains a broader set of categories than the five annotated benchmark labels. Predictions outside the annotated label space are treated as FAILED. The complete prompt and labelmapping rules are provided in Appendix B.1. We evaluate each model–harness configuration three times and report the mean and standard deviation across runs. To prevent excessively long or non-terminating executions, we impose a maximum runtime limit of 10 minutes per task. Executions that exceed this limit are terminated and recorded as timeouts. Overall, 98.2% of executions finish within this limit; timeouts are counted as failures and concentrate in MiniMax-M2.7 (12.0%) and Kimi-K2.6 (3.1%). Detailed runtime statistics are provided in Appendix D.4.
Performance consistently drops from vulnerability-status prediction to justification prediction. For all nine configurations, justification accuracy is lower than status accuracy. The largest drop occurs for GPT-5.4 mini, from 82.2% status accuracy to 66.7% justification accuracy, a gap of 15.5 percentage points. GPT-5.5 and Claude Opus 4.6 also show clear drops of 11.1 and 12.0 points, respectively. Thus, reporting only the binary affected/not-affected status can overstate performance relative to the finer-grained reasoning needed for software supply-chain triage. 6
Table 3: Performance of traditional SCA baselines on VEX-B ENCH. Cov., Acc., Prec., and Rec. denote coverage, accuracy, precision, and recall, respectively. Coverage is the percentage of repositories successfully analyzed. OSV (detect.) uses OSV-Scanner for dependency detection, whereas OSV (reach.) enables its vulnerablefunction reachability analysis. The full benchmark contains all 75 cases, and the Go subset contains 30 cases. All values are percentages.
80
OpenCode Codex CLI Claude Code
GPT-5.5
Opus-4.6
Performance (%)
GLM-5.1
75 DS-V4-Pro
Kimi-K2.6 Sonnet-4.6
70
GPT-5.4-mini
65 MiniMax-M2.7
60
Method
Cov.
Acc.
Prec.
Rec.
F1
Full benchmark (n = 75) OSV-Scanner 93.3 33.3 Trivy 96.0 38.7
27.4 26.0
77.3 59.1
40.5 36.1
Go subset (n = 30) OSV (detect.) 100.0 OSV (reach.) 100.0 govulncheck 63.3
26.7 46.7 36.7
24.1 28.6 22.2
100.0 85.7 28.6
DS-V4-Flash
$0.03 $0.05 $0.1 $0.3 Cost per case (USD, log scale)
Figure 3: Performance versus per-case inference cost on VEX-B ENCH. The x-axis uses a log scale, and the line connects Pareto-efficient configurations.
38.9 42.9 25.0
4.4 4.4.1 4.3
$1
Comparison with Traditional SCA Tools
Analysis Results Cost-Performance Analysis
Performance generally improves with inference cost, but the gains show diminishing returns. Figure 3 plots per-case cost against a balanced performance score P = 12 ((As + Fs )/2 + (Aj + Mj + Wj )/3), where As and Fs denote status accuracy and status F1, and Aj , Mj , and Wj denote justification accuracy, macro-F1, and weighted-F1. GPT5.5 achieves the highest performance at 81.2%, but is also the most expensive configuration at $1.082 per case. Claude Opus 4.6 follows at 78.5% and costs $0.595 per case. In contrast, DeepSeek-V4Pro reaches 74.4% at only $0.075 per case, and GLM-5.1 reaches 74.9% at $0.095 per case. These results suggest that inference cost is an incomplete proxy for exploitability-analysis quality: GPT-5.5 and Claude Opus 4.6 are strongest in absolute performance, while DeepSeek-V4-Pro and GLM-5.1 provide better cost-performance trade-offs.
To assess the performance of traditional software composition analysis (SCA) tools, we select Trivy (Aqua Security) and OSV-Scanner (Google), two widely used tools, as baselines on the full dataset. We further evaluate OSV-Scanner’s reachability mode and govulncheck (The Go Authors) on the 30 Go cases, as reachability analysis requires mappings from CVEs to vulnerable functions, and reliable function-level records are available only for Go. We predict affected if a tool reports the target CVE and not affected otherwise. Table 3 reports the performance of traditional SCA baselines on VEX-B ENCH. Traditional SCA tools can flag a broad set of potential vulnerabilities, but their low precision shows that they cannot reliably distinguish affected projects from false alerts. Reachability analysis does not close this gap. Govulncheck requires the target project to build successfully and therefore completes only 19 of the 30 cases. Moreover, reachability tools determine whether a vulnerable function appears in the call graph, but this does not establish whether the corresponding path is feasible or whether its branch conditions can actually trigger the vulnerable behavior. This limitation, together with the difficulty of constructing call graphs that are both sound and precise, explains why 70% of their false positives are still labeled code_not_reachable and prevents reachability analysis from reliably assessing exploitability.
4.4.2
Language Analysis
Performance varies across programming languages. Figure 5 breaks down performance by language for four representative configurations. Averaged over these configurations, status F1 ranges from 71.3% on Python to 81.3% on Go, a 10.0point spread. In contrast, justification macro-F1 ranges from 56.5% on Python to 75.5% on Go, a 19.0-point spread. Go achieves the highest average justification macro-F1, while Python is the weakest language on both justification macro-F1 and average status F1. This suggests that language ecosystems affect agents’ ability to identify the correct exploitability reason. One possible explanation 7
GPT-5.5 (n_runs=3)
Sonnet-4.6 (n_runs=3)
8
CNR (38)
14.0
6
Vuln (22)
2.3 0
5
10
15
0
5
Mis-classified per run
6.7
2.7
1
ReqCfg (7)
DS-V4-Pro (n_runs=3)
10.3
5.3
1.3
CNP (6)
Opus-4.6 (n_runs=3)
10
15
7.0
1.3
1
1.3
1.3
0
5
10
15
0
5
10
15
Predicted as
Vuln — vulnerable
FAILED
CNP — code_not_present
ReqCfg — requires_configuration
CNR — code_not_reachable
Figure 4: Error breakdown by ground-truth output category for four representative configurations. Each bar shows the average number of incorrect predictions per run, stacked by the category predicted by the agent. Categories with fewer than three benchmark cases are omitted.
(a) Status F1
non-default configuration can activate the corresponding component. Under our precedence rule, these cases are labeled code_not_reachable; nevertheless, requires_configuration captures an additional condition supported by the evidence. These predictions therefore violate the required precedence but may still be factually valid. Claude Opus 4.6, in contrast, often over-calls CNR cases as vulnerable, indicating a more aggressive tendency to treat dependency evidence as exploitable risk. DeepSeek-V4-Pro exhibits a more balanced but less decisive profile, with comparable error counts on vulnerable and CNR cases, suggesting an unstable boundary between truly affected cases and non-exploitable dependency paths. Across models, errors cluster near the operational boundary between filtering false dependency alerts and identifying truly affected projects.
(b) Justification macro-F1
1.0
F1 score
0.8 0.6 0.4 0.2 0.0
Java
Go
Python
Codex CLI · GPT-5.5 Claude Code · Sonnet-4.6
Java
Go
Python
Claude Code · Opus-4.6 OpenCode · DeepSeek-V4-Pro
Figure 5: Per-language F1 for four representative configurations. (a) Status F1. (b) Justification macro-F1.
is that Go has a comparatively mature and manageable package ecosystem, whereas Java projects often involve more complex multi-module dependency structures and Python projects frequently rely on loosely specified dependencies. 4.4.3
4.4.4
Error Analysis
Repository Size Analysis
Repository size is not the primary bottleneck for strong agents, but it makes fine-grained reasoning less stable. Figure 6 examines 25-task sliding windows ranging from about 44K to 1.9M lines of code. GPT-5.5 and Claude Opus 4.6 maintain high status F1 in the largest windows, reaching 85.0% and 84.5%, respectively. DeepSeek-V4-Pro is more sensitive to repository size, dropping from 83.2% status F1 in the smallest window to 71.0% in the largest window. Across models, the justification macro-F1 curves are lower and more variable than the status-F1 curves, especially in large repositories. This indicates that scaling to larger codebases mainly stresses the agent’s ability to localize and explain the relevant dependency path, rather
Models share where they fail but differ in how they fail. Figure 4 shows that errors across representative configurations concentrate in code_not_reachable (CNR) and vulnerable cases, yet the misclassification patterns diverge by model. FAILED denotes runs that do not produce a valid, evaluable output because of a timeout, a parsing failure, or a prediction outside the five annotated classes. GPT-5.5 frequently predicts requires_configuration for CNR cases. We manually reviewed the five cases with this error pattern and found that four admit both explanations: the default configuration does not activate a path to the vulnerable functionality, whereas a 8
(a) Status F1
1.0
0.9
Macro F1
0.8
Status F1
more than classification performance.
(b) Justification macro-F1 1.0
0.6 0.4
5
0.8
0.6
We introduced VEX-B ENCH, the first benchmark designed to evaluate whether LLM agents can assess the exploitability of software supply chain vulnerabilities. The benchmark comprises 75 realworld cases drawn from Go, Python, and Java projects, with ground-truth labels manually annotated by security experts. Our evaluation of nine model–harness configurations shows that these systems perform better on binary vulnerability-status classification than on the more fine-grained task of determining justification status. Overall, the results suggest that LLM agents are effective at identifying whether a project is affected by a vulnerability but still lack the depth of analysis required to provide accurate justifications.
0.5 0.2
0.4 105 106 Repository LoC
105 106 Repository LoC
Codex CLI · GPT-5.5 Claude Code · Sonnet-4.6
Claude Code · Opus-4.6 OpenCode · DeepSeek-V4-Pro
Figure 6: Status F1 and justification macro-F1 as a function of target repository size for four representative configurations. Each point is computed over a 25-task sliding window sorted by language-specific repository LOC. Shaded regions show run-to-run standard deviation when available. Codex CLI
GPT-5.4-mini Score (%)
Status Acc. Status F1 100
82.2 80.0
75
OpenCode
Just. Acc.
Just. F1
Tokens (K) 947
63.1 65.5
66.7 68.4
1000
Limitations
60.4 55.7
50
500 233
25 0
GPT-5.5 Score (%)
100 75
VEX-B ENCH has three main limitations. First, its scale and coverage are constrained by the high cost of expert annotation. The benchmark contains 75 cases across three language ecosystems, and some fine-grained justification categories contain only a small number of examples. Moreover, the current dataset covers only four types of not-affected justifications. This dataset also does not include other widely used ecosystems, such as JavaScript and npm. Consequently, the reported results may not fully capture the diversity of exploitability assessment challenges encountered in practice, particularly at the fine-grained justification level.
0
89.3
82.7
79.9
78.2 68.4
1000 72.0
73.5
50
67.4
700
331
500
25 0
0
Figure 7: Ablation on the harness for GPT-5.4 mini and GPT-5.5 on VEX-B ENCH. Each row shows one model evaluated under the Codex CLI and OpenCode harnesses, reporting vulnerability-status accuracy/F1, justification accuracy/F1, and average per-case token usage (in thousands).
Second, a single case may admit multiple valid explanations. We apply a predefined precedence rule to assign one gold label and make evaluation unambiguous. Nevertheless, this single-label formulation may omit valid secondary reasons for why a project is not affected.
than only its ability to make a binary affected/notaffected decision. 4.4.5
Conclusion
0.7
Harness Ablation Study
Harness frameworks yield modest performance differences but substantially different token costs. Figure 7 compares GPT-5.4 mini and GPT5.5 under Codex CLI and OpenCode. For GPT-5.5, Codex CLI consistently outperforms OpenCode across all metrics, but at the cost of more than doubling per-case token usage. For GPT-5.4 mini, the two harnesses are comparable in performance, yet Codex CLI consumes roughly 4× more tokens. Overall, harness choice affects inference cost far
Third, VEX-B ENCH assesses repository snapshots using the source code and configuration information available in the repository. Such static evidence cannot fully capture dynamic user inputs or deployment-specific environment settings. The resulting judgments therefore approximate exploitability under the repository’s documented or default conditions and may not reflect every realworld deployment. 9
Ethical Considerations
Yonas Atinafu and Robin Cohen. 2026. Rewardhackingagents: Benchmarking evaluation integrity for llm ml-engineering agents. In 2026 IEEE 42nd International Conference on Data Engineering Workshops (ICDEW), pages 27–34.
Research on vulnerability exploitability has an inherent dual-use risk. In principle, the ability to determine whether a vulnerable dependency is reachable in a downstream project could help malicious actors prioritize known vulnerabilities for further investigation. However, VEX-B ENCH uses only publicly disclosed CVEs and does not involve zeroday discovery. Its tasks ask agents to classify exploitability and justify their decisions with code and advisory evidence; they do not require exploit generation, payload construction, or attacks against running services. In addition, all experiments analyze source-code snapshots inside isolated Docker environments without access to deployed systems, reducing their direct operational value for attacks and the risk of affecting real-world infrastructure. VEX-B ENCH is intended for defensive vulnerability triage, helping defenders prioritize genuine risks, reduce false-positive alerts, and evaluate the capabilities of LLM agents.
CISA. 2022. Vulnerability exploitability exchange (vex) – status justifications. Technical report, Cybersecurity and Infrastructure Security Agency (CISA). Accessed: 2026-05-24. DeepSeek-AI. 2026. Deepseek-v4: Towards highly efficient million-token context intelligence. Dependabot. 2026. Dependabot: Automated dependency updates built into GitHub. GitHub organization, https://github.com/dependabot. Verified GitHub organization operated by GitHub, Inc. Accessed: 2026-05-18. Trevor Dunlap, John Speed Meyers, Bradley Reaves, and William Enck. 2024. Pairing security advisories with vulnerable functions using open-source llms. In International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment, pages 350–369. Springer. GLM-5-Team, Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, Chenzheng Zhu, Congfeng Yin, Cunxiang Wang, Gengzheng Pan, Hao Zeng, Haoke Zhang, Haoran Wang, Huilong Chen, and 167 others. 2026. Glm-5: from vibe coding to agentic engineering. Preprint, arXiv:2602.15763.
Acknowledgments We sincerely thank the anonymous reviewers for their constructive feedback. We also thank Theodor Mihalache, Shimon Tanny, Tamar Weisskopf, Michelle DiPalma, and Jude Niroshan at Red Hat for their contributions to this work. This work was supported in part by NSF Proto-OKN Award 2333736.
Shahriar Golchin and Mihai Surdeanu. 2024. Time travel in llms: Tracing data contamination in large language models. In International Conference on Learning Representations, volume 2024, pages 43008–43029.
References
Google. OSV-Scanner. https://github.com/ google/osv-scanner. Accessed: 2026-08-30.
Anomaly Innovations (SST). 2025. opencode: The AI coding agent built for the terminal. https: //opencode.ai/. Source repository: https:// github.com/sst/opencode. Anthropic. 2025. Claude Code: An agentic coding tool that lives in your terminal. https://www. anthropic.com/product/claude-code. Source repository: https://github.com/anthropics/ claude-code. Anthropic. 2026a. Introducing Claude Opus 4.6. https://www.anthropic.com/news/ claude-opus-4-6. Accessed: 2026-05-20. Anthropic. 2026b. Introducing Claude Sonnet 4.6. https://www.anthropic.com/news/ claude-sonnet-4-6. Accessed: 2026-05-16.
Runzhi He, Hao He, Yuxia Zhang, and Minghui Zhou. 2023. Automating dependency updates in practice: An exploratory study on github dependabot. IEEE Trans. Softw. Eng., 49(8):4004–4022. Nasif Imtiaz, Seaver Thorn, and Laurie Williams. 2021. A comparative study of vulnerability reporting by software composition analysis tools. In Proceedings of the 15th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), ESEM ’21, New York, NY, USA. Association for Computing Machinery. Zhizhuang Jia, Chao Yang, Pengbin Feng, Xiaoyun Zhao, Xinghua Li, and Jianfeng Ma. 2025. Impact assessment of third-party library vulnerabilities through vulnerability reachability analysis. Computers & Security, 157:104546.
Aqua Security. Trivy. https://github.com/ aquasecurity/trivy. Accessed: 2026-08-30.
10
Hwiwon Lee, Ziqi Zhang, Hanxiao Lu, and LINGMING ZHANG. 2026. SEC-bench: Automated benchmarking of LLM agents on real-world software security tasks. In The Thirty-ninth Annual Conference on Neural Information Processing Systems.
33rd Annual Network and Distributed System Security Symposium, NDSS 2026, San Diego, California, USA, February 23-27, 2026. The Internet Society. Benjamin Rombaut, Filipe R. Cogo, Bram Adams, and Ahmed E. Hassan. 2023. There’s no such thing as a free lunch: Lessons learned from exploring the overhead introduced by the greenkeeper dependency bot in npm. ACM Trans. Softw. Eng. Methodol., 32(1).
Microsoft. 2026. Azure AI Model Inference REST API. https://learn.microsoft.com/en-us/ rest/api/aifoundry/modelinference/. Accessed: 2026-05-26. MiniMax. 2026. MiniMax M2.7: Early Echoes of Self-Evolution. https://www.minimax.io/news/ minimax-m27-en. Accessed: 2026-05-20.
Yijun Shen, Xiang Gao, Hailong Sun, and Yu Guo. 2025. Understanding vulnerabilities in software supply chains. Empirical Software Engineering, 30(1):20.
Moonshot AI. 2026. Kimi K2.6: Advancing opensource coding. https://www.kimi.com/blog/ kimi-k2-6. Accessed: 2026-05-20.
Sonatype. 2024. 2024 state of the software supply chain report: A 10-year look back. Industry report, Sonatype. Accessed: 2026-05-13.
NTIA. 2021. Vulnerability-exploitability eXchange (VEX) – an overview. Technical report, National Telecommunications and Information Administration (NTIA). Multistakeholder Process for Software Component Transparency.
Kunvar Thaman. 2026. Reward hacking benchmark: Measuring exploits in llm agents with tool use. Preprint, arXiv:2605.02964.
NVD. 2021. CVE-2021-44228: Apache Log4j2 remote code execution vulnerability. https://nvd.nist. gov/vuln/detail/CVE-2021-44228. Known as Log4Shell; CVSS 10.0; accessed 2026-05-05. OpenAI. 2025. Codex: A cloud-based software engineering agent and CLI. https://openai.com/ index/introducing-codex/. First released as Codex CLI on April 16, 2025; cloud agent research preview announced May 16, 2025. Source repository: https://github.com/openai/codex.
The Go Authors. govulncheck. https://pkg.go. dev/golang.org/x/vuln/cmd/govulncheck. Accessed: 2026-08-30. Zhun Wang, Tianneng Shi, Jingxuan He, Matthew Cai, Jialin Zhang, and Dawn Song. 2026. Cybergym: Evaluating AI agents’ real-world cybersecurity capabilities at scale. In The Fourteenth International Conference on Learning Representations. Yulun Wu, Zeliang Yu, Ming Wen, Qiang Li, Deqing Zou, and Hai Jin. 2023. Understanding the threats of upstream vulnerabilities to downstream projects in the maven ecosystem. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 1046–1058.
OpenAI. 2026a. Introducing GPT-5.4 mini and nano. https://openai.com/index/ Acintroducing-gpt-5-4-mini-and-nano/. cessed: 2026-05-20.
Andy K Zhang, Joey Ji, Celeste Menders, Riya Dulepet, Thomas Qin, Ron Yifeng Wang, Junrong Wu, Kyleen Liao, Jiliang Li, Jinghan Hu, Sara Hong, Nardos Demilew, Shivatmica Murgai, Jason Khiem Tran, Nishka Kacheria, Ethan Jun shen Ho, Denis Liu, Lauren McLane, Olivia Beyer Bruvik, and 15 others. 2026. Bountybench: Dollar impact of AI agent attackers and defenders on real-world cybersecurity systems. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
OpenAI. 2026b. Introducing GPT-5.5. https:// openai.com/index/introducing-gpt-5-5/. Accessed: 2026-05-20. OWASP Foundation. 2025. OWASP Top 10:2025 – A03: Software supply chain failures. https://owasp.org/Top10/2025/A03_ 2025-Software_Supply_Chain_Failures/. Accessed: 2026-05-04. Ivan Pashchenko, Henrik Plate, Serena Elisa Ponta, Antonino Sabetta, and Fabio Massacci. 2022. Vuln4Real: A Methodology for Counting Actually Vulnerable Dependencies . IEEE Transactions on Software Engineering, 48(05):1592–1609.
Andy K Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, Justin W Lin, Eliot Jones, Gashon Hussein, Samantha Liu, Donovan Jasper, Pura Peetathawatchai, Ari Glenn, Vikram Sivashankar, Daniel Zamoshchin, Leo Glikbarg, Derek Askaryar, Mike Yang, Teddy Zhang, Rishi Alluri, and 8 others. 2025. Cybench: A framework for evaluating cybersecurity capabilities and risks of language models. In The Thirteenth International Conference on Learning Representations.
Serena Elisa Ponta, Henrik Plate, and Antonino Sabetta. 2020. Detection, assessment and mitigation of vulnerabilities in open source dependencies. Empirical Software Engineering, 25(5):3175–3215. Yingyuan Pu, Lingyun Ying, and Yacong Gu. 2026. From noise to signal: Precisely identify affected packages of known vulnerabilities in npm ecosystem. In
Lida Zhao, Sen Chen, Zhengzi Xu, Chengwei Liu, Lyuye Zhang, Jiahui Wu, Jun Sun, and Yang Liu.
11
2023. Software composition analysis for vulnerability detection: An empirical study on java projects. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2023, page 960–972, New York, NY, USA. Association for Computing Machinery.
by this project. Example. CVE-2024-7254 in apache/dubbo at commit 218aab1: although the underlying library appears in upstream advisories, the affected module is not part of the project’s resolved dependencies, so no call path can reach the vulnerable code.
Li Zhou, Marc Dacier, and Charalambos Konstantinou. 2026. A reality check on sbom-based vulnerability management: An empirical study and a path forward. In Proceedings of the Sixteenth ACM Conference on Data and Application Security and Privacy, CODASPY ’26, page 255–268, New York, NY, USA. Association for Computing Machinery.
code_not_reachable The vulnerable function is present somewhere in the dependency tree but is never invoked along any reachable call path from first-party code. The vulnerable surface may live in a transitive dependency, in an unused submodule of a direct dependency, or behind an import path that the project does not exercise. Assigning this label requires identifying the vulnerable functions named in the advisory or its fix and verifying that no import, reflection edge, or call site in first-party source leads to them. Example. CVE-2025-55752 in alibaba/nacos at commit 20caf99: the affected library is included in the dependency graph, but the specific APIs that the patch modifies are not referenced anywhere in the project’s source.
Yuxuan Zhu, Antony Kellermann, Dylan Bowman, Philip Li, Akul Gupta, Adarsh Danda, Richard Fang, Conner Jensen, Eric Ihli, Jason Benn, Jet Geronimo, Avi Dhir, Sudhit Rao, Kaicheng Yu, Twm Stone, and Daniel Kang. 2025. CVE-bench: A benchmark for AI agents’ ability to exploit real-world web application vulnerabilities. In Forty-second International Conference on Machine Learning.
A
Justification Category Definitions
VEX-B ENCH reports four justification categories that explain why a downstream project is not affected by a known vulnerability in one of its dependencies. Table 4 summarizes the four categories at a glance; the paragraphs that follow describe, for each category, the underlying condition, the evidence required to assign it, and a representative case from VEX-B ENCH. Although VEX-B ENCH reports four justification categories in the main benchmark, we considered a broader VEX-aligned category space when designing the task prompt. The additional categories listed in Table 5 represent theoretically possible not-affected reasons and boundary cases drawn from the status-justification vocabulary maintained by CISA for VEX (CISA, 2022). They document the prompt design, but are not introduced as separate benchmark classes. All dataset statistics and justification metrics in the paper are reported over the four categories above.
requires_configuration The vulnerable code path exists and is reachable, but exploitation requires a non-default configuration option, feature flag, or runtime setting that the project does not enable. The controlling configuration may live in a YAML or properties file shipped with the project, an environment variable read at startup, or a hard-coded constant in first-party code. Assigning this label requires locating the controlling configuration source, confirming that its disabling value is in effect by default, and checking that no other code path overrides it at runtime. Example. CVE-2023-47248 in apache/airflow at commit 3dbad4f: the vulnerable serialization behavior is gated by a configuration switch whose default value disables the affected functionality. requires_environment The vulnerable code is reachable but its trigger depends on a runtime environment that the project does not establish. The required environment may be an operating system, kernel feature, hardware capability, or external service that the project’s deployment manifests, Dockerfiles, or installation instructions do not provide. Assigning this label requires reading the advisory’s precondition description and confirming that the host project’s declared environment lies outside the affected range. Example. CVE-2024-47535 in prestodb/presto at commit 8136f7c: the exploit’s required runtime environment is not part of
code_not_present The vulnerable package or module is absent from the project’s resolved dependency graph: not declared in any manifest, not pinned in any lockfile, not vendored, and not pulled in transitively. This label is assigned only after inspecting every dependency manifest, the corresponding lockfiles, and any vendored or generated sources, since a vulnerability database entry for the broader package family does not by itself imply that the affected module is consumed 12
Table 4: The four not_affected justification categories used in VEX-B ENCH, adapted from the VEX statusjustification vocabulary (CISA, 2022). Category
Definition
code_not_present
The vulnerable package or module is absent from the project’s resolved dependency graph: not declared in any manifest, not pinned in any lockfile, not vendored, and not pulled in transitively.
code_not_reachable
The vulnerable function exists somewhere in the dependency tree but is never invoked along any reachable call path from first-party code.
requires_configuration
The vulnerable code path is reachable, but exploitation requires a non-default configuration option, feature flag, or runtime setting that the project does not enable.
requires_environment
The vulnerable code is reachable, but its trigger depends on a runtime environment (operating system, kernel feature, hardware capability, or external service) that the project does not establish.
Table 5: Broader VEX-aligned categories considered during prompt construction. These categories represent theoretically possible not-affected reasons and boundary cases, but are not separate benchmark classes in the reported evaluation. Category
Definition
requires_dependency
Exploitation requires an additional optional dependency (library, plugin, or module) that the project does not declare.
compiler_protected
Compile-time hardening configured in the project’s build pipeline (stack canaries, PIE/ASLR-enforced builds, sanitizer instrumentation) prevents the exploit primitive.
runtime_protected
Runtime mechanisms instantiated by the project’s own code (sandboxing, in-process isolation, seccomp filters) prevent exploitation.
perimeter_protected
Network, authentication, or perimeter controls shipped with the project (authentication middleware, allowlists, deployment-level network policies) block the attack surface.
mitigating_control_protected Other in-repository safeguards not covered above (input validation, output encoding, custom guards) reduce risk to negligible.
the project’s supported deployment surface.
B
Prompts
B.1
Task Instruction Prompt
<CVE> {cve_id} </CVE> <CLASSIFICATION_CATEGORIES> The 12 categories are listed below in STRICT LOGICAL PRECEDENCE ORDER. When evaluating, walk through the list from top to bottom and select the FIRST category that applies. Do not skip ahead.
The prompt below reflects the broader VEXaligned category space considered during prompt construction. It should be read as the instruction template used to make the agent’s decision process explicit; the benchmark label space reported in the paper remains the four categories defined in Table 4.
1. "false_positive" - The CVE-to-package mapping is wrong (named package is not what the CVE applies to, or the CVE is withdrawn/malformed). Use only with concrete evidence of mismatch. 2. "code_not_present" - The vulnerable package/module is absent from the repository: not declared in any manifest, not in any lockfile, no vendored copy.
Listing 1: This is task instruction prompt template.
3. "code_not_reachable" - The vulnerable code is present (manifest, lockfile, or vendored copy) in the codebase but is never executed at runtime, e.g., never imported, referenced, or called from first-party source. - Only applicable when code IS present AND call-chain/reachability analysis confirms no execution path leads to it.
<ROLE> You are a security analyst. Decide whether the given CVE actually affects current codebase, and classify the result into one of 12 fixed categories. </ROLE> <CONTEXT> The working directory is a source tree. There is no running container, no running process, and no deployment context: only files you can read and search, plus any public CVE/advisory information you look up.
4. "requires_configuration" - Exploitation requires a specific configuration option, feature flag, or setting that is currently disabled by default in this repository.
Treat this as a static analysis task. Do not assume runtime behavior you cannot back with evidence visible in the source tree. </CONTEXT>
5. "requires_dependency" - Exploitation requires an additional dependency (library, plugin, module) that this repository does
13
not declare.
Reasoning must be grounded in concrete evidence — cite file paths, manifest entries, version numbers, function names, or advisory fragments. Be specific; avoid vague claims like "the code looks safe". Newlines inside the reasoning string must be escaped (`\\n`) to keep the object valid JSON. </OUTPUT_FORMAT>
6. "requires_environment" - Exploitation requires a specific runtime environment (OS, architecture, kernel version, hardware feature) that this repository does not establish or rely on. 7. "compiler_protected" - Compile-time hardening configured in this repository's build (stack canaries, PIE/ASLR-required builds, sanitizer instrumentation, etc.) prevents the exploit primitive.
<EXAMPLE> {"category": "code_not_reachable", "reasoning": "GHSA-44wm-f244-xhp3 describes a vulnerability in PIL.ImageMath.eval for Pillow < 10.3.0. pyproject.toml pins Pillow to ^9.5, and poetry.lock records 9.5.0 as the resolved version — within the affected range, so the vulnerable code is in scope.\\n\\nSearching first-party source (`rg \\\"ImageMath\\\" src/`) returns no matches. The repository imports PIL.Image only, and the only call sites are `Image.open()` and `Image.thumbnail()` in src/img/loader.py:14-37. Neither reaches ImageMath.eval, so the vulnerable function is unreachable."} </EXAMPLE>
8. "runtime_protected" - Runtime mechanisms set up by this repository's own code (sandboxing, in-process isolation, seccomp filters) prevent exploitation. 9. "perimeter_protected" - Network, authentication, or perimeter controls shipped in this repository (auth middleware, allowlists, network policies in deployment manifests) block the attack surface. 10. "mitigating_control_protected" - Other in-repo mitigations not covered by 7-9 (input validation, output encoding, custom guards) reduce risk to negligible.
Mapping to Benchmark Labels. Although the prompt allows 12 categories, VEX-B ENCH evaluates only five annotated classes: vulnerable and the four not-affected categories defined in Appendix A. A vulnerable prediction maps to the affected status, while the four annotated justification categories map to the not-affected status. Predictions in any of the remaining categories are marked as FAILED; they are neither remapped to an annotated class nor excluded from evaluation.
11. "uncertain" - Investigation cannot establish presence, reachability, or mitigation status with the available evidence. Use as a true fallback, not a hedge. 12. "vulnerable" - All of: - an affected version of the vulnerable package is present, - the vulnerable surface is imported and called from first-party (non-test) source, - no mitigation from categories 4-10 applies. </CLASSIFICATION_CATEGORIES> <DECISION_RULES> 1. A CVE is classified as "vulnerable" if and only if ALL of the following hold: - The vulnerable code is PRESENT in the container/codebase. - The vulnerable code is USED or CALLED by the application. - The vulnerable code is REACHABLE from an attack surface (user input, network input, file processing, IPC, etc.). - No effective mitigations or protections are in place.
C
Model-Side Leakage Analysis
Because VEX-B ENCH is constructed from public repositories, the source PRs may have appeared in model training data even though the benchmark labels are not public. We assess this potential source of leakage through a manual audit of the source PRs and a guided-completion probe for memorized PR content.
2. If ANY of the above conditions fails, select the SINGLE most appropriate non-vulnerable category by walking the precedence list from top (1) to bottom (11) and choosing the first matching category. For example: - If the vulnerable code is not present → "code_not_present" (do NOT also consider "code_not_reachable" or environment factors). - If the code is present but unreachable → "code_not_reachable" (do NOT fall through to "requires_environment"). - If a required dependency is missing → "requires_dependency". - If the vulnerable code is prevented by a default or clearly set configuration → "requires_configuration".
Source PR Audit We manually review the titles, descriptions, and discussion threads of the source PRs associated with all 75 cases. Only three discuss whether the corresponding vulnerability is exploitable, and none contains the vulnerability status or justification assigned by our annotators. The public source PRs therefore do not directly reveal the benchmark answers.
3. Use "uncertain" only when the investigation genuinely lacks the evidence needed to reach any conclusion. </DECISION_RULES>
Guided-Completion Probe Following prior work (Golchin and Surdeanu, 2024), we further probe whether the evaluated models reproduce source PR content from memory. We select PRs that were created before the models’ knowledge cutoff dates, written by humans rather than bots, non-boilerplate, and sufficiently long to support a meaningful completion test. Nineteen distinct
<OUTPUT_FORMAT> Output a single JSON object on stdout — no surrounding prose, no markdown fences, no comments. Schema: { "category": "<one of the 12 category names, exact snake_case>", "reasoning": "<evidence-based explanation; multi-line strings are fine>" }
14
PRs satisfy these criteria. We test all of them with GPT-5.5 and Claude Opus 4.6, with web access and tools disabled, and classify each response as an exact match, a verified near-exact match, a nonmatching completion, or a refusal/empty response. Across the 38 completions, we observe no verified near-exact matches and only one exact tentoken continuation. This match occurs because the PR body repeats its title, which is already provided in the prompt, rather than because the model reproduces unseen PR content. Together with the source PR audit, these results provide no meaningful evidence that the evaluated models memorized the source PR contents or could recover the benchmark answers from them.
D
Experimental Details
D.1
Environment
backends are reported separately from the tool versions. D.4
Overall, 98.2% of executions complete within the 10-minute limit, indicating that the cap accommodates most agent trajectories. Runtime nevertheless varies substantially across configurations, particularly in the upper tail. Most timeouts occur for Kimi K2.6 and MiniMax-M2.7. Manual inspection shows that Kimi’s timeouts mainly arise from long analysis trajectories on large repositories, whereas MiniMax’s timeouts are primarily associated with slow API responses. Because these measurements cover the complete execution, they reflect the combined latency of the model backend, agent harness, and tool-use trajectory rather than model inference alone.
Table 7 summarizes the language-level runtime environments used in our Docker containers. We report only the base image, operating system, and primary runtime or build tool versions; common utilities such as git, curl, and ca-certificates are inherited or installed as part of the container setup. D.2
E
Responsible Research Details
E.1
Artifact Use and Intended Use
VEX-B ENCH is constructed from public opensource repositories and public vulnerability records for research evaluation of software supply chain vulnerability triage. This use is consistent with the public and research-facing nature of the source artifacts: the repositories are selected from permissively licensed open-source projects, and the vulnerability identifiers and advisories describe already disclosed security issues. The benchmark artifacts we create, including task metadata, labels, prompts, and evaluation scripts, are intended for defensive triage research, evaluation of LLM-agent reliability, and analysis of failure modes. They are not intended to support exploit generation, unauthorized vulnerability testing, or attacks against deployed systems. Users of the benchmark are responsible for complying with the licenses and terms of the upstream repositories, tools, models, and vulnerability-data sources they use.
Model Backends
All model API calls in our experiments were routed through Azure’s model-serving APIs. We used the Azure AI Model Inference REST API as the common provider interface for deployed foundation models (Microsoft, 2026). Table 8 lists the model names used in the paper, the corresponding provider-facing model card or release source, and the public release date of each model. D.3
Runtime and Timeout Analysis
Harness Framework
Table 9 reports the harness frameworks used in our experiments. For each agent, we list the corresponding official website or documentation page, the exact CLI version installed in the experimental containers, and the effort setting used when the harness exposes such a control. We used the strongest available effort setting in our setup: Codex was run with xhigh, and Claude Code was run with high. For the Claude 4.6 models, high is the maximum supported effort level. OpenCode was run without an effort override because no analogous effort setting was configured in our experiments. These versions and settings identify the agent-side execution environment; model identifiers and provider
E.2
Data Privacy and Offensive Content
The benchmark is derived from public source-code repositories and public vulnerability records rather than from private user data. During curation, we do not collect user profiles, private communications, issue discussions, code-review threads, or other person-centered content. The code snapshots used in evaluation remove Git history, which avoids retaining commit-author timelines and email metadata. The released benchmark metadata is intended 15
Table 6: Results of the guided-completion probe on the 19 source PRs selected for model-side leakage analysis. Model
Exact
Verified Near-exact
Nonmatching
Refusal/Empty
GPT-5.5 (n = 19) Claude Opus 4.6 (n = 19)
1 0
0 0
12 19
6 0
Combined (n = 38)
1
0
31
6
Table 7: Language runtime environments used in our Docker containers. Language
Base image
OS
Runtime / Build tool
Python Go Java
python:3.13-slim-trixie golang:1.26-bookworm eclipse-temurin:21-jdk
Debian GNU/Linux 13 Debian GNU/Linux 12 Ubuntu 26.04 LTS
Python 3.13.13 Go 1.26.3 OpenJDK 21.0.11; Maven 3.9.12
to contain task identifiers, repository references, CVE identifiers, labels, annotations, prompts, and evaluation scripts, rather than personal information about project contributors or annotators. We do not intentionally collect offensive content. Because the source artifacts are real software projects and public security advisories, they may contain security-related terminology or public project/license notices, but such content is included only insofar as it is necessary for vulnerability triage research. Before release, we review benchmark metadata to avoid including unnecessary personal identifiers or unrelated sensitive content. E.3
dency alerts, help researchers measure the reliability of LLM agents for security analysis, and identify current failure modes before such agents are used in high-stakes software maintenance workflows. E.4
Annotation Instructions and Consent
Annotators were instructed to determine whether a downstream project is actually affected by a known vulnerability in one of its third-party dependencies. For each candidate case, annotators were given the target repository snapshot, the relevant CVE identifier, and the dependency-update pull request from which the case was mined. They were asked to inspect the dependency manifest or lockfile, the public vulnerability advisory, the upstream patch when available, and the downstream project code needed to assess whether the vulnerable functionality is present, called, and reachable from the project. The annotation task had two outputs. First, annotators assigned a binary vulnerability status indicating whether the downstream project is affected by the dependency vulnerability. Second, annotators assigned a justification label explaining the status. A case should be labeled as affected only when the vulnerable dependency version is present, the vulnerable functionality is used by the downstream project, the relevant code path is reachable from a plausible input or execution path, and no in-repository mitigation blocks exploitation. If any of these conditions is not met, annotators selected the most specific not-affected justification in the reported benchmark label space. These four categories cover missing vulnerable code, unreachable code, required configuration, and required environment. The additional categories shown in the prompt are part of the broader design space used to formulate the task, not additional reported bench-
Potential Risks
This work studies software supply chain vulnerability exploitability, so it has an inherent dualuse aspect. A benchmark that evaluates whether agents can reason about vulnerable dependencies and downstream reachability could, in principle, help a malicious actor prioritize which known vulnerabilities deserve further manual investigation. However, our work is designed as a defensive evaluation resource rather than an exploitation system. The tasks are based on already disclosed vulnerabilities, and the benchmark asks agents to determine exploitability status and provide evidence for the decision; it does not require agents to generate exploit payloads, compromise running services, or perform end-to-end attacks. We further reduce operational risk by framing the task as static analysis of source-code snapshots. The experimental environments are isolated containers without deployment context, and the prompts explicitly instruct agents to ground their answers in code and advisory evidence rather than to execute attacks. The intended use of VEXB ENCH is to improve defensive triage of depen16
Table 8: Model backends used in our experiments. Model name
Model card / source name
Release date
Claude Opus 4.6 Claude Sonnet 4.6 GPT-5.5 GPT-5.4 mini Kimi K2.6 DeepSeek-V4-Pro DeepSeek-V4-Flash MiniMax-M2.7 GLM-5.1
Claude Opus 4.6 (Anthropic, 2026a) Claude Sonnet 4.6 (Anthropic, 2026b) Introducing GPT-5.5 (OpenAI, 2026b) GPT-5.4 mini (OpenAI, 2026a) Kimi K2.6 (Moonshot AI, 2026) DeepSeek-V4-Pro (DeepSeek-AI, 2026) DeepSeek-V4-Flash (DeepSeek-AI, 2026) MiniMax M2.7 (MiniMax, 2026) GLM-5.1 (GLM-5-Team et al., 2026)
Feb. 5, 2026 Feb. 17, 2026 Apr. 23, 2026 Mar. 17, 2026 Apr. 20, 2026 Apr. 24, 2026 Apr. 24, 2026 Mar. 18, 2026 Apr. 7, 2026
Table 9: Agent harness versions and effort settings used in our experiments. Agent
Website
Version
Effort setting
Codex Claude Code OpenCode
https://developers.openai.com/codex/cli https://code.claude.com/docs/en/setup https://opencode.ai/docs/
0.131.0 2.1.150 1.15.5
xhigh high Not set
Table 10: End-to-end wall-clock runtime across all executions. Median, P90, and maximum runtimes are reported in seconds. Timeout is the percentage of executions that reach the 600-second limit; consequently, a maximum of 600 seconds indicates at least one timed-out execution. Model Claude Opus 4.6 Claude Sonnet 4.6 GPT-5.5 GPT-5.4 mini GLM-5.1 DeepSeek-V4-Pro Kimi K2.6 DeepSeek-V4-Flash MiniMax-M2.7
Median (s)
P90 (s)
Max (s)
Timeout (%)
105 108 162 176 79 99 135 151 91
217 177 253 256 246 215 364 456 600
336 456 442 380 562 600 600 600 600
0.0 0.0 0.0 0.0 0.0 0.4 3.1 0.4 12.0
E.5
mark classes.
Use of AI Assistants
The authors used AI assistants for language polishing, manuscript revision, and literature-search assistance. All AI-generated suggestions were reviewed, edited, and verified by the authors. AI assistants were not used to create ground-truth labels, conduct final exploitability judgments, generate experimental results, or make unverified scientific claims.
Annotators were instructed to base labels on concrete evidence, including file paths, dependency versions, function or class names, configuration values, and advisory or patch details. They were asked not to infer exploitability from the existence of a vulnerable dependency alone, and not to label a case as affected unless they could identify a downstream path that uses the vulnerable functionality under the required conditions. Ambiguous cases were discussed during calibration meetings, and final labels were assigned after reconciling disagreements among annotators. No screenshotbased annotation interface was used. Annotators were members of the research team and were informed that their annotations would be used to construct the benchmark labels and to report aggregate dataset statistics. We do not release annotator identities, annotator-level performance, or other personal information about annotators. 17