Conceptio › Archive › arXiv CS

arXiv CSopen access

Code-Centric Detection of Vulnerability-Fixing Commits: A Unified Benchmark and Empirical Study

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

cryptography cybersecurity privacy security

cryptography, security, privacy, cybersecurity

arXiv:2605.13138v1 [cs.SE] 13 May 2026

Code-Centric Detection of Vulnerability-Fixing Commits: A Unified Benchmark and Empirical Study Nils Loose

Joseph Bienhüls

Kristoffer Hempel

[email protected] University of Lübeck Institute for IT Security Lübeck, Germany

Felix Mächtle

Thomas Eisenbarth

[email protected] University of Lübeck Institute for IT Security Lübeck, Germany

Abstract Automated detection of vulnerability-fixing commits (VFCs) is critical for timely security patch deployment, as advisory databases lag patch releases by a median of 25 days and many fixes never receive advisories. We present a comprehensive evaluation of code language model based VFC detection through a unified framework consolidating over 20 fragmented datasets spanning more than 180 000 commits. Across over 180 experiments with fine-tuned models from 125M to 14B parameters, we find no evidence that models acquire transferable security-relevant code understanding from code changes alone. When commit messages are available, they dominate model attention, and when removed, an attribution analysis shows that enriching diffs with additional intra-procedural semantic context does not shift model attention toward the code changes. Group-stratified evaluation exposes approximately 17% performance drops compared to random splits, while temporal splits on aggregated datasets prove unreliable due to compositional shift in the underlying project distributions. At a false positive rate of 0.5% all fine-tuned code-only models miss over 93% of vulnerabilities. Larger and more diverse training data or generative approaches show preliminary improvements but do not resolve the underlying limitations. To support future research on code-centric VFC detection, we release our unified framework and evaluation suite.

Introduction

Modern software supply chains create cascading security dependencies where vulnerabilities impact numerous downstream projects. Organizations must identify and apply patches before attackers exploit the disclosure window. However, Imtiaz et al. [18] reveal that advisory publication lags patch release by a median of 25 days, leaving systems vulnerable to attacks when developers rely on vulnerability databases such as the National Vulnerability Database (NVD) [30]. Silent security patches present an even greater challenge, as they never receive public advisories, leaving downstream software vulnerable. As the vast majority of reused software is maintained in open-source repositories, an accurate analysis of each commit across upstream dependencies could result in safer software ecosystems with faster patch deployment times. However, as manual analysis is infeasible at scale, automated techniques for identifying

vulnerability-fixing commits (VFCs) have become essential. More broadly, as code changes in diff format have become the standard unit of review in modern development, from manual code review to AI-assisted programming, understanding the reasoning capabilities of current models on software changes is relevant well beyond the security domain. Code LMs [13, 15, 49] have shown promise across code understanding tasks, leading to their adoption for Security Patch Detection (SPD), the task of identifying commits that fix vulnerabilities. While existing approaches report strong results when combining code with commit messages [5, 27, 36, 39, 42, 43, 48], the actual capabilities of these models for understanding securityrelevant code changes without relying on textual information remain largely unclear. Chen et al. [5] and Sun et al. [39] performed ablation studies evaluating their systems with and without the commit message and observed substantial drops in performance, suggesting a strong reliance on textual signals. Wang et al. [45] proposed the first graph-based approach that relies exclusively on code information, but requires expensive precomputations with limited demonstrated success [45, 51]. For the related task of function-level vulnerability detection, recent studies have shown that models exploit superficial dataset correlations rather than learning vulnerability semantics, with simple word-count classifiers matching deep learning models on standard benchmarks [34, 35]. To enable a systematic and large-scale evaluation of code LM performance for SPD we first address the data fragmentation challenge. Over the past decade, a variety of research groups have collected and labeled VFCs with different target languages, labelling techniques and ground truths, resulting in fragmented datasets that hinder largescale comparative evaluation. We build a comprehensive framework that unifies access to 20 VFC datasets, enabling systematic evaluation across different labeling strategies, programming languages, and vulnerability classes. To evaluate whether additional semantic context can improve model behavior, we additionally implement a lightweight intra-procedural context enrichment method that identifies semantically relevant code context around modifications. Using this framework, we conduct the first comprehensive evaluation of SPD using code LMs, spanning over 180 experiments across models from 125M to 14B parameters, multiple architectures, input representations, and training strategies. Among the fine-tuned models and representations tested, we find no evidence that models acquire

Nils Loose, Joseph Bienhüls, Kristoffer Hempel, Felix Mächtle, and Thomas Eisenbarth

Figure 1: Temporal overview of existing VFC datasets, their label source, size and dependent datasets. The advisory label class includes other, similar, sources of information such as bug trackers. The synthetic label indicates that at least a subset of samples is synthetic. Manual verification indicates that some effort was spent validating at least a subset of the samples. VulnPP: VulnPatchPairs. transferable security-relevant code understanding. When commit messages are available, they dominate model attention, confirming prior observations at larger scale. When messages are removed, an attribution analysis shows that the actual code changes consistently receive modest attention. Under group-stratified evaluation that separates repositories between training and test sets, performance drops by approximately 17% compared to random splits, confirming project-specific memorization consistent with findings by Steenhoek et al. [38] for cross-project vulnerability detection. Temporal splits show an even larger degradation but prove unreliable on aggregated datasets, as a sliding window analysis reveals that performance variation is dominated by shifting project compositions rather than genuine temporal generalization. Neither scaling fine-tuned model capacity from 125M to 14B parameters nor enriching diffs with intra-procedural data-flow context changes this picture. Prompt-based generative classification without task-specific training achieves modest improvements over fine-tuned models on the same CVE-confirmed test set, though data contamination from internet-scale pretraining cannot be excluded. At a false positive rate of 0.5%, all fine-tuned code-only models miss over 93% of vulnerabilities (PD-S = FNR @ FPR ≤ 0.005), indicating that none can reliably detect vulnerability-fixing commits under strict false-positive constraints. In summary, we make the following contributions. First, we develop a unified framework consolidating and preprocessing 20 fragmented VFC datasets, enabling systematic comparative evaluation across different labeling strategies and programming languages [44].1 Second, through a comprehensive evaluation we find that fine-tuned code LMs up to 14B parameters do not acquire transferable securityrelevant code understanding under rigorous evaluation for SPD, producing concrete insights on evaluation methodology for aggregated VFC datasets and on the limitations of current approaches.

Related Work

1 https://github.com/UzL-ITS/vfc_datasets

2.1

VFC Datasets

Security research focusing on SPD or vulnerability detection (VD) requires code samples that are annotated with accurate vulnerability information. To approximate the underlying structural distribution of the data, a sufficiently large sample size is required. Yet, obtaining positive ground truth samples is difficult. Over the past decade, an increasingly popular method of obtaining security-annotated samples revolves around VFCs. To obtain code samples, open source repositories hosted on public version control systems are the primary source. The process of identifying which commit is security relevant has seen three major approaches: algorithmic pattern matching, often on the commit message [32, 33, 40, 50, 60, 61]; linking vulnerability information from public advisories, bug trackers or similar sources to commits [1, 3, 8, 9, 20, 24, 28, 29, 31, 33, 40, 47, 52, 61]; and using other tools, algorithms or machine learning to perform code-based identification of potential VFCs [16, 31, 34, 40, 47, 61]. Additionally, synthetic generation of VFCs has also been explored [47]. Regardless of the technique used for generating labels, it has been shown [9] that the quality of labels is a significant concern. Some works have attempted to quantify this by using manual verification [31, 32, 40, 52, 60, 61]. An overview of dataset relations and key characteristics is shown in Figure 1. While some connections are present, overall several fragmented datasets exist, hindering large scale comparative evaluations. An in-depth summary of the identified datasets, their size, label distribution and included programming languages can be found in Table 1.

2.2

VFC Detection

Recent work has raised fundamental questions about what ML models actually learn for security-related code understanding. Risse and Böhme [34] show that top-performing vulnerability detection models overfit to label-unrelated features and cannot distinguish vulnerable functions from their patched versions. Risse et al. [35] survey 81 ML4VD papers and demonstrate that a simple word-count classifier achieves an F1 score comparable to deep learning models

Code-Centric Detection of Vulnerability-Fixing Commits: A Unified Benchmark and Empirical Study

Table 1: Overview of research-based data sets that contain VFCs. Size Year

Dataset Repos

VFCs

¬VFCs

CVEs

Included Languages 𝐷𝑆 1 𝐷𝑆 2 𝐷𝑆 3 𝐷𝑆 4

2017

Secbench [32]

112

639

183

13 PLs

✓

2019

Project KB [31] Devign [60]

189 2

1045 11 091

14 587

551 386

Java C/C++

✗ ✓

✓ ✓

2020

Big-Vul [8]

413

4235

3808

C/C++

✗

✓

2021

CVEfixes [3] SPI [61] PatchDB [47] TQRG21 [33] CrossVul [29] CodeXGLUE [25]† 900Repo [20]

3925 2 391 1298 1518 4 910

11 329 11 086 9583 7627 5432 10 894 3189

14 573 22 806 85 442 5475

11 156 386 2338 6079 5166 2545

27 PLs C/C++ C/C++ 20 PLs 48 PLs* C/C++ 20 PLs

✗ ✓ ✓ ✗ ✗ ✗ ✗

✓ ✓ ✓ ✓ ✓ ✗ ✓

2022

VCMatch [46] Tracer [52] VUDENC [50]

10 727 784

1568 2389 1009

1631 1308 -

C/C++, Java, PHP ✗ 7+ PLs ✓ Python ✗

✓ ✓ ✗

2023

DiverseVul [6] PySecDB [40] Sven [16]

797 476 261

7514 668 532

1823 -

302 417

C/C++ Python C/C++, Py

✗ ✗ ✓

✓ ✗ ✓

✓ ✓ ✓

2024

MoreFixes [1] MegaVul [28] VulnPatchPairs [34] PrimeVul [8] RepoSPD [51]

7037 1393 2 755 363

32 358 9108 6352 6827 17 341

30 447

26 197 8575 5369 2323

54 PLs C/C++,Java C/C++ C/C++ C/C++

✗ ✗ ✗ ✗ ✓

✗ ✓ ✗ ✗ ✓

✓ ✓ ✗ ✗ ✓

2025

ICVul [24]

658

3828

3730

C/C++

✗

✓

* Unique file extensions.

† Defect detection set is identical to Devign [60], CodeXGLUE contains several other tasks

on popular benchmarks, suggesting that reported progress reflects dataset correlations rather than vulnerability understanding. These findings motivate a careful examination of SPD approaches. Detecting VFCs has been explored under several assumptions, from SPD where the goal is to identify if a commit is security relevant [5, 39, 45, 51, 56, 58, 61], to ranking-based approaches trying to identify the VFC belonging to a specific advisory [10, 46]. For ranking-based approaches, a submodule is sometimes utilized that provides an SPD prediction indicator [10] but that is then embedded into a larger prediction framework that relies on additional metadata that is generally used for ranking VFCs. For SPD, several works have also explored the prediction capabilities under the utilization of the code changes along with the commit message [27, 36, 42, 43, 48]. In this work we evaluate SPD under the restriction that only the code is utilized for the classification to understand the security-related code reasoning capabilities. Sequence-based approaches typically rely on representing code changes as token sequences that are used for training a machine learning model or ensemble of models for supervised classification [10, 39, 58]. Zhou et al. [57] explore a

contrastive learning approach to pretrain a BERT model with a specialized embedding space that is used in downstream SPD systems. Wang et al. [45] introduced PatchCPG, a semantics-aware patch representation based on code property graphs [53]. To generate a PatchCPG, the CPG is generated for the pre- and post-patch versions using Joern [54]. Then, dependence-guided forward and backward slicing, seeded by the added and deleted lines, is performed to confine context. Finally, it feeds a multi-attributed graph into a graph neural network (GNN) to classify security patches directly from graph structure [45]. Building on this idea, Wen et al. [51] construct a RepoCPG that preserves semantic changes while augmenting them with cross-file dependencies (e.g., function-level call relations), and fuses graph-based and sequence-based representations with progressive learning to capture relationships among multiple code changes at repository scale [51]. Recently, a dynamic approach to evaluating safe patches by integrating dynamic symbolic execution has also been proposed [26]. Additionally, Yang et al. [56] propose an LLM-based generative approach that utilizes retrieval augmented

Nils Loose, Joseph Bienhüls, Kristoffer Hempel, Felix Mächtle, and Thomas Eisenbarth

generation (RAG) through embedding-based matching of historic VFC information to detect VFCs.

VFC Collection Framework

The fragmented landscape of VFC datasets (Section 2.1) presents challenges for comparative evaluation, as accessibility issues, format inconsistencies, and varying labeling methodologies across individual collections make it difficult to compare results and reproduce findings across studies. To address these challenges, we develop a framework that unifies access to existing VFC datasets through systematic parsing, normalization, and enrichment. The framework consists of three stages: data ingestion and normalization across diverse formats and platforms, enrichment with commit metadata and repository information, and deduplication and filtering for dataset customization. The framework currently integrates 20 different data sources, enabling researchers to combine and filter collections based on specific research requirements. To manage the overlap during dataset merging, we implement two complementary deduplication strategies. First, identical entries (same commit hash and repository) are merged to maximize metadata retention while removing conflicts where labels disagree. Second, semantic matches based on diff content and modified files are removed, primarily catching commits across repository mirrors. The framework offers extensive filtering capabilities, allowing researchers to create custom datasets based on, among others, different programming languages, vulnerability classes, labeling strategies, source datasets, and temporal constraints. It also supports multiple data splitting strategies including random, temporal, and group-stratified splits that account for project identity by grouping related repositories to prevent undesired cross-split biases. Table 2: Evaluation dataset compositions. Commits Total VFCs

Projects 100% 75%

D1 C/C++ MR D2 C/C++ Adv D3 C/C++ All

55 931 88 687 96 021

36% 32% 38%

480 1341 1698

7 35 46

183 106

32%

8092

314

PLs

Multi

Label

All

Using our framework, we systematically construct four datasets with increasing scope to evaluate the impact of data quality, quantity, and diversity on SPD performance. An overview of the size and label distribution for each dataset is shown in Table 2. A detailed breakdown of all contributing source datasets to the respective target datasets can be seen in Table 1 and the source set overlap of the five largest contributors in D2 is shown in the alluvial diagram in Figure 2a. D1 - Manually reviewed C/ C++ : Combines all C/ C++ commits from datasets where the original authors report some manual validation of label quality. D2 - D1 + Advisory-based C/ C++ : Our primary evaluation dataset. D2 extends D1 by including all advisory-mapped C/ C++ commits without requiring additional manual verification.

D3 - D2 + Automated tooling C/ C++ : All C/ C++ commits, including those labeled using machine-learning approaches and traditional tools. D4 - All commits: The largest dataset, removing all filtering constraints including the target programming language.

RepoSPD

FFmpeg

1262 others

PatchDB TQRG21 Devign

Linux QEMU

SPI-DB

(a) Source-dataset overlap in D2 .

(b) Project composition of D2 .

Figure 2: Dataset characterization for D2 . (a) Alluvial diagram showing source-dataset overlap across the five largest contributing sources. Left bars represent individual sources; right bar shows the deduplicated dataset, with flows sized proportionally to shared entries. (b) Project treemap of D2 : block area is proportional to the number of commits from each repository. Hatched regions indicate the VFC fraction within each project. To understand the structure and potential biases in D2 , we analyze dataset provenance, project composition, token distributions, and latent representations. The project treemap (Figure 2b) reveals the imbalance in the distribution of commits to source repositories. Additionally, the hatched regions show that per-project VFC ratios also vary considerably. This imbalance motivates split strategies that account for project identity and label ratios to prevent data leakage through project-specific patterns (Section 4.3) and balance each split. Project-specific biases are further highlighted by a t-SNE analysis of CodeBERT [13] embeddings (Figure 3) providing further insights into the dataset distribution. Source repositories exhibit clear clustering patterns that indicate project-specific coding patterns or commit styles (Figure 3 (a) and (b)). The visualized embeddings reveal that this structure is preserved even after fine-tuning on the downstream detection task. When colored by commit timestamp, the temporal structure reflects some patterns visible in the project composition. This is an expected artifact caused by different repositories being crawled for different time periods by respective source datasets. This overlap between temporal and project structure raises concerns about the reliability of temporal splits on aggregated datasets, which we investigate in Section 4.3. The token-level analysis (Figure 4) shows that a substantial fraction of diffs exceeds the 512-token limit of smaller commonly used encoder models, while the actual code changes account for only a modest fraction of each diff’s total length. A significant portion of tokens consists of context lines, file headers, and syntax elements. This becomes particularly relevant when inputs must be truncated to fit model sequence limits, as naive end-truncation disproportionately discards vulnerability-relevant change tokens, unnecessarily impacting model performance [11]. We therefore implement a context-aware truncation strategy that removes context lines in order of decreasing distance from the nearest code change, preserving the security-relevant signal while fitting within

diffdf2 diffdf1 diffcf diff Commit Message

Evaluation

Naive

8k 6k

60%

4k 2k

40%

4 16 64 28 210 212

210

211

212

Token limit

Token count (a) Token composition by line type.

Ctx-aware

80%

Samples truncated

the model’s token budget. As shown in Figure 4b, this approach substantially reduces the fraction of discarded change tokens compared to naive truncation across all token limits. Beyond standard diffs, we evaluate enriched input representations that replace the physically neighbouring context lines with semantically targeted context derived from control-flow and data-flow analysis (Section 4.2).

Removed changes

Code-Centric Detection of Vulnerability-Fixing Commits: A Unified Benchmark and Empirical Study

(b) Truncation comparison.

Table 3: Overview of evaluated models. Model

Param

Tokens

Arch

CodeBERT [13] UniXcoder [15] CodeBERT C++ [59] CodeT5 Large [49] CommitBART [22] Qwen2.5-Coder [55]

125M 125M 125M 770M 140M 14B

512 512/1024 512 512 1024 40961

Enc Enc-Dec Enc Enc-Dec Enc-Dec Dec

GLM-4.7 [14]2

355B

81921 Dec

1 The model’s max capacity is larger. 2 Evaluated in a prompt-based setting.

The dataset characterization in Section 3 revealed pronounced project-level clustering and temporal confounds in the data structure. We now leverage these observations to design an evaluation that systematically tests how models interact with these properties across multiple datasets, architectures, and splitting strategies.

(a) Stock, colored by project.

Figure 4: Token-level analysis of D2 . (a) Stacked ridge plots showing token distributions for six input representations. Each ridge decomposes the total count into changed lines (red), file headers/ hunks (gray), context (green), and commit message (blue). Dashed lines mark medians, red vertical lines indicate common sequence limits. (b) Fraction of discarded tokens that are code changes under naive end-truncation vs. context-aware truncation at four token limits. Gray bars show the number of affected samples (right axis).

(b) Finetuned, colored by project.

Figure 3: t-SNE visualizations of D2 based on CodeBERT [13] embeddings (perplexity = 150, 2,000 iterations, PCA pre-reduction to 50 dimensions). Top row: points colored by the five most frequent source repositories, remainin projects in gray. Bottom row: points colored by commit timestamp with lighter colors representing more recent commits. Left column uses stock CodeBERT, right column uses fine-tuned model on diff.

4.1

Models and Training Setup

We evaluate popular code LMs (Table 3) using 60/20/20 train/validation/test splits to ensure sufficiently large evaluation sets under group-stratified conditions where repository constraints reduce effective sample sizes. We train all models for 10 epochs and select the best validation F1 checkpoint for testing. For group-stratified splits, related repositories (e.g. forks) are grouped so that all commits from related projects always appear in the same partition, preventing cross-split data leakage. Groups are assigned to splits via a greedy algorithm refined by local search to approximate the target ratio, applied hierarchically. First separating test (20%) from train and validation (80%), then splitting the remainder into train (60%) and validation (20%). All experiments are trained on NVIDIA A100, H100 and H200 GPUs. All models except Qwen2.5-Coder are trained using an effective batch size 64, AdamW [23] and a learning rate of 2 × 10−5 using the classification heads provided by the HuggingFace *ForSequenceClassification model implementations. To train Qwen2.5-Coder [55] we use low rank adaptation [17] with rank 16 and 𝛼 = 32, a batch size of 2 with 32 gradient accumulation steps matching the same effective batch size and a learning rate of 5 × 10−5 . For all models, class imbalance is handled via inverse frequency weighting. We report F1-scores as a well recognized metric and adapt the vulnerability detection score (VD-S) introduced by Ding et al. [9] as the patch detection score PD-S (= VD-S) = 𝐹 𝑁 𝑅@(𝐹 𝑃𝑅 ≤ 𝑟 ) where 𝑟 ∈ [0, 1] [9] to quantify detection rates under constrained misclassification rates to help mitigate base rate fallacy [2]. Following the evaluation protocol of Ding et al. [9] we set 𝑟 = 0.005. For random and group-stratified splits, we report mean and standard deviation across three seeds. Temporal and CVE splits are inherently unique splits.

4.2

Input Representations and Context Enrichment

Standard diffs from version control systems capture what was modified but may lack sufficient semantic context for accurate

Nils Loose, Joseph Bienhüls, Kristoffer Hempel, Felix Mächtle, and Thomas Eisenbarth

Algorithm 1: Context enriched diff Input: Pre-patch 𝑐 𝑝 and target commit 𝑐𝑡 . Output: Enriched diff D𝑐𝑡𝑥 D ← diff(𝑐 𝑝 , 𝑐𝑡 ) F ← changed_functions(D) for each 𝑓 in F do // Concrete syntax tree 𝑓

𝐶𝑆𝑇𝑝 ← TreeSitter(𝑐 𝑝 , 𝑓 ) 𝑓

𝐶𝑆𝑇𝑡 ← TreeSitter(𝑐𝑡 , 𝑓 ) // Statement level IR 𝑓

𝑓

(𝐼𝑅𝑝 , 𝐼𝑅𝑡 ) ← build_ir(𝐶𝑆𝑇𝑝 , 𝐶𝑆𝑇𝑡 ) // Compute structural diff Δ 𝑓 ← GumTree(𝐶𝑆𝑇𝑝 , 𝐶𝑆𝑇𝑡 ) // Changed statement extraction 𝑝 (𝑆 Δ , 𝑆 Δ𝑡 ) ← extract_statements(Δ 𝑓 ) for each 𝑘 in {𝑝, 𝑡 } do // Bi-directional slicing 𝑓

𝑓

𝑏𝑤 ← backward_slice(𝑆 𝑘 , 𝐶𝑆𝑇 , 𝑑) 𝑆𝑐𝑡𝑥 Δ 𝑘 𝑓𝑤 𝑓 𝑆𝑐𝑡𝑥 ← forward_slice(𝑆 Δ𝑘 , 𝐶𝑆𝑇𝑘 , 𝑑) 𝑓 𝑓 𝑓𝑤 𝑏𝑤 𝐼𝑅𝑘 ← update_ctx(𝐼𝑅𝑘 , 𝑆𝑐𝑡𝑥 , 𝑆𝑐𝑡𝑥 )

end // Merge and align IRs 𝐼𝑅 𝑓 ← merge_ir(𝐼𝑅𝑝 , 𝐼𝑅𝑡 , Δ 𝑓 ) 𝑝 𝐼𝑅 𝑓 ← add_control_flow(𝑆 Δ , 𝑆 Δ𝑡 ) D𝑐𝑡𝑥 ← append(D𝑐𝑡𝑥 , 𝐼𝑅 𝑓 ) 𝑓

𝑓

end return D𝑐𝑡𝑥

Figure 5: Example VFC from FFmpeg with integrated gradients attribution (absolute values) overlaid on the enriched diff representation. Changed lines (yellow) show the actual modification. Context lines (green) are added by the enrichment method through data-flow slicing. Color intensity indicates attribution magnitude.

classification. Consider the memory leak fix in Figure 5, where the actual changes (green) specify what was modified but without additional context the impact remains unclear. To clean the diffs, we strip comments by retrieving the full file versions before and after each commit and using Tree-sitter [4] to parse them into concrete syntax trees, precisely identifying comment nodes rather than relying on regex heuristics that cannot distinguish comments from comment-like patterns in string literals. To systematically evaluate whether additional semantic context improves SPD, we implement a lightweight intra-procedural context enrichment method. For each changed function, we generate concrete syntax trees for both pre- and post-patch versions using Tree-sitter and compute a structural diff using GumTree [12]. From the changed nodes, we identify relevant statements through forward and backward slicing along definition-use chains at depths 𝑑 ∈ {1, 2}, and augment the result with immediate control-flow enclosures (if-statements, loops). This produces three progressively enriched representations: controlflow context (cf), single data-flow pass (df1 ), and two data-flow passes (df2 ). Each context level is colored accordingly in Figure 5. The formal algorithm is provided in Algorithm 1.

Our method averages 1.72 seconds per sample on 500 PatchDB [47] samples, compared to 57.36 seconds for the Joern-based [54] analysis used by GraphSPD [45]. Unlike standard unified diffs that include physically neighbouring context lines regardless of their semantic relevance, the enriched representations replace these with targeted context based on control-flow and data-flow dependencies. As a result, the control-flow representation (cf) has a median size of 280 tokens2 , smaller than the baseline diff at 394 tokens (0.71×), because non-dependent neighbouring lines are removed and only the enclosing control-flow structures are retained. Adding additional data-flow context through two incremental data-flow passes increases sizes to 417 (df1 , 1.06×) and 747 (df2 , 1.9×) tokens.

4.3

Experimental Results

We structure our evaluation to progressively isolate the factors that determine SPD performance. We first quantify the relative contributions of commit messages and code changes, then evaluate the impact of splitting strategies on performance estimates. Using group-stratified splits we assess the impact of model scale and context enrichment across seven models. Finally, we investigate the effect of training data diversity and compare fine-tuned models to prompt-based classification. Commit Message Impact Following existing observations [5, 39] we systematically quantify the relative contributions of the commit message (NL) and diff (PL) for SPD. We train CodeBERT [13] and 2 Token counts use the CodeBERT tokenizer but are similar across tokenizers.

Code-Centric Detection of Vulnerability-Fixing Commits: A Unified Benchmark and Empirical Study

Table 4: SPD performances for different data splitting strategies and input contexts. Values in parentheses denote std. in the last decimal place over 3 seeds.

Model

Context

Random F1 ↑ PD-S ↓

Temporal Group-stratified F1 ↑ PD-S ↓ F1 ↑ PD-S ↓

CVE F1 ↑ PD-S ↓

CodeBERT [13]

Commit Message diffc diff

0.88 (0) 0.86 (0) 0.65 (1) 0.64 (1)

0.39 (1) 0.47 (2) 0.85 (1) 0.85 (0)

0.58 0.59 0.43 0.43

0.88 0.98 0.98 0.99

0.76 (1) 0.77 (0) 0.54 (2) 0.53 (3)

0.69 (4) 0.67 (4) 0.98 (2) 0.99 (1)

0.68 0.69 0.41 0.41

0.69 0.72 0.92 0.91

Commit Message CodeT5 Large [49] diffc diff

0.88 (0) 0.87 (0) 0.65 (1) 0.64 (1)

0.39 (0) 0.42 (3) 0.86 (1) 0.86 (0)

0.59 0.65 0.46 0.43

0.92 0.87 0.97 0.95

0.76 (5) 0.76 (3) 0.47 (8) 0.54 (4)

0.82 (7) 0.62 (3) 0.98 (1) 0.97 (2)

0.65 0.68 0.45 0.44

0.81 0.69 0.92 0.99

diffc retains source code comments; diff has comments stripped.

Splitting Strategy Impact Data leakage and spurious correlations are well-documented challenges in machine learning for security [2] that can cause severe overestimation of performance. Additionally, distribution shifts over time must be considered when evaluating detection systems [7, 19]. A t-SNE analysis of CodeBERT embeddings reveals clear repository-specific clusters in D2 (Figure 3), motivating an evaluation that explicitly tests generalization across project boundaries. This is further supported by Steenhoek et al. [38] who report a decrease in performance for various systems on VD when using cross-project evaluations. Results for random, temporal, group-stratified and CVE splits are shown in Table 4. Group-stratified splits reveal F1 decreases of approximately 14% for commit-level and up to 17% for diff-only representations compared to random splits across both models, confirming that models learn projectspecific patterns. Temporal splits show an even larger degradation with commit-level F1 dropping from 0.88 to 0.58 for CodeBERT, a decrease of 34%. To understand this temporal sensitivity, we conduct a sliding window analysis following Chow et al. [7] (Figure 6). A fixed-size window (20% each for train, validation and test) is moved across the chronologically ordered dataset in 5% increments, training a fresh CodeBERT model at each position. As the window advances, the Jensen-Shannon divergence between the project distributions in training and test roughly doubles from 0.28 to 0.62, while the Test F1 Test PD-S

JSD (train→test) Test vul. rate

1.0 0.8 0.6 0.4 0.2 15 20 09 16 20 05 16 20 12 17 20 06 17 20 12 18 20 08 19 -1 20 2 20 20 08 20 -1 1

0.0 20

CodeT5-Large [49] on D2 under random split conditions (Table 4). Removing the commit message drops F1 by roughly a quarter while PD-S more than doubles, indicating a substantial loss in detection capability. Conversely, training on the commit message alone yields performance within a few percentage points of the full-commit baseline. This indicates that models primarily exploit the textual signal for classification, masking any code reasoning capability, consistent with findings by Chen et al. [5] and Sun et al. [39]. We use integrated gradients [41] at the word embedding layer with a PAD-token baseline, 50 interpolation steps, and L2-norm summarization across the embedding dimension to compute per-token attribution scores. Figure 7 aggregates these scores by line type across the test set for CodeBERT on multiple representations. When the commit message is available (message+diff), it dominates the attribution budget, causing a substantial decrease in attention allocated to all other line types including the actual code changes. This effect is particularly pronounced for vulnerable commits, where messages receive a higher attribution share than for benign samples, suggesting that the model relies on message content as the primary discriminative signal for identifying VFCs. When messages are removed, file headers receive the highest per-token attribution. However, training and evaluating models on a representation with file headers removed yields comparable performance within standard deviations, indicating that the model does not causally depend on headers and that the attribution signal reflects informational density of header tokens rather than exclusive reliance. Across all representations, the actual added and deleted lines consistently receive modest attribution. We note that integrated gradients measures input sensitivity rather than learned internal representations, and the header ablation demonstrates that high attribution does not imply causal reliance. Training dynamics for all experiments (Figure 9) underline the training stability across representations. While not directly comparable due to different training sets, full commit classification performance is generally in line with Zhou et al. [61] and higher than other systems [5, 42, 43] that also utilize the commit message. These findings strengthen existing observations about message reliance and extend them through attribution analysis, raising fundamental questions about what features models actually use for code-only classification and whether current architectures engage with the security-relevant code modifications at all.

Test set midpoint

Figure 6: Sensitivity of CodeBERT on the diff representation to temporal split placement. A fixed-size window (20% train, 20% val, 20% test) slides across the chronologically ordered dataset in 5% increments. Dotted markers indicate three non-overlapping windows. All metrics are on a shared [0, 1] scale.

Nils Loose, Joseph Bienhüls, Kristoffer Hempel, Felix Mächtle, and Thomas Eisenbarth

Table 5: Code-centric SPD performances across different model classes, sizes and pre-training tasks. Values in parentheses denote std. in the last decimal place over 3 seeds. Metric

Context

CB++

UC512

UC1024

CT5L

CBART

Q2.5-C

F1 ↑

diff cf df1 df2

0.53 (3) 0.49 (3) 0.49 (3) 0.50 (2)

0.51 (2) 0.51 (4) 0.51 (4) 0.50 (4)

0.52 (4) 0.52 (5) 0.50 (4) 0.52 (2)

0.53 (3) 0.52 (3) 0.52 (3) 0.52 (2)

0.54 (4) 0.54 (3) 0.52 (3) 0.52 (2)

0.51 (3) 0.51 (2) 0.50 (1) 0.48 (0)

0.63 (3) 0.60 (4) 0.60 (4) 0.58 (3)

PD-S ↓

diff cf df1 df2

0.99 (1) 0.99 (1) 0.97 (2) 0.98 (2)

0.99 (1) 0.99 (1) 0.99 (1) 0.98 (2)

0.99 (0) 0.97 (2) 0.96 (3) 0.98 (1)

0.99 (0) 0.97 (2) 0.98 (2) 0.97 (3)

0.97 (2) 0.95 (2) 0.96 (2) 0.96 (2)

0.98 (0) 0.97 (2) 0.97 (2) 1.00 (0)

0.93 (5) 0.94 (5) 0.94 (4) 0.94 (5)

CB(++): CodeBERT (C++), UC: UniXcoder, CT5L: CodeT5 Large, CBART: CommitBART, Q2.5-C: Qwen2.5-Coder 14B [55]. diff: diff without comments. cf: diff+ control flow enclosure, dfi : cf + 𝑖 data-flow pass(es).

Attribution share (top-3 mean)

fraction of test commits from unseen projects grows from 6% to 22%. The test vulnerability rate drops as well, reflecting that later periods contain proportionally fewer advisory-linked commits. Test F1 tracks both trends, falling from 0.62 to 0.29, a 33 percentage-point swing caused entirely by shifting the date boundary. While the test F1 can be at least partially explained due to the base rate fallacy, the full saturation of the PD-S demonstrates that on these aggregated VFC datasets, temporal split performance is dominated by dataset composition biases rather than a temporal shift of the underlying distribution. The changing project mix and label prevalence over time confound concept drift with compositional shift. We therefore adopt group-stratified splitting for all subsequent experiments, which evaluates cross-project generalization without conflating multiple sources of distribution shift. 1.0 0.8

diff diffdf2 Msg+diff diff ( D4 )

Benign (left) Vulnerable (right)

0.6 0.4 0.2 0.0 Added

Deleted

Context

File headers

Message

Figure 7: Integrated gradients attribution (absolute values, top-3 token mean) across line types for CodeBERT on diff (blue), enriched diff (red), message+diff (green), and diff trained on D4 (brown) representations, split by benign (left) and vulnerable (right) samples. CVE Split Only a subset of VFCs in D2 are directly mapped to a specific CVE identifier. These CVE-mapped commits arguably carry the highest-confidence labels and most closely approximate the distribution of real-world security patches. To evaluate model performance on this subset, we distribute all CVE-mapped VFCs equally across validation and test sets, excluding them entirely from training. Benign samples are added to match the vulnerability ratio of the other splits. Results in Table 4 show that diff-only performance drops substantially on this split, with F1 falling close to random

and PD-S approaching 1.0. Even commit-level representations that include the message achieve notably lower F1 than under random splitting. This degradation likely reflects two compounding effects. The model is neither trained on these high-confidence samples nor evaluated under conditions that match the training distribution. The remaining training data, which lacks CVE confirmation, may contain weaker or noisier vulnerability patterns that do not transfer to the more clearly defined CVE-mapped patches. The CVE split thus provides a lower bound on detection performance on realistic samples, complementing the group-stratified evaluation that measures crossproject generalization. Note that this lower bound can be inflated when training data contains samples from projects that also appear in the CVE test set, as we investigate in the cross-dataset evaluation below. Model Impact To investigate how model architecture, size, context length and specialized pretraining affect SPD performance we evaluate all models shown in Table 3 on the diff representation under group-stratified splitting. We establish baselines using standard CodeBERT [13] and UniXcoder [15] models (both 125M parameters) and include CodeBERT C++ [59] and CommitBART [22] to assess specialized pre-training impact. As shown in Table 5, baseline models plateau with F1 scores between 0.49 and 0.54 with specialized models showing no advantage. Extended context capacity (UniXcoder with 1024 tokens) yields no improvement despite many diffs exceeding 512 tokens (see Figure 4a). Scaling from 125M to 14B parameters produces only marginal gains, with only Qwen2.5-Coder [55] showing a notable advantage in both F1 and PD-S. However, even at 14B parameters PD-S remains above 0.93, while smaller models remain above 0.95, indicating that none achieve reliable detection under strict false-positive constraints. These results suggest that models fail to learn meaningful representations that capture the underlying patterns. To investigate whether this limitation lies in the input representation, we systematically enrich the semantic context provided to the model. Context Enrichment Incrementally increasing the intra-procedural context by control-flow enclosures (cf), a single data-flow pass (df1 ) and two data-flow passes (df2 ) across all models does not yield consistent performance improvements (see Table 5). Conversely, the added context enrichment consistently reduces performance, indicating that trained models fail to utilize the additional signal. The attribution analysis (Figure 7) supports this observation. The

Code-Centric Detection of Vulnerability-Fixing Commits: A Unified Benchmark and Empirical Study

enriched representation (df2 , red) exhibits a nearly identical attribution distribution to the baseline diff (blue), with the added context lines receiving minimal attention. Rather than shifting focus to the semantically relevant data-flow statements, models maintain the same attention pattern regardless of the richer context available. Manual inspection of the top 50 cases where all models unanimously predict the wrong label reveals the nature of the code-level patterns models rely on. False positives are dominated by small changes exhibiting defensive hardening (47%) or memory operation (36%) patterns, changes that address bugs and may be security-dependent but are indistinguishable from vulnerability fixes without broader context. False negatives tend to involve larger multi-file changes or complex semantic modifications that exceed the local context window. Together with the attribution and ablation findings, these observations paint a consistent picture. Models exploit commit messages when available, and without them consistently fail to attend to the actual code changes. The features models rely on in the code-only setting remain difficult to localize. The code-level patterns that do drive unanimous failure cases are superficial structural patterns rather than security semantics. Having found no evidence that current models engage with security-relevant code semantics regardless of architecture or context, we next investigate whether the composition of training data can alter this pattern.

Table 7: F1 scores for prompt-based SPD using GLM-4.7 AWQ 355B [14] on the CVE split. DA: Direct Answer, CoT: Chain-ofThought prompting. Method

diff

df1

df2

DA CoT

0.61 0.59

0.58 0.55

(Table 7). Without any task-specific training, DA outperforms all finetuned models on the same CVE split by a consistent margin, while CoT achieves slightly lower F1. Context enrichment reduces performance for both strategies, mirroring the pattern observed for fine-tuned models. PD-S cannot be computed for prompt-based approaches as the discrete classification output does not allow calibration of the false-positive rate to the required constraint. While this result is noteworthy, it must be interpreted with caution. The model was trained on large scale datasets that may include public repositories and CVE advisories, making data contamination difficult to exclude. The performance advantage may therefore reflect pretraining exposure rather than genuine reasoning about code semantics. Nevertheless, the result is interesting and generative approaches that incorporate dynamic context fetching or tool-based code exploration represent a promising direction for future investigation.

Table 6: Cross-dataset evaluation: CodeBERT trained on different dataset compositions, evaluated on the CVE test split of D2 . Metric

F1 ↑ 0.33 0.41 0.44 0.62 PD-S ↓ 0.98 0.91 0.92 0.82 Cross-Dataset Evaluation To investigate whether training data quality or diversity drives performance on realistic samples, we train CodeBERT on four dataset compositions of increasing scope and evaluate on the CVE test split of D2 (Table 6). Performance improves monotonically with dataset scope, with the most substantial gain coming from cross-language training (D4 ). However, the attribution analysis (Figure 7) reveals that the D4 -trained model exhibits the same distribution of attribution across line types as the D2 -trained baseline, with code changes continuing to receive modest attention. While this aggregated view does not reveal whether specific patterns within input categories differ, the unchanged distribution suggests that the model has not shifted toward attending to code semantics. Combined with the observation that the training set expansion from D1 to D4 substantially increases the number of training samples from projects that also appear in the CVE test set, the improved performance is more consistent with broader project coverage reducing the out-of-distribution gap than with qualitatively different learned representations. Prompt-based Classification To compare fine-tuned models to prompt-based classification, we evaluate GLM-4.7 [14] (355B parameters) with activation aware quantization (AWQ) [21] on the CVE test split of D2 . Based on observations by Sprague et al. [37] that chain-of-thought prompting is not universally beneficial, we evaluate two strategies, direct answer (DA) classification and chainof-thought (CoT) reasoning, across all context enrichment levels

(a) All entries from D2 , colored by VFC label.

(b) Test set samples from D2 , colored by VFC label.

Figure 8: t-SNE visualizations of D2 based on fine-tuned CodeBERT [13] embeddings and colored by VFC label. Left column shows all samples from D2 , right column shows only test-set samples. Overall, among the fine-tuned models and representations tested we find no evidence that models acquire transferable security-relevant code understanding. Models exploit commit messages when available. When messages are removed, code changes consistently receive modest attention in the integrated gradients analysis, and performance degrades substantially under strict evaluation criteria. Neither scaling fine-tuned models up to 14B parameters nor adding intra-procedural context changes this picture. The t-SNE analysis of fine-tuned embeddings provides direct visual evidence for this conclusion (Figure 8). When the fine-tuned model is applied to all samples in D2 , including those from projects seen during training, clear label separation emerges in the embedding space. When restricted to test time samples from unseen projects, this separation largely vanishes, indicating that the learned discriminative structure is tied to project-specific patterns rather than transferable vulnerability semantics. The cross-dataset

Nils Loose, Joseph Bienhüls, Kristoffer Hempel, Felix Mächtle, and Thomas Eisenbarth

0.8

0.6

0.6 0.4

0.4

0.4 0.2 0.0

Eval F1

0.8

Message diff_c 2 3 4

diff Msg + diff 5 6 7 8 9

0.2 0.0

CT5L CBART Q2.5-C 7

Epoch

Epoch (a) Training dynamics for different representations

CB CB++ UC UC-1024 4 5 6

(b) Training dynamics for different models

0.2 0.0

D2 (ref.) D1 D3 D4 6 7 8 9

Epoch (c) Training dynamics for different datasets

Figure 9: Training dynamics across different evaluation settings. Each plot shows the evolution of the F1 on the validation set after each training epoch. attribution analysis reinforces this interpretation. Although training on D4 substantially improves CVE-split F1, the distribution of attribution across line types remains unchanged, suggesting broader project coverage rather than a shift toward code understanding. The same project-level memorization that makes evaluation methodology critical also explains the consistent failure to generalize identified through attribution and ablation. Our findings suggest that groupstratified splitting provides a more reliable evaluation strategy for SPD on aggregated datasets. Temporal evaluation remains essential for understanding distribution shift over time, but care must be taken during sampling to ensure that confounding factors from dataset construction, such as shifting project compositions or changes in collection methodology, do not dominate the observed performance variation.

Conclusion

We present a comprehensive evaluation of code LM-based SPD through a unified framework spanning 20 datasets and over 180 trained models. Among the fine-tuned models and representations tested, we find no evidence that models acquire transferable securityrelevant code understanding from code changes alone. Models exploit commit messages when available. Without messages, code changes consistently receive modest attention in the integrated gradients analysis, performance degrades under cross-project evaluation, and label separation in the embedding space vanishes on unseen projects. Neither model scaling, intra-procedural context enrichment, nor increased data diversity changes this picture. Even when more diverse training data improves F1 on CVE-confirmed vulnerabilities, the attribution distribution remains unchanged, indicating broader project coverage rather than a shift toward code understanding. Prompt-based generative classification outperforms fine-tuned models on CVEconfirmed samples, though data contamination from internet-scale pretraining cannot be excluded. Generative approaches with dynamic context fetching represent a promising direction for future work. Moving forward, identifying representations that capture the security semantics of code changes well remains a core challenge. The evaluated, intra-procedural diff-based inputs appear insufficient, and inter-procedural context, richer semantic representations, or fundamentally different input formulations may be necessary. Additionally, a better understanding of what models actually learn from training data is needed to ensure that reported performances reflect the desired

properties and are not artifacts of the training process or data distribution. Our integrated gradients analysis suggests that models do not primarily attend to code changes, but finer-grained interpretability methods including representation-level probing are required to better characterize model behavior. On the methodological side, our analysis suggests that group-stratified splitting provides a reliable and realistic evaluation strategy for SPD on aggregated datasets. Temporal evaluation remains essential for understanding distribution shift over time, but researchers must ensure that confounding factors from dataset construction, such as shifting project compositions or changes in collection methodology, do not dominate the observed performance variation. To support future work in these directions, we release our unified framework providing standardized access to the integrated datasets alongside the training and enrichment pipelines used in this study. Data Availability. To facilitate reproducibility and future research, we release a comprehensive artifact archive [44]; the toolkit is available at https://github.com/UzL-ITS/vfc_datasets. The archive contains: (1) the unified VFC framework with automated pipelines that download, normalize, deduplicate, and split all integrated datasets after users obtain access to the respective source collections, (2) the lightweight context enrichment tool for generating enriched diff representations, (3) the full training and evaluation pipeline including preprocessing, context-aware truncation, split generation, and model training configurations, and (4) complete training logs for all experiments. Due to licensing restrictions on several source datasets and the code contained in the collected commits, we are unable to redistribute the raw data or trained model weights directly. Instead, the framework automates the full data generation process once access to the gated source datasets has been obtained by the researcher.

References [1] Jafar Akhoundali, Sajad Rahim Nouri, Kristian F. D. Rietveld, and Olga Gadyatskaya. 2024. MoreFixes: A Large-Scale Dataset of CVE Fix Commits Mined through Enhanced Repository Discovery. In Proceedings of the 20th International Conference on Predictive Models and Data Analytics in Software Engineering, PROMISE 2024. ACM. https://doi.org/10.1145/3663533.3664036 [2] Daniel Arp, Erwin Quiring, Feargus Pendlebury, Alexander Warnecke, Fabio Pierazzi, Christian Wressnegger, Lorenzo Cavallaro, and Konrad Rieck. 2022. Dos and Don’ts of Machine Learning in Computer Security. In 31st USENIX Security Symposium, USENIX Security 2022, Boston, MA, USA, August 10-12, 2022, Kevin R. B. Butler and Kurt Thomas (Eds.). USENIX Association, 3971–3988. https://www.usenix.org/conference/usenixsecurity22/presentation/arp [3] Guru Prasad Bhandari, Amara Naseer, and Leon Moonen. 2021. CVEfixes: automated collection of vulnerabilities and their fixes from open-source software. In PROMISE ’21: 17th International Conference on Predictive Models and Data

Code-Centric Detection of Vulnerability-Fixing Commits: A Unified Benchmark and Empirical Study

Analytics in Software Engineering, 2021. ACM. https://doi.org/10.1145/3475960. 3475985 [4] Max Brunsfeld. [n. d.]. Tree-sitter. https://github.com/tree-sitter/tree-sitter [5] Tianyu Chen, Lin Li, Taotao Qian, Jingyi Liu, Wei Yang, Ding Li, Guangtai Liang, Qianxiang Wang, and Tao Xie. 2024. CompVPD: Iteratively Identifying Vulnerability Patches Based on Human Validation Results with a Precise Context. arXiv:2310.02530 [cs.CR] https://arxiv.org/abs/2310.02530 [6] Yizheng Chen, Zhoujie Ding, Lamya Alowain, Xinyun Chen, and David A. Wagner. 2023. DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection. In Proceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses, RAID 2023. ACM. https://doi.org/10.1145/3607199.3607242 [7] Theo Chow, Mario D’Onghia, Lorenz Linhardt, Zeliang Kan, Daniel Arp, Lorenzo Cavallaro, and Fabio Pierazzi. 2026. Beyond the TESSERACT: Trustworthy Dataset Curation for Sound Evaluations of Android Malware Classifiers. In Proceedings of the 4th IEEE Conference on Secure and Trustworthy Machine Learning. https://discovery.ucl.ac.uk/id/eprint/10220473/ [8] Yangruibo Ding, Yanjun Fu, Omniyyah Ibrahim, Chawin Sitawarin, Xinyun Chen, Basel Alomair, David A. Wagner, Baishakhi Ray, and Yizheng Chen. 2024. Vulnerability Detection with Code Language Models: How Far Are We? CoRR abs/2403.18624 (2024). https://doi.org/10.48550/ARXIV.2403.18624 arXiv:2403.18624 [9] Yangruibo Ding, Yanjun Fu, Omniyyah Ibrahim, Chawin Sitawarin, Xinyun Chen, Basel Alomair, David A. Wagner, Baishakhi Ray, and Yizheng Chen. 2025. Vulnerability Detection with Code Language Models: How Far are We?. In 47th IEEE/ACM International Conference on Software Engineering, ICSE 2025, Ottawa, ON, Canada, April 26 - May 6, 2025. IEEE, 1729–1741. https: //doi.org/10.1109/ICSE55347.2025.00038 [10] Trevor Dunlap, Elizabeth Lin, William Enck, and Bradley Reaves. 2024. VFCFinder: Pairing Security Advisories and Patches. In Proceedings of the 19th ACM Asia Conference on Computer and Communications Security, ASIA CCS 2024, Singapore, July 1-5, 2024, Jianying Zhou, Tony Q. S. Quek, Debin Gao, and Alvaro A. Cárdenas (Eds.). ACM. https://doi.org/10.1145/3634737.3657007 [11] Jonathan Evertz, Niklas Risse, Nicolai Neuer, Andreas Müller, Philipp Normann, Gaetano Sapia, Srishti Gupta, David Pape, Soumya Shaw, Devansh Srivastav, Christian Wressnegger, Erwin Quiring, Thorsten Eisenhofer, Daniel Arp, and Lea Schönherr. 2025. Chasing Shadows: Pitfalls in LLM Security Research. CoRR abs/2512.09549 (2025). https://doi.org/10.48550/ARXIV.2512.09549 arXiv:2512.09549 [12] Jean-Rémy Falleri and Matias Martinez. 2024. Fine-grained, accurate and scalable source differencing. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024. ACM, 231:1–231:12. https://doi.org/10.1145/3597503.3639148 [13] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020 (Findings of ACL, Vol. EMNLP 2020), Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, 1536–1547. https://doi.org/10.18653/V1/2020.FINDINGS-EMNLP.139 [14] GLM. 2025. GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models. CoRR abs/2508.06471 (2025). https://doi.org/10.48550/ARXIV.2508.06471 arXiv:2508.06471 [15] Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. UniXcoder: Unified Cross-Modal Pre-training for Code Representation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (Eds.). Association for Computational Linguistics, 7212–7225. https://doi.org/10.18653/V1/2022.ACLLONG.499 [16] Jingxuan He and Martin T. Vechev. 2023. Large Language Models for Code: Security Hardening and Adversarial Testing. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, CCS 2023, Copenhagen, Denmark, November 26-30, 2023. ACM. https://doi.org/10.1145/ 3576915.3623175 [17] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. https://openreview.net/forum?id=nZeVKeeFYf9 [18] Nasif Imtiaz, Aniqa Khanom, and Laurie A. Williams. 2023. Open or Sneaky? Fast or Slow? Light or Heavy?: Investigating Security Releases of Open Source Packages. IEEE Trans. Software Eng. 49, 4 (2023), 1540–1560. https://doi.org/ 10.1109/TSE.2022.3181010 [19] Zeliang Kan, Shae McFadden, Daniel Arp, Feargus Pendlebury, Roberto Jordaney, Johannes Kinder, Fabio Pierazzi, and Lorenzo Cavallaro. 2024. TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time (Extended Version). CoRR abs/2402.01359 (2024). https://doi.org/10.48550/

ARXIV.2402.01359 arXiv:2402.01359 [20] Jian Yi David Lee and Hai Leong Chieu. 2021. Co-training for Commit Classification. In Proceedings of the Seventh Workshop on Noisy User-generated Text, W-NUT 2021, Online, November 11, 2021, Wei Xu, Alan Ritter, Tim Baldwin, and Afshin Rahimi (Eds.). Association for Computational Linguistics, 389–395. https://doi.org/10.18653/V1/2021.WNUT-1.43 [21] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. 2023. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. CoRR abs/2306.00978 (2023). https://doi.org/10.48550/ARXIV. 2306.00978 arXiv:2306.00978 [22] Shangqing Liu, Yanzhou Li, and Yang Liu. 2022. CommitBART: A Large Pre-trained Model for GitHub Commits. CoRR abs/2208.08100 (2022). https: //doi.org/10.48550/ARXIV.2208.08100 arXiv:2208.08100 [23] Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. https://openreview.net/ forum?id=Bkg6RiCqY7 [24] Chaomeng Lu, Tianyu Li, Toon Dehaene, and Bert Lagaisse. 2025. ICVul: A Welllabeled C/C++ Vulnerability Dataset with Comprehensive Metadata and VCCs. In 22nd IEEE/ACM International Conference on Mining Software Repositories, MSR@ICSE 2025, Ottawa, ON, Canada, April 28-29, 2025. IEEE, 154–158. https://doi.org/10.1109/MSR66628.2025.00034 [25] Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. 2021. CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021. https://datasets-benchmarks-proceedings.neurips.cc/paper/ 2021/hash/c16a5320fa475530d9583c34fd356ef5-Abstract-round1.html [26] Changhua Luo, Wei Meng, and Shuai Wang. 2024. Strengthening Supply Chain Security with Fine-grained Safe Patch Identification. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024. ACM, 89:1–89:12. https://doi.org/10.1145/3597503. 3639104 [27] Giang Nguyen-Truong, Hong Jin Kang, David Lo, Abhishek Sharma, Andrew E. Santosa, Asankhaya Sharma, and Ming Yi Ang. 2022. HERMES: Using CommitIssue Linking to Detect Vulnerability-Fixing Commits. In IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2022, Honolulu, HI, USA, March 15-18, 2022. IEEE, 51–62. https://doi.org/10.1109/ SANER53432.2022.00018 [28] Chao Ni, Liyu Shen, Xiaohu Yang, Yan Zhu, and Shaohua Wang. 2024. MegaVul: A C/C++ Vulnerability Dataset with Comprehensive Code Representations. In 21st IEEE/ACM International Conference on Mining Software Repositories, MSR 2024, Lisbon, Portugal, April 15-16, 2024, Diomidis Spinellis, Alberto Bacchelli, and Eleni Constantinou (Eds.). ACM, 738–742. https://doi.org/10.1145/3643991. 3644886 [29] Georgios Nikitopoulos, Konstantina Dritsa, Panos Louridas, and Dimitris Mitropoulos. 2021. CrossVul: a cross-language vulnerability dataset with commit data. In ESEC/FSE ’21: 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM. https://doi.org/10.1145/3468264.3473122 [30] NIST. [n. d.]. National Vulnerability Database. https://nvd.nist.gov/ accessed 2025-09. [31] Serena Elisa Ponta, Henrik Plate, Antonino Sabetta, Michele Bezzi, and Cédric Dangremont. 2019. A manually-curated dataset of fixes to vulnerabilities of open-source software. In Proceedings of the 16th International Conference on Mining Software Repositories, MSR 2019. IEEE / ACM. https://doi.org/10.1109/ MSR.2019.00064 [32] Sofia Reis and Rui Abreu. 2017. SECBENCH: A Database of Real Security Vulnerabilities. In Proceedings of the International Workshop on Secure Software Engineering in DevOps and Agile Development co-located with the 22nd European Symposium on Research in Computer Security (ESORICS 2017) (CEUR Workshop Proceedings, Vol. 1977). CEUR-WS.org. https://ceur-ws.org/Vol-1977/paper6.pdf [33] Sofia Reis and Rui Abreu. 2021. A ground-truth dataset of real security patches. CoRR abs/2110.09635 (2021). arXiv:2110.09635 https://arxiv.org/abs/2110. 09635 [34] Niklas Risse and Marcel Böhme. 2024. Uncovering the Limits of Machine Learning for Automatic Vulnerability Detection. In 33rd USENIX Security Symposium, USENIX Security 2024. USENIX Association. https://www.usenix.org/conference/ usenixsecurity24/presentation/risse [35] Niklas Risse, Jing Liu, and Marcel Böhme. 2025. Top Score on the Wrong Exam: On Benchmarking in Machine Learning for Vulnerability Detection. Proc. ACM Softw. Eng. 2, ISSTA (2025), 388–410. https://doi.org/10.1145/3728887 [36] Antonino Sabetta and Michele Bezzi. 2018. A Practical Approach to the Automatic Classification of Security-Relevant Commits. In 2018 IEEE International Conference on Software Maintenance and Evolution, ICSME 2018,

Nils Loose, Joseph Bienhüls, Kristoffer Hempel, Felix Mächtle, and Thomas Eisenbarth

Madrid, Spain, September 23-29, 2018. IEEE Computer Society, 579–582. https://doi.org/10.1109/ICSME.2018.00058 [37] Zayne Rea Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. 2025. To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. https://openreview.net/forum?id=w6nlcS8Kkn [38] Benjamin Steenhoek, Md Mahbubur Rahman, Richard Jiles, and Wei Le. 2023. An Empirical Study of Deep Learning Models for Vulnerability Detection. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023. IEEE, 2237–2248. https://doi.org/10. 1109/ICSE48619.2023.00188 [39] Jiamou Sun, Zhenchang Xing, Qinghua Lu, Xiwei Xu, Liming Zhu, Thong Hoang, and Dehai Zhao. 2023. Silent Vulnerable Dependency Alert Prediction with Vulnerability Key Aspect Explanation. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023. IEEE, 970–982. https://doi.org/10.1109/ICSE48619.2023.00089 [40] Shiyu Sun, Shu Wang, Xinda Wang, Yunlong Xing, Elisa Zhang, and Kun Sun. 2023. Exploring Security Commits in Python. In IEEE International Conference on Software Maintenance and Evolution, ICSME 2023. IEEE. https: //doi.org/10.1109/ICSME58846.2023.00027 [41] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic Attribution for Deep Networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 (Proceedings of Machine Learning Research), Doina Precup and Yee Whye Teh (Eds.). PMLR, 3319–3328. http://proceedings.mlr.press/v70/sundararajan17a.html [42] Xunzhu Tang, Zhenghan Chen, Saad Ezzini, Haoye Tian, Yewei Song, Jacques Klein, and Tegawendé F. Bissyandé. 2023. Multilevel Semantic Embedding of Software Patches: A Fine-to-Coarse Grained Approach Towards Security Patch Detection. CoRR abs/2308.15233 (2023). https://doi.org/10.48550/ARXIV.2308. 15233 arXiv:2308.15233 [43] Xunzhu Tang, Kisub Kim, Saad Ezzini, Yewei Song, Haoye Tian, Jacques Klein, and Tegawende Bissyande. 2025. Just-in-Time Detection of Silent Security Patches. ACM Trans. Softw. Eng. Methodol. (July 2025). https://doi.org/10.1145/3749370 Just Accepted. [44] VFCDetective Artifact 2026. https://doi.org/10.5281/zenodo.19250701 Zenodo artifact archive. [45] Shu Wang, Xinda Wang, Kun Sun, Sushil Jajodia, Haining Wang, and Qi Li. 2023. GraphSPD: Graph-Based Security Patch Detection with Enriched Code Semantics. In 2023 IEEE Symposium on Security and Privacy (SP). IEEE Computer Society, Los Alamitos, CA, USA, 2409–2426. https://doi.org/10.1109/SP46215.2023. 00035 [46] Shichao Wang, Yun Zhang, Liagfeng Bao, Xin Xia, and Minghui Wu. 2022. VCMatch: A Ranking-based Approach for Automatic Security Patches Localization for OSS Vulnerabilities. In IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2022, Honolulu, HI, USA, March 15-18, 2022. IEEE, 589–600. https://doi.org/10.1109/SANER53432.2022.00076 [47] Xinda Wang, Shu Wang, Pengbin Feng, Kun Sun, and Sushil Jajodia. 2021. PatchDB: A Large-Scale Security Patch Dataset. In 51st Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2021. IEEE. https://doi.org/10.1109/DSN48987.2021.00030 [48] Xinda Wang, Shu Wang, Pengbin Feng, Kun Sun, Sushil Jajodia, Sanae Benchaaboun, and Frank Geck. 2021. PatchRNN: A Deep Learning-Based System for Security Patch Identification. In 2021 IEEE Military Communications Conference, MILCOM 2021, San Diego, CA, USA, November 29 - Dec. 2, 2021. IEEE, 595–600. https://doi.org/10.1109/MILCOM52596.2021.9652940 [49] Yue Wang, Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi. 2021. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation. In EMNLP. Association for Computational Linguistics, 8696–8708. [50] Laura Wartschinski, Yannic Noller, Thomas Vogel, Timo Kehrer, and Lars Grunske. 2022. VUDENC: Vulnerability Detection with Deep Learning on a Natural Codebase for Python. Inf. Softw. Technol. 144 (2022). https://doi.org/10.1016/J. INFSOF.2021.106809 [51] Xin-Cheng Wen, Zirui Lin, Cuiyun Gao, Hongyu Zhang, Yong Wang, and Qing Liao. 2024. Repository-Level Graph Representation Learning for Enhanced Security Patch Detection. arXiv:2412.08068 [cs.SE] https://arxiv.org/abs/2412. 08068 [52] Congying Xu, Bihuan Chen, Chenhao Lu, Kaifeng Huang, Xin Peng, and Yang Liu. 2022. Tracking patches for open source software vulnerabilities. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, Singapore, Singapore, November 14-18, 2022, Abhik Roychoudhury, Cristian Cadar, and Miryung Kim (Eds.). ACM, 860–871. https://doi.org/10.1145/3540250.3549125 [53] Fabian Yamaguchi, Nico Golde, Daniel Arp, and Konrad Rieck. 2014. Modeling and Discovering Vulnerabilities with Code Property Graphs. In 2014 IEEE Symposium on Security and Privacy, SP 2014, Berkeley, CA, USA, May 18-21,

2014. IEEE Computer Society, 590–604. https://doi.org/10.1109/SP.2014.44 [54] Fabian Yamaguchi, Nico Golde, Daniel Arp, and Konrad Rieck. 2014. Modeling and Discovering Vulnerabilities with Code Property Graphs. In 2014 IEEE Symposium on Security and Privacy, SP 2014, Berkeley, CA, USA, May 18-21, 2014. IEEE Computer Society, 590–604. https://doi.org/10.1109/SP.2014.44 [55] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2024. Qwen2.5 Technical Report. CoRR abs/2412.15115 (2024). https://doi.org/10.48550/ARXIV.2412.15115 arXiv:2412.15115 [56] Xu Yang, Wenhan Zhu, Michael Pacheco, Jiayuan Zhou, Shaowei Wang, Xing Hu, and Kui Liu. 2025. Code Change Intention, Development Artifact, and History Vulnerability: Putting Them Together for Vulnerability Fix Detection by LLM. Proc. ACM Softw. Eng. 2, FSE (2025), 489–510. https://doi.org/10.1145/3715738 [57] Jiayuan Zhou, Michael Pacheco, Jinfu Chen, Xing Hu, Xin Xia, David Lo, and Ahmed E. Hassan. 2023. CoLeFunDa: Explainable Silent Vulnerability Fix Identification. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023. IEEE, 2565– 2577. https://doi.org/10.1109/ICSE48619.2023.00214 [58] Jiayuan Zhou, Michael Pacheco, Zhiyuan Wan, Xin Xia, David Lo, Yuan Wang, and Ahmed E. Hassan. 2021. Finding A Needle in a Haystack: Automated Mining of Silent Vulnerability Fixes. In 36th IEEE/ACM International Conference on Automated Software Engineering, ASE 2021, Melbourne, Australia, November 15-19, 2021. IEEE, 705–716. https://doi.org/10.1109/ASE51524.2021.9678720 [59] Shuyan Zhou, Uri Alon, Sumit Agarwal, and Graham Neubig. 2023. CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, 13921–13937. https://doi.org/10.18653/V1/2023.EMNLP-MAIN.859 [60] Yaqin Zhou, Shangqing Liu, Jing Kai Siow, Xiaoning Du, and Yang Liu. 2019. Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019. https://proceedings.neurips.cc/paper/2019/hash/ 49265d2447bc3bbfe9e76306ce40a31f-Abstract.html [61] Yaqin Zhou, Jing Kai Siow, Chenyu Wang, Shangqing Liu, and Yang Liu. 2022. SPI: Automated Identification of Security Patches via Commits. ACM Trans. Softw. Eng. Methodol. 31, 1 (2022), 13:1–13:27. https://doi.org/10.1145/3468854

Record · ID 180588 · SHA-256 99ca38caed33fbf2
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.