Reassessing Dataset Quality and Fine-Tuning Practices for Vulnerability Detection on PrimeVul dataset

Reassessing Dataset Quality and Fine-Tuning Practices for Vulnerability Detection on PrimeVul dataset Norbert Sándor Szolnoki∗

Gábor Antal∗†

szolnoki@inf.u-szeged.hu

antal@inf.u-szeged.hu

University of Szeged Szeged, Hungary

University of Szeged Szeged, Hungary

arXiv:2611.06373v1 [cs.SE] 4 Nov 2026

Abstract Recent studies report promising results from supervised fine-tuning of pre-trained code language models for automated vulnerability detection on benchmarks such as PrimeVul. In this paper, we present a large-scale empirical evaluation of both model performance and dataset quality in this setting. We systematically fine-tune three widely used encoder models (CodeBERT, GraphCodeBERT, and UniXCoder) across five imbalance-mitigation strategies, extensive hyperparameter sweeps, and multiple random seeds, resulting in 770 independent training runs. Our results reveal a consistent pattern. While standard metrics such as F1 and ROC-AUC occasionally suggest improvements, security-critical and calibration-aware measures tell a different story. Matthews Correlation Coefficient remains near zero, predicted probabilities are poorly calibrated, and evaluation under strict false-positive constraints reveals high false-negative rates. Neither imbalance handling, threshold optimization, nor post-hoc calibration recovers a meaningful discriminative signal. We also conduct a data quality assessment of the PrimeVul benchmark and identify structural issues such as minimal semantic differences between vulnerable and fixed functions and annotation noise introduced by commit-level labeling. These findings suggest that the PrimeVul dataset, in its current form, presents significant challenges for reliable AI-based vulnerability detection in software engineering with the methodologies employed in this study. Our study highlights the need for more robust dataset design, improved evaluation protocols, and a rethinking of fine-tuning practices for security-critical tasks.

CCS Concepts • Security and privacy → Software security engineering; Vulnerability scanners; • Computing methodologies → Machine learning.

Keywords vulnerability detection, fine-tuning, QLoRA, dataset quality ∗ Both authors contributed equally to this research. † Also with FrontEndART Ltd..

This work is licensed under a Creative Commons Attribution 4.0 International License. EASE ’26, Glasgow, United Kingdom © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2348-3/2026/06

https://doi.org/10.1145/3816483.3816594

ACM Reference Format: Norbert Sándor Szolnoki and Gábor Antal. 2026. Reassessing Dataset Quality and Fine-Tuning Practices for Vulnerability Detection on PrimeVul dataset. In International Conference on Evaluation and Assessment in Software Engineering (EASE ’26), June 09–12, 2026, Glasgow, United Kingdom. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3816483.3816594

1

Introduction

Vulnerability detection in source code is a central task in software security and has become a critical research frontier due to the increasing size, complexity, and exposure level of modern software systems. Traditional static analysis techniques frequently have high false positive rates and are difficult to scale across heterogeneous codebases [13, 16]. Recent transformer-based code models suggest that fine-tuning pre-trained language models on vulnerability datasets may allow them to learn patterns of insecure behavior and outperform traditional static analysis [6, 9, 10]. As a result, fine-tuning models such as CodeBERT, GraphCodeBERT, and UniXCoder has emerged as a promising approach in vulnerability classification research [15, 22]. However, despite promising results in prior work, the effectiveness of these models in real-world security settings remains uncertain. Given the increasing role of AI in software engineering, especially in security-critical domains, understanding the correlation between model behavior and dataset properties is essential. Recent studies suggest that high reported performance may reflect dataset artifacts rather than generalizable learning. Examples include inconsistent labeling, duplicated functional patterns across train and test splits, and minimal semantic differences between vulnerable and fixed code versions [1, 24]. Moreover, commonly used metrics such as F1 score or ROC-AUC quantify statistical separability but fail to reflect operational security concerns such as false negatives under strict false-positive constraints [12, 21]. These observations raise broader concerns about both model reliability and dataset quality in SE-specific AI tasks. This paper presents a systematic fine-tuning and diagnostic evaluation of CodeBERT, GraphCodeBERT, and UniXCoder on the PrimeVul dataset, a widely used benchmark for function-level vulnerability detection. Our study jointly assesses model performance and dataset quality, focusing on the real-world relevance, robustness, and calibration of the learned classifiers. Specifically, we investigate the following research questions: • RQ1: Does supervised fine-tuning on PrimeVul produce consistent improvements in standard metrics (F1, ROC-AUC) across models, seeds, and imbalance mitigation strategies? • RQ2: Are these improvements robust and security-relevant when evaluated using calibration error (ECE), Matthews

EASE ’26, June 09–12, 2026, Glasgow, United Kingdom

correlation coefficient (MCC), and VD-S (false negative rate under strict false positive constraints)? • RQ3: Which characteristics of the PrimeVul dataset may help explain unstable or misleading model behavior under the fine-tuning setups evaluated in this study? Our findings reveal that across more than one hundred training configurations, standard metrics occasionally suggest performance gains, while security-critical metrics indicate unreliable or nearrandom behavior. In several cases, ROC-AUC remains close to 0.5 despite high F1 scores, and models fail to generalize beyond trivial threshold adjustments. Furthermore, we identify multiple dataset characteristics that may help explain the observed learning difficulties, including near-identical function pairs and noise introduced by commit-level labeling heuristics. Contrary to expectations from prior literature, our results show no evidence that existing encoder models can be reliably fine-tuned on PrimeVul for practical vulnerability detection. We provide a negative but actionable outcome that highlights limitations in both model pipelines and dataset construction. Our diagnostic evaluation contributes to a growing body of work emphasizing the need for rigorously designed benchmarks and evaluation protocols in software engineering AI. Contributions. This work presents a comprehensive empirical evaluation of three widely used pre-trained encoder models (CodeBERT, GraphCodeBERT, and UniXCoder) under supervised fine-tuning on the PrimeVul benchmark. We systematically evaluate five commonly adopted imbalance mitigation strategies and a broad range of hyperparameter configurations, resulting in a largescale, multi-seed experimental study. Beyond standard performance metrics, we provide a calibration-aware and risk-aware analysis showing that metrics such as F1 and ROC-AUC may give misleading impressions of model effectiveness. Finally, we contribute a data quality assessment of PrimeVul, highlighting structural characteristics that may affect learnability and reliability. Under the experimental settings considered in this study, we find no evidence that the evaluated standard encoder-based fine-tuning pipelines yield dependable vulnerability detection performance on PrimeVul. These results motivate further investigation into dataset design choices and evaluation practices in security-focused SE tasks. The remainder of the paper reviews relevant background and related work (Sections 2–3), details the experimental methodology (Section 4), presents results (Section 5), discusses implications and limitations (Sections 6–7), and concludes with directions for future research (Sections 8–9).

2 Background 2.1 Vulnerability Detection in Source Code Software vulnerability detection aims to identify code fragments that introduce security weaknesses exploitable by an adversary. Formally, given a function or method 𝑥, the task is to learn a mapping 𝑓 : 𝑥 → {0, 1}, where 1 denotes a vulnerability. The task is challenging due to incomplete specifications of what constitutes a vulnerability, semantic diversity, and dependency on execution context [22].

Norbert Sándor Szolnoki and Gábor Antal

2.2

Transformer-based Code Models

Recent advances in pretrained transformer encoders have shaped the direction of static vulnerability detection. CodeBERT [6], GraphCodeBERT [9] and UniXcoder [10] extend masked language modelling with code-specific objectives such as data-flow prediction or cross-modal alignment. These models are frequently adopted as general-purpose feature extractors in downstream classification. Their success on tasks such as code search, summarization and clone detection has motivated their application to security tasks.

2.3

The PrimeVul Benchmark

PrimeVul [4] is a function-level vulnerability detection dataset designed to mitigate biases found in earlier resources. It employs temporal splits, removes exact duplicates and aligns samples with documented CVE and CWE identifiers. Importantly, PrimeVul introduces risk-aware metrics such as VD-S to reflect deployment contexts where false positives must remain below strict operational tolerances.

2.4

Evaluation Metrics in Security Contexts

Security classification is inherently imbalanced and cost-asymmetric. Accordingly, traditional metrics such as accuracy and F1 may provide misleading conclusions [27]. Matthews Correlation Coefficient is recommended for imbalanced settings and has been shown to be more stable [2]. Calibration metrics such as Expected Calibration Error and Brier Score measure probabilistic reliability, which is critical for operational decision-making [19]. PrimeVul further proposes VD-S to assess failure modes under strict false-positive budget constraints. These metrics jointly provide a multi-perspective evaluation complementary to the conventional F1 score.

3 Related Work 3.1 Encoder Models for Vulnerability Detection Pretrained transformer encoders are widely deployed in vulnerability classification. Studies based on CodeBERT [6], GraphCodeBERT [9] and UniXcoder [10] commonly report improvements on benchmarks such as BigVul and Devign [1]. However, subsequent analyses indicate that these gains often fail to generalize beyond benchmark-specific artifacts, including duplicated patterns, spurious correlations, and dataset-specific biases [20]. Recent work explores architectural extensions or alternative learning objectives, such as heterogeneous source-level intermediate representations in HGVul [23] or hierarchical supervised contrastive learning for CWE classification [11], reporting performance improvements under controlled experimental conditions.

3.2

PrimeVul-based Studies

PrimeVul has enabled research on vulnerability detection in more realistic learning settings by introducing temporal splits, functionlevel labels, and risk-aware evaluation metrics. Li et al. propose CleanVul, an LLM-assisted data filtering pipeline designed to mitigate annotation noise in commit-based datasets [14]. Wen et al. introduce a reinforcement-based curriculum learning approach using synthetic preference signals to improve encoder robustness [26].

Reassessing Dataset Quality and Fine-Tuning Practices for Vulnerability Detection on PrimeVul dataset

Other approaches combine auxiliary semantic signals, such as commit metadata or documentation cues, with representation learning to improve detection performance. These studies suggest that model behavior on PrimeVul is highly sensitive to dataset curation, auxiliary supervision, and training protocol choices. Straightforward supervised fine-tuning of encoder models often yields inconsistent or degraded results when compared to earlier benchmarks.

3.3

Evaluation and Metric Reliability

Previous research on evaluation methodology indicates that model rankings may change when transitioning from accuracy or F1 score to correlation-based, calibration-aware, or risk-sensitive metrics [17]. Wakili et al. demonstrate that calibration-aware evaluation is essential to avoid overestimating the operational utility of machine learning models in security monitoring systems [2]. Ding et al. formalize the VD-S metric to expose failure modes under strict falsepositive constraints, reflecting realistic deployment settings [4]. Across the literature, there is increasing recognition that evaluation protocols significantly shape reported findings, particularly in the presence of dataset noise, class imbalance and temporal variance, which can lead to contradictory conclusions under standard finetuning setups.

3.4

Methodological Differences in Prior PrimeVul Studies

Many previous studies show positive outcomes using transformerbased models for PrimeVul. However, these outcomes are often achieved under experimental conditions that significantly differ from ours. Common methodological choices include reporting single-seed results, tuning decision thresholds on test data, focusing exclusively on accuracy or F1 score, or omitting security-relevant constraints such as low false positive rates. In addition, few studies analyze the impact of near-duplicate functions or paired vulnerable and fixed samples on learning stability. Our work complements these efforts by systematically evaluating robustness across seeds and configurations, incorporating calibration- and risk-aware metrics, and explicitly analyzing dataset characteristics that influence model behavior.

4

Methodology

This section describes the experimental design used to evaluate standard fine-tuning practices under controlled and reproducible conditions.

4.1

Problem Formulation

We consider function-level vulnerability detection as a binary classification task. Each instance is a function (or method) body 𝑥 extracted from a C or C++ project, and the label 𝑦 ∈ {0, 1} indicates whether 𝑥 contains a vulnerability (𝑦 = 1) or not (𝑦 = 0). 𝑁 , the goal is to learn a classifier Given a training set {(𝑥𝑖 , 𝑦𝑖 )}𝑖=1 𝑓𝜃 : 𝑥 ↦→ 𝑝𝜃 (𝑦 = 1 | 𝑥) that outputs a probability estimate for the vulnerable class. A decision threshold 𝜏 ∈ [0, 1] is then used to  obtain a binary prediction 𝑦ˆ = I 𝑝𝜃 (𝑦 = 1 | 𝑥) ≥ 𝜏 .

4.2

EASE ’26, June 09–12, 2026, Glasgow, United Kingdom

Dataset and Preprocessing

We use the PrimeVul dataset [4] in its JSONL format, which provides three disjoint splits: training, validation, and test. Each entry contains at least: • func: the function-level source code, • target: the binary vulnerability label, • commit_message: the commit message (when available). We parse the JSONL files line by line using the Python standard library, convert them into pandas DataFrames, and then into datasets.Dataset objects. For each instance, we construct a single input field code: • by default, code is set to the raw function text (func); • when commit context is enabled, code is the concatenation of commit_message, two newline characters, and func. The numeric label is stored in a field labels. We do not perform any additional filtering, de-duplication or manual cleaning beyond the original PrimeVul preparation. The validation split is used only for model selection, threshold tuning, and calibration; the test split is held out for final evaluation.

4.3

Models and Tokenization

We evaluate three widely used pre-trained transformer encoders for source code: • CodeBERT (microsoft/codebert-base) [6], • GraphCodeBERT (microsoft/graphcodebert-base) [9], • UniXcoder (microsoft/unixcoder-base) [10]. For each encoder, we instantiate a binary classifier using AutoModelForSequenceClassification with num_labels = 2. We employ the corresponding AutoTokenizer for each model. The code field is tokenized with truncation and maximum-length padding. We explore three maximum sequence lengths, 𝐿 ∈ {128, 256, 512}, which determines the amount of code (and optional commit context) visible to the model.

4.4

Imbalance Handling Strategies

PrimeVul exhibits an imbalanced class distribution with significantly fewer vulnerable functions than benign functions. To study the effect of common imbalance mitigation techniques, we define five training strategies: Baseline The original training distribution is used without resampling, and the loss is the standard cross-entropy. Class weights We compute inverse-frequency class weights on the original training labels. Let 𝑛 0 and 𝑛 1 denote the number of benign and vulnerable examples, and 𝑛 = 𝑛 0 + 𝑛 1 . The weight for class 𝑐 is 𝑤𝑐 =

𝑛 , 2𝑛𝑐

𝑐 ∈ {0, 1},

and we use these weights in the cross-entropy loss. Undersampling All vulnerable samples are retained. Benign samples are randomly undersampled without replacement such that the final ratio is approximately 3 : 1 (benign to vulnerable). The resulting training set is shuffled before each run.

EASE ’26, June 09–12, 2026, Glasgow, United Kingdom

Oversampling All benign samples are retained. Vulnerable samples are oversampled with replacement until the vulnerable class accounts for approximately 25% of the training set. When the original proportion of vulnerable samples already exceeds this value, no oversampling is applied. Focal loss The original training distribution is preserved, but the loss is replaced by focal loss with focusing parameter 𝛾 = 2.0. For a sample with true class 𝑦 and predicted probability 𝑝 𝑦 , the per-sample loss is ℓfocal = −(1 − 𝑝 𝑦 )𝛾 log(𝑝 𝑦 + 𝜀), where 𝜀 is a small constant to avoid numerical issues. Resampling is applied only to the training split; the validation and test splits remain unchanged.

4.5

temperature parameter 𝑇 > 0 and define scaled logits as 𝑧/𝑇 . The value of 𝑇 is estimated by minimizing the negative log-likelihood on the validation set using L-BFGS (up to 50 iterations). The calibrated model is then applied to validation and test logits, and probabilities are recomputed. We repeat the threshold search on calibrated validation probabilities and obtain a calibrated optimal threshold 𝜏˜★. As before, we compute all metrics at the default threshold 𝜏 = 0.5 and at 𝜏˜★ for both the validation and test sets.

4.7

Calibration and Threshold Selection

We evaluate each trained model both before and after post-hoc calibration. Uncalibrated evaluation. We first obtain logits for the validation and test sets using the trained model and convert them to class probabilities. We retain 𝑝𝜃 (𝑦 = 1 | 𝑥) as the score for the vulnerable class. On the validation set we perform a one-dimensional search over decision thresholds 𝜏 ∈ {0.01, 0.02, . . . , 0.99} and select the threshold 𝜏 ★ that maximizes the validation F1 score. We then compute all metrics on both validation and test sets at (i) the default threshold 𝜏 = 0.5 and (ii) the validation-optimal threshold 𝜏 ★. Temperature scaling. To improve probability calibration, we apply temperature scaling on the validation set. We introduce a scalar

Evaluation of Metrics

Let 𝑦 denote the vector of true labels and 𝑝 the predicted probabilities for the vulnerable class. For a fixed decision threshold 𝜏, we derive binary predictions 𝑦ˆ and compute the following metrics: • accuracy and balanced accuracy, • F1 score, • Matthews Correlation Coefficient (MCC) [24], which measures the correlation between predicted and true labels and remains informative under class imbalance; values near zero indicate performance statistically indistinguishable from random guessing, • ROC-AUC[5], treating 𝑝 as continuous scores and measuring threshold-independent separability, • Brier Score [7], defined as the mean squared error between 𝑝 and 𝑦, which captures the overall quality of probabilistic predictions, • Expected Calibration Error (ECE)[8], computed using 15 equal-width probability bins, which quantifies the mismatch between predicted confidence and empirical accuracy; lower values indicate better-calibrated probabilities, • VD-S, a risk-aware metric defined as the false negative rate at the threshold that maximizes the true positive rate while enforcing a false positive rate below 0.5% [4], reflecting deployment scenarios where false positives are tightly constrained and missed vulnerabilities represent the dominant failure mode.

Training Procedure

We used the Hugging Face Trainer API. A custom trainer which publicly available, ImbalanceTrainer, extends the base class and overrides the compute_loss method to select between unweighted cross-entropy, class-weighted cross-entropy and focal loss based on the chosen strategy. Apart from this loss selection, the training loop follows the standard implementation. For each model and strategy we perform a small hyperparameter grid search over: • learning rate 𝑙𝑟 ∈ {1 · 10−5, 2 · 10−5, 3 · 10−5 }, • maximum sequence length 𝐿 ∈ {128, 256, 512}, • training batch size 𝐵 train ∈ {8, 16}. The evaluation batch size is fixed to 𝐵 eval = 16 and the number of epochs is fixed to 𝐸 = 5. Each configuration is run with three random seeds 𝑠 ∈ {1, 2, 3} using set_seed. We enable mixed-precision training on GPUs (fp16) in all experiments. We do not apply early stopping. We do not retain intermediate checkpoints in order to reduce storage and, the model is trained for the full five epochs and evaluated once at the end. The hyperparameters above were set as static variables to evaluate all the models with the same settings and make our results reproducible. We have not tested what the best hyperparameters are, and it is not the scope of our research. A combination of model, strategy, hyperparameters and seed defines a single run. For each run we store all metrics and predictions in a dedicated directory that encodes the configuration.

4.6

Norbert Sándor Szolnoki and Gábor Antal

All metrics are computed on both validation and test sets, using uncalibrated and calibrated probabilities, and evaluated at the default threshold 𝜏 = 0.5 as well as at thresholds optimized on the validation split.

4.8

Experiment Pipeline

Figure 1 provides an overview of our experimental pipeline. The pipeline is designed to trace the full lifecycle of a single experimental configuration, from data ingestion to final metric reporting. Each experiment is defined by the tuple (𝑚, 𝑠, ℓ, 𝑏, 𝐿, 𝛾, 𝑟 ), where 𝑚 is the pretrained model (CodeBERT, GraphCodeBERT, UniXcoder), 𝑠 is the imbalance strategy (baseline, class weights, undersample, oversample, focal), ℓ is the maximum sequence length, 𝑏 is the batch size, 𝐿 is the learning rate, 𝛾 is the focal loss parameter (if applicable), and 𝑟 is the random seed. Step 1. Dataset loading. We load the official PrimeVul JSONL files (train, validation, test). Fields func and target are used as inputs and labels. Optional commit-level context is prepended when –use_commit_context is enabled.

Reassessing Dataset Quality and Fine-Tuning Practices for Vulnerability Detection on PrimeVul dataset

PrimeVul JSONL loading

Tokenization + optional commit context

Imbalance strategy (baseline

Model fine-tuning

Temperature scaling

Threshold optimisation

Final evaluation MCC, AUC, ECE, VD-S, etc. Figure 1: End-to-end experimental pipeline used in this study. Step 2. Preprocessing and tokenization. Each function is tokenized using the model-specific tokenizer with fixed value of max_length and padding="max_length". Step 3. Imbalance strategy application. Depending on configuration, we apply one of: (i) no modification, (ii) class-weighted loss, (iii) undersampling of majority class, (iv) oversampling of minority class, (v) focal loss. Resampling is applied only to the training split to avoid distribution shift. Step 4. Fine-tuning. We fine-tune the model using the Trainer API with 5 epochs, AdamW optimizer, and the specified hyperparameters. Checkpointing is disabled to prevent accidental model selection and enforce strict comparability. Step 5. Prediction and calibration. We evaluate the model on validation and test sets to obtain logits. We then fit a temperature scaling layer on the validation logits to produce calibrated probability estimates. Step 6. Threshold optimisation. For both calibrated and uncalibrated predictions, we search over thresholds in [0.01, 0.99] to maximise 𝐹 1 on the validation set. Step 7. Evaluation. We compute the following metrics on the test set: accuracy, balanced accuracy, MCC, ROC-AUC, ECE, Brier score, and VD-S (FNR under FPR ≤ 0.005). All predictions, probabilities, thresholds, losses, and metric values are written to disk for post-hoc analysis. Step 8. Aggregation. Results are aggregated across seeds. We report mean and standard deviation to capture model stability.

4.9

EASE ’26, June 09–12, 2026, Glasgow, United Kingdom

Design Rationale and Methodological Scope

The experimental design of this study prioritizes breadth, control and diagnostic coverage over architectural novelty. Rather than proposing a new model or training objective, we intentionally adopt widely used pretrained encoder architectures and standard fine-tuning pipelines in order to evaluate a common experimental practice under controlled and reproducible conditions. This choice allows observed performance limitations to be attributed to task formulation, data properties and evaluation protocol, rather than to idiosyncratic modeling decisions. Hyperparameter ranges, imbalance mitigation strategies and calibration procedures were selected to reflect established practice in the vulnerability detection literature. At the same time, design choices that could introduce optimistic bias, such as test-set threshold tuning, early stopping on test metrics or selective reporting across seeds were deliberately avoided. As a result, the study emphasizes robustness and interpretability of outcomes over peak performance, with the goal of providing a reliable empirical assessment of what standard fine-tuning using the PrimeVul dataset.

4.10

Data Quality Analysis of PrimeVul

In addition to evaluating model performance, we conduct a diagnostic analysis of the PrimeVul dataset to examine dataset characteristics that may help explain the observed model behavior. Our results suggest that several structural characteristics of PrimeVul may limit its effectiveness as a training resource for vulnerability detection under the setups studied here. First, we observe a high degree of lexical and syntactic similarity between paired vulnerable and fixed functions. Using TF-IDF vectorization over character-level 𝑛-grams (𝑛 = 3 to 5), we compute cosine similarity between each vulnerable function and its corresponding fixed version across all paired instances in the dataset. Across 4,416 function pairs, the median similarity is 0.992 (IQR: 0.967–0.999), indicating near-identical lexical structure in most cases. Notably, 89.4% of pairs exhibit similarity above 0.90 and 81.2% exceed 0.95. These results indicate that vulnerable and fixed variants often differ only by minimal edits in the analyzed pairs (e.g., constant values, log messages, or localized line-level patches). Such extreme structural overlap may introduce substantial ambiguity into the learning task, because global sequence representations can be dominated by unchanged context rather than vulnerability-indicative features, potentially limiting separability for token-based encoder models. Second, PrimeVul derives function-level binary labels from commits that fixes vulnerabilities. While this approach improves scalability and alignment with documented CVEs, it inherently relies on commit-level signals to infer fine-grained vulnerability semantics. As a result, individual functions modified in a fixing commit may include changes unrelated to the root vulnerability, incomplete fixes, or refactorings. Combined with high structural similarity between vulnerable and fixed versions and substantial class imbalance, this weak supervision setting may reduce the effective learning signal available to fine-tuned models. Taken together, these findings highlight dataset characteristics that may help explain why standard supervised fine-tuning performs unreliably on PrimeVul in our experiments. The dataset’s minimal semantic variation between vulnerable and fixed functions,

EASE ’26, June 09–12, 2026, Glasgow, United Kingdom

0.56 0.5440.542

0.54 ROC-AUC

coarse-grained function-level labeling, and reliance on commitderived annotations may weaken the effective supervision signal available to models. These characteristics make it difficult for encoder-based classifiers to learn robust and interpretable decision boundaries. We therefore advocate for improved annotation granularity, context-aware labeling, and change-level supervision to better support reliable AI-based vulnerability detection.

Norbert Sándor Szolnoki and Gábor Antal

0.52

0.5

0.523

0.498

0.5230.522

0.524

0.499

0.48 CB -b as e C lin CB B-c e - w C B fo c GC CB -ov al B- -un er ba de s GC elin r GC B e -c GCB-fo w GC B- ca l UX B- ove -b und r as er e U lin UX X-c e w U -foc UXX-o al -u ver nd er

Experimental Sufficiency. To ensure that our findings are not an artifact of insufficient tuning, we conducted an extensive and systematic hyperparameter exploration. Across all experiments, we varied pretrained model architectures, random seeds, imbalance mitigation strategies, learning rates, sequence lengths, batch sizes, calibration settings, and decision thresholds. In total, this resulted in over 770 independent training runs and corresponding evaluations, all performed using fixed dataset splits and controlled execution environments. The selected hyperparameter ranges (learning rates in the order of 10−5 , batch sizes of 8–16, sequence lengths up to 512 tokens, and 3–5 training epochs) follow the standard finetuning regime established for BERT-style transformer models [3] and widely adopted in subsequent studies on fine-tuning stability and code representation models [6, 9, 18]. Further expansion of this search space would risk overfitting to PrimeVul rather than improving generalizable performance. We therefore consider the experimental coverage sufficient to draw reliable conclusions.

0.521 0.517 0.514 0.5110.51

0.5420.54

Figure 2: Test ROC-AUC for each model and imbalance strategy, using the best hyperparameter configuration selected on validation MCC (after calibration and threshold optimisation).

5.2

Robustness and Hyperparameter Sensitivity

Overall Performance Across Models and Strategies

To assess robustness, we analyze the distribution of performance across all hyperparameter configurations and random seeds. Figure 3 shows the distribution of test MCC values for each model. For all three architectures, MCC values are narrowly concentrated around zero, with limited variance across configurations. UniXCoder exhibits a slightly higher central tendency, but the distributions remain highly overlapping. For CodeBERT, MCC remains effectively zero across all imbalance strategies and hyperparameter settings. Despite moderate F1 scores, this indicates predictions that are statistically indistinguishable from random guessing. Inspection of predicted probabilities reveals a strong concentration around 0.5, suggesting that the model fails to learn meaningful decision boundaries and instead converges toward majority-class behavior under class imbalance. Representative ROC curves further illustrate this behavior. Figures 4 and 5 compare baseline and best-performing configurations for CodeBERT and UniXCoder, respectively. Substantial overlap across the full operating range indicates limited discriminative improvement despite extensive hyperparameter tuning.

Table 1 reports the best-performing configuration for each (model, imbalance strategy) pair, selected by validation MCC. Across all evaluated settings, performance remains close to random-guessing baselines. UniXCoder achieves the highest ROC-AUC values, reaching up to 0.549, while CodeBERT and GraphCodeBERT generally remain near or below 0.52. However, these differences do not translate into meaningful improvements in MCC or balanced accuracy, which remain close to zero across all models. Figure 2 provides an overview of ROC-AUC across imbalance mitigation strategies. No strategy consistently outperforms the unmodified baseline across architectures. Observed differences are small and model-dependent, indicating a lack of systematic benefit from class weighting, oversampling, undersampling or focal loss.

We next examine whether post-hoc calibration and decision threshold optimization recover a useful signal. Figure 6 shows the reliability diagram for the best UniXCoder configuration. Temperature scaling slightly reduces expected calibration error, but predicted probabilities remain tightly concentrated around 0.5. Threshold optimization increases F1-scores on the validation split but does not improve ROC-AUC, MCC or VD-S on the test set. Low–false-positive operating points remain characterized by high false negative rates, indicating that calibration and thresholding cannot compensate for the lack of separability in the learned representations.

5

Results

We report results for 770 fine-tuning configurations spanning three pretrained models (CodeBERT, GraphCodeBERT and UniXCoder), five imbalance mitigation strategies, multiple hyperparameter settings and three random seeds (Section 4). All test results use calibrated probabilities and decision thresholds selected exclusively on the validation split. In our analysis, MCC, calibration error and VDS are treated as primary indicators of security-relevant performance, while accuracy, F1 and ROC-AUC are reported for completeness.

5.1

5.3

Calibration and Confidence Behaviour

Reassessing Dataset Quality and Fine-Tuning Practices for Vulnerability Detection on PrimeVul dataset

Model

Strategy

codebert codebert codebert codebert codebert

EASE ’26, June 09–12, 2026, Glasgow, United Kingdom

F1

MCC

AUC

Brier

ECE

VD-S FNR

Thr.

baseline class_weights focal oversample undersample

0.666 0.667 0.667 0.667 0.666

-0.004 0.000 0.000 0.000 0.000

0.511 0.510 0.514 0.517 0.521

0.250 0.250 0.250 0.250 0.250

0.010 0.009 0.008 0.008 0.014

0.990 0.992 0.994 0.992 0.992

0.170 0.010 0.010 0.010 0.157

graphcodebert graphcodebert graphcodebert graphcodebert graphcodebert

baseline class_weights focal oversample undersample

0.667 0.666 0.667 0.665 0.667

0.000 -0.005 0.000 0.009 0.017

0.498 0.523 0.499 0.523 0.522

0.250 0.250 0.250 0.250 0.250

0.002 0.019 0.004 0.014 0.016

0.997 0.993 0.996 0.994 0.992

0.170 0.310 0.010 0.167 0.130

unixcoder unixcoder unixcoder unixcoder unixcoder

baseline class_weights focal oversample undersample

0.665 0.665 0.666 0.666 0.664

0.035 0.021 0.004 0.018 0.024

0.544 0.542 0.524 0.542 0.540

0.249 0.249 0.250 0.249 0.249

0.036 0.024 0.014 0.029 0.026

0.993 0.991 0.995 0.992 0.993

0.433 0.437 0.330 0.433 0.313

Table 1: PrimeVul test performance for the best configuration of each model and imbalance strategy. Values are averaged over three seeds. F1, MCC and AUC are threshold-optimised on the validation set using calibrated probabilities. VD-S FNR is the false negative rate at the best achievable true positive rate under FPR ≤ 0.005. graphcodebert

0.1

0.1

0.05 MCC

MCC 0

0

1

0.1

0.05 MCC

0.05

ROC

unixcoder

0

True Positive Rate

codebert

0.8 0.6 0.4 0.2

Ba se UnCW de O r Fover ca l

Ba se UnCW de O r Fover ca l

Ba se UnCW de O r Fover ca l

0

Figure 3: MCC across imbalance strategies and models. Distributions are computed across all seeds and hyperparameter combinations.

Taken together, these results are consistent with models relying on shallow lexical or distributional cues rather than vulnerabilityrelevant semantic distinctions.

6 Threats to Validity 6.1 Internal Validity The training pipeline contains several design choices that may impact the measured outcomes. First, model performance reflects the behavior of the default Hugging Face fine-tuning procedure with minor modifications to the loss functions; alternative optimization schedules or regularization may lead to different results. Second, the learning rate and sequence-length grids, while covering commonly used values in prior work, are not exhaustive. It is therefore

codebert baseline codebert best(undersample)

0

0.2

0.4 0.6 False Positive Rate

0.8

1

Figure 4: ROC curves for representative runs. Substantial overlap indicates limited discriminative improvement across imbalance mitigation strategies and hyperparameter sweeps.

possible that configurations outside our search space perform better. Third, temperature scaling and threshold selection rely exclusively on the validation split; if the validation set is not representative, the calibrated results may be misaligned with deployment expectations. To mitigate these threats, we fix random seeds, apply a systematic grid search for all key hyperparameters, and store all intermediate outputs for reproducibility. However, we cannot exclude the possibility that the observed negative results are partially influenced by implementation-specific choices.

6.2

External Validity

Our conclusions are based solely on the PrimeVul dataset. While PrimeVul was designed to reduce bias and improve ecological validity, it does not fully capture the diversity of real-world development

EASE ’26, June 09–12, 2026, Glasgow, United Kingdom

Norbert Sándor Szolnoki and Gábor Antal

ROC

execution context, or usage patterns outside the function boundary. As a result, labels may not always reflect the true presence or absence of a vulnerability in isolation. Prior analyses indicate that PrimeVul contains mislabeled or ambiguous instances due to commit-level annotation heuristics, incomplete fixing patches, and functions that are syntactically distinct but semantically related across commits [4, 14]. Such label noise may disproportionately harm models relying on local patterns, leading to unstable predictions and calibration degradation. To reduce construct bias, we do not introduce additional cleaning or re-annotation that could alter dataset assumptions, and we report multiple metrics (including MCC, calibration, and risk-aware VD-S) to avoid over-interpreting single-score trends.

True Positive Rate

1 0.8 0.6 0.4 0.2 0

unixcoder baseline unixcoder best(oversample)

0

0.2

0.4 0.6 False Positive Rate

0.8

1

7

Figure 5: ROC curves for representative runs. Substantial overlap indicates limited discriminative improvement across imbalance mitigation strategies and hyperparameter sweeps. Reliability diagram

Empirical accuracy

1 0.8 0.6

Perfect calibration unixcoder uncal unixcoder calibrated

0.2 0

0

0.2

0.4 0.6 0.8 Mean predicted probability

1

Figure 6: Reliability diagram (15 equal-width bins) for representative runs. Temperature scaling improves probability calibration only marginally, consistent with residual ECE. environments. In particular, PrimeVul is dominated by C and C++ projects and focuses on function-level granularity. Our findings may not generalize to other languages (e.g., Python, JavaScript), other granularities (e.g., project-level or basic-block-level analysis) or settings where dynamic behavior, build configurations, or third-party dependencies influence vulnerability manifestation. Furthermore, our study considers only three pretrained encoder models (CodeBERT, GraphCodeBERT, UniXcoder). Transformer architectures with specialized objectives (e.g., contrastive alignment, syntax-aware, or graph-guided encoders) might exhibit different behavior.

6.3

Scope and Boundaries of the Study. This work does not claim that pretrained code language models are inherently unsuitable for vulnerability detection, nor that fine-tuning is ineffective in all settings. Rather, our findings are scoped to supervised whole-function binary classification on the PrimeVul dataset under standard fine-tuning protocols. Alternative formulations, such as change-level modeling, pairwise learning, or task-specific supervision, may yield different results. Our goal is to diagnose limitations in both the dataset and standard training pipeline, and to clarify their interplay in shaping reported outcomes.

7.1

0.4

Construct Validity

PrimeVul assigns binary labels per function, but vulnerability is ultimately a semantic property that can depend on program state,

Discussion

Implications for Dataset Design in AI for SE

Our results suggest several data-related factors that may contribute to the limitations observed in encoder-based fine-tuning for vulnerability detection. Most notably, PrimeVul’s function pairs (vulnerable and fixed versions) are often nearly identical, differing only by small lexical edits such as a modified constant or a logging statement. As quantified in Section 4.10, this extreme structural overlap limits effective separability at the function level. When vulnerable and fixed variants share nearly identical lexical structure, global sequence representations are dominated by unchanged context rather than vulnerability-indicative features. This provides one plausible structural explanation for the near-zero MCC values observed across models in Section 5. In addition, the use of commit-level labeling introduces significant annotation noise. Vulnerabilities are labeled based on changes committed together, which may include unrelated modifications or incomplete fixes. This weak supervision may misguide the model during training, especially in edge cases where vulnerable behavior is subtle or distributed across the program. Taken together, these issues highlight the need for better dataset design practices in AI for software engineering. Specifically, future datasets should: • adopt vulnerability annotation strategies that isolate and highlight the actual source of insecure behavior, • provide change-level or region-specific supervision rather than treating entire functions as binary-labeled units, • encourage structural and semantic variation between positive and negative samples to support effective representation learning.

Reassessing Dataset Quality and Fine-Tuning Practices for Vulnerability Detection on PrimeVul dataset

Datasets that reflect these principles are more likely to support robust, interpretable, and practically relevant AI models for securityfocused tasks in SE.

7.2

Implications for Evaluation Practice and Reproducibility

Beyond data quality, our findings underscore the critical importance of evaluation protocol design in model assessment. Prior work often relies on single-seed experiments, fixed decision thresholds, and performance metrics that do not reflect operational requirements. In contrast, our large-scale multi-seed study reveals that seemingly strong results are frequently sensitive to threshold tuning and fail to generalize across configurations. Specifically, we find that threshold optimization can inflate F1 scores without improving actual class separability, and that uncalibrated probability estimates yield poor reliability in low–falsepositive regimes. Metrics such as MCC, ECE, and VD-S are more indicative of security-relevant behavior and should be prioritized in future work. From a reproducibility standpoint, we recommend that future studies adopt evaluation practices that: • aggregate results across multiple seeds and configurations, • report confidence intervals or standard deviations to indicate performance stability, • include calibration- and risk-aware metrics that align with real-world constraints. Such practices are essential for producing robust and verifiable claims about model effectiveness in SE contexts.

7.3

Lessons Learned for Data-Centric AI in Software Security

Our results suggest that the observed limitations of encoder finetuning on PrimeVul are consistent with both modeling limitations and data-related challenges. Apparent improvements under commonly used metrics such as F1 or ROC-AUC do not necessarily translate into security-relevant behavior, especially when label noise, class imbalance, and weak supervision dominate the learning signal. Even with extensive hyperparameter tuning across more than 770 runs, the models consistently fail to generalize beyond trivial thresholds or baseline separability. This suggests that improvements to model architectures alone may be insufficient to address the core issue, namely that the learning task may be difficult under the current dataset and labeling formulation. We conclude that progress in learning-based vulnerability detection will require a more integrated focus on dataset design, supervision quality, and evaluation rigor. In particular, aligning data representations with the semantics of vulnerability and the operational realities of secure software development is critical to enabling reliable AI systems in SE.

8

Conclusion

This paper presented a large-scale empirical evaluation of supervised fine-tuning of pretrained code language models for functionlevel vulnerability detection on the PrimeVul benchmark. Across

EASE ’26, June 09–12, 2026, Glasgow, United Kingdom

770 configurations spanning three widely used encoder architectures, multiple imbalance mitigation strategies and extensive hyperparameter sweeps, we found no evidence that standard finetuning yields reliable or security-relevant performance improvements. While some conventional metrics suggested improvement, these gains did not persist under calibration-aware or risk-sensitive evaluation. Taken together, our results suggest that commonly reported performance improvements on PrimeVul should be interpreted with caution. In particular, strong results under single metrics or fixed thresholds may not reflect practical effectiveness in securitycritical settings. By applying rigorous evaluation protocols and deployment-oriented metrics, this study clarifies the current limitations of baseline encoder fine-tuning on realistic vulnerability benchmarks. We hope that these findings help redirect future research toward problem formulations, datasets and learning objectives that are more closely aligned with the semantic and operational characteristics of real-world software vulnerabilities. In particular, our findings highlight several dataset-level issues with high lexical and semantic similarity between fixed and vulnerable variants, noise introduced by function-level labels, and limited diversity of vulnerability patterns—that hinder effective model training. These insights underscore the need for more semantically rich, precisely annotated, and representative datasets to advance AI-based vulnerability detection in realistic SE settings.

9

Data Availability

All data and code required to reproduce the experiments reported in this study are publicly available via Zenodo [25]. The archive includes the processed PrimeVul splits used in our experiments, training and evaluation scripts, configuration files, and scripts for metric computation and result aggregation. To support reproducibility, we also provide all random seeds and hyperparameter settings used in the study.

10

Future Work

Our findings suggest multiple directions for future research. Architectural Adaptations. A natural extension is to evaluate architectures that incorporate structural priors beyond token sequences. Examples include models with explicit data-flow representations, code property graphs, or syntax-guided attention mechanisms. Several recent studies indicate that combining pretrained encoders with contrastive or graph-based objectives may mitigate noise sensitivity [11], and systematic evaluation of such hybrid approaches on PrimeVul remains open. Siamese and Pair-aware Training. As PrimeVul contains semantically linked pairs of vulnerable and fixed code instances originating from the same commit, future work could leverage this implicit pairing with Siamese or triplet networks. These approaches could directly model the relationship between vulnerable and nonvulnerable variants of the same function, potentially improving discrimination under label noise and representation drift.

EASE ’26, June 09–12, 2026, Glasgow, United Kingdom

Data Quality and Re-annotation. Our results highlight the influence of dataset noise and inconsistent annotation granularity. Future efforts could investigate semi-automated re-annotation workflows, such as LLM-assisted validation or human-in-the-loop correction, building on recent work in dataset cleaning for vulnerability detection [14]. A cleaner and more semantically coherent dataset would permit more fine-grained conclusions about architectural contributions. Robust Training Regimes. Several training strategies remain unexplored in this context. Promising directions include curriculum learning that orders samples by estimated difficulty, noise-robust loss functions, uncertainty-aware optimization or test-time adaptation for out-of-distribution robustness. Moreover, hard negative mining or active learning may help address the imbalance between benign and vulnerable instances more effectively than static resampling. Evaluation Protocols. While this work focuses on function-level classification, future research could consider multi-level evaluation across files, classes or projects, and integration with contextual signals such as build configuration or dependency graphs. Additionally, adapting risk-aware metrics to deployment-specific tolerances (e.g., alternative VD-S thresholds or cost-sensitive metrics) may provide further insight into practical applicability. Benchmark Diversity. Our conclusions are tied to PrimeVul. Extending the study to other benchmarks (e.g., BigVul, Devign, Vul4J or CleanVul) could help determine whether the observed instability is specific to PrimeVul’s properties or a broader limitation of baseline encoder fine-tuning. Cross-dataset evaluation would also clarify the transferability of calibration and imbalance strategies.

11

Acknowledgment

This work was supported in part by the European Union through the European Cybersecurity Competence Centre under Grant no. 101128082 (APPTake).

References [1] Saikat Chakraborty et al. 2021. Deep Learning based Vulnerability Detection: Are We There Yet?. In ICSE. [2] Davide Chicco and Giuseppe Jurman. 2020. The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics 21, 1 (2020), 6. doi:10.1186/s12864-019-6413-7 [3] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). 4171–4186. [4] Yangruibo Ding, Yanjun Fu, Omniyyah Ibrahim, Chawin Sitawarin, Xinyun Chen, Basel Alomair, David Wagner, Baishakhi Ray, and Yizheng Chen. 2024. Vulnerability detection with code language models: How far are we? arXiv preprint arXiv:2403.18624 (2024). [5] Tom Fawcett. 2006. An introduction to ROC analysis. Pattern Recognition Letters 27, 8 (2006), 861–874. doi:10.1016/j.patrec.2005.10.010 [6] Zhangyin Feng et al. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In NeurIPS. [7] Tilmann Gneiting and Adrian E. Raftery. 2007. Strictly proper scoring rules, prediction, and estimation. J. Amer. Statist. Assoc. 102, 477 (2007), 359–378. doi:10.1198/016214506000001437 [8] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On Calibration of Modern Neural Networks. In Proceedings of the 34th International Conference on Machine Learning (ICML), Vol. 70. PMLR, 1321–1330. http://proceedi

ngs.mlr.press/v70/guo17a.html

Norbert Sándor Szolnoki and Gábor Antal

[9] Daya Guo et al. 2021. GraphCodeBERT: Pre-training Code Representations with Data Flow. In ICLR. [10] Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. Unixcoder: Unified cross-modal pre-training for code representation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 7212–7225. [11] Chen Ji, Su Yang, Hongyu Sun, and Yuqing Zhang. 2024. Applying Contrastive Learning to Code Vulnerability Type Classification. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 11942–11952. [12] Roberto Jordaney, Zhi Wang, Davide Papini, Ilia Nouretdinov, and Lorenzo Cavallaro. 2016. Misleading metrics: On evaluating machine learning for malware with confidence. Tech. Rep. [13] B Kitchenham, David Budgen, Pearl Brereton, Mark Turner, S Charters, and S Linkman. 2007. Large-scale software engineering questions–expert opinion or empirical evidence? IET software 1, 5, 161–171. [14] Yikun Li, Ting Zhang, Ratnadira Widyasari, Yan Naing Tun, Huu Hung Nguyen, Tan Bui, Ivana Clairine Irsan, Yiran Cheng, Xiang Lan, Han Wei Ang, et al. 2024. CleanVul: Automatic Function-Level Vulnerability Detection in Code Commits Using LLM Heuristics. arXiv preprint arXiv:2411.17274 (2024). [15] Zhen Li et al. 2021. VulDeePecker: A Deep Learning based System for Vulnerability Detection. In NDSS. [16] Zhen Li, Deqing Zou, Shouhuai Xu, Hai Jin, Hanchao Qi, and Jie Hu. 2016. Vulpecker: an automated vulnerability detection system based on code similarity analysis. In Proceedings of the 32nd annual conference on computer security applications. 201–213. [17] Swaroop Mishra and Anjana Arunkumar. 2021. How Robust are Model Rankings: A Leaderboard Customization Approach for Equitable Evaluation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. AAAI Press, 13561–13569. doi:10.1609/aaai.v35i15.17599 [18] Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2020. On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines. arXiv preprint arXiv:2006.04884 (2020). [19] Mahdi Pakdaman Naeini, Gregory Cooper, and Miloš Hauskrecht. 2015. Obtaining Well Calibrated Probabilities Using Bayesian Binning. In AAAI. [20] Kollin Napier, Tanmay Bhowmik, and Shaowei Wang. 2023. An empirical study of text-based machine learning models for vulnerability detection. Empirical Software Engineering 28, 2 (2023), 38. [21] Adriana Sejfia, Satyaki Das, Saad Shafiq, and Nenad Medvidović. 2024. Toward improved deep learning-based vulnerability detection. (2024), 1–12. [22] Abubakar Omari Abdallah Semasaba, Wei Zheng, Xiaoxue Wu, and Samuel Akwasi Agyemang. 2020. Literature survey of deep learning-based vulnerability analysis on source code. IET Software 14, 6 (2020), 654–664. [23] Zihua Song, Junfeng Wang, Shengli Liu, Yu Cui, Yawen Xu, and Yanming Fang. 2022. HGVul: A Code Vulnerability Detection Method Based on Heterogeneous Source-Level Intermediate Representation. Security and Communication Networks 2022 (2022), 1–13. doi:10.1155/2022/1919907 [24] Benjamin Steenhoek, Md Mahbubur Rahman, Richard Jiles, and Wei Le. 2023. An empirical study of deep learning models for vulnerability detection. (2023), 2237–2248. [25] N. S. Szolnoki and G. Antal. 2026. Reassessing Dataset Quality and Fine-Tuning Practices for Vulnerability Detection on PrimeVul dataset (dataset and script). doi:10

.5281/zenodo.18154077 [26] Xin-Cheng Wen, Yijun Yang, Cuiyun Gao, Yang Xiao, and Deheng Ye. 2025. Boosting Vulnerability Detection of LLMs via Curriculum Preference Optimization with Synthetic Reasoning Data. arXiv preprint arXiv:2506.07390. [27] Jingxiu Yao and Martin Shepperd. 2021. The impact of using biased performance metrics on software defect prediction research. Information and Software Technology 139 (2021), 106664.

Related documents

• VICBench: A Multi-Language Benchmark for Code Vulnerability Detection#447872
• RAG-PIBench: A Leakage-Aware Benchmark for Prompt-Injection Detection in Trustworthy RAG Systems#1218306
• RAG-PIBench: A Leakage-Aware Benchmark for Prompt-Injection Detection in Trustworthy RAG Systems#1210189
• TrainSDC: Characterizing and Mitigating Silent Data Corruption in Large Language Model Training#622547
• Dataset Watermarking with Provable Black-Box Detection#1130546