A comparative analysis of automated techniques for security bug report identification Muhammad Laiq
arXiv:2607.27893v1 [cs.SE] 30 Jul 2026
Blekinge Institute of Technology, Karlskrona, Sweden [email protected]
Abstract—Timely identification of security-related bug reports is essential to minimize the window of vulnerabilities in software systems. Manually screening incoming bug reports to identify security-related issues is time-consuming, error-prone, and non-scalable for large-scale software systems. Thus, a variety of automatic techniques, including traditional machine learning (ML) techniques and large language models, have been proposed to facilitate this task. However, the literature remains fragmented. Most studies introduce or optimize a particular technique and evaluate it against a limited set of baselines, often under different experimental setups. As a result, it is difficult to compare their results and draw reliable conclusions about the effectiveness of existing approaches, leaving researchers and practitioners without clear guidance on which techniques are most suitable for the task. To address this gap, we conducted a comparative analysis of several promising automated techniques to identify securityrelated bug reports using benchmark datasets. We evaluated Logistic Regression, Support Vector Machines, Random Forest, OpenAI’s GPT-5.2, BERT-base, RoBERTa, and SetFit (a state-of-the-art few-shot learning framework). Our results indicate that SetFit achieves the best overall performance, achieving an F1-score of 0.80 and outperforming other techniques on three of the four datasets. RoBERTa performs competitively and approaches SetFit in some projects, while traditional ML techniques, particularly Logistic Regression, remain a strong baseline in certain contexts. In contrast, GPT5.2 performs poorly in both zero-shot and few-shot settings. In addition, cross-project experiments demonstrate that transfer learning can improve performance for projects with limited data, but may degrade results for projects with strong projectspecific characteristics. Keywords–Few-shot learning; Issue classification; Bug report classification; Security bug reports 1. I NTRODUCTION Software organizations rely on bug tracking systems, such as Bugzilla1 and Jira2 , to manage bug reports throughout the software development lifecycle. These systems receive bug reports from various stakeholders, including developers, testers, and end-users, and typically encompass a wide range of quality concerns such as performance degradation, functional failures, stability issues, and security vulnerabilities. Among these categories, security-related bug reports are particularly 1 https://www.bugzilla.org/ 2 https://www.atlassian.com/software/jira
critical. In contrast to non-security issues, security vulnerabilities have the potential to expose systems to unauthorized access, data breaches, or service disruption. Consequently, the timely and accurate identification of security-related bug reports is essential to prevent exploitation and mitigate risk in software ecosystems. Manually inspecting large volumes of incoming bug reports to identify potential security issues is a laborious and errorprone process [1]. Contemporary software projects, especially those that are community-driven or large-scale, may receive hundreds or even thousands of bug submissions, making the manual process increasingly infeasible. Thus, to assist practitioners in identifying security-related bug reports, several automated techniques have been proposed in the literature, for example, [2], [3], [4], [5], [6]. Most of these approaches rely on traditional machine learning (ML) techniques, such as Logistic Regression and Random Forest, to classify bug reports into security-related and non-security-related bug reports based on textual features. More recently, researchers have begun exploring the use of more advanced large language models (LLMs), which offer enhanced capabilities for understanding contextual and semantic information in natural language [7], [6], [8]. Despite this growing body of work and the increasing diversity of proposed techniques, the literature remains fragmented and lacks a systematic comparison of existing automated techniques for identifying security-related bug reports. Prior studies typically focus on proposing or refining individual techniques and evaluating them against a limited set of baselines, often under differing datasets and experimental setups. This makes it difficult to compare the results of these studies and to draw reliable conclusions about the effectiveness of these techniques, leaving researchers unable to determine which methodological advances are truly effective and practitioners without clear guidance on which techniques are most suitable for adoption in practice. To address the aforementioned gap, this study conducted a comparative analysis of several promising automated techniques to identify security-related bug reports using benchmark datasets (Ambari, Camel, Wicket, and Derby). We systematically evaluated traditional ML techniques (Logistic Regression, Support Vector Machines, and Random Forest), BERTbased LLMs (BERT-base and RoBERTa), a state-of-the-art few-shot learning framework (SetFit), and a GPT-based LLM (GPT-5.2). The contributions of this study are as follows. • We summarize existing work on automatic identification of security-related bug reports. • We comprehensively evaluate several promising automatic
techniques under a consistent experimental setup on benchmark datasets. • We investigate transfer learning through cross-project prediction to assess when training on other projects helps or hurts performance. • We investigate the impact of the commonly used data imbalance handling technique (i.e., Synthetic Minority Oversampling Technique [9]) on the prediction accuracy of classical ML techniques. • We explore and compare prompt engineering strategies for the GPT-based model, specifically zero-shot, few-shot with two examples, and few-shot using 1% of the training data. • We evaluate RoBERTa (an additional) technique for identifying security bug reports and compare it with existing approaches. This paper is organized as follows. Section 2 describes the related work on automatic identification of security-related bug reports. Section 3 presents the research method of the study. Section 4 presents the results of the study. Section 5 discusses the findings of the study. Section 6 describes the validity threats to the study and, finally, Section 7 concludes the paper.
TABLE I: An overview of the techniques, features, evaluation strategies, and evaluation metrics used in previous work Techniques
Studies
Random Forest
[8], [6], [10], [11], [7], [5], [2], [12], [3] [8], [6], [5], [2], [12], [3] [8], [6], [13], [12], [3] [5], [2], [12], [3] [5], [2], [12], [3] [5], [2], [12] [10], [11], [7] [14] [8] [11] [3]
5 4 4 3 3 1 1 1 1
[7] [6]
1 1
Logistic Regression Support Vector Machines K-Nearest Neighbor Naive Bayes Multilayer Perceptron BERT-base Fasttext SetFit CNN and BiLSTM AdaBoost, Gradient boosting, and Stacking-based Ensemble ChatGPT* GPT-based models: GPT4AllFalcon, Instruct, Open-Orca (Mistral), and Wizard
Count 9 6
Features
Studies
Description of a bug report
[8], [6], [10], [11], [7], [14], [5], [2], [12], [3] [13], [10], [14], [5], [2], [12] [8], [11], [3] [13], [12], [3]
10
[3]
1
2. R ELATED WORK
Summary of a bug report
Bug reports can be classified according to different criteria, such as validity, priority, or whether they represent securityrelated or non-security-related issues. Consequently, several studies have explored different aspects of bug report classification [15]. For example, previous work has examined classification based on report validity [16], [17] and on differentiating among tasks such as feature requests, questions, and documentation updates [18], [19]. Similarly, other studies have focused on grouping or clustering bug reports to identify shared underlying causes [20], [21]. In this study, we focus specifically on classifying bug reports to distinguish between security-related and non-security-related reports. Table I presents an overview of the identified related work on classifying bug reports as security-related or non-securityrelated. The studies are grouped by the techniques employed, the types of features used to train a model, and the evaluation strategies and metrics used. In terms of techniques, as shown in Table I, classical ML techniques are predominant. Random Forest, Logistic Regression, Support Vector Machines, and K-Nearest Neighbor are the most commonly applied techniques in these studies. More recent work has expanded the methodological landscape by incorporating deep learning and LLMs, including models based on BERT [11], [5], [2], [12], models based on GPT [7], [6], and few-shot learning approaches such as SetFit [8]. In these studies, various features, including the title, summary, and description of a bug report, have been utilized to train models to classify bug reports to identify security-related bug reports. Most studies only use the summary and description of the bug report. Only a few studies have used additional
Title/heading of a bug report Meta features, such as reported time, priority, and creation time. Commit messages and comments Evaluation strategies
Studies
Test on a fixed set 10-fold cross-validation 3-fold cross-validation 10 repetitions 5-fold cross-validation 10-fold cross-validation TemplateStickiness
[13], [10], [11], [5], [2], [12] [14] [7] [6] [8] [3] [6]
Evaluation metrics
Studies
Precision
[8], [6], [13], [10], [11], [7], [14], [5], [2], [3] [8], [6], [13], [10], [11], [7], [14], [5], [12], [3] [8], [6], [13], [10], [11], [7], [14], [2], [12] [10], [14], [5], [2], [12] [6], [13], [11], [7] [10], [5], [12] [8]
Recall F – Score G-measure Accuracy False Positive Rate AUC and MCC
Count
6 3 3
Count 6 1 1 1 1 1 1 Count 10 10 9 5 4 3 1
* ChatGPT model was not reported in the study.
features such as report timestamps, priority levels, or commit
messages. The evaluation approaches in these studies are highly heterogeneous. As summarized in Table I, most studies rely on fixed test sets, although several adopt cross-validation or repeated trials. F1-score, Precision, and Recall are the primary evaluation metrics, while other complementary metrics (e.g., AUC, MCC, or G-measure) are less frequently used. The diversity of features, evaluation metrics, and evaluation strategies employed in these studies makes it challenging to compare their findings. In addition, the literature remains fragmented: most studies introduce or refine a particular technique and evaluate it against a limited set of baselines. For example, recent work by Laiq [8] introduced a SetFitbased few-shot learning approach to identify security bug reports. However, it only compares SetFit with classical ML techniques. Similarly, the study by Francca et al. [6] compares classical ML techniques with GPT-based models. These two studies do not compare their work with BERT-based models. Similarly, studies that use BERT-based models [10], [11], [7] do not compare their work with SetFit and GPT-based models. Additionally, the experimental settings (e.g., evaluation strategy and metrics) used in these studies are not directly comparable. These inconsistencies underscore the need for a comprehensive and unified evaluation of existing techniques to facilitate more reliable comparisons and informed adoption decisions. Motivated by the aforementioned gap, this study systematically and comprehensively evaluates a wide range of techniques for automatically identifying security-related bug reports. We evaluated traditional ML techniques, BERT-based models, OpenAI’s GPT-5.2, and SetFit (a state-of-the-art few-shot learning framework) on benchmark datasets (Ambari, Camel, Wicket, and Derby). 3. R ESEARCH METHOD This study aims to comprehensively evaluate the effectiveness of several promising automatic techniques in identifying security bug reports. To achieve this goal, we posed the following research question: How do automatic techniques perform to identify security bug reports? To answer this research question, we perform comparative experiments [22], [23]. In the following subsections, we describe the key decisions made in designing the experiments, including the selection of automatic techniques, the choice of datasets, the evaluation approach, and the selection of evaluation metrics.
In addition to classical ML techniques, recent work (see Table I) has explored the use of LLMs to identify security bug reports. We therefore included the BERT-base and OpenAI’s GPT-5.2 models. Beyond existing approaches, this study uses an additional technique (i.e., RoBERTa). Although RoBERTa [24] has achieved state-of-the-art performance in classifying bug reports [15] into bug and non-bug issues, it has not yet been applied to identify security-related bug reports. Therefore, we include it to assess its performance in this task. Finally, motivated by the limited availability of labeled security bug reports, we incorporate SetFit [25], a state-of-the-art few-shot learning framework that has previously been studied in isolation (without comparison with other techniques for this task). Together, these techniques enable a unified evaluation across classical ML techniques, LLMs, and the SetFit-based few-shot learning approach, addressing the fragmentation observed in prior work. 3.2 Selection of datasets To evaluate the selected techniques, we utilized four benchmark datasets widely adopted in prior research on security bug report identification [14], [7], [6]: Ambari, Camel, Derby, and Wicket. These datasets have been manually labeled Wu et al. [1]. The validity threats associated with the chosen datasets are discussed in detail in Section 6. An overview of the datasets is provided in Table II. All four datasets are derived from open-source software projects and were originally compiled and made available by Wu et al. [1]. Ambari is a Hadoop management platform, Camel is an integration framework, Derby is a relational database management system, and Wicket is a Java-based web application framework. Collectively, these datasets span diverse application domains, helping mitigate project-specific bias and supporting a more generalizable evaluation of the selected techniques. TABLE II: Evaluated datasets of security bug reports (BRs), manually labeled by Wu et al. [1] Project
Security BRs
Non-Security BRs
Total BRs
Camel Ambari Derby Wicket
74 (7.4%) 56 (5.6%) 179 (17.9%) 47 (4.7%)
926 944 821 953
1000 1000 1000 1000
3.3 Evaluation approach 3.1 Selection of techniques As shown in Table I, the majority of existing studies on identifying security bug reports are based on classical (traditional) ML techniques. Similar to previous work (e.g., [6], [8], [5]), we selected the following three classical ML techniques as baselines: Support Vector Machines, Logistic Regression, and Random Forest. These techniques have been widely used for identifying security bug reports (See Table I).
We use a five-fold cross-validation strategy to evaluate the selected techniques. Compared to evaluation in a fixed train–test split, cross-validation reduces potential sampling bias and provides a more reliable estimate of model performance [26]. To perform the five-fold cross-validation, we split our datasets into five approximately equal folds. In each iteration, we trained the models using four of these folds and evaluated them on the remaining fold. This procedure was repeated five times to ensure that different subsets were used for testing in
each cycle. Ultimately, we averaged the results of all five folds for each technique evaluated. Evaluation approach for the GPT-based model: For GPT-5.2, we use the following approach. (a) Zero-shot setting: In this setting, we use only the test sets for prediction. Table III presents a zero-shot prompt template in which the model is instructed to assume the role of a software security expert and to assign a given bug report to one of two predefined categories: security bug or non-security bug. The template provides only task instructions, category definitions, and the target bug report, without including any labeled examples. In addition, the prompt explicitly restricts the model’s output by prohibiting explanations or extraneous text, thus enforcing a concise single-label response. This design assesses the model’s ability to perform the classification task solely based on prior knowledge encoded during pretraining, without requiring task-specific demonstrations. (b) Few-shot setting with 2 examples: In this setting, we include two labeled examples from the training folds as in-context examples, and then evaluate performance on the corresponding test fold. Table IV presents a quick snippet template that enhances the same classification task by using labeled examples derived from the dataset. Before presenting the target bug report, the prompt includes one non-security bug report labeled 0 and one security bug report labeled 1. These examples serve as in-context demonstrations that illustrate both the expected input format and the desired output labels. By providing explicit mappings between bug report descriptions and their corresponding classes, the few-shot template guides the model toward the intended decision boundary and output structure. The final segment of the prompt then requests the classification of the target bug report, leveraging the contextual information provided by the preceding examples. (c) Few-shot setting with 1% training data: In this setting, we include 1% (8 examples) of data from the training folds as in-context examples, and then evaluate performance on the corresponding test fold. The same template shown in Table IV was used for the few-shot setting with 1% training data; instead of 2 examples, we provided 8 examples for the model. The GPT model was used with the above-mentioned setting, primarily following the recent work by Francca et al. [6]. Although their study focuses on a zero-shot setting, we additionally explored the few-shot setup. Due to the cost associated with these experiments, the number of examples was limited to a maximum of eight. This constraint may affect comparisons with other techniques in this study, which leverage the entire training dataset before testing. However, we consider this a minor limitation, as GPT models are inherently capable of interpreting and generating bug reports and can function as general-purpose assistants for tasks such as text summarization and feature extraction.
TABLE III: Zero-shot prompt template for identifying security-related bug reports, adapted from [7] #
Prompt content
1
# Task: You are a software security expert. Classify the given software bug report into exactly one of the two categories below. Categories: (a) security bug (b) non-security bug Do not provide any explanation or additional text.
2 3 4 5 6 7
# Description of software bug report: (Target bug report) # Answer:
TABLE IV: Few-shot prompt template for identifying securityrelated bug reports, adapted from [7] #
Prompt content
1
# Task: You are a software security expert. Classify the given software bug report into exactly one of the two categories below. Categories: * security bug: 1 * non-security bug: 0
2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
# Description of software bug report: (Non-security bug report from the dataset) # Answer: 0 # Description of software bug report: (Security bug report from the dataset) # Answer: 1 # Description of software bug report: (Target bug report) # Answer:
3.4 Performance evaluation metrics Similar to the existing work (see Table I), this study measures Precision, Recall, and F1-score to evaluate the performance of the selected techniques. Recall is calculated as the proportion of correctly identified positive instances among all actual positive instances. Precision is calculated by measuring the proportion of correctly predicted positive instances among all predicted positives. F-score is calculated by computing the harmonic mean of Precision and Recall. Recall =
TP TP + FN
(1)
TP TP + FP
(2)
P recision =
P recision ∗ Recall F − score = 2 ∗ P recision + Recall
(3)
3.5 Data preprocessing Similar to previous work (see Table I), we utilize bug report descriptions to train our models. We implement a standard data preprocessing approach for these descriptions, which includes removing special characters, numbers, and hyperlinks. For our classical ML techniques (Logistic Regression, Random Forest, and Support Vector Machines), we utilized Term Frequency-Inverse Document Frequency (TF-IDF) to convert textual data into sparse matrices. For the SetFit model, we employed the all-mpnet-base-v2, which is one of the top-performing pre-trained models for generating embeddings [27]. For the RoBERTa and BERT-base models, we used the RoBERTa tokenizer and the BERT-base tokenizer from Hugging Face3 , respectively. 3.6 Implementation details We used the scikit-learn4 library to implement the selected classical ML techniques. To identify the optimal parameters for these techniques, we performed a grid search using scikitlearn. Initially, we fine-tuned the parameters of each chosen technique through this grid search. After determining the best parameters, we built a model using them. For implementing SetFit 5 , RoBERTa, and BERT-base, we follow the official documentation from Hugging Face. To use OpenAI’s GPT-5.2 model, we used the OpenAI Python API6 . To perform our experiments, we used a machine equipped with multiple Nvidia H200 NVL GPUs and a couple of AMD EPYC 9565 CPUs, each with cores exceeding 70. The code and data associated with this work are available at the following links. (1) Dataset provided by Soltaniani et al. [10]: https://zenodo.org/records/15240583. (2) Code for this study: https://figshare.com/s/dfedf56689f638c7b35e. 4. R ESULTS AND ANALYSIS To answer the posed research question (i.e., How do automatic techniques perform to identify security bug reports?), we evaluated the performance of the selected techniques, Logistic Regression, Support Vector Machines, Random Forest, RoBERTa, BERT-base, OpenAI’s GPT-5.2 (in zero-shot and few-shot settings), and SetFit. These techniques are assessed using four datasets of security bug reports (Camel, Ambari, Derby, and Wicket, see Table II). In the following, we present the evaluation results of these techniques. Table V and Figure 1 summarize the performance of all the techniques evaluated. Overall, SetFit achieves the best performance across datasets, except Ambari, particularly in terms of F-score, Recall, Precision, and Accuracy. 3 https://huggingface.co/ 4 https://scikit-learn.org/ 5 https://huggingface.co/docs/setfit/en/index 6 https://pypi.org/project/openai/
Camel project: For the Camel dataset, SetFit clearly outperforms all other techniques, achieving the highest F-score (0.6776), Recall (0.5562), Precision (0.9118), and Accuracy (0.9629). Among the other baselines, RoBERTa is the strongest performer (F-score = 0.5881), followed by Logistic Regression (F-score = 0.5527). Other traditional techniques, Random Forest and Support Vector Machines, perform poorly. GPT-5.2 (both in zero-shot and few-shot settings) shows limited effectiveness, with low F-scores below 0.20, indicating that zero-shot and few-shot prompting are insufficient for accurately identifying security bug reports in this dataset. Overall, SetFit demonstrates a strong balance between Precision and Recall. Ambari project: The Ambari dataset is particularly challenging for most techniques. Logistic Regression achieves the highest F-score (0.4651) and Recall (0.4818), outperforming all other techniques in this dataset. GPT-5.2 (zero-shot and few-shot) and Support Vector Machines obtain moderate Fscores around 0.32, while BERT-base, RoBERTa, and SetFit perform less effectively in terms of F-score. In particular, Random Forest completely fails to identify any security bug reports. These results confirm the difficulty of identifying security bug reports in the Ambari project, as also observed in prior work [6]. Derby project: In the Derby dataset, SetFit achieves the best overall performance, with the highest F-score (0.7816), Precision (0.8081), Recall (0.7657), and Accuracy (0.9240). RoBERTa performs competitively, obtaining a similar F-score (0.7802) and slightly higher Recall (0.7989), but with lower Precision. Traditional ML techniques, Logistic Regression, Support Vector Machines, and Random Forest, show reasonable performance but remain clearly below SetFit and RoBERTa. GPT-5.2 again underperforms, particularly in Recall, indicating difficulty in consistently identifying securityrelated reports without task-specific fine-tuning. Wicket project: The Wicket dataset further highlights the effectiveness of SetFit, which achieves the highest F-score (0.7959), Recall (0.7133), Precision (0.9381), and Accuracy (0.9850). Logistic Regression and RoBERTa perform similarly in terms of F-score (approximately 0.60), but with noticeably lower Recall. Random Forest exhibits high Precision but extremely low Recall, limiting its usefulness in practice. The variants of GPT-5.2 (zero-shot and few-shot) perform poorly, with F-scores below 0.15, indicating that prompting alone is insufficient for this dataset. SetFit’s strong Recall while maintaining very high Precision is particularly important in security contexts, where missing relevant bug reports can be costly. 4.1 Generalizability in terms of average F1-score and standard deviation across datasets Table VI shows the average F1-score and its standard deviation for all techniques in the evaluated datasets. In general, SetFit achieves the best average F1-score (0.6223), indicating the strongest overall performance, though it also exhibits the highest variability (STD = 0.2288). The highest variability
TABLE V: Performance of Logistic Regression, Random Forest, Support Vector Machines, RoBERTa, BERT-base, GPT-5.2, and SetFit for identifying security bug reports Dataset
Camel
Ambari
Derby
Wicket
Technique
F-Score
Recall
Precision
Accuracy
Logistic Regression Support Vector Machines Random Forest BERT-base RoBERTa GPT-5.2 (Zero-shot) GPT-5.2 (Few-shot, 2 examples) GPT-5.2 (Few-shot, 1%) SetFit
0.5527 0.3433 0.3217 0.3416 0.5881 0.1793 0.1318 0.1828 0.6776
0.4857 0.4848 0.2286 0.2695 0.5400 0.1352 0.0933 0.1219 0.5562
0.6740 0.2915 0.7306 0.6140 0.6581 0.2683 0.2467 0.4143 0.9118
0.9410 0.8620 0.9350 0.9270 0.9440 0.9130 0.9160 0.9240 0.9629
Logistic Regression Support Vector Machines Random Forest BERT-base RoBERTa GPT-5.2 (Zero-shot) GPT-5.2 (Few-shot, 2 examples) GPT-5.2 (Few-shot, 1%) SetFit
0.4651 0.3289 0.0000 0.2047 0.2489 0.3227 0.3202 0.3438 0.2340
0.4818 0.3773 0.0000 0.1621 0.1939 0.2864 0.3364 0.3197 0.1789
0.4840 0.2995 0.0000 0.3133 0.3476 0.3782 0.3195 0.3788 0.3936
0.9350 0.9180 0.9440 0.9360 0.9370 0.9340 0.9210 0.9330 0.9380
Logistic Regression Support Vector Machines Random Forest BERT-base RoBERTa GPT-5.2 (Zero-shot) GPT-5.2 (Few-shot, 2 examples) GPT-5.2 (Few-shot, 1%) SetFit
0.6850 0.5538 0.5363 0.6793 0.7802 0.2674 0.3535 0.3106 0.7816
0.6195 0.5019 0.4190 0.6652 0.7989 0.1733 0.2459 0.2122 0.7657
0.7707 0.6712 0.7531 0.7099 0.7652 0.6136 0.6417 0.5968 0.8081
0.8990 0.8580 0.8710 0.8910 0.9180 0.8360 0.8430 0.8350 0.9240
Logistic Regression Support Vector Machines Random Forest BERT-base RoBERTa GPT-5.2 (Zero-shot) GPT-5.2 (Few-shot, 2 examples) GPT-5.2 (Few-shot, 1%) SetFit
0.5996 0.3048 0.3508 0.3479 0.5997 0.1329 0.0472 0.1358 0.7959
0.5311 0.3600 0.2311 0.2978 0.4956 0.1089 0.04222 0.1089 0.7133
0.7057 0.2787 0.8333 0.4444 0.7759 0.1786 0.05357 0.1915 0.9381
0.9660 0.9270 0.9610 0.9500 0.9670 0.9340 0.9350 0.9340 0.9850
in SetFit is primarily due to its poor performance for one of the four datasets (i.e., Ambari). Logistic Regression shows stable performance with low variability (Avg = 0.5756, STD = 0.0795), while RoBERTa achieves competitive results with higher variability. GPT-5.2 configurations yield lower average F1-scores overall, with the few-shot (1%) setting, based on 8 examples, slightly outperforming the zero-shot and 2-example setups while maintaining relatively low variance. We note that although these results are based on 5-fold crossvalidation, we do not claim statistical significance for the observed differences. In particular, performance differences between top-performing techniques may not be statistically meaningful and should therefore be interpreted with caution.
TABLE VI: Average (Avg) and standard deviation (STD) of F1-score across datasets Technique
Avg F1
Std F1
Logistic Regression Support Vector Machines Random Forest BERT-base RoBERTa GPT-5.2 (Zero-shot) GPT-5.2 (Few-shot, 2 examples) GPT-5.2 (Few-shot, 1%) SetFit
0.5756 0.3827 0.3022 0.3934 0.5542 0.2256 0.2132 0.2433 0.6223
0.0795 0.0997 0.1930 0.1746 0.1920 0.0740 0.1279 0.0864 0.2288
4.2 Impact of cross-project prediction We also conducted cross-project prediction experiments using SetFit and RoBERTa to assess the impact of transfer learning on the identification of security bug reports. We apply a leave-one-dataset-out strategy: we train a classifier on the full
datasets of three projects and test it on the remaining fourth project. This procedure is repeated with each project serving as the target in turn.
Derby
Wicket
0.8
0.8
0.6
0.6
0.4 0.2 0.0
LR
SVM
RF
se
-ba ERT
Derby
Wicket
0.4
B
Ambari
0.0
) ) a t) Fit ERT ot-2 ot-1% sho Set h RoB (Zero- ew-sh s (F ew -5.2 5.2 -5.2 (F GPT GPTT GP Technique Project Camel Derby Wicket
0.8
0.8
0.6
0.6
0.4 0.2
LR
SVM
LR
SVM
1.0
Accuracy
Precision
Project Camel
0.2
1.0
0.0
Ambari
1.0
Recall
F-Score
Project Camel
Ambari
1.0
Fit %) ot) RTa ase t-2) Set T-b RoBE ero-sh -sho shot-1 w BER Z ( (Fe ew -5.2 5.2 -5.2 (F GPT GPTT GP Technique Project Ambari Camel Derby Wicket RF
0.4 0.2
LR
SVM
RF
se
B
-ba ERT
) ) a t) ERT ot-2 ot-1% sho h RoB (Zero- ew-sh s (F ew -5.2 5.2 -5.2 (F GPT GPTT GP Technique
Fit Set
0.0
RF
%) ot) RTa ase t-2) T-b RoBE ero-sh -sho shot-1 w BER Z ( (Fe ew -5.2 5.2 -5.2 (F GPT GPTT GP Technique
Fit
Set
Figure 1: An overview of the evaluation results. Logistic Regression (LR), Random Forest (RF), Support Vector Machines (SVM), GPT-5.2 (Few-shot-2): GPT-5.2 with 2 examples and GPT-5.2 (Few-shot-1%): GPT-5.2 with 1% of the training data TABLE VII: Cross-project prediction using a leave-one-dataset-out setup: training on three datasets and testing on the remaining dataset Dataset Camel
Ambari
Derby
Wicket
Approach
Technique
F-Score
Recall
Precision
Accuracy
Default Cross-prediction
RoBERTa RoBERTa
0.5881 0.6917
0.5400 0.6216
0.6581 0.7797
0.9440 0.9590
Default Cross-prediction
SetFit SetFit
0.6776 0.6984
0.5562 0.5946
0.9118 0.8462
0.9629 0.9620
Default Cross-prediction
RoBERTa RoBERTa
0.2489 0.4071
0.1939 0.4107
0.3476 0.4035
0.9370 0.9330
Default Cross-prediction
SetFit SetFit
0.234 0.4248
0.1789 0.4286
0.3936 0.4211
0.9380 0.9350
Default Cross-prediction
RoBERTa RoBERTa
0.7802 0.7349
0.7989 0.6816
0.7652 0.7974
0.9180 0.9120
Default Cross-prediction
SetFit SetFit
0.7816 0.6232
0.7657 0.4804
0.8081 0.8866
0.9240 0.8960
Default Cross-prediction
RoBERTa RoBERTa
0.5997 0.5902
0.4956 0.7660
0.7759 0.4800
0.9670 0.9500
Default Cross-prediction
SetFit SetFit
0.7959 0.6372
0.7133 0.7660
0.9381 0.5455
0.9850 0.9590
Table VII shows the results of the cross-project prediction experiments. Generally, the impact of the prediction between
projects varies between datasets and models. For Camel, cross-project training improves performance for
both approaches. RoBERTa exhibits a notable increase in the F-score from 0.5881 to 0.6917, along with improvements in recall, precision, and accuracy. SetFit also benefits from cross-project prediction, achieving a higher F-score (0.6984 vs. 0.6776) and recall, while maintaining comparably high precision and accuracy. For Ambari, cross-project prediction yields the most significant gains. Both models show substantial improvements in F-score, increasing from 0.2489 to 0.4071 for RoBERTa and from 0.2340 to 0.4248 for SetFit. These improvements are primarily driven by significant increases in recall, indicating that leveraging data from other projects helps mitigate the limited availability of security bug reports in Ambari. In contrast, cross-project prediction negatively impacts performance in Derby. Although precision increases slightly for both models, recall drops considerably, resulting in lower F-scores compared to the default within-project setting. This indicates that the characteristics of security bug reports in Derby may be more project-specific and less transferable. A similar pattern is observed for Wicket, where cross-project prediction leads to reduced F-scores for both RoBERTa and SetFit. Although recall improves, particularly for RoBERTa, the accompanying decrease in precision leads to overall performance degradation. This suggests that cross-project training introduces more false positives for Wicket. 4.3 Impact of class rebalancing on prediction accuracy The datasets used in this study are highly imbalanced (see details in Table II). To investigate whether the performance of classical ML techniques can be improved by addressing class imbalance, we applied the widely used Synthetic Minority Over-sampling Technique (SMOTE) [9]. SMOTE was applied after transforming the textual data of bug reports using TFIDF. Following prior work [28], we tuned the (k) parameter of SMOTE and applied the technique only to the training data. Table VIII presents the results for classical ML techniques under two settings: (i) the Default approach without class rebalancing, and (ii) the Rebalanced approach using SMOTE. Overall, the results show that SMOTE does not consistently improve performance across datasets or techniques. For the Camel dataset, SMOTE slightly improves Logistic Regression (F-score increases from 0.4368 to 0.4634), but degrades performance for both SVM and Random Forest. In Ambari, results remain largely poor across all techniques, with SMOTE leading to no improvement for Logistic Regression and SVM, and only a small gain for Random Forest (from 0.0000 to 0.0851). In the Derby data set, SMOTE provides only a marginal improvement for SVM (from 0.4300 to 0.4339), while reducing performance for Logistic Regression and Random Forest. Similarly, for Wicket, SMOTE improves Random Forest (from 0.2951 to 0.3636), but decreases performance for Logistic Regression and SVM. Overall, these findings indicate that SMOTE has a limited and inconsistent impact on improving F-score performance for identifying security bug reports. These results may be
explained by SMOTE’s reliance on nearest-neighbor interpolation, which can be less effective in high-dimensional feature spaces, such as TF-IDF representations of text. We also note that the observed differences between the Default and Rebalanced approaches are relatively small and inconsistent; therefore, we do not claim statistical significance for these differences. TABLE VIII: Data imbalance handling with SMOTE Dataset
Camel
Ambari
Derby
Wicket
Approach
Technique
F-Score
Default Rebalanced
Logistic Regression Logistic Regression
0.4368 0.4634
Default Rebalanced
SVM SVM
0.2500 0.1935
Default Rebalanced
Random Forest Random Forest
0.2718 0.2626
Default Rebalanced
Logistic Regression Logistic Regression
0.1429 0.0000
Default Rebalanced
SVM SVM
0.0000 0.0000
Default Rebalanced
Random Forest Random Forest
0.0000 0.0851
Default Rebalanced
Logistic Regression Logistic Regression
0.6280 0.5980
Default Rebalanced
SVM SVM
0.4300 0.4339
Default Rebalanced
Random Forest Random Forest
0.4688 0.4000
Default Rebalanced
Logistic Regression Logistic Regression
0.3562 0.3385
Default Rebalanced
SVM SVM
0.2308 0.1667
Default Rebalanced
Random Forest Random Forest
0.2951 0.3636
5. D ISCUSSION This study aimed to address the lack of consolidated empirical evidence in prior research on automatically identifying security bug reports by comparing several promising techniques across four benchmark datasets. We evaluated classical ML techniques (Logistic Regression, Support Vector Machines, Random Forest), Bert-based models (RoBERTa and BERTbase), OpenAI’s GPT-5.2, and a state-of-the-art few-shot learning framework (SetFit). The results provide insights into how different techniques behave under realistic constraints, such as limited labeled data, severe class imbalance, and crossproject distribution shift. In particular, the findings clarify which approaches are robust when security reports are scarce, when transfer learning is beneficial, and which methodological trade-offs matter most for practical bug triage scenarios. In evaluations across various projects, few-shot learning with SetFit has proven to be the most effective approach, achieving the highest F1 Scores on three of the four datasets. It demonstrates a favorable balance between precision and recall, which
is especially important in security contexts where overlooking vulnerabilities can be costly, while excessive false positives can overwhelm reviewers. RoBERTa generally performs well and surpasses BERT-base, confirming that stronger pretraining objectives lead to better representations for this task. Meanwhile, classical Logistic Regression remains a surprisingly strong baseline, achieving the best performance on the Ambari dataset. This highlights that simpler models can still be effective, even in situations with extreme class imbalance and limited data. In contrast, the prompt-based OpenAI’s GPT-5.2 model performs poorly in both zero-shot and fewshot scenarios, suggesting that prompting is insufficient to capture the nuanced decision boundaries necessary for reliably identifying security issues. The results of the cross-project prediction reveal that transfer learning is highly dependent on the dataset. Cross-project training improves performance on Camel and Ambari, particularly for Ambari, where the minority class is extremely small, suggesting that exposure to additional security examples can compensate for data scarcity. However, the same strategy degrades performance in Derby and Wicket, primarily due to recall losses or sharp drops in precision. These patterns may indicate significant changes in the distribution between projects, particularly in terms of vocabulary, reporting style, and the composition of security issues. Consequently, “more data” is not inherently beneficial unless it is well aligned with the target project, underscoring the need for careful adaptation and calibration when deploying cross-project models. From a practical perspective, the results suggest that model selection should be guided by operational priorities. SetFit’s high precision makes it well-suited for triage assistance scenarios where the goal is to flag high-confidence security reports for expedited review, while RoBERTa may be preferable in settings that prioritize recall and can tolerate more false positives. Logistic Regression offers a lightweight, interpretable, and cost-effective option, particularly attractive for organizations with limited computational resources. Importantly, all models are better viewed as decision-support tools rather than fully automated gatekeepers, given the asymmetric costs of errors in security triage. 6. T HREATS TO VALIDITY Generalizability of the results: To improve the generalizability of our findings, we used multiple datasets (i.e., Camel, Ambari, Derby, and Wicket; see Section 3-B for details). Although these datasets originate from different software projects, they may not fully represent all software development contexts, particularly those that involve proprietary systems. Validity of the evaluation approach: Various evaluation strategies, such as cross-validation and fixed train–test splits, can be adopted to assess the performance of ML techniques. To mitigate potential evaluation bias and improve the reliability of the results, we applied a five-fold cross-validation approach (see Section 3-C). This approach helps to reduce experimental bias and improve the reliability of the findings [26].
Reliability of training data: The quality of training data can introduce validity threats. For example, bug reports may be incorrectly labeled. To address this concern, we relied on four widely used datasets that have been manually validated [1] for assessing the effectiveness of the selected techniques for identifying security-related bug reports. Another potential risk related to data is data leakage, i.e., the data used for testing models may have been exposed to the models during their training. This is particularly relevant for pre-trained LLMs such as GPT-5.2 used in this study. However, given the relatively low performance of the GPT5.2 model, such leakage appears minimal. Therefore, while this risk cannot be entirely dismissed, its practical impact on our findings is likely limited. 7. C ONCLUSION In this study, we conducted a comparative analysis of several automatic techniques to identify security-related bug reports using four benchmark datasets (Ambari, Camel, Derby, and Wicket). We compared traditional ML techniques (Logistic Regression, Support Vector Machines, and Random Forest), BERT-base, RoBERTa, GPT-5.2 (in zero-shot and fewshot settings), and SetFit (a state-of-the-art few-shot learning framework). Among the evaluated techniques, SetFit demonstrated the overall best performance, achieving the highest F1 scores on three of the four datasets. Within traditional ML techniques, Logistic Regression remained a strong baseline, achieving the best F1-score on the Ambari dataset. In contrast, GPT-5.2 performed poorly in both zero-shot and few-shot settings, suggesting that prompt-only classification is not a reliable substitute for supervised learning in this task. We further examined cross-project prediction using a leaveone-dataset-out setup for SetFit and RoBERTa. Cross-project training improved performance for Camel and Ambari, particularly for Ambari, where the minority class is very small, indicating that utilizing labeled data from other projects can mitigate data scarcity. However, cross-project training degraded performance in Derby and Wicket, which may indicate differences across projects in reporting style, vocabulary, and the manifestation of security-related issues. Overall, our findings suggest that SetFit is a strong candidate for practical deployment for identifying security bug reports, particularly when labeled data are scarce. At the same time, the dataset-dependent behavior observed in cross-project prediction highlights the need for project-aware model selection and calibration. R EFERENCES [1] X. Wu, W. Zheng, X. Xia, and D. Lo, “Data quality matters: A case study on data label correctness for security bug report prediction,” IEEE Transactions on Software Engineering, vol. 48, no. 7, pp. 2541–2556, 2021.
[2] F. Peters, T. T. Tun, Y. Yu, and B. Nuseibeh, “Text filtering and ranking for security bug report prediction,” IEEE Transactions on Software Engineering, vol. 45, no. 6, pp. 615–631, 2017. [3] Y. Zhou and A. Sharma, “Automated identification of security issues from commit messages and bug reports,” in Joint meeting on foundations of software engineering, 2017, pp. 914–919. [4] K. Goseva-Popstojanova and J. Tyo, “Identification of security related bug reports via text mining using supervised and unsupervised classification,” in International conference on software quality, reliability and security. IEEE, 2018, pp. 344–355. [5] R. Shu, T. Xia, L. Williams, and T. Menzies, “Better security bug report classification via hyperparameter optimization,” arXiv preprint arXiv:1905.06872, 2019. [6] H. L. França, K. Goseva-Popstojanova, C. Teixeira, and N. Laranjeiro, “Gpts are not the silver bullet: Performance and challenges of using gpts for security bug report identification,” Information and Software Technology, p. 107778, 2025. [7] D. Yokoyama, K. Nishiura, and A. Monden, “Identifying security bugs in issue reports: Comparison of bert, ngram idf and chatgpt,” in International Conference on Software Engineering Research, Management and Applications. IEEE, 2024, pp. 328–333. [8] M. Laiq, “Few-shot learning for security bug report identification,” arXiv preprint arXiv:2601.02971, 2026. [9] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “Smote: synthetic minority over-sampling technique,” Journal of Artificial Intelligence Research, vol. 16, pp. 321–357, 2002. [10] F. Soltaniani, M. Ghafari, and M. Sayagh, “Security bug report prediction within and across projects: A comparative study of bert and random forest,” in Proceedings of the 21st International Conference on Predictive Models and Data Analytics in Software Engineering, 2025, pp. 85–94. [11] X. Cao, T. Liu, J. Zhang, M. Feng, X. Zhang, W. Cao, H. Sun, and Y. Zhang, “Sbrpbert: A bert-based model for accurate security bug report prediction,” in 2022 52nd Annual IEEE/IFIP International Conference on Dependable Systems and Networks Workshops (DSN-W). IEEE, 2022, pp. 129–134. [12] Y. Jiang, P. Lu, X. Su, and T. Wang, “Ltrwes: A new framework for security bug report detection,” Information and Software Technology, vol. 124, p. 106314, 2020. [13] D. Zou, Z. Deng, Z. Li, and H. Jin, “Automatically identifying security bug reports via multitype features analysis,” in Australasian Conference on Information Security and Privacy. Springer, 2018, pp. 619–633. [14] S. S. Alqahtani, “Security bug reports classification using fasttext,” International Journal of Information Security, vol. 23, no. 2, pp. 1347–1358, 2024. [15] M. Laiq and F. Dobslaw, “Automatic techniques for issue
report classification: A systematic mapping study,” arXiv preprint arXiv:2505.01469, 2025. [16] M. Laiq, N. b. Ali, J. Böstler, and E. Engström, “Early identification of invalid bug reports in industrial settings– a case study,” in International Conference on ProductFocused Software Process Improvement. Springer, 2022, pp. 497–507. [17] M. Laiq, N. bin Ali, J. Börstler, and E. Engström, “A comparative analysis of ml techniques for bug report classification,” Journal of Systems and Software, p. 112457, 2025. [18] M. Laiq, “An intelligent tool for classifying issue reports,” in 2023 IEEE/ACM 2nd International Workshop on Natural Language-Based Software Engineering (NLBSE). IEEE, 2023, pp. 13–15. [19] R. Kallis, A. Di Sorbo, G. Canfora, and S. Panichella, “Ticket tagger: Machine learning driven issue classification,” in 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2019, pp. 406–409. [20] M. Laiq, N. bin Ali, J. Börstler, and E. Engström, “A data-driven approach for understanding invalid bug reports: An industrial case study,” Information and Software Technology, vol. 164, p. 107305, 2023. [21] M. M. Rahman, F. Khomh, and M. Castelluccio, “Why are some bugs non-reproducible?:–an empirical investigation using data fusion–,” in 2020 IEEE international conference on software maintenance and evolution (ICSME). IEEE, 2020, pp. 605–616. [22] E. Alpaydin, Introduction to machine learning. MIT press, 2020. [23] C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, A. Wesslén et al., Experimentation in software engineering. Springer, 2012, vol. 236. [24] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692, 2019. [25] G. Koch, R. Zemel, R. Salakhutdinov et al., “Siamese neural networks for one-shot image recognition,” in ICML deep learning workshop, vol. 2, no. 1. Lille, 2015. [26] I. H. Witten and E. Frank, “Data mining: practical machine learning tools and techniques with java implementations,” Acm Sigmod Record, vol. 31, no. 1, pp. 76– 77, 2002. [27] G. Colavito, F. Lanubile, and N. Novielli, “Few-shot learning for issue report classification,” in International Workshop on Natural Language-Based Software Engineering. IEEE, 2023, pp. 16–19. [28] R. Shu, T. Xia, J. Chen, L. Williams, and T. Menzies, “How to better distinguish security bug reports (using dual hyperparameter optimization),” Empirical Software Engineering, vol. 26, pp. 1–37, 2021.