arXiv CSopen access

A systematic literature Review for Transformer-based Software Vulnerability detection

Graphical Abstract A systematic literature Review for Transformer-based Software Vulnerability detection

arXiv:2604.24822v1 [cs.SE] 27 Apr 2026

Fiza Naseer, Javed Ali Khan, Muhammad Yaqoob, Alexios Mylonas, Ishaya Gambo

Highlights A systematic literature Review for Transformer-based Software Vulnerability detection Fiza Naseer, Javed Ali Khan, Muhammad Yaqoob, Alexios Mylonas, Ishaya Gambo • A detailed critical analysis of 80 transformer-based software vulnerability detection approaches to identify insightful information. • Identify frequently used transformer architectures, evaluation matrices, and datasets, providing a holistic overview to the software vendors and researchers about the state-of-the-art. • Analysing existing approaches to identify different vulnerability types explored to date and mapping them with the CWE. • Categorising existing transformer-based software vulnerability-based approaches to different levels of granularity. • Identifying state-of-the-art research on multi-lingual software vulnerability detection using transformers.

A systematic literature Review for Transformer-based Software Vulnerability detection Fiza Naseera , Javed Ali Khana,∗ , Muhammad Yaqooba , Alexios Mylonasa and Ishaya Gambo b a Department of Computer Science, Cybersecurity and Computing Systems Research Group, University of Hertfordshire, Hertfordshire, College

Lane, Hatfield, AL10 9AB, Hertfordshire, UK b Department of Software Engineering, Obafemi Awolowo University Ile-Ife, Ile-Ife, Nigeria

ARTICLE INFO

Abstract

Keywords: Software Vulnerability Transformer SLR CodeBERT Multi-lingual Software Vulnerability

Context: Software vulnerabilities pose significant security threats to software systems, especially as software is increasingly used across many areas of daily life, including health, government, and finance. Recently, transformer-based models have demonstrated promising results in automatic software vulnerability identification due to their robust contextual modelling and representation learning capabilities. Objectives: While numerous systematic literature reviews (SLRs) have examined machine learning and deep learning methods for identifying vulnerabilities, a more transformercentric analysis remains to be explored. This SLR critically analysed 80 studies published between 2021 and 2025 that utilised transformer models to identify software vulnerabilities. Methods: Using Kitchenham’s SLR guidelines, we methodically evaluate current research from various perspectives, encompassing study trends, datasets and sources, programming languages, transformer frameworks, detection detail levels, assessment metrics, reference models, types of vulnerabilities, and experimental configurations. Results: We classify transformer models into encoder, decoder, and combined architectures and analyse both pre-trained and fine-tuned versions utilized on source code, logs, and smart contracts. The results emphasise prevailing research trends, frequently utilised benchmarks, and main baselines. It also uncovers crucial technical issues like data imbalance, interpretability, scalability, and generalization across programming languages. Conclusion: By integrating current evidence and recognising unaddressed research areas, this SLR provides a consolidated resource for researchers and professionals seeking to develop more reliable, precise, and interpretable transformerbased vulnerability identification systems.

1. Introduction Software vulnerabilities are potential loopholes in software that attackers can exploit to cause harm [12]. Code vulnerability detection plays an important role in software security. Its main goal is to find and fix weaknesses in software code to reduce the risk of cyberattacks and system failures [54]. These vulnerabilities may include coding errors, design flaws or insecure programming practices that can lead to issues such as data breaches, denial-of-service attacks, and information leakage. As software systems become larger and more complex, security risks and vulnerabilities have become more serious concerns. Detecting these vulnerabilities is a challenging task for developers as it requires analysing large volumes of code [57]. On the other hand, increased reliance on software applications in critical domains, such as health [4, 62] and finance [27], software vulnerabilities pose significant threats, including privacy breaches, service disruptions, and unauthorized access [34]. As reported on the CWE site, the complete CWE list contains 943 weaknesses as of version 4.17 [16]. The majority of security flaws today arise from insecure coding practices [5], which can lead to significant financial losses. For example, an unpatched Apache Struts vulnerability in 2017 caused a major Equifax ∗ Javed Ali Khan

[email protected] (F. Naseer); [email protected] (J.A. Khan); [email protected] (M. Yaqoob); [email protected] (A. Mylonas); [email protected] (I.G. ) ORCID (s):

Naseer et al.: Preprint submitted to Elsevier

data breach, exposing the personal information of 147 million users [69]. To address these issues, researchers have developed a variety of approaches to detect code vulnerabilities and their types. Methods include conventional static, dynamic, and hybrid code analysis approaches, as well as ML [81] and DL [34] based approaches that can automatically identify potential vulnerabilities. The conventional approaches can be divided into three categories. In the Static Analysis category, a program is analysed from its source code without executing it [28]. In the Dynamic Analysis category, a given program is analysed by executing it with specific input data and monitoring its runtime behaviour [1]. In the Hybrid Analysis category, a given program is analysed with a mixture of static analysis and dynamic analysis techniques[77] However, static analysis techniques often suffer from high false positives. Dynamic analysis techniques suffer from low code coverage, and hybrid analysis techniques suffer from the limitations of both approaches and are inefficient to operate in practice [28, 54]. To improve the effectiveness and efficiency of vulnerability detection and reduce manual effort, many learningbased vulnerability detection methods that use ML [28, 99] and DL [67] have been proposed recently. However, MLbased detection methods are limited by the quality of feature engineering and the ability to extract deep features, and they often exhibit high false alarm rates in practice, making them challenging to meet the needs of practical applications. DL has the advantage of processing large amounts of data and Page 1 of 27

SLR for transformer-based Software Vulnerability detection

mining deep features, and is thus increasingly used for vulnerability detection tasks [107, 72]. DL-based approaches still focus on coarse-grained vulnerability prediction, where models only point out vulnerabilities at the file or function level, which remains coarse-grained [26]. Self-attentionbased automated vulnerability models (transformers) are particularly effective. Transformer-based pre-trained language models extract features from long sequences and perform well on natural language tasks, making them suitable for programming language analysis [107, 17, 22]. Several experimental studies highlight their effectiveness at capturing complex code patterns [5, 9, 10, 15] . For example, in a research experiment, the transformer model consistently outperformed graph neural network models with improvements in average F1, precision and recall scores. The reason for this gap is the transformer classifier, which generates multiple attention patterns at each layer, yielding contextualised vectors that combine multiple weighted graph structures. This argument supports the use of transformer-based models, such as DetectBERT, for classifying vulnerable statements, particularly when many encoder layers are needed to capture complex data patterns [30]. To highlight the importance of transformers in detecting and classifying software vulnerabilities in software code, it is essential to present a state-of-the-art systematic literature review (SLR) that provides opportunities for researchers and software practitioners to improve existing approaches. Considering its importance, researchers have developed several SLRs for automatic software vulnerability detection; however, they mainly discuss ML/DL for vulnerability detection [77, 49, 81], they do not specifically address transformerbased models and their interpretation, as summarised in Table 1), leaving a research gap. This study analysed 80 research articles covering a wide range of transformer architectures (encoder-based, decoder-based, hybrid) and pretrained/fine-tuned variants applied to source code analysis. We investigate the following research questions: • RQ1: What types of studies are currently gaining attention in transformer-based software vulnerability detection? • RQ2: What are the commonly used datasets, their sources, and the programming languages involved? • RQ3: Which transformer models are frequently used for software vulnerability detection, and what are their reported accuracies? • RQ4: What evaluation metrics are commonly used to evaluate transformer-based approaches’ performances? • RQ5: What types of software vulnerabilities are most frequently detected? • RQ6: What hyperparameters and environment settings are commonly used for transformer-based experiments?

Naseer et al.: Preprint submitted to Elsevier

• RQ7: At what level of granularity is vulnerability detection performed? • RQ8: What baseline models are used for comparing transformer-based approaches’ performances? • RQ9: How many existing approaches focus on multilingual software vulnerability detection using transformers? The structure of the paper is as follows: Section 2 elaborates on the related work. Section 3 describes the methodology adopted for the proposed systematic literature review (SLR). Section 4 presents the answers to the research questions. Section 5 discusses the threats to validity. Section 6 outlines open challenges and potential future work on software vulnerability detection, and Section 7 presents the conclusion.

2. Related Work In this section, we elaborate on existing SLRs on software vulnerabilities and explain how the proposed SLR differs from them. The comparative study of the proposed SLR with the existing SLRs is depicted in analyze existing reviews in Table 1. Whereas Table 1 compares existing SLRs in terms of their coverage of key aspects related to transformer-based software vulnerability detection. Specifically, it evaluates whether each SLR addresses transformer models, hyper-parameter analysis, detection granularity, vulnerability types, baseline models, evaluation metrics, and data sources. Senanayake et al. [77] examine 118 research articles focused on detecting and preventing vulnerabilities in Android source code. It explores both ML-based and conventional methods, showing the relative frequency with which each approach is used. The study reviews various analysis techniques, including static, dynamic, and hybrid analyses. It also discusses various tools and repositories that support vulnerability detection and compares different tools and frameworks used in Android application analysis, highlighting their strengths and limitations. Similarly, Harzevili et al. [81] analysed 138 research papers and explored ML techniques for detecting vulnerabilities in automated software. It provides a detailed review of benchmark datasets, repositories, and data types used in these studies. The survey also examines how datasets are represented and embedded, the types of models applied, trends over time, the top 18 vulnerability types, and the tools commonly used for vulnerability detection in the reviewed articles. While Le et al. [49] primarily focus on software vulnerability assessment, they highlight the characteristics of vulnerabilities identified during the discovery phase and their prioritisation by severity. It reviews 84 research articles and provides a detailed analysis of data sources and data-driven approaches, particularly those involving Natural Language Processing (NLP), ML and DL techniques. The survey also summarises commonly used data sources, features, models, evaluation Page 2 of 27

SLR for transformer-based Software Vulnerability detection Table 1 Comparison Between Contribution of Our Survey and Existing Vulnerability Detection SLRs SLR Senanayke et al.[77] H.M. Le et al.[49] Harzevili et al.[81] Croft et al.[13] Ghaffarian et al.[28] Eberendu et al.[20] Bassi and Singh [6] Sohan and Basalamah[83] Our Survey

Transformers Hyper-parameter Granularity Vul.Types Baseline Eval.Metric Data Source Multi-language ✗ ✗ Partial ✗ ✗ ✗ ✗ ✗ ✓

✗ ✗ ✗ ✗ ✗ ✗ ✓ ✗ ✓

methods, and metrics for software vulnerability assessment and prioritisation. Moreover, Croft et al. [13] SLR mainly focuses on data preparation processes and their role in software vulnerability prediction. They review 61 relevant studies and highlight the most commonly used programming languages in this research area. The survey also discusses the main types of data sources and compares their frequency of use. Additionally, it outlines common data labelling methods and data cleaning techniques. The study presents a taxonomy of data-related challenges in software vulnerability research, identified from the reviewed papers. Additionally, Ghaffarian et al. [28] explore the use of ML and data mining techniques for detecting software vulnerabilities. It reviews two main categories of research: one focusing on the analysis of program syntax and semantics, and the other on software metrics-based approaches. The survey summarizes recent work on vulnerability prediction models, anomaly detection methods, recognition of vulnerable code patterns, and other miscellaneous techniques. On the other hand, Eberebdu et al. [20] reviewed 55 articles related to software vulnerability detection. It discusses trends in detection methods, various detection approaches, topics commonly addressed, and the characteristics and causes of software vulnerabilities. The study covers techniques such as neural networks, machine learning, code clone detection, and static and dynamic analysis methods. Bassi et al. [6] reviewed 77 articles focused on software vulnerability prediction. It explores DL and ML techniques, tools, and feature types used in prediction models. The survey also covers data balancing techniques, crossvalidation methods, feature extraction methods and datasets, evaluation metrics, and parameter tuning approaches. Finally, Sohan et al. [83] focus specifically on malware detection in JavaScript. It reviews 32 articles and addresses research questions about trends in the field, dataset types and sizes, data analysis methods, detection techniques, performance metrics, and common challenges. The survey also examines the machine learning and data analysis methods employed in these studies. Most existing SLRs focus on general vulnerabilitydetection techniques and provide limited or no coverage of transformer-based approaches, as shown in Table 1. For example, Senanayake et al. [77] and H. M. Le et al. [49] cover Naseer et al.: Preprint submitted to Elsevier

✗ ✗ ✗ ✓ ✓ ✗ ✗ ✗ ✓

✗ ✗ ✓ Partial ✗ Partial ✗ ✗ ✓

✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✓

✗ ✗ ✗ ✗ ✓ ✗ ✓ ✓ ✓

✓ ✓ ✓ ✗ Partial Partial ✓ Partial ✓

✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓

data sources but do not address transformers, hyperparameters, granularity, vulnerability types, baseline models, or evaluation metrics in detail. Harzevili et al. [34] partially discuss transformer-based approaches and vulnerability types, but do not examine hyperparameters, granularity, baseline models, or evaluation metrics comprehensively. Similarly, Croft et al. [13], and Ghaffarian et al. [28] address certain aspects such as granularity and evaluation metrics. However, their coverage remains incomplete or partial across other dimensions. On the other hand, Eberendu et al. [20] and Bassi and Singh [6] provide partial or selective coverage of vulnerability types, evaluation metrics, and data sources, yet lack a focused analysis of transformer models. Sohan and Basalamah [83] address baseline models and evaluation metrics but do not discuss transformers or detection granularity in detail. In contrast, the proposed SLR approach provides comprehensive coverage across all evaluated dimensions, including transformer models, hyperparameters, detection granularity, vulnerability types, baseline comparisons, evaluation metrics, and data sources. This highlights the novelty and contribution of the proposed work, as it offers the first systematic and in-depth review dedicated to transformerbased vulnerability detection, to the best of our knowledge. The proposed SLR provides opportunities for software developers, researchers, and vendors to equip themselves with state-of-the-art transformer-based vulnerability approaches to further improve the performance of existing vulnerabilitybased approaches. In contrast, the proposed research explores the existing research focused on vulnerability detection and prediction. The review emphasizes studies that use transformerbased approaches for identifying software vulnerabilities. Harzevili et al. [81] briefly discuss transformer-based approaches, but the study’s scope is limited and does not provide an in-depth analysis. We examine the use of pretrained models, their fine-tuning strategies, and key hyperparameter configurations. The proposed SLR also unfolded the reported performance evaluation methods, results, commonly addressed vulnerability types and the diversity of data sources and datasets employed in the literature. In addition, we consider articles that use multiple programming languages and assess how different techniques contribute to effective software vulnerability detection and prediction. Page 3 of 27

SLR for transformer-based Software Vulnerability detection

Figure 1: SLR methodology stages following the Kitchenham guidelines [48]

3. Methodology 3.1. Studies Source In this article, we conduct a systematic literature review following Kitchenham’s guidelines [48]. Figure 1 shows the Kitchenham methodology stages, which define a structured methodology for conducting systematic literature reviews in software engineering through three phases: planning phase, conducting phase that includes search, selection, extraction and Reporting phase that includes results report writing and discussion, to ensure rigour, transparency, and reproducibility. We collect and examine research published between 2021 and 2025 that focuses on software vulnerability detection using Transformer-based models. We examine articles from 2021 because Transformer models [94] were introduced in 2017, and researchers began actively exploring their applications shortly thereafter; however, a substantial body of mature and promising research has been available since 2021, based on knowledge and search. The overall workflow of the systematic methodology is illustrated in Figure 2. To ensure comprehensive coverage, we draw the SLR data from several widely used and reputable digital libraries, including the ACM Digital Library, ScienceDirect, IEEE Xplore, Springer, and Google Scholar. We included Google Scholar to ensure that any related and important transformer-based vulnerability detection approaches are not missed in the proposed SLR, as it searches through various research databases to identify potential papers for further analysis.

3.2. Search String For the proposed SLR, we developed the following research string to identify transformer-based vulnerability detection approaches: (“software vulnerability detection” OR “code vulnerability detection” OR “software vulnerability prediction” OR “software flaw detection” OR “security vulnerability detection” OR “software anomaly detection”) AND (“transformer model” OR “transformer-based model”)

3.2.1. Inclusion Criteria The following criteria were used to include studies in the proposed systematic literature review: • Studies that address software vulnerability detection, classification, or prediction. • Studies that employ transformer-based approaches. Naseer et al.: Preprint submitted to Elsevier

• When multiple versions of a study exist, only the most recent and comprehensive version was included. • Studies published in the English language. • Studies published between 2021 and 2025. • Articles focusing on the significance, strategies, techniques, application domains, or challenges of transformerbased models for software vulnerability detection.

3.2.2. Exclusion Criteria The following criteria were used to exclude studies from this review: • Studies that do not specifically address software vulnerabilities. For example, works on network security. • Duplicate publications. • Pre-print articles. • Studies not published in English. • Literature that is not directly related to transformerbased methods.

3.3. Data Extraction Figure 2 shows the publication collection process for the proposed SLR. After conducting a search in the ACM Digital Library using relevant keywords, we found 43 articles. From these, we selected 15 articles that satisfy the inclusion criteria defined for the proposed SLR. The remaining articles were excluded based on titles and abstracts, as they mainly addressed general topics in machine learning or artificial intelligence rather than software vulnerability detection using a transformer. Additionally, when we run the search query on the ACM Digital Library, it returns names of various conference proceedings that were not relevant to the search query. When we explored further, we found that the papers published under these themes or conference titles were not relevant, so we excluded them from the final count. After conducting a search on Springer, a total of 247 articles were retrieved. From these, only 6 articles were found to be relevant and selected for inclusion in the proposed SLR. A total of 203 articles were excluded based on their titles because they primarily focused on unrelated domains, such as medical applications for disease detection, computer networking, time-series forecasting, and businessrelated software. The remaining 53 articles were rejected Page 4 of 27

SLR for transformer-based Software Vulnerability detection

Figure 2: Overall workflow of our systematic survey

after reviewing their abstracts, which did not align with the objectives of the proposed study. We searched IEEE Xplore using the keywords "Software vulnerability detection using transformers" because they yielded the most relevant results compared to the main search query. We found 87 papers in total. Of these, we selected 36 papers for the proposed SLR and excluded 51. We removed 33 papers because their titles were unrelated to our topic, and 18 more were excluded after reading their abstracts, as they did not align with the research objectives. We applied a search query on ScienceDirect and initially retrieved 73 articles. We then filtered the results to include only research articles within the computer science domain, reducing the set to 50 articles for further screening. After a detailed review of titles and abstracts, we excluded 33 additional articles and selected only 17 articles relevant to the proposed study based on the inclusion criteria. We conducted a search on Google Scholar and initially found 78 articles. Following a screening process, we shortlisted 16 articles for detailed evaluation. Out of the remaining, 44 articles were excluded based on irrelevant titles, and 18 were rejected based on abstract content. After removing duplicates from the shortlisted set, we finalized 5 articles for inclusion in the proposed survey. Precisely, we had a total of 80 articles that are included in this review paper. Figure 3 Naseer et al.: Preprint submitted to Elsevier

Figure 3: Number of Publications per Year

illustrates the number of research articles published during the period of 2021 to 2025. It shows an increasing trend of transformer-based approaches for software vulnerability detection, with the highest number of papers published in 2025. It highlights the importance of a detailed SLR on transformer-based approaches to software vulnerability detection by identifying the key findings to date. Additionally, before submitting the SLR paper, we quickly reviewed the latest studies published in 2026 [45, 74, 19] and included them for the SLR, but we did not examine them in detail.

Page 5 of 27

SLR for transformer-based Software Vulnerability detection

3.4. Quality Assessment Before performing a detailed analysis of the research articles, we evaluate the relevance and rigor of the selected studies. We establish specific quality criteria for assessment and formulate a set of questions. These questions help determine whether a study should be included for further analysis. Only studies that receive positive answers to these questions are considered relevant. • Q1: Does the paper clearly state a research goal related to software vulnerability detection in its introduction? • Q2: Does the proposed approach employ specifically transformer/attention-based models? • Q3: Is the vulnerability detection technique clearly defined and described in a way that makes it repeatable or reproducible? • Q4: Does the study present an explicit contribution to the field of software vulnerability detection? • Q5: Is there a clear and well defined methodology for validating the proposed transformer based approach? • Q6: Are the subject projects or software systems used for validation appropriate and aligned with the stated research objectives? • Q7: Are the datasets used in the study relevant to software vulnerability detection? • Q8: Does the study include baseline models or control techniques to demonstrate the effectiveness of the proposed approach? • Q9: Are the evaluation metrics appropriate and relevant to measuring vulnerability detection performance? • Q10: Do the reported results align with the stated research objectives, and are they presented clearly and in a meaningful manner?

4. Results and Discussion In this section, we present the detailed analysis of the selected research articles. The findings are discussed with respect to all research questions and are supported by Tables and Figures to better illustrate them for potential readers.

4.1. What are the current research foci in software vulnerabilities using transformers? To answer this research question, we analysed each research article included in the proposed SLR to identify the current foci in software vulnerability detection. The holistic foci of current research trends are shown in Table 2. By analysing recent research studies on software vulnerability detection, we can observe a clear study framework and methodological diversity. Among the reviewed works, 39

Naseer et al.: Preprint submitted to Elsevier

papers primarily address binary vulnerability detection, aiming to distinguish between vulnerable and non-vulnerable code segments. In contrast, a group of 38 papers focuses on multi-class detection, identifying distinct categories of vulnerabilities such as buffer overflows, injection flaws, and access control issues, reflecting a growing trend toward more fine-grained vulnerability classification. Moreover, only 3 studies explore vulnerability prediction, there aims to forecast the likelihood of vulnerabilities before they occur. For example, Liu et al. [55] methodology starts by extracting multidimensional code representations, including plain text, flattened control/data-flow sequences, and structural program graphs. In the second step, they use a combination of CodeBERT and bidirectional LSTM (BLSTM) models in a multi-model training regime to capture semantic and syntactic patterns. At the final step, they evaluate the approach across multiple real-world datasets. Similarly, Le et al. [50] predict vulnerability by constructing vulnerability datasets using CVEfixes and fine-tuning a CodeBERT model for both function-level and line-level vulnerability prediction with and without data sampling techniques to evaluate performance under data scarcity, and additionally explore using ChatGPT as an alternative predictive model, finding that CodeBERT performance deteriorates in low resource settings while ChatGPT yields substantial gains in prediction accuracy. On the other hand, Fu and Tantithamthavorn [26] apply a CodeBERT with token embeddings to source code to learn contextual semantic representations and then uses a classifier on top of those embeddings to perform vulnerability prediction at both function and fine grained line levels, leveraging self attention mechanisms to identify vulnerable code patterns and rank likely vulnerable lines via attention scores, resulting in improved line level vulnerability detection over prior sequence or graph based models. RQ1 Research Finding Researchers have shown interest in software binary vulnerability detection, multi-class software vulnerability detection and software vulnerability prediction. Among these, Binary vulnerability detection is comparatively dominant. However, the trends show that fine-grained software vulnerability analysis is receiving equal attention from the research community to better understand frequently occurring software vulnerabilities. Moreover, we identify only three research papers focusing on software vulnerability prediction, indicating an alternative research domain that researchers can further explore for early software vulnerability detection.

4.2. What are the commonly used datasets, their sources, and the programming languages involved? Software vulnerability detection and classification are emerging issue in software development that needs efforts from software researchers and vendors to minimise their Page 6 of 27

SLR for transformer-based Software Vulnerability detection Table 2 Current Research Trends of Studies Category

Count

Articles

Binary Vulnerability Classification

39

Vulnerability Detection on Specific Types / Multi-class Type Classification Vulnerability Prediction

38

[52, 11, 60, 7, 108, 36, 32, 53, 101, 57, 107, 41, 106, 2, 64, 35, 96, 8, 40, 102, 104, 103, 66, 18, 58, 75, 82, 43, 98, 97, 86, 31, 51, 93, 14, 92, 68, 71] [76, 5, 73, 23, 59, 88, 47, 30, 70, 109, 46, 65, 100, 89, 91, 9, 61, 33, 37, 105, 79, 42, 38, 87, 15, 3, 29, 56, 63, 21, 95, 39, 44, 80, 25, 85, 78, 90]

3

[50, 55, 26]

effects on existing software systems. For this purpose, researchers and software vendors have begun developing various datasets and baseline approaches to improve the detection and classification of software vulnerabilities. In this research question, we investigate the datasets used in transformer-based software vulnerability detection research articles, their frequency of use, the programming languages included in each dataset, and their corresponding links, aiming to provide a holistic overview of these resources for the research community and software vendors. Table 3 summarizes the datasets used in recent software vulnerability detection studies, including their programming languages, sources, and corresponding references. The table 3 shows that most research is based on C/C++ datasets, with BigVul, Devign, Reveal, and SARD being the most frequently used benchmarks. It also highlights the growing use of multi-language datasets such as CVEfixes and CodeXGLUE, as well as the increasing importance of Solidity datasets in smart contract vulnerability detection. Overall, the Table 3 indicates that current research is still centred on traditional C/C++ vulnerability detection, while Python, Java, Go, and Solidity are receiving increasing attention. It also shows that most vulnerability detection studies rely on publicly available datasets derived from real-world software repositories and vulnerability databases. Widely used datasets include Devign, Big-Vul, Reveal and SARD, which are constructed by mining source code from GitHub and linking it to vulnerability labels from the National Vulnerability Database (NVD) and Common Vulnerabilities and Exposures (CVE) records [84]. For Example, Saimbhi and Akpinar [76] use a custom dataset of PHP code snippets that cover 17 types of vulnerabilities. The process involved parsing PHP code to extract individual functions, which were then analysed using prompt engineering techniques. These prompts were submitted to the GPT-3.5 Turbo model through the OpenAI API to identify possible software vulnerabilities. Similarly, Li et al. [52] use a custom dataset consisting of C programs from publicly available vulnerability datasets, while the model itself is trained on unlabelled code to learn normal coding patterns. Each function or code snippet is encoded into a representation capturing syntactic and semantic features, and an anomaly attention mechanism is applied to highlight unusual patterns indicative of potential vulnerabilities.

Naseer et al.: Preprint submitted to Elsevier

RQ2 Research Finding To date, researchers use over 40 types of vulnerability datasets, most of which are publicly available, showing researchers growing interest. The BigVul (13) and Devign (13) is the most frequently used, followed by Reveal (12) and SARD (9), primarily for C/C++ code, while smart contract research relies on Solidity-based datasets. Also, there is a trend of developing a vulnerability dataset for multiple programming languages.

4.3. RQ3: Which transformer models are used in existing studies, and what are their reported results? Transformer-based approaches have been widely used for various natural language processing tasks, achieving comparatively better results than deep and machine learningbased approaches. In the literature, researchers have used various baseline transformers and transformer combinations to identify software vulnerabilities in different programming languages. Through this RQ, we examine the types and combinations of transformer-based models used by software researchers in their studies, along with the best reported results. With this, we aim to provide a holistic overview of the state-of-the-art in transformer-based vulnerability detection for software vendors and researchers to identify models that best suit the problem. Also, this will give opportunities to software researchers to identify areas that need further research. We consider only the best-performing results reported in each study, as determined by the proposed analysis. Table 4 shows the details of the models used in the research articles along with the reported results of the evaluation metric. Table 4 shows that transformer models have become the dominant paradigm in software vulnerability detection research. Among them, CodeBERT is the most frequently used backbone, often serving either as a stand-alone encoder or as the semantic foundation of hybrid architectures. A major trend is that researchers increasingly combine transformers with CNNs, BiLSTMs, GNNs, and graph-based reasoning mechanisms, indicating that plain token-sequence modeling is often considered insufficient for capturing the structural complexity of vulnerable code. Recent work also shows

Page 7 of 27

SLR for transformer-based Software Vulnerability detection Table 3 The datasets used in the research articles, along with their links and the programming languages. Dataset

Language

URL

Articles

BigVul

C/C++

https://www.kaggle.com/datasets/kaggler10240/msr-data

SARD

https://samate.nist.gov/SARD

VulDeePecker FormAI Devign

C, C++, Java, PHP, and C# C/C++ C C/C++

https://github.com/CGCL-codes/VulDeePecker https://github.com/FormAI-Dataset/FormAI https://github.com/epicosy/devign

Reveal

C/C++

https://huggingface.co/datasets/claudios/ReVeal

LVDAndro FFmpeg FFmpeg+QEMU OpenSSL 1.0.1e

Android C/C++ C/C++ C, C++, & assembly as above as above C/C++ Multi Multi Multi Python C/C++ Solidity Multi Multi C/C++ Solidity Solidity Java Web C/C++ Solidity Multi C/C++ C/C++ C/C++/Python /Java Solidity Solidity Solidity Java Python Go Solidity C/C++ C C Log Text Log Text Java Python Solidity Java Multi

https://github.com/softwaresec-labs/LVDAndro https://github.com/ffmpeg/ffmpeg https://github.com/ffmpeg/ffmpeg; https://github.com/qemu/qemu https://github.com/openssl/openssl

[59, 70, 53, 75, 15, 35, 96, 8, 55, 103, 66, 51, 43, 85] [5, 60, 47, 53, 37, 79, 35, 14, 90] [73, 58, 91, 33] [24] [59, 7, 88, 57, 107, 38, 35, 96, 8, 55, 103, 18, 97, 98] [59, 58, 107, 33, 38, 96, 103, 92, 97, 98, 43, 85] [68] [90, 82] [43, 82] [90]

https://github.com/postgres/postgres https://github.com/apache/subversion https://github.com/wagner-group/diversevul https://github.com/fkie-cad/nvd-json-data-feeds https://www.debian.org/download https://github.com/secureIT-project/CVEfixes https://huggingface.co/datasets/DetectVul/Vudenc https://github.com/davidhin/linevd https://github.com/smartbugs/smartbugs-curated https://github.com https://github.com/microsoft/CodeXGLUE https://github.com/IBM/D2A https://github.com/DependableSystemsLab/SolidiFI-benchmark https://github.com/smartbugs/smartbugs-wild https://github.com/WebGoat/WebGoat https://github.com/SySeVR/SySeVR https://figshare.com/s/fbaf47e3ac2a9581dbd7 https://github.com/github/CodeSearchNet https://osf.io/d45bw/ https://samate.nist.gov/SARD/test-suites/112 https://github.com/mvd-dataset/MVD

[90] [90] [85] [60, 47, 56, 21] [108] [30, 105] [30, 109, 105] [36] [46, 65] [63, 21, 3] [58, 33, 41] [58, 33, 38] [65] [65, 100] [89] [91, 79] [9] [26] [11, 58, 33, 42, 85] [41] [38]

https://github.com/MetaTrustLabs/GPTScan-Top200 https://github.com/MetaTrustLabs/GPTScan-Web3Bugs https://github.com/MetaTrustLabs/GPTScan-DefiHacks https://github.com/apache/hadoop https://opendev.org/openstack https://github.com/kubernetes/kubernetes https://github.com/wuhongjun15/Peculiar https://github.com/Icyrockton/MegaVul https://github.com/torvalds/linux https://github.com/SoftSec-KAIST/BinKit https://github.com/logpai/loghub https://github.com/logpai/loghub https://github.com/rjust/defects4j https://github.com/soarsmu/BugsInPy https://github.com/s00ne/SmartConDetect https://github.com/TQRG/VDET-for-Java https://www.kaggle.com/datasets/andrewkronser/ cve-common-vulnerabilities-and-exposures https://www.kaggle.com/datasets/shashwatwork/ android-malware-dataset-for-machine-learning

[87] [87] [87] [106] [106] [106] [29] [8] [55] [40] [102] [102] [104] [104] [39] [61] [44]

PostgreSQL 9.2.4 Apache Subversion 1.8.3 DiverseVul NVD Debian CVEfixes VUDENC LineVD SB Curated GitHub CodeXGLUE D2A SolidiFI-Benchmark SmartBugs Wild OWASP WebGoat SeVC dataset OverflowGen CodeSearchNet Draper VDISC Juliet Test Suite v1.3 MVD Top200 Web3Bugs DeFiHacks HDFS (project) OpenStack Kubernetes Peculiar MegaVul Linux Kernel BinKit HDFS (LogHub) Blue Gene/L (BGL) Defects4J BugsInPy SmartCon VDET-for-Java CVE Drebin

Android applications

a shift toward specialized transformer frameworks, graphaware models, and multi-model collaboration, while LLMs are increasingly explored as complementary components rather than complete replacements for task-specific code Naseer et al.: Preprint submitted to Elsevier

[2]

transformers. Researchers use baseline Transformer models and a combination of models to achieve the best results. For example, Alqarni and Azim [3] proposed an improved version of BERT to detect vulnerabilities in source code by Page 8 of 27

SLR for transformer-based Software Vulnerability detection

extending its architecture with deeper layers tailored for code analysis. Firstly, they preprocess the dataset and address the class imbalance by resampling to create a balanced training set. After that, the improved BERT algorithm was finetuned with the most effective hyperparameter to maximise accuracy. Bui and Do [7] have proposed a Vulnerability detection approach for converting the source code into a code property graph (CPG), a unified representation that combines the syntax, control flow, and data flow of the program. Then, each node in this graph, such as statements, variables, or control structures, is converted into a vector using a pre-trained codeBERT model. The hybrid model Adaptive Transformer-GCN (AT-GCN) uses the Transformer’s self-attention to understand long-range relationships across the code and a Graph Convolutional Network (GCN) to capture local graph structure around each node. By learning from both global and local features, the model can more accurately identify the software vulnerabilities. In another research, Jian-Jie and Le [41] cleaned the source code and converted it into a more concise form. In the first step, they embed code as tokens, AST tokens, and dependency graph nodes using BERT, convert tokens into high-dimensional vector representations, and then process them with the DL models TextCNN, BiLSTM, BiLSTM CNN, and GCN to extract different types of features. The outputs of all five models are combined via a stacking ensemble, and the resulting predictions are fed into a final classifier that determines whether a software defect is present in the code. Similarly, Bahaa et al. [5] introduce the DB-CBIL model. It is a hybrid deep learning approach developed to automatically detect vulnerabilities in software code. It combines DistilBERT, a compact version of the BERT transformer model, to generate contextual word embeddings from source code functions represented as Abstract Syntax Trees (ASTs). These embeddings capture both the syntax and semantics of the code. The model then employs two neural networks: a CNN to extract local features and a Bidirectional Long Short-Term Memory (BiLSTM) network to capture sequential dependencies in the code. By integrating these components, DB-CBIL effectively identifies vulnerable code patterns. While, Kaanan et al. [42] propose VulBERTDense, a vulnerability detection approach that combines multiple large language models with a custom neural network layer. In the first step, they preprocess the data from the Draper VDISC dataset, labelling code snippets and applying padding and truncation. Then they use a dataloader to batch them for training. LLM is used as a feature extractor, turning code into meaningful embeddings, which are then refined by the added dense neural layer. The combined model predicts the vulnerability of CWE-120 as ’YES’ or ’NO’. Furthermore, Sun et al. [87] use a hybrid model, GPTScan, to detect logic vulnerabilities in Solidity smart contracts by combining a GPT-based transformer with traditional static analysis. In the first step, they apply static reachability filtering to highlight candidate functions. Then they break down vulnerability types into scenarios and properties, which are matched using GPT prompts to assess semantic Naseer et al.: Preprint submitted to Elsevier

code patterns and identify key variables and statements. Finally, these GPT-informed findings are validated using static program analysis to eliminate false positives. Mylläri et al. [64] proposed the Ladle technique that uses a sentence transformer, a pre-trained language model to embed overlapping short segments of log entries from multiple log types into a vector space. It calculates the anomaly score by comparing its embedding to a reference distribution for its log type. The system supports data drift adaptation by updating the reference collection with new log segments without retraining the model, allowing it to stay accurate as log behaviour evolves. Tested on a real-world dataset. Ladle showed high accuracy and outperformed traditional singlelog anomaly detection approaches. Similarly, He et al.[35] introduce the VulTR model to detect software vulnerabilities by enhancing and refining key features across multiple layers of analysis. First, it extracts static features from source code functions, including code metrics and semantic representations. Then, a multi-layer enhancement module processes these features to better capture structural and contextual information. These enriched features are then fed into a deep learning classifier to predict whether a function is vulnerable or not. Furthermore, Wu et al. [102] proposed a vulnerability detection methodology that employs a Transformer encoderbased model to detect software anomalies in system log sequences. Their designed model accepts log sequences of varying lengths, incorporating special tokens, each with temporal embeddings. These embeddings are combined with event representations and fed into Transformer encoder layers, with an attention mask ensuring the model ignores padded tokens. During supervised training, only the output token is used as a sequence-level representation and optimized with a binary classification objective using binary cross-entropy loss to distinguish between normal and anomalous sequences. Khan et al. [45] use a CodeBERTbased transformer model as the core architecture for vulnerability detection on code slices. Reza et al. [74] explicitly evaluates four transformer-based models CodeBERT, CodeT5+, PLBART and UniXcoder. Among them, UniXcoder gives the best overall results in their JavaScript vulnerability detection experiments. Do et al. [19] proposes a custom transformer/LLM-inspired ensemble architecture called RoS-Dex. RQ3 Research Finding Transformer based models, especially CodeBERT and GraphCodeBERT variants, dominate recent software vulnerability detection research, with a clear trend toward hybrid, graph-aware, and taskspecific architectures rather than plain transformeronly models. Table 5 groups the reviewed articles according to the main transformer backbone or family they use. It shows that CodeBERT and its variants are the most commonly used Page 9 of 27

SLR for transformer-based Software Vulnerability detection

Table 4 Transformer models used in articles along with reported results (Part 1) Article Saimbhi and [76] Li et al. [52]

Akpinar

Chen and Liu [11] Bahaa et al. [5] Purba et al. [73] Ferrag et al. [24] Lu et al. [59] Mahyari [60] Bui and Do [7] Sun et al. [88]

Models VulnerAI, Transformer (GPT)

Results Accuracy 70.00%, Precision 100.00%, Recall 68.00%

Transformer with Anomaly Attention Mechanism LSTM and Transformer (HLT) CNN and BiLSTM with BERT tokenizer (DB-CBIL) GPT-3.5-Turbo, CodeGen (Transformer Ensemble) Self-Attention + MLP (SecureFalcon) CodeBERT, CodeT5, and CodeGPT

Accuracy 87.73%, FPR 13.60%, TPR 85.26%, F1-score 93.23%

Kim et al. [47]

BERT and DistilBERT GCN + Transformer CodeBERT (12-layer Transformer Encoder) SCDetect with improved self-attention mechanism (Si-AS) BERT

Gujar [30]

DetectBERT

Peng et al. [70] Zhao et al. [109] Hin et al. [36]

CodeBERT Self-Attention (PTLVD) CodeBERT CodeBERT (LineVD)

Kim et al. [46]

Transformer-based LLMs (BERT, GPT3, DistilBERT, PruneBERT, etc.) Custom BERT (PDBERT) MANDO-HGT (Heterogeneous Graph Transformer) DL-VulBERT (LSTM + Attention + BERT) CodeBERT + Adaptive GNN Peculiar (GraphCodeBERT-based) CodeBERT with LIME and SHAP Transformer (Multi-Head Attention + FFN) Vul-GPT

Zhao and Liu [108]

Liu et al. [58] Nguyen et al. [65] Gupta et al. [32] Liang et al. [53] Wu et al. [100] Rusinova et al. [75] Wu et al. [101] Liu et al. [57] Tanko et al. [89] Zhang et al. [107] Thapa et al. [91]

Transformer Encoder + Transformerbased CNN + GGNN VulD-Transformer + FastText

Cao [9] Fu and Tantithamthavorn [26] Mamede [61] Hanif and Maffeis [33] Hou et al. [37] Zahid [105] Omar and Shiaeles [79] Kaanan et al. [42] Jianjie and Le [41]

CodeBERT, GraphCodeBERT, PLBART, CodeT5 VDTransformer LineVul (BPE + CodeBERT + Attention) javaBERT VulBERTa-MLP/CNN (BPE) Transformer (Attention + MLP) DetectBERT, MiniLM, MPNet GPT-based VulDetect VulBERTDense (CodeBERT-based) BERT + GCN + CNN + BiLSTM

Islam et al. [38]

RoBERTa-PFGCN

Sun et al. [87] Curto et al. [15] Alqarni and Azim [3] Le et al. [50]

GPTScan (GPT + Static Analysis) Multitask CodeBERT BERT CodeBERT + ChatGPT (GPT-3.5)

Naseer et al.: Preprint submitted to Elsevier

Accuracy 67.85% and 67.00%; F1-score 70.17% and 72.03% Recall 100.00%, Accuracy 99.81%, Precision 99.51%, AUC 99.84%, F1-score 99.75% FPR 74.22%, FNR 3.96%, TPR 96.04%, Precision 57.40%, F1-score 71.85% Accuracy 94.00% (binary), up to 92.00% (multiclass) Accuracy 97.98%, Precision 91.55%, Recall 70.50%, F1-score 79.66% (Big-Vul dataset) Accuracy 98.25% (BERT), 98.17% (DistilBERT) Accuracy 54.20%, Precision 38.70%, Recall 39.90%, F1-score 41.10% Accuracy 70.00%, F1-score 68.00% Accuracy 96.00%, Precision 98.00%, Recall 91.00%, F1-score 95.00% CWE-119: FPR 0.30%, FNR 4.00%, TPR 96.00%, Precision 99.90%, F1-score 97.90% F1-score 64.88 ± 5.46%, Precision 60.71 ± 8.08%, Recall 73.08 ± 2.65% Accuracy 60.05%, Precision 54.90%, Recall 80.99%, F1-score 65.44% Accuracy 95.89–98.63%, F1-score 94.74–98.33% F1-score 36.00%, Recall 53.30%, Precision 27.10%, ROC-AUC 91.30%, PR-AUC 64.20% Accuracy 97.00%, Precision 97.00%, Recall 97.00%, F1-score 97.00% (Dataset 3) Accuracy 67.61%, F1-score 59.41% F1-score 95.40% (Time Manipulation vulnerability) Accuracy 94.43%, Precision 94.39%, Recall 94.64%, F1-score 94.33% Accuracy 68.50%, Precision 70.70%, Recall 92.00%, F1-score 82.90% Precision 91.80%, Recall 92.40%, F1-score 92.10% LIME 94.00%, SHAP 99.00% Accuracy 94.30%, Precision 85.20%, F1-score 90.20%, FPR 6.20%, FNR 4.20% TF-IDF: Accuracy 49.82%, Precision 46.47%, Recall 71.90%, F1score 56.45%; BM25: Accuracy 49.35%, F1-score 56.18% CWE-78: Accuracy 74.07%, Precision 79.59%, Recall 75.58%, F1score 77.53% Accuracy improvement 1.42–6.70%, Recall improvement 2.72– 10.11%, F1-score improvement 1.82–5.64% FPR 3.63%, FNR 9.06%, Precision 89.14%, Recall 90.94%, F1-score 90.03% (BERT-base) Accuracy 92.80%, Precision 93.90%, Recall 89.00%, F1-score 91.40% F1-score 91.00%, Precision 97.00%, Recall 86.00% Accuracy 99.00%, Precision 95.00%, Recall 93.00% VulBERTa-MLP: Precision 95.76% Precision 95.04%, Recall 88.89%, F1-score 91.86% Precision 60.71%, Recall 73.08%, F1-score 64.88% F1-score 92.40%, TPR 91.30%, AUC 90.30% (SARD dataset) Accuracy 90.10%, Precision 91.10%, Recall 89.20%, F1-score 89.70% CodeXGLUE: Accuracy 62.87%, F1-score 60.51%; Juliet: Accuracy 95.98%, F1-score 87.80% VulF: Accuracy 96.24%, F1-score 95.85%; MVB: Accuracy 98.23%, F1-score 98.01% Top200: FP 13; Web3Bugs: TP 40, FP 30; DeFiHacks: TP 10, FN 4 Accuracy 99.03%, Precision 97.31%, Recall 93.87%, F1-score 95.51% Accuracy 99.30% F1-score 43.00%, Precision 44.00%, Recall 43.00% (best on Rust)

Page 10 of 27

SLR for transformer-based Software Vulnerability detection Table 4 Transformer models used in articles along with reported results (Part 2) Zhang et al. [106]

TWLog

Almakayeel [2] Myllari et al. [64] Gong et al. [29] Liu et al. [56] He et al. [35] Mechri et al. [63] Wang et al. [96] Ehrenberg et al. [21] Cao and Dong [8] Liu et al. [55] Jiang et al. [40] Wu et al. [102] Yang et al. [104] Xuan et al. [103] Tian and Zhang [92]

DLBITM-AMD (Transformer + RNN) Ladle (Transformer-based) GRATDet (Transformer-GP) BiVulD (CodeBERT + LSTM) VulTR (CodeBERT + BiLSTM) SecureQwen (CodeQwen1.5, Qwen2) SCL-CVD (GraphCodeBERT + MLP) CodeBERT, RoBERTa, DistilBERT MSVD (CodePTM + CodeBERT) VulPCL (BiLSTM + CodeBERT) HAformer Transformer-BERT LLMAO (CodeGen-16B, GPT-2) FG-CVD (BiSelf-Attention + MLP) EFVD (CodeBERT + EA-GGNN + MLP + focal loss) SIExVulTS (SentBERT + CodeQL + GraphCodeBERT)

Katz et al. [44]

Cui et al. [14]

VulGTDA (GraphTransformer + domain adaptation)

Vanam et al. [93] Sun et al. [86]

OAssAI / SF-TransBiLSTM HgtJIT (Graph-transformer model)

Oladokun and Rice [68] Wang et al. [97]

CodeBERT, GraphCodeBERT CSLS (Transformer-based line-level semantic structure learning model)

Shir et al. [80]

CodeBERT, Longformer, RNN, LSTM, GRU

Wang et al. [98]

M2CVD (transformer-based code models and LLMs)

Ferretti et al. [25]

BERT, DistilBERT, CodeBERT, Gemini, RF, G-NB, SVC, GBoost, DT

Perera et al. [71]

CodeBERT-based embeddings + neural network classifier CodeVul+ (GraphCodeBERT embeddings plus graph neural network reasoning) CEGT (GCN-Transformer)

Sultan et al. [85]

Shang et al. [78] Kalouptsoglou et al. [43] Tao et al. [90]

Smaili et al. [82]

CodeBERT fine-tuning / CodeBERT embeddings CodeBERT-based bimodal Transformer + BiGRU CodeGATNet (CodeBERT + CNN + gated attention)

models in software vulnerability detection research, followed by the BERT/RoBERTa/DistilBERT family, GPT/LLMbased models, GraphCodeBERT and graph-aware transformers, other code language models such as CodeT5, PLBART, and UniXcoder, and custom transformer architectures. Overall, the table indicates that recent studies rely heavily on pretrained code-oriented transformer models, Naseer et al.: Preprint submitted to Elsevier

HDFS F1-score 95.40%; OpenStack F1-score 96.30%; Kubernetes F1score 90.90% Accuracy 99.26%, Precision 99.39%, Recall 99.26%, F1-score 99.32% Accuracy 99.76% Accuracy 95.22%, Precision 95.59%, Recall 95.17%, F1-score 95.16% CWE-119 F1-score 91.50%; CWE-399 F1-score 90.40% SARD+NVD: Accuracy 95.71%, F1-score 96.06%, Recall 96.18% Accuracy 95.00%, Precision 99.00%, Recall 99.00% Accuracy 91.36%, F1-score 46.59% Accuracy 97.53%, Precision 92.65%, Recall 92.60%, F1-score 92.61% Accuracy 82.00%, F1-score 71.62%, AUC 86.78% CWE-264: Accuracy 100.00%, F1-score 90.91% AUC 99.73% Precision 99.00%, Recall 99.50%, F1-score 99.30% Top-1 22.30%, Top-3 37.70%, Top-5 46.30% Accuracy 96.19%, F1-score 57.34% (BigVul) Maximum absolute improvement of 35.63% in accuracy and 289.32% in F1-score over baselines across three benchmark datasets Attack-surface detection: average F1-score 93.00%; Exposure analysis: F1-score 85.71%; Flow verification: Precision 87.23%, Accuracy 95.50% QEMU: Accuracy 64.10%, Precision 64.00%, Recall 67.30%, F1score 65.60%; FFmpeg: Accuracy 63.90%, Precision 64.70%, Recall 63.50%, F1-score 64.10%; SARD: Accuracy 83.60%, Precision 88.90%, Recall 80.10%, F1-score 84.30% Accuracy 98.20% Temporal split: Precision 61.00%, Recall 51.00%, F1-score 55.00%, AUC 83.00%; Cross-project split: Precision 62.00%, Recall 64.00%, F1-score 63.00%, AUC 80.00% 99.00% Accuracy Devign: Accuracy 70.57%, Recall 59.36%, Precision 71.70%, F1-score 64.95%; Reveal: Accuracy 91.86%, Recall 39.91%, Precision 65.46%, F1-score 49.59% LLVM-IR: Accuracy 93.60% and 94.20%; Assembly: Accuracy 91.00% and 89.80%; single-compilation LLVM-IR CodeBERT: 86.60% multiclass accuracy Devign: Accuracy 69.25%, Recall 61.51%, Precision 68.38%, F1-score 64.77%; Reveal: Accuracy 91.78%, Recall 45.18%, Precision 62.42%, F1-score 52.54% Best source-code single model: Accuracy 79.00%, micro-F1 88.00%; best byte-code single model: Accuracy 77.00%, micro-F1 86.26%; best meta-classifier: Accuracy 83.46%, weighted F1 91.07% Accuracy 97.37%, Precision 96.77%, Recall 98.36%, F1-score 97.56% Main multiclass result: AUC 89.60%, Accuracy 67.10%, Precision 66.90%, Recall 66.10%, F1-score 66.30%; Juliet F1-score 89.10% Reentrancy F1-score 93.47%; Timestamp dependence F1-score 89.33%; Integer overflow F1-score 91.27% Big-Vul: F1-score 91.60% (fine-tuning), 91.40% (word-level embeddings) SARD coarse-grained: Accuracy 97.40%, Precision 96.70%, Recall 92.70%, F1-score 94.70%; fine-grained localization average IOU 84.50% FFmpeg: Accuracy 76.25%, F1-score 75.63%, MCC 52.64%; QEMU: Accuracy 89.74%, F1-score 87.48%, MCC 78.82%; FFmpeg+QEMU: Accuracy 78.66%, F1-score 76.80%, MCC 57.05%

while also showing growing interest in graph-aware and LLM-based approaches.

4.4. RQ4: What are the commonly used evaluation metrics? For improved transformer-based approaches, it is important to validate them using various machine learning Page 11 of 27

SLR for transformer-based Software Vulnerability detection Table 5 Popular Transformer Model Categories Used in Articles Transformer Backbone / Family CodeBERT and CodeBERT-based variants BERT / RoBERTa / DistilBERT family GraphCodeBERT / Graph-aware pretrained code transformers GPT / LLM family CodeT5 / PLBART / UniXcoder / other code LMs Custom Transformer Architectures

Articles [59, 88, 70, 109, 36, 32, 53, 75, 89, 91, 26, 42, 15, 50, 56, 35, 92, 44, 68, 80, 98, 25, 71, 43, 90, 82, 8] [5, 60, 108, 47, 30, 46, 58, 101, 61, 33, 37, 105, 38, 3, 21, 102, 63, 68] [100, 91, 96, 44, 68, 85, 71, 14, 65]

[76, 73, 46, 57, 79, 87, 50, 104, 98, 25, 63, 43] [59, 91, 97, 98, 25, 71, 43, 8, 85, 63]

[52, 24, 7, 65, 107, 9, 29, 40, 92, 14, 93, 86, 97, 78]

evaluation metrics. To identify the evaluation metrics commonly used in transformer-based approaches, we examine the metrics adopted in software vulnerability detection research and analyse their frequency of use in measuring vulnerability detection performances. Table 6 summarizes the evaluation metrics employed in the analysed vulnerability detection studies. The Table 6 shows that Accuracy, Precision, Recall, and F1-score are the most widely used measures for assessing model performance, reflecting their suitability for binary and multi-class vulnerability detection tasks. Several studies additionally report specificity (TNR), sensitivity (TPR), false positive rate (FPR) and false negative rate (FNR) to provide a more detailed analysis of detection errors, particularly in security-critical contexts where false alarms and missed vulnerabilities have significant consequences. To address class imbalance and provide more robust performance evaluation, some research studies employ Matthews Correlation Coefficient (MCC), Area Under the Curve (AUC), and AUROC, which capture overall predictive quality beyond accuracy alone. While fewer studies in the proposed SLR report macro- and weighted averaged metrics, macro and weighted F1-scores and precision/recall-based measures such as PRAUC. Additionally, metrics such as Top-k Accuracy and Receiver Operating Characteristic (ROC) curves are used in a limited number of studies to evaluate rankingbased predictions and threshold independent performance. For Example, Kim et al. [46] prepared several labelled ethereum smart contract datasets, addressed class imbalance through over sampling and fine tuning transformer encoders using supervised learning to capture both syntactic and semantic code patterns. They evaluate the model’s performance using standard classification metrics such as accuracy, precision, recall, F1-score, and ROC. They use the ROC curve to assess the model’s discriminative ability across different decision thresholds. Specifically, ROC plots the true positive rate against the false positive rate, and the corresponding AUC value is used to demonstrate how Naseer et al.: Preprint submitted to Elsevier

effectively the fine tuned models distinguish vulnerable contracts from non vulnerable ones, particularly under imbalanced data conditions. On the other hand, Liu et al. [57] investigates the use of GPT for software vulnerability detection without traditional model fine tuning. They use Top-K accuracy as an evaluation metric because GPT often produces a ranked list of possible vulnerability predictions rather than a single deterministic output. Top-K accuracy measures whether the correct vulnerability appears within the top K predicted candidates e.g Top-1, Top-3, Top-5, which is especially suitable for LLM based and multi class vulnerability detection tasks. This metric reflects realistic usage scenarios where security analysts can review several high confidence predictions and it better captures the practical usefulness of GPT when multiple plausible vulnerability types exist for a given code snippet. RQ4 Research Finding Most researchers use conventional metrics such as Accuracy, Precision, Recall and F1-score to evaluate experimental results. Additionally, some studies employ measures like FPR, FNR, MCC, AUC, Top-K Accuracy, and ROC to address class imbalance and provide a more detailed performance analysis. These metrics can help software researchers and vendors to evaluate their vulnerability detection approaches for more generalised results.

4.5. RQ5: What are the fine-grained software vulnerability types identified in the literature? In RQ-1, we classified the research papers into binary and fine-grained vulnerability types. However, considering the growing interest of software researchers and vendors in classifying software vulnerabilities into more fine-grained types, we examine how different studies detect various CWE vulnerability types, identify their corresponding CWE numbers and analyse the categories to which these vulnerabilities belong. Table 7 summarizes the diversity of vulnerability types identified in the surveyed articles. For each study, Table 7 lists the distinct vulnerability categories identified and maps them to their corresponding CWE [16] identifiers, to the best of our knowledge. For example, Shiaeles et al. [79] present a framework to identify 124+ vulnerabilities. They leverage pre-trained language models to detect multiple CWE vulnerability types in source code. In the first step, they tokenize and normalize code snippets to create textual representations suitable for input to the language model, which generates contextual embeddings capturing both syntax and semantic patterns relevant to vulnerabilities. Model VulDetect treats vulnerability detection as a multi classification task, where each class corresponds to a specific CWE identifier. A classification layer on top of the language model outputs probabilities over these CWE classes, allowing the model to predict one or more vulnerability types for each code snippet. During training, the model is fine Page 12 of 27

SLR for transformer-based Software Vulnerability detection Table 6 Evaluation metrics reported in the reviewed articles Evaluation Metric

Articles

Accuracy

[76, 52, 11, 5, 24, 59, 60, 7, 88, 108, 70, 109, 58, 32, 53, 75, 101, 57, 107, 41, 106, 2, 64, 35, 96, 8, 55, 103, 66, 18, 21, 63, 15, 3, 89, 91, 9, 42, 38, 40, 102, 104, 92, 44, 14, 93, 31, 51, 68, 97, 80, 98, 25, 71, 85, 78, 43, 90, 82] [95, 103, 102, 8, 21, 96, 63, 35, 56, 29, 2, 106, 50, 15, 38, 41, 42, 79, 37, 61, 26, 9, 91, 89, 57, 101, 100, 53, 32, 46, 36, 109, 70, 30, 47, 108, 7, 59, 24, 73, 5, 76, 66, 39, 92, 44, 14, 31, 51, 86, 68, 97, 80, 98, 25, 71, 85, 78, 43, 90, 82] [76, 5, 24, 59, 7, 88, 108, 47, 30, 70, 109, 36, 46, 32, 53, 100, 101, 57, 89, 107, 91, 9, 26, 61, 37, 105, 79, 42, 41, 38, 21, 15, 50, 106, 2, 29, 35, 63, 96, 8, 95, 103, 102, 40, 66, 39, 92, 44, 14, 31, 51, 86, 68, 97, 80, 98, 25, 71, 85, 78, 43, 90, 82] [76, 52, 11, 5, 73, 24, 59, 7, 88, 108, 47, 30, 70, 109, 36, 46, 58, 65, 32, 53, 100, 101, 57, 89, 107, 91, 9, 26, 61, 33, 37, 105, 79, 41, 42, 38, 15, 50, 106, 96, 2, 21, 29, 56, 35, 63, 8, 95, 103, 102, 66, 39, 92, 44, 14, 31, 51, 86, 68, 97, 80, 98, 25, 71, 85, 78, 43, 90, 82] [30, 33, 105, 2, 24, 85, 82] [76, 105, 102, 33] [105, 52, 73, 89, 33, 87, 56, 35, 76, 66] [52, 73, 89, 9, 33, 87, 56, 35, 61, 76, 5, 66, 90] [5, 79, 8, 104, 40, 66, 86, 71, 85] [30, 36, 33, 105] [73, 89, 9, 61, 33, 87, 35, 76, 5, 90] [24] [24] [59] [59] [70] [36] [46, 79] [57, 104] [31, 85] [31, 85] [85] [85] [85] [85] [43] [90]

Precision Recall F1-score

MCC Specificity (TNR) Sensitivity (TPR) FPR AUC AUROC FNR Macro avg Weighted avg Weighted F1 Macro F1 MFR PRAUC ROC Top-k Accuracy MSE MAE Kappa SP SN MK F2 IOU

tuned on labelled datasets using suitable loss functions for multi class classification, enabling it to learn CWE specific code patterns. By combining contextual embeddings with CWE based classification, VulDetect can identify a wide range of vulnerability types, including subtle or previously unseen weaknesses, in a single unified framework. Similarly, Ni et al. [66] propose a function-level vulnerability detection framework that leverages multiple representations of source code to improve detection accuracy. It integrates three complementary modalities: textual features extracted using a pre-trained code language model UniXcoder to capture semantic patterns, graph-based features from control flow and data flow structures encoded via a Graph Neural Network to capture structural dependencies and optional image or matrix-based representations to learn additional structural patterns. The embeddings from these modalities are fused through a multi-modal network to create a unified functionlevel representation, which is then fed into a classification layer to predict CWE-based vulnerability types. Table 8 indicates that the literature addresses a wide range of software security issues, with the most common categories including injection vulnerabilities, buffer or memory errors, input and

Naseer et al.: Preprint submitted to Elsevier

validation problems, access control issues, cryptographyrelated weaknesses, path or file traversal, numeric and arithmetic errors, resource management problems, race conditions, and information disclosure. The table 8 also highlights a separate group of smart-contract-specific vulnerabilities, showing that blockchain and Solidity security form an important part of recent research. Overall, the table provides a high-level view of how vulnerability research is distributed across major weakness categories and which types are most frequently studied. RQ5 Research Finding Software researchers have shown interest in detecting a wide range of fine-grained vulnerabilities in code, which are mostly categorized into nine groups based on our knowledge and understanding. Among these, Injection (SQL, OS, XSS, Code), and Buffer/Memory Errors/Corruption are the most dominant.

Page 13 of 27

SLR for transformer-based Software Vulnerability detection

Table 7 Total number of Vulnerability types detected in each article Article Saimbhi Akpinar [76]

Vulnerability Types and

Bahaa et al.[5] Purba et al.[73] Ferrag et al.[23] Lu et al.[59] Sun et al.[88] Kim et al.[47] Gujar [30] Peng et al.[70] Zhao et al.[109] Kim et al.[46] Nguyen et al.[65] Wu et al.[100] Tanko et al.[89] Thapa et al.[91] Cao [9] Mamedi [61] Hanif and Maffeis [33] Hou et al.[37] Zahid [105] kaanan et al.[42] Islam et al.[38]

Sun et al.[87] Curto et al.[15]

Alqarni and Azim [3] Gong et al.[29] Liu et al.[56] Mechri et al.[63]

Ehrenberg et al.[21] Wang et al.[95] Shiaeles et al.[79] Katz et al.[44] Shir et al.[80]

Ferretti et al.[25] Sultan et al.[85] Shang et al.[78] Tao et al.[90]

Jeon et al.[39]

Injection (CWE-79, CWE-89, CWE-77, CWE-78),Auth / Access Control(CWE-287, CWE-284, CWE-639),Cryptography(CWE-327, CWE330, CWE-338, CWE-311),Input Validation (CWE-22, CWE-434, CWE-502),Web Logic (CWE-352, CWE-601),Information Disclosure (CWE-200),Configuration (CWE-16) Injection (CWE-78), Buffer/memory corruption (CWE-121, 122, 124, 126, 127, 59), Format string (CWE-134),Numeric/Integer errors (CWE-194, 195, 197), Null pointer / memory management (CWE-690) Buffer Errors / Memory Corruption (CWE-120), Injection / SQL Injection (CWE-89) Buffer/memory corruption (CWE-119, 120, 121, 122, 787), Injection: (CWE-78), Numeric errors: (CWE-190), Input validation (CWE-20), Memory management / null dereference (CWE-476, 762) Buffer/memory corruption (CWE-119, 125, 787),Injection / XSS:(CWE-79),Input validation (CWE-20) Buffer/memory corruption (CWE-787) ,Injection (CWE-89 (SQL), 79 (XSS))CSRF (CWE-353),Path / File Traversal (CWE-22) Buffer/memory corruption (CWE-119), Resource management / DoS (CWE-399) Path / File Traversal (CWE-22),OS / Command Injection (CWE-77),Injection (CWE-79 (XSS), 89 (SQL), 94 (Code Injection)),CSRF (CWE-352),Open Redirect (CWE-601) Buffer/memory corruption (CWE-119, 125, 416, 476),Input validation (CWE- 20),Information disclosure (CWE-200), Access control / race conditions (CWE-264, 362),Integer / numeric errors (CWE-189, 190) etc Injection (CWE-78 (OS), 79 (XSS), 89 (SQL), 94 (code injection))CSRF (CWE-352),File upload / RCE (CWE-434)Information disclosure / sensitive data (CWE-200, 319),Cryptography / weak algorithms (CWE-326, 327),Open redirect (CWE-601) Arithmetic errors (CWE-189, 190, 191),Logic errors (CWE-840) series,Input / unexpected events (CWE-20, 359), Resource / timing (CWE-362, 399) Access control (CWE-284),Integer / numeric (CWE-190),Concurrency / race condition (CWE-362),Input validation / error handling (358, 252),Logic/business errors (CWE-841),Resource / unsafe function usage (CWE-829, 682) application logic CWE 841 Injection / command (CWE-78),Injection / SQL (CWE-89),Injection / XSS (CWE-79),Path / File Traversal (CWE-22),Cryptography / weak crypto (CWE-327) Buffer/memory corruption (CWE-119),Resource management / DoS (CWE-399) buffer/memory corruption issue (CWE-120) Resource Management / Denial of Service (CWE-400) others(not specified) Integer / Numeric Errors (CWE-190,191) Buffer / Memory Errors (CWE-120,121) Path / File Traversal (CWE-22),Cross-Site Scripting (XSS) (CWE-79),Command / OS Injection (CWE-77), SQL Injection (CWE-89), Code Injection (CWE-94),Cross-Site Request Forgery (CSRF)( CWE-352),Open Redirect / URL Manipulation (CWE-601) Buffer / Memory Errors( CWE-120) others(not specified) Input / Validation Errors( CWE-20,74),Resource Management / Denial of Service (CWE-400,404),Path / File Manipulation / Directory Traversal (CWE-221),Cryptography / Sensitive Data Handling (CWE-311),Integer / Numeric Errors (CWE-190,187),Information Exposure / Security Flaw (CWE-138,467,469) Access Control / Privileges (CWE-285,284,346),Integer / Numeric Errors / Calculation Errors (CWE-682),Concurrency / Race Conditions(CWE-362),Logic / Business Errors (CWE-841) Access Control / Privileges(CWE-732,284, 264),Buffer / Memory Errors ( CWE-787,125,416, 476),Input / Validation Errors(CWE254,20),Integer / Numeric Errors (CWE-190,189),Concurrency / Race Conditions(CWE-362),Resource Management / DoS (CWE399),Information Disclosure / Sensitive Data (CWE-200,199) Buffer / Memory Errors ( CWE-120, 476, 805),Resource Management / Memory Leak (CWE-401),Input / Validation Errors (CWE-469) Logic / Business Errors (CWE-841) Buffer / Memory Errors (CWE-119), Resource Management / DoS (CWE-399) Resource management / DoS (CWE-400, 703), Path / File Traversal (CWE-22), Command / OS Injection (CWE-78), Cryptography / Weak Algorithms (CWE-327, 330), SQL Injection (CWE-89), Authentication / Credential Management, CWE-259, Input / Validation Errors(CWE-20), Concurrency / Race Conditions (CWE-377), Certificate / Trust Issues (CWE-295) Buffer / Memory Errors (CWE-787) ,Cross-Site Scripting (XSS)( CWE-79),SQL Injection (CWE-89),Input / Validation Errors (CWE20),Path / Directory Traversal (CWE-24) Injection (CWE- 78 (OS), 89 (SQL), 94 (code), Web vulnerabilities: 79 (XSS), 352 (CSRF), 601 (Open Redirect),File / resource access: 706 Library / API Function Calls (CWE-119, 120, 665, 672, 676),Array Usage (CWE-125, 126, 127, 787, 20),Pointer Usage (CWE-476, 457, 590, 761, 119/787),Arithmetic Expression (CWE-189, 190, 191, 680, 840/845) Information Disclosure / Sensitive Information Exposure (CWE-200, 201, 203, 204, 208, 209, 214, 215, 532, 535, 536, 537, 538, 550, 598, 615) Buffer / Memory Errors (CWE-121, 122, 124, 126, 127), Integer / Numeric Errors (CWE-190, 191, 194, 195, 197, 680), Resource / Memory Management (CWE-401, 415, 590, 690, 762), Input / Path / Command Issues (CWE-23, 36, 78, 134), Logic / Runtime / Validation Errors (CWE-369, 400, 457) Smart Contract Vulnerability Classes: Access Control, Arithmetic, Reentrancy, Unchecked Calls, Other Buffer / Memory Errors (CWE-119, CWE-120), Pointer / Reference Errors (CWE-476), Pointer Subtraction / Addressing Issues (CWE469), Other / Miscellaneous Vulnerabilities (CWE-other) Reentrancy (CWE-841), Timestamp Dependence (CWE-829), Integer Overflow (CWE-190) Injection Vulnerabilities (CWE-78, 88, 89), Buffer / Memory Errors (CWE-124, 127, 129, 416, 476, 805), Integer / Numeric Errors (CWE-190, 191, 195), Resource Management / DoS (CWE-400, 789, 835), Race / Concurrency / TOCTOU (CWE-363, 367), Resource Lifetime / Handle Issues (CWE-773, 775), Other Memory / Runtime Weaknesses (CWE-401) Gas exhaustion (CWE-400), Unchecked function call (CWE-252), balance Equality check point (CWE-697), Incorrect return type (CWE628 / CWE-704), , Misuse of visibility ( CWE-284), Array length manipulation (CWE-400 / CWE-770), Use of insecure math functions (CWE-190 / CWE-191), Locked ether ( CWE-667 / CWE-703), Data leakage when using private (CWE-200), Misuse of approve function in ERC20 library ( CWE-362), Misuse of var (vCWE-190 / CWE-681), Misuse of multiple return values in internal/private functions ( CWE-393 / CWE-703), Misuse of transfer function in loop ( CWE-400), Misuse of inline assembly (CWE-710), Hardcode of the address ( CWE-798), Deprecated constructions ( CWE-477), False return of ERC20 (CWE-252), Misuse of revert require ( CWE-670)

Naseer et al.: Preprint submitted to Elsevier

Page 14 of 27

SLR for transformer-based Software Vulnerability detection Table 8 Major types of vulnerabilities covered in the reviewed articles Vulnerability Description

Articles

Injection (SQL, OS, XSS, Code, Command) Buffer / Memory Errors / Corruption Input / Validation Errors

[76, 5, 73, 23, 59, 88, 30, 109, 89, 105, 95, 26, 70, 63, 90] [5, 73, 23, 59, 88, 47, 91, 9, 26, 3, 56, 79, 42, 70, 15, 80, 85, 90] [76, 23, 59, 70, 65, 26, 21, 38, 15, 3, 63] [76, 65, 87, 15, 25, 71]

Access Control / Privileges / Authentication Cryptography / Weak Algorithms / Sensitive Data Path / File / Directory Traversal Cross-Site Request Forgery (CSRF) Numeric / Arithmetic / Integer Errors Resource Management / Denial of Service Race Conditions / Concurrency / TOCTOU Information Disclosure / Exposure Smart Contract Specific Vulnerabilities Logic / Business / API / Miscellaneous

[76, 109, 89, 38, 63, 44] [76, 88, 30, 89, 26, 105, 63, 21, 38, 95, 80] [76, 88, 30, 109, 105, 95] [5, 23, 46, 65, 33, 15, 38, 79, 70, 80, 78, 90, 39] [47, 91, 61, 56, 15, 38, 63, 3, 46, 80, 90, 39] [70, 46, 65, 87, 15, 63, 90, 39] [76, 70, 109, 38, 15, 44, 39] [25, 71, 78, 39] [100, 29, 79, 63, 46, 65, 87, 39]

4.6. RQ6: What Hyper-parameters and environment settings are typically used? To optimise existing transformer-based vulnerability approaches for improved software vulnerability detection and classification, software researchers employ different hyperparameters to fine-tune transformer models. Although hyperparameters are algorithm and dataset-dependent. However, in this section, we critically review the hyperparameter configurations employed in the existing transformer-based studies and discuss the corresponding experimental environment settings. These frequently hyperparameter settings can be employed to evaluate the performance of newly developed transformer-based approaches to identify and classify software vulnerabilities, as it is considered one of the important steps in improving their performance. Furthermore, it can help software researchers and vendors to identify frequently used experimental setups required for source-intensive vulnerability detection. Table 9 presents the hyperparameter and experimental environment setups used in the surveyed articles. For each study, we list the key training hyperparameters along with the hardware and software environments used in the experiments. Hyperparameters are critically important in ML and DL because they directly influence how a model learns from data, how well it generalizes and how efficiently it trains. For Example, Mechri et al. [63] employ hyperparameters extensively to optimize LLM for identifying CWE based vulnerabilities. They use pre trained CodeQwen models and

Naseer et al.: Preprint submitted to Elsevier

configures model specific hyperparameter such as 32 decoder layers, 4 attention heads, a hidden size of 4096, an intermediate MLP size of 13,440, and maximum position embeddings of 65,536 to define the model’s capacity. Training hyperparameters include a learning rate of 0.0001, the Adam optimizer with standard betas, batch sizes of 4 for training and 8 for evaluation and one epoch. The model predicts 15 CWE vulnerability classes using a classification head, and distributed multi GPU training with 16 NVIDIA A100 40GB GPUs is employed to handle memory and computational demands. These hyperparameters, spanning model architecture, training, and distributed execution, are crucial for ensuring high accuracy, F1-score, and reliable multi class vulnerability detection in Python codebases. Similarly, Chen and Liu [11] divide code into statements, functions, files, and then encodes each level using transformer encoders to capture both local and global contextual information. These embeddings are aggregated through a hierarchical attention mechanism, allowing the model to combine fine grained and coarse grained features for vulnerability prediction. For better performance, stable convergence and high detection accuracy across different code structures; they tuned hyperparameters such as learning rate, batch size, number of transformer layers, hidden size, attention heads, and training epochs. Table 10 lists the most important hyperparameters for transformer models and their typical ranges. For each hyperparameter, the table provides a brief description and the values commonly used in experiments. Overall, the Table 10 provides a concise overview of the configuration ranges for transformer models, helping guide experimental setup and reproducibility. We believe these values are not standard and can vary across approaches, depending on the nature of the software vulnerability data. However, these values can serve as a baseline, as they are frequently reported in the literature for software vulnerability detection and classification. Moreover, the experimental environment can help software researchers and developers identify in advance the resources required for vulnerability detection and classification by considering transformer-based algorithms and datasets before actually running the experiments. RQ6 Research Finding Researchers explore a wide range of hyperparameters for fine-tuning their models and environment settings. The most important hyperparameters are the learning rate, batch size, number of transformer layers, hidden size, number of attention heads and training epochs.

4.7. RQ7: At what level of granularity is the vulnerability detection performed? For more fine-grained analysis of software vulnerability detection, this RQ explores the granularity levels at which researchers identify vulnerabilities in their experiments.

Page 15 of 27

SLR for transformer-based Software Vulnerability detection Table 9 Hyperparameters, fine-tuning, pre-training, and platform/environment setup reported in the reviewed articles(Part 1) Reference Saimbhi Akpinar [76]

and

Li et al. [52] Chen and Liu [11] Bahaa et al. [5] Purba et al. [73] Ferrag et al. [24] Lu et al. [59] Mahyari [60] Sun et al. [88] Zhao and Liu [108] Kim et al. [47] Hanif and Maffeis [33]

Hou et al. [37]

Zahid [105] Shiaeles et al. [79] Kaanan [42] Jianjie and Le [41] Islam et al. [38] Sun et al. [87] Curto et al. [15] Jeon et al. [39]

Alqarni and Azim [3] Le et al. [50] Li[51] Sun et al.[86] Oladokun & Rice [68] Wang et al.[97] Shir et al.[80] Wang et al.[98] Ferretti et al.[25] Perera et al.[71] Sultan et al.[85] Shang et al.[78] Kalouptsoglou et al.[43] Tao et al.[90] Smaili et al.[82]

Hyperparameters / Fine-tuning / Pre-training N/A

Adam optimizer, learning rate 0.0001, 5 epochs, batch size 32 RAdam optimizer, learning rate 0.001, 6 encoder layers, 6 multi-head selfattention heads Adam optimizer, learning rate 3e-5, binary cross-entropy loss, 20 epochs, batch size 20 28 layers, 16 attention heads, batch size 8, 4 epochs AdamW optimizer; learning rate 1.85e-4 (warm-up); Z-loss 1e-4; batch size 256; later effective batch size 1152; 100 billion tokens AdamW optimizer, learning rate 5e-5 30 epochs, learning rate 0.01, batch size 64 AdamW optimizer, batch size 16, input length 512, learning rate 2e-5, 12 attention heads 4 encoder layers, dropout 0.2, 4 attention heads, learning rate 1e-3, 50 epochs, batch size 16 24 transformer layers, hidden size 1024, 16 self-attention heads Hidden layer with 768 neurons; output layer with 2 or 41 neurons; 10 epochs; learning rate 3e-5

Learning rate {0.01, 0.001, 0.02}, batch size {64, 128, 256}, embedding size {64, 128, 256}, number of heads {2, 4, 8}, transformer blocks {1, 2, 4} 24 attention heads, hidden size 768 N/A Maximum input length 512 tokens Batch size 512, later modified to 1 × 1024 Word embedding size 768, hidden layer 128, learning rate 5e-4, 100 epochs, batch size 512, token length 400 4k context token size Single linear layer with 15 output neurons, 10 epochs, learning rate 2e-5 Learning rate search {1e-6, 1e-5, 1e-4, 1e-3}; batch sizes {16, 24, 32}; epochs {2, 3, 4}; final setup: learning rate 1e-5, batch size 24, 3 epochs, random seed 2018, Adam optimizer, cross-entropy loss, BERT-base with 12 transformer blocks, hidden size 768, 12 attention heads Learning rate 2e-5, batch size 64, 10 epochs

N/A 32 NVIDIA A100 40 GB GPUs OpenPrompt server with NVIDIA GeForce RTX 4090 N/A N/A PyTorch on a server equipped with NVIDIA RTX 3090 GPU PyTorch, NVIDIA GeForce Titan PyTorch 1.7 with CUDA 10.2 on Python 3.7. Pretraining: GCP VMs with 48 vCPUs, 240 GB RAM, 2 NVIDIA Tesla A100 40 GB GPUs. Fine-tuning: 48-core Intel Xeon Silver CPU, 292 GB RAM, 2 NVIDIA GTX TITAN Xp GPUs (12 GB each) ANTLR4 for AST; NVIDIA GeForce GTX 1080

N/A ASUS TUF Gaming laptop with Intel Core i7 8th-generation CPU; 6 cores; max frequency 2.2 GHz N/A N/A 8 DGX-A100 NVIDIA GPUs; training and testing took 4–6 hours N/A NVIDIA RTX A6000 GPU Intel Xeon Silver 4114 2.20 GHz CPU, 256 GB RAM, NVIDIA GeForce Titan RTX, PyTorch

Intel Core i9 processor, Tesla K80 GPU, 12 GB RAM

10 epochs, learning rate 1e-5, feature embedding size 768 Batch 16, epochs 10, lr 2e-5, AdamW, cross-entropy 4 HGT layers, hidden 768, 8 heads, dropout 0.2, batch 64, lr 1e-4, Adam, max 30 epochs, early stop 4 epochs; pretrained on 6 PLs; other hyperparameters not stated

N/A RTX 3070 8GB, Ubuntu 20.04.6, PyTorch Ubuntu 18.04, Tesla T4 16GB, PyTorch, DGL

Batch 12, lr 2e-5, seed 123456, 8 Transformer layers, 8 heads, p = 20, k = 100 Batch 16, lr 1.1e-5, weight decay 3e-4, 30 epochs, early stop, CodeBERTbase, 512-token model limit, 800-token function threshold 4 epochs, max length 1024, batch 12, lr 2e-5, grad clip 1.0, seed 42, temp 0 Adam, lr 1e-5, batch 8, dropout 0.3, BCEWithLogitsLoss, GridSearchCV Best: 3 layers [256,128,128], 4000 epochs, Sigmoid, Adam Emb dim 768, hidden 512, out 768, 2 GCN layers, MSE alignment Adam, cross-entropy, 50 epochs, 5-fold CV Fine-tune LR 2e-5 AdamW max len 512; best Big-Vul classifier BiGRU

Not explicitly stated

Rusinova et al. [75] Wu et al. [101]

Emb dim 768, dropout 0.1, batch 20, LR 5e-5, Adam LR 1e-3, batch 64, max 50 epochs, early stopping 6, L2 1e-4, Q/K/V 128, hidden 256 12 encoder blocks, hidden size 768, feed-forward/filter size 3072, model sizes 14.5M and 125M parameters Learning rate 0.0015, token dictionary size 20000, 6 encoder blocks

Liu et al. [57]

N/A

Zhang et al. [106]

N/A

Almakayeel [2]

Learning rate 0.01, ReLU activation, 50 epochs, dropout 0.5, batch size 5

Myllari et al. [64] Gong et al. [29]

Context length 256; sliding window length 𝑤 = 4 6 layers, embedding dimension 512, hidden dimension 512, learning rate 0.0001, Adam optimizer, batch size 16 N/A

Liu et al. [56]

Platform / Environment Setup Lenovo IdeaPad with AMD Ryzen 5 3500U processor, 8 GB RAM, 256 GB SSD; Python 3.10.4, Pandas library, OpenAI tools NVIDIA GeForce RTX 8 GB GPU with 16 GB RAM Server with Tesla P100 SXM2 16 GB; PyTorch 1.8.0; 8 NVIDIA P100 GPUs NVIDIA A100 GPU with 34 GB RAM

Naseer et al.: Preprint submitted to Elsevier

Google Colab, NVIDIA A100, HF Transformers, PyTorch

Linux x86-64, ARM/x86-64, GCC/Clang, GPU-enabled 3× V100 32GB GPUs 2-core AMD VM, RTX 2080 Ti 12GB Google Colab, Tesla K80, 12.7GB RAM Not explicitly stated Xeon 3204, RTX 4070, 128 GB RAM RTX 4080 Super GPU Threadripper 3960X, RTX 4090D, 32GB RAM RTX 4070, 32 GB RAM, Ryzen 5700X, PyTorch N/A Ubuntu Linux 18.04; NVIDIA GeForce RTX 2080Ti GPU; Intel Xeon Silver 4214 CPU @ 2.20 GHz GPT-3.5-turbo; 11th Gen Intel Core i7-11800H @ 2.30 GHz; 32 GB RAM; Windows 11 Linux server; 20-core CPU, 128 GB RAM, NVIDIA Tesla P4 GPU; Python 3.9.18 and PyTorch 1.9.0 Python 3.6.5 on PC with Intel i5-8600K, 250 GB SSD, GeForce 1050Ti 4 GB, 16 GB RAM, 1 TB HDD Python 3, PyTorch Ubuntu 18.04; TITAN RTX GPU; 32 GB RAM; Intel Xeon Gold 5120 CPU @ 2.20 GHz; PyTorch GNU GCC 4.8.2 compiler supporting Intel x86, x64, and ARM; Windows GCC (MinGW) and Linux GCC at optimization levels -O0, -O1, -O2, -O3; Windows 10 Enterprise with Intel Xeon W-2133 CPU @ 3.60 GHz and 32 GB memory; Ubuntu 16.04.3 with 16 vCPUs and 32 GB memory

Page 16 of 27

SLR for transformer-based Software Vulnerability detection Table 9 Hyperparameters, fine-tuning, pre-training, and platform/environment setup reported in the reviewed articles (Part 2) He et al. [35] Mechri et al. [63]

Wang et al. [96] Ehrenberg et al. [21] Cao and Dong [8] Liu et al. [55]

Jiang et al. [40]

Wu et al. [102] Yang et al. [104]

Xuan et al. [103]

Wang et al. [95] Ni et al. [66]

Do et al. [18] Gujar [30] Peng et al. [70]

Hin et al. [36] Zhao et al. [109]

100 epochs, cross-entropy loss, ReLU activation, AdamW optimizer, batch size 32, learning rate 0.001, dropout 0.5 Pretrained model CodeQwen1.5-7B; BOS token ID 2; EOS token ID 2; hidden activation SiLU; hidden size 4096; intermediate size 13,440; max position embeddings 65,536; model type Qwen2; 32 decoder layers; 4 key-value heads; RMSNorm epsilon 1e-5; RoPE theta 1,000,000; dtype bfloat16; cache enabled; vocab size 92,416; learning rate 1e-4; 15 labels; train batch size 4; eval batch size 8; seed 42; total train batch size 32; total eval batch size 64; Adam optimizer with betas (0.9, 0.999), epsilon 1e-8; 1 epoch N/A Learning rate 2e-5, 5 epochs, batch size 8, AdamW optimizer; 2–3 h per training session Explored all possible hyperparameter combinations (not specified) Embedded 768-dimensional vector; FCDS as 1024dimensional vector; 3 dense layers; dropout 0.5; batch size 8; ReLU; 20 epochs Embedding dimension 768, input length 512, 12 attention heads, 12 hidden layers, contrastive learning loss 0.05, 10 epochs 12 attention heads, learning rate 5e-4, transformer block size 2048 Max learning rate 5e-6, min learning rate 1e-8, model dimension 256, 8 layers, batch size 64, 2000 epochs Encoder: max length 512, encoding dimension 768, output dimension 512, MLP layers 4; KA: 12 layers, feedforward dimension 768, output dimension 512; training epochs 9, gradient accumulation steps 12, learning rate 3e-5 12 layers, 12 self-attention heads, attention head size 64, hidden size 768, intermediate size 3072, 125M parameters Tested hyperparameter values 0, 0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1, 2, and 5 Token length set to 32 or 64 AdamW optimizer, learning rate 1e-5, 100 epochs Learning rate 2e-5, AdamW optimizer, cross-entropy loss, batch size 16, 10 epochs, input size 512, hidden dimension 768, output [0, 1], 12 attention heads, dropout 0.1 N/A

Wu et al. [100]

Epochs 1–10, hidden size 768, 12 hidden layers, 12 attention heads, batch size 16 Input feature 1, padding 256, batch size 8, AdamW optimizer, 10 epochs, learning rate 1e-5 10 epochs, batch size 128, learning rate 1e-4 Embedding size 128; learning rate 0.0005–0.01 for coarsegrained classification and 0.0002–0.005 for fine-grained classification; 8 attention heads; hidden size 128; 100 and 50 epochs Adam optimizer, batch size 32, 50 epochs, learning rate 0.001, input shape (code) (2048,1), input shape (text) (2048,1), dense layer units 64, dropout 0.3 Adam optimizer; learning rates 1e-4, 1e-3, 1e-2; hidden channels 64–512; number of heads 1–8 N/A

Tanko [89]

N/A

Zhang et al. [107]

Learning rate 0.001, batch size 64, maximum 50 epochs, 1 transformer encoder layer, word length 512 110M parameters, 12 layers, hidden size 768, 12 attention heads N/A

Kim et al. [46] Liu et al. [58] Nguyen et al. [65]

Gupta et al. [32]

Liang et al. [53]

Thapa et al. [91] Cao [9] Fu and Tantithamthavorn [26] Mamede [61] Tian & Zhang[92]

Katz et al.[44]

12 Transformer encoder blocks, hidden size 768, 12 attention heads BERT maximum length 512 tokens Pre-trained CodeBERT; 256-dim CodeBERT output; 128dim Word2Vec; 10 attention heads; 256 hidden dim; lr 0.0001; early stopping LR {1e-5, 1e-4, 1e-3, 1e-2}, dropout {0.2, 0.3}, activations {ReLU, ELU, Sigmoid}, batch {32, 64}, epochs {50, 60}, 50-trial random search, early stopping, 5-fold CV, pretrained transformer models

Naseer et al.: Preprint submitted to Elsevier

128 GB RAM, 20-core CPU, NVIDIA GeForce RTX 4090 Multi-GPU distributed setup; Transformers 4.41.1; PyTorch 2.1.0; 16× A100 40 GB GPUs

Ubuntu 18.04; Intel Xeon Gold 6130 CPU @ 2.10 GHz; 80 GB RAM; NVIDIA Tesla V100 32 GB GPU; Python 3.8; PyTorch 1.7.0; CUDA 11.0 N/A Two Intel Xeon Gold 6230R CPUs; NVIDIA GeForce RTX 3090 GPU; Python 3.8; PyTorch 1.9.0 N/A

Hugging Face Transformers, PyTorch 2.0; Ubuntu 20.04 server with 2 Intel Xeon 4216 CPUs, 256 GB memory, 8 RTX 3090 GPUs Python 3.11, PyTorch 2.2.1, Intel Gold 6148 Skylake @ 2.4 GHz CPUs, NVIDIA V100 GPUs Intel Xeon 6248R CPU @ 3.00 GHz running Debian GNU/Linux and a single NVIDIA Quadro RTX 8000 GPU; largest model LLMAO with CodeGen16B NVIDIA Tesla T4 GPU with 2,560 CUDA cores and 16 GB GDDR6 memory

Linux 5.11.0 on AMD EPYC 7502P 32-Core Processor @ 3.31 GHz with 128 GB RAM and Tesla T4 16 GB GPU 32-core workstation with Intel Xeon Platinum 8358P CPU @ 2.60 GHz, 768 GB RAM, 4× NVIDIA GeForce RTX 2080 GPUs, Ubuntu 20.04.6 LTS; Python libraries: Transformers and PyTorch N/A N/A Ubuntu 20.04 server with 251 GB memory, 32-core CPU, two Tesla V100 32 GB GPUs; Joern, Python, PyTorch, Transformers, Captum, Scikit-learn, Pandas Computing cluster with multiple NVIDIA Tesla V100 GPUs and Xeon E52698v3 CPUs @ 2.30 GHz N/A NVIDIA Tesla V100 DGX, 32 GB memory; NVIDIA-SMI 450.142.00; CUDA 11.7 Pre-trained on 4 NVIDIA RTX 3090 GPUs N/A

Google Colab; Intel Xeon CPU; Tesla T4 GPU with 16 GB memory; 51 GB RAM Linux server with 128 GB memory, 16-core Intel Xeon processor, NVIDIA RTX A4000 GPU with 16 GB VRAM Ubuntu 18.04 system, 64 GB RAM, Intel i7-9700 CPU, NVIDIA 1080Ti GPU Intel Core i5 CPU, V100 GPU, 32 GB memory, 2 TB disk, Windows 11, Python 3.9.18, Pandas 2.1.3, PyTorch 2.1.0, Scikit-learn 1.3.0, Joern 2.0.201 Windows 10, 32 GB RAM, GeForce RTX 3060 GPU, PyTorch GPU internal RAM e.g. 16 GB; GPT-2 with 1.5B parameters also discussed PyTorch 1.9.0, NVIDIA Quadro RTX 6000 GPU, Intel Xeon Gold 6126 CPU NVIDIA RTX 3090 GPU

VDet for Java, a VS Code extension RTX 3090, CUDA 12.7, PyTorch 1.8.1

N/A

Page 17 of 27

SLR for transformer-based Software Vulnerability detection Table 9 Hyperparameters, fine-tuning, pre-training, and platform/environment setup reported in the reviewed articles (Part 3) Cui et al.[14] Vanam et al.[93] Gunda et al.[31]

Batch 128, lr 0.001, dropout 0.2, BCE + CosineSimilarity + BCE loss SFOA tuning, cross-entropy loss, 100-epoch training curves; numeric hyperparameters not given Pretrained CodeBERT, BCE + MSE, Adam, threshold 0.4, Gaussian noise, 10 epochs

Linux for Joern; hardware not stated N/A N/A

Table 10 Most important transformer hyperparameters and their observed ranges in the reviewed articles Hyperparameter

Description

Common Value

Observed Range

d model / Hidden Size

Dimensionality of hidden representations or embeddings used in transformer or related encoder modules. Number of complete passes over the training set. Number of samples processed in one training step. Step size used by the optimizer to update model weights. Size of tokenizer vocabulary; often inherited from pretrained models and not always reported. Regularization rate used to reduce overfitting during training. Number of parallel self-attention heads in transformer layers. Depth of the network, usually transformer encoder/decoder blocks or related stacked layers. Maximum number of tokens or context size accepted by the model. Regularization strength applied to model weights during training. Number of steps accumulated before one optimizer update; used to simulate larger batches. Maximum norm/value used to stabilize training updates. Fixed seed used for reproducibility across runs. Size of token or feature embeddings before deeper processing. Size of inner feed-forward network in transformer blocks or related modules. Final projected feature dimension used before classification or fusion. Algorithm used to update model parameters during training. Objective function optimized during training.

768

128 – 4096

10 16–32 2e-5 to 1e-4 Pretrained / not reported 0.2 – 0.3

1 – 2000 4 – 1152 1e-6 – 0.02 20,000 – 92,416

12

1 – 24

12

1 – 32

512

32 – 65,536

1e-4 to 3e-4

1e-4 – 3e-4

12

12

1.0 42 / 2018 768

1.0 42 – 123456 64 – 768

3072

768 – 13,440

512

512 – 768

Adam / AdamW Cross-entropy

Adam, AdamW, RAdam

Epochs Batch Size Learning Rate Vocabulary Size Dropout Number of Attention Heads Number of Layers Maximum Input Length / Token Length Weight Decay / L2 Regularization Gradient Accumulation Steps Gradient Clipping Random Seed Embedding Dimension Feed-forward / Intermediate Size Output Dimension Optimizer Loss Function

Activation Function Early Stopping Cross-validation

Non-linear function used in hidden/output layers. Stops training when validation performance stops improving. Repeated train-validation splitting for robust evaluation and tuning.

Table 11 presents the levels of granularity at which previous research has identified vulnerabilities in datasets. Granularity of vulnerability refers to the degree of detail or precision in presenting and analysing a security vulnerability. We compiled this information based on details from research articles and our own understanding. Most of the studies focus on a single granularity level; however, a few of them, such as [60, 65, 53, 26, 61, 41, 87, 3, 50, 55, 95, 103], identify vulnerabilities at multiple levels. For example, Nguyen et al. [65] propose a framework that models Naseer et al.: Preprint submitted to Elsevier

ReLU Used in several studies 5-fold CV

0.1 – 0.5

Cross-entropy, Binary Cross-Entropy, BCEWithLogitsLoss, BCE+MSE, BCE+CosineSimilarity +BCE, MSE alignment, Contrastive loss, Z-loss ReLU, ELU, Sigmoid, SiLU Present / not reported 5-fold CV

smart contracts as heterogeneous graphs, in which nodes represent entities such as functions, variables, statements, and contracts, and edges capture relations such as function calls, data dependencies, and control flow. Each node and edge is embedded with features encoding its type and semantic role, allowing the model to distinguish different interactions within the contract. A heterogeneous graph transformer (HGT) is applied to these graphs, using type specific multi head attention to aggregate contextual information from neighbours while respecting node and edge types. By Page 18 of 27

SLR for transformer-based Software Vulnerability detection Table 11 Vulnerability granularity levels in the reviewed articles Category

Count

Articles

Function Level

40

Statement Level

9

Line-Level

13

Task Workflow Level Binary Code File/Program Level Slice Level Code Gadget Contract-Level Code Snippet-Level Feature Level Log Entry Level Log Sequence-Level Commit-Level Flow/Program Analysis Level App Level

1 2 3 2 2 5 4 2 1 1 1 1

[11, 73, 24, 59, 7, 88, 108, 109, 46, 32, 100, 89, 9, 26, 33, 79, 41, 87, 18, 39, 3, 50, 29, 35, 63, 96, 8, 55, 103, 95, 66, 14, 31, 51, 80, 98, 85, 43, 82] [30, 36, 58, 61, 105, 42, 38, 87, 90] [60, 70, 65, 75, 57, 26, 15, 3, 50, 55, 104, 103, 97] [106] [56, 40] [52, 60, 93] [5, 107] [47, 91] [53, 65, 25, 71, 78] [37, 21, 95, 92] [2, 76] [64] [102] [86] [44]

1

[68]

leveraging these node embeddings and a readout function, MANDO-HGT can detect vulnerabilities at multiple granularity levels, including function level and contract level, enabling fine grained identification of unsafe operations as well as coarse grained assessment of vulnerable functions or contracts. Their model is trained with multi level supervision on labelled smart contract datasets, allowing it to capture complex structural and semantic patterns and achieve robust vulnerability detection across various levels of granularity. Similarly, Xuan et al. [103] propose a methodology that leverages pretrained LLMs to detect software vulnerabilities at multiple levels of granularity, including line- and function-level. The approach treats vulnerability detection as a hierarchical classification problem, enabling the model to capture both fine grained and coarse grained vulnerability patterns. Figure 4 presents a top-down breakdown of software granularity, along with the number of research articles reporting vulnerabilities at each level. According to the best of our knowledge, levels progress from conceptual and process level views (applications and workflows), to the functional structure (features and functions) and finally to low level execution details (statements and binary code).

Naseer et al.: Preprint submitted to Elsevier

RQ7 Research Finding Researchers detect vulnerabilities at fine grained and coarse grained levels. While over 12 studies have explored multi granularity detection, most focus on single granularity detection, with function level vulnerability detection being the most dominant.

4.8. RQ8: What baseline models are used for comparison in these studies? For software researchers, it is always challenging to identify baseline approaches to compare against their proposed transformer-based approach. For this purpose, we evaluated existing transformer-based approaches to identify the baseline methods commonly used by researchers to evaluate and compare their methods. Table 12 summarizes the baseline models and approaches used in the surveyed articles for software vulnerability detection. Baseline is a reference method or model used for comparison to evaluate the performance of a new approach. It provides a point of reference so researchers can judge whether their approach is actually improving performance in software vulnerability detection or classification. For example, Islam et al [38] combines transformer based source code representation with a semantic vulnerability graph (SVG) to detect software vulnerabilities more accurately. The approach constructs a graph capturing semantic relationships between code elements, such as function calls, variable dependencies and control flow interactions, which is then integrated with a transformer encoder to generate contextual embeddings of code snippets. To evaluate their method, they use baseline comparison techniques including BiLSTM, TextCNN, RoBERTa, CodeBERT, Devign, VulDeeP-ecker, and VELVET. By comparing against these baselines, they demonstrate that combining semantic graph information with transformer embeddings significantly improves performance metrics such as accuracy, precision, recall and F1-score, highlighting the effectiveness of their approach in capturing subtle and complex vulnerability patterns in source code. Similarly, Cao and Dong [8] use pretrained code models to detect vulnerabilities across different programming languages and domains. In the first step, they pretrain a language model on multiple source code repositories to capture general code semantics and syntactic patterns. After that, source code from various domains is tokenized and represented using the pretrained embeddings, which are fed into a classification layer to predict vulnerabilities. To enable cross domain detection, the model incorporates domain adaptation techniques, aligning features from source and target domains to handle variations in coding styles and domain specific constructs. For evaluation, the study uses baseline methods such as single domain pretrained models, standard transformer models without cross domain adaptation and classical machine learning classifiers like SVM or LSTM-based models. Comparisons against these baselines demonstrate that the multi source pretrained model Page 19 of 27

SLR for transformer-based Software Vulnerability detection Table 12 The baseline Approaches Used to Evaluate Vulnerability Approaches Article Saimbhi and Akpinar [76] li et al [52] Chen and liu [11] Bahaa et al. [5] Purba et al. [73] Ferrag et al. [24] lu ey al. [59] Mahyari [60] Bui et al. Sun et al. [88] Zhao and liu [108] Kim et al.[47] Hanif and Maffeis [33] Hou et al. [37] Zahid [105] Shiaeles at al.[79] Kannan [42] Jianjie and le [41] Islam et al.[38] Sun et al.[87] Curto et al.[15] Jeon et al.[39] Alqarni and Azim [3] Le et al. [50] Zhang et al. [106] Almakayeel [2] Gong et al.[29] Liu et al. [56] He et al.[35] Mechri et al.[63] Cao [9] Wang et al. [96] Ehrenberg et al. [21] Cao and Dong [8] Liu et al. [55] Jiang et al. [40] Wu et al. [102] Yang et al. [104] Xuan et al. [103] Wang et al. [95] Ni et al.[66] Do et al.[18] Gujar [30] Peng et al. [70] Hin et al. [36] Kim et al. [46] Liu et al.[58] Nguyen et al.[65] Gupta et al.[32] Liang et al.[53] Wu et al.[100] Rusinova et al. [75] Wu et al.[101] Liu et al.[57] Tanko [89] Zhang et al.[107] Thapa et al.[91] Fu & Tantithamthavorn [26] Tian & Zhang[92] Katz et al.[44] Cui et al.[14] Gunda et al. [31] Li [51] Sun et al.[86] Oladokun & Rice [68] Wang et al. [97] Shir et al. [80] Wang et al. [98] Ferretti et al. [25] Perera et al. [71] Sultan et al. [85] Shang et al. [78] Kalouptsoglou et al. [43] Tao et al. [90] Smaili et al. [82]

Baseline GPT-4 vs GPT-3.5, GPT-4 vs GPT-4 Turbo CNN, LSTM, Bi-LSTM, and CNN-LSTM. Transformer and Embedding-Transformer SOTA approaches: VulDeePecker, Flawfinder, Static Analysis,RATS ,Checkmarx VulDeePecker,CodeGen,Davinci RoBERTa, BERT,CodeBERT etc CodeBERT, CodeT5, and CodeGPT. (Each Model fine tunning campare with prompt tunning) LSTM, VulDeeLocator Russell, VulDeePecker,SySeVR,Devign LineVUL,Reveal,Devign,SySeVR Siamese, TokenCNN,VulDeePecker,CodeBERT BiLSTM Baseline-BiLSTM, Baseline-TextCNN VulDeepecker AE-KNN LineVD LSTM, CodeBERT, yseVR and VulDeBERT RoBERTa,CodeBERT,EnsembleLLM,GPT-2 BiLSTM, TextCNN,GCNN,Devign BiLSTM,TextCNN,RoBERTa,CodeBERT,Devign, VulDeePecker,VELVET GPT LineVul SVM ,Eth2Vec ,DR-GCN RNN+LSTM, Bi-LSTM C/C++ language results PCA,SVM, deeplog RHSODL-AMD , AAMD-OELAC ,GBWODL-AMC , DBN , J48 ,Naïve–Bayes ,Linear-SVM,LSTM classifier , KNN , RF, CNN encoder-RNN , DNN , CNN classifier, Dense model Mythril, Smartcheck , Oyente,Slither, Securify, Manticore, Peculiar,BLSTM-ATT, TextCNN, CGE Ins2vec-TCNN, VDiscover,A-BiLSTM,SAFE,SAFE+ROS VulDeePecker,SySeVR,Devign, VulCNN, IVDetect, and mVulPreter VulDeePecker„Devign,SecureFalcon etc Cppcheck ,SyseVR,Deepwukong ReGVD,CodeBERT ContraBERT ,GraphCodeBERT compare results of models with each other SySeVR, devign,CodeBERT, CodeBERT-mmd, CodeBERT-dann Devign , SySeVR,REVEAL Gemini, Asm2Vec , SAFE,jTrans ,Trex ,Asteria-Pro K-Nearest Neighbor (KNN), Decision Tree (DT), and Multi-layer Perception (MLP) DeepFL , DeepRL4FL , and TRANSFER-FL GRD,AMPLE,Reveal,MAGNET,LIVABLE,SySeVR,DeepWukong, LineVul,ReVeal (re), IVDetect Compare models with each other UniXcoder,IVDetect,Devign,Reveal,LineVD,LineVul,VulCNN, VulDeeLocator,Cppcheck CodeT5,Bert GCN,GAT SAC,IG,Saliency,Deeplift,Deeplift Shap,GuidedBackprop,InputxGradient IVDetect Different dataset Bi-LSTM, Transformer,VulDeePecker ,Devign, SySeVR, ReVeal etc GNNs etc Bert,CodeBert,GraphCodeBert,TextCNN,TextGCN, Devign(AST),DL-VulBERT Checkmark,Flawfinder,RATS,LineVul,LineVD,DeeplineDP ,VulSniper Honeybadger,Manticore, MythrilOsiris ,Oyente, Securify, Slither ,Smartcheck, DR-GCN, TMP LIME and SHAP BLSTM, BGRU VulDeePeckker,SySeVR,Original GPT Different Graph representation techniques VulDeePecker,SySeVR-BGRU,SySeVR-ABGRU,Russell VulDeePecker Original, BiLSTM, BiGRU, BERTBase, GPT-2 Base, CodeBERT, DistilBERT, RoBERTa etc CodeBERT,IVDect,Reveal,SyseVR,Devign,BoW+RF, VulDeePecker etc VulDeePecker, SySeVR, Devign, Reveal, GGNN, GCN, R-GCN, SVM, RF, MLP, BGNN4VD ChatGPT-4o, CodeBERT, CodeT5; discussed DeepWukong, VulDeePecker, SonarQube, LineVul Devign, VulCNN, VulDeePecker Logistic Regression IVDetect, ReVeal, Devign, SySeVR, VulDeePecker VCCFinder, DeepJIT, CC2Vec, CCT5, CodeJIT AutoML (RF, SVC, MLP) ChatGPT 3.5/4o, Devign, ReGVD, CodeBERT, UniXcoder-base, CodeT5+, UniXcoder-nine, TRACED, DeepDFA, PDBERT RNN, LSTM, GRU, Longformer, prior LLVM-IR/assembly works ChatGPT 3.5/4o, Devign, ReGVD, CodeBERT, CodeT5, UniXcoder-base, UniXcoder-nine, TRACED BERT, DistilBERT, CodeBERT, Gemini, stacking classifiers CodeBERT variants; prior smart-contract tools in related comparison CodeBERT, GraphCodeBERT, CodeT5, SantaCoder, Devign, ReVeal, IVDetect, BGNN4VD, ContraFlow, 𝜇VulDeePecker, SySeVR, MANDO Oyente, Mythril, Slither, LSTM, RNN, GCN, BiLSTM, ReChecker, DR-GCN, TMP, CGE, DL4SC VulDeePecker, SySeVR, Devign, ReVeal, LineVul, traditional embeddings Flawfinder, Checkmarx, SySeVR, IVDetect, VDTC, VulDeeLocator, LineVul VulDeePecker, Devign, Reveal, GCL4SVD, GSVD, MGVD, DGVD, CodeT5

Naseer et al.: Preprint submitted to Elsevier

Page 20 of 27

SLR for transformer-based Software Vulnerability detection

Figure 4: Granularity Level

Table 13 Methodological Taxonomy of Baseline Used in Vulnerability Detection Category

Representative Methods / Tools

Traditional Machine Learning Deep Learning (Sequence-based) Transformer-based Models Graph-based Models

SVM, KNN, Decision Trees, Naïve Bayes, Random Forest LSTM, BiLSTM, GRU, CNN, CNN– LSTM BERT, RoBERTa, CodeBERT, GraphCodeBERT, UniXcoder, CodeT5 GCN, GAT, DR-GCN, AST-based Devign Flawfinder, RATS, Cppcheck, Checkmarx Mythril, Oyente, Slither, Manticore

Static Analysis Tools Symbolic / Formal Analysis Tools Large Language Models (LLMs) State-of-Art frameworks

GPT-2, GPT-3.5, GPT-4, GPT-4 Turbo VulDeePecker, SySeVR, Devign, LineVul etc

significantly improves detection accuracy, F1-score, and robustness across unseen domains. Table 13 shows the summary of baseline categories and associated values. Table 13 highlights the diversity of baselines across conceptual, functional and low level software analysis techniques. These baselines include widely adopted deep learning frameworks such as Devign, ReVeal, VulDeePecker, SecureFalcon, SySeVR, IVDetect and LineVul, which serve as benchmarks for evaluating the effectiveness of new vulnerability detection approaches.

Naseer et al.: Preprint submitted to Elsevier

Figure 5 shows the popular baseline methods and their proportion of use in studies and research articles. VulDeePecker and Devign holds the largest share at 20.0%, followed by SySeVR at 17.0% and CodeBERT at 16.0%, Reveal at 11.0%, LineVul and IVDetect 8.0% indicating that these approaches contribute the most among the models compared. RQ8 Research Finding Researchers mostly use seven categories of models and tools as baselines for their experimental results. Among these, VulDeePecker, Devign, SySeVR are the most frequently used baselines.

4.9. RQ9: How many existing approaches focus on multi-lingual software vulnerability detection using transformers? Recently, researchers have started exploring multi-lingualbased vulnerability detection using a transformer. For example, Le et al. [50] compare two transformer models, CodeBERT and ChatGPT, for vulnerability prediction in low-resource programming languages. In the First step, they prepare a labelled dataset of vulnerable and non-vulnerable code snippets and apply the same samples to both models. Then, CodeBERT is fine-tuned on the dataset, while ChatGPT is evaluated using prompt-based in-context learning. Table 14 identifies the studies in the review that address multi-language vulnerability detection rather than focusing on a single programming language. It shows that only a small number of papers explicitly work across multiple languages, using either multi-language benchmark datasets or custom Page 21 of 27

SLR for transformer-based Software Vulnerability detection

Figure 5: Popular Baselines

Table 14 Papers addressing multi-language vulnerability detection Articles

Multi-language Methodology

Mahyari [60]

Uses multi-language vulnerability data rather than restricting the task to one programming language. Uses datasets such as CodeXGLUE / Juliet, which cover more than one programming language. The study is explicitly centered on vulnerability prediction in lowresource languages using CodeBERT and ChatGPT Uses mixed-source datasets, such as SARD + NVD, that include vulnerabilities across multiple programming languages. Uses CVEfixes and related sources, so the detection setting is not restricted to a single language. Uses a custom dataset containing both Python and C code, making it clearly multi-language.

Jianjie and Le [41] Le et al. [50]

He et al. [35]

Gujar [30] Gunda et al. [31]

Detection

Languages C/C++, Java, PHP, C# C/C++, Java

Rust,Kotlin, Swift

C/C++, Java, Python, PHP, and others 50+ languages through CVEfixes Python, C

datasets containing code from more than one language. The covered languages include C/C++, Java, PHP, C#, Python and C, and low-resource languages such as Rust, Kotlin, and Swift, while some studies rely on broader datasets like Naseer et al.: Preprint submitted to Elsevier

CVEfixes, which contain vulnerabilities from more than 50 programming languages. Overall, Table 14 highlights that multi-language vulnerability detection is still less common in the literature compared with single-language or domainspecific studies. RQ9 Research Finding A fewer number of reviewed papers work across multiple programming languages. Most existing studies still focus on single-language or domainspecific vulnerability detection. This encourages researchers to develop software vulnerability approaches

5. Threats to Validity This review may not include all relevant studies because the selection of papers depended on the databases searched, the search strings and keywords used, and the inclusion and exclusion criteria applied during screening. As a result, some relevant studies may have been missed, especially if they described transformer models, vulnerability detection, or software security using different terminology. Another possible threat is publication bias, since the review mainly includes published papers, which are more likely to report positive or stronger results, while Page 22 of 27

SLR for transformer-based Software Vulnerability detection

negative findings, replication failures, workshop drafts, and unpublished studies may be underrepresented. In addition, several parts of the review required manual classification, including transformer family, granularity level, vulnerability type coverage, single-language versus multi-language scope, and baseline grouping. Because these categories involve interpretation, some studies may reasonably fit more than one group, and a few borderline cases may have been simplified. A further limitation is that many papers did not clearly report important details such as exact baselines, hyperparameters, granularity, language scope, vulnerability type, experimental setup, or dataset composition. In such cases, some entries had to be inferred from the datasets used, the experiments described, or the overall problem setting, which introduces a risk of misinterpretation. There is also a dataset-based inference threat, since some papers used datasets that contained multiple CWE types or programming languages, whereas the model itself only performed binary vulnerability detection; interpreting dataset diversity as model capability may therefore overestimate the study’s actual contribution. Finally, granularity was not consistently reported across papers, and levels such as function-level, line-level, statement-level, contract-level, and flow-level had to be grouped into broader categories. Because some studies span multiple granularity levels, assigning them to one or more categories may also require subjective judgment.

6. Open Directions Future research on transformer-based software vulnerability detection should move beyond benchmark-oriented classification and focus more on practical and fine-grained security solutions. In particular, future studies should give equal importance to both vulnerability detection and vulnerable code localization. Although many existing approaches aim to determine whether a code sample is vulnerable, such predictions are of limited practical value unless the model can also identify the exact vulnerable region, such as the function, statement, line, contract fragment, or execution path responsible for the weakness. Therefore, future systems should be designed to jointly support classification and localization so that they can provide more actionable outputs for developers. This would improve their usefulness in debugging, remediation, patch generation, and vulnerability triage, and would make transformer-based methods more relevant for real-world software engineering practice [52, 26, 36, 90]. Another important direction is the development of richer code representations and more generalized detection models. The findings of this review indicate that the way code is represented has a significant impact on detection performance. Sequence-based representations are effective for capturing lexical and contextual patterns, while graph-based representations better encode structural and dependency information. Future work should therefore explore hybrid representations that combine token sequences with abstract

Naseer et al.: Preprint submitted to Elsevier

syntax trees, control-flow graphs, data-flow graphs, and program dependence information in order to capture both local and global vulnerability patterns. At the same time, future models should move beyond narrow settings limited to a single language or a small number of vulnerability categories. More generalized approaches are needed to support multiple programming languages and broader vulnerability coverage, which would improve the scalability, transferability, and practical relevance of vulnerability detection systems [11, 53, 65, 5, 60, 47]. A further promising research direction is the integration of transformer models and large language models with traditional program analysis techniques. Although transformerbased methods have demonstrated strong contextual understanding, they still face limitations in semantic precision and may produce predictions based on statistical patterns rather than verifiable security reasoning. In contrast, traditional static analysis, dynamic analysis, symbolic execution, and taint analysis provide program-level evidence that can validate or refine model outputs. Future work should therefore investigate hybrid frameworks that combine the learning ability of transformers and LLMs with the precision of traditional analysis methods. In addition, explainability should become a central goal of future research. Vulnerability detection systems should not only identify security flaws but also explain why the code is vulnerable, highlight the suspicious region, and help developers understand the root cause. Improved explainability would enhance trust, usability, and adoption in real-world software security workflows [73, 87, 75, 79]. Dataset quality and evaluation design also require substantial improvement. This review shows that many studies rely on a limited number of benchmark datasets, some of which may contain noisy labels, duplicated samples, synthetic examples, class imbalance, or hidden data leakage. These limitations can lead to overly optimistic performance results and weak real-world generalization. Future research should focus on constructing balanced, diverse, and highquality datasets with transparent collection procedures, accurate labels, realistic code contexts, and broader vulnerability coverage. More challenging and realistic evaluation settings are also needed, including cross-project, cross-version, cross-language, and temporally separated testing. In addition, future studies should more clearly distinguish between dataset diversity and actual model capability, since a dataset may include multiple languages or vulnerability types even when the model itself only performs limited binary detection [59, 30, 46, 43]. Finally, future work should focus on improving robustness, adaptability, and real-world usability. Current models often remain sensitive to tokenization strategies, preprocessing choices, hyperparameter settings, and dataset composition, which means that further research is needed on security-aware pretraining, effective fine-tuning, expressive embeddings, and systematic ablation studies. Future systems should also be better equipped to detect previously unseen

Page 23 of 27

SLR for transformer-based Software Vulnerability detection

or zero-day vulnerabilities through approaches such as selfsupervised learning, anomaly detection, continual learning, and transfer learning. In addition, research should explore cross-representation learning across source code, binaries, commits, logs, and runtime traces, as well as stronger integration into practical developer workflows such as IDEs, code review systems, and CI/CD pipelines. Overall, the next generation of transformer-based vulnerability detection systems should aim not only for high predictive accuracy, but also for localization capability, explainability, robustness, reproducibility, and deployment readiness [47, 36, 30, 46, 80, 25, 104].

current knowledge and identifying the most promising directions for future work. In particular, future progress is likely to depend on stronger benchmark standardization, broader multi-language datasets, better reporting practices, explainable transformer architectures, and deeper integration of transformer models and large language models with graph-based analysis and program analysis techniques. Such advances will be essential for building more accurate, robust, and practically deployable software vulnerability detection systems.

CRediT authorship contribution statement

Fiza Naseer: Data curation, Conceptualization of this study, Methodology, Analaysis, Writing - Original draft Software vulnerabilities continue to pose a serious threat preparation. Javed Ali Khan: Methodology, Analaysis, to modern software systems as applications grow in size, Revising - Original draft preparation, Supervision. Muhamcomplexity, and interconnectivity. In recent years, transformer- mad Yaqoob: Methodology, Analaysis, Revising - Original based models have emerged as a powerful direction for softdraft preparation, Supervision. Alexios Mylonas: Revising ware vulnerability detection due to their ability to capture - Original draft preparation, Supervision. Ishaya Gambo : semantic, structural, and contextual information from source Revising - Original draft preparation, Supervision. code more effectively than many traditional machine learnAcknowledgements: The authors acknowledge the use ing and deep learning techniques. Despite this rapid growth, of the writing assistance tool (Grammarly and ChatGPT) to prior systematic literature reviews have mainly focused improve the writing quality of this paper. Following its use, on conventional vulnerability detection approaches, with the authors thoroughly reviewed and revised the content, and limited attention to transformer-based methods as a distinct they take full responsibility for the final version of the paper. research area. To address this gap, this study presented a Conflict of Interest:The authors declare no conflicts of systematic literature review of transformer-based software interest. vulnerability detection research, guided by eight research questions. References Following Kitchenham’s SLR guidelines, we reviewed [1] Alaoui, R.L., Nfaoui, E.H., 2022. Deep learning for vulnerability and 80 primary studies published between 2021 and 2025 and attack detection on web applications: A systematic literature review. analyzed them across multiple dimensions, including task Future Internet 14, 118. type, datasets, vulnerability types, granularity levels, base[2] Almakayeel, N., 2024. Deep learning-based improved transformer model on android malware detection and classification in internet of lines, evaluation metrics, hyperparameter, and language covvehicles. Scientific Reports 14, 25175. erage. The findings show that transformer-based approaches [3] Alqarni, M., Azim, A., 2022. Low level source code vulnerability are now widely applied to binary vulnerability detection and detection using advanced bert language model., in: Canadian AI. vulnerability type detection. The review also reveals that [4] Ameh, J.E., Otebolaku, A., Shenfield, A., Ikpehai, A., 2025. C3Devign and Big-Vul are the most frequently used datasets, vulmap: A dataset for privacy-aware vulnerability detection in healthcare systems. Electronics 14, 2703. while Devign and VulDeePecker appear as the most com[5] Bahaa, A., Kamal, A.E.R., Fahmy, H., Ghoneim, A.S., 2024. Dbmon baseline models in comparative experiments. Most cbil: A distilbert-based transformer hybrid model using cnn and studies operate at the function level, although recent work bilstm for software vulnerability detection. IEEE Access . increasingly explores finer granularities such as line-level, [6] Bassi, D., Singh, H., 2023. A systematic literature review on statement-level, and commit-level detection. The review fursoftware vulnerability prediction models. IEEE Access 11, 110289– 110311. ther shows that the literature covers a broad range of vul[7] Bui, V.C., Do, X.C., 2023. Detecting software vulnerabilities nerability categories, yet only a limited number of studies based on source code analysis using gcn transformer, in: 2023 explicitly address multi-language vulnerability detection. RIVF International Conference on Computing and Communication Overall, this survey provides a focused and comprehenTechnologies (RIVF), IEEE. pp. 112–117. sive synthesis of transformer-based vulnerability detection [8] Cao, Y., Dong, Y., 2025. Multi-source cross-domain vulnerability detection based on code pre-trained model. Information and Softresearch and highlights several important research gaps. ware Technology , 107764. These include the lack of standardized benchmarks, limited [9] Cao, Y., Dong, Y., Peng, J., 2024. Vulnerability detection based on support for multi-language detection, inconsistent reporting transformer and high-quality number embedding. Concurrency and of hyperparameter and baselines, insufficient attention to Computation: Practice and Experience 36, e8292. explainability, and limited evaluation in real-world devel[10] Chan, A., Kharkar, A., Moghaddam, R.Z., Mohylevskyy, Y., Helyar, A., Kamal, E., Elkamhawy, M., Sundaresan, N., 2023. Transformeropment settings. The results of this study provide a useful based vulnerability detection in code at edittime: Zero-shot, fewreference for researchers and practitioners by consolidating

7. Conclusion

shot, or fine-tuning? arXiv preprint arXiv:2306.01754 .

Naseer et al.: Preprint submitted to Elsevier

Page 24 of 27

SLR for transformer-based Software Vulnerability detection [11] Chen, Y., Liu, Z., 2022. Hlt: A hierarchical vulnerability detection model based on transformer, in: 2022 4th International Conference on Data Intelligence and Security (ICDIS), IEEE. pp. 50–54. [12] Chernis, B., Verma, R., 2018. Machine learning methods for software vulnerability detection, in: Proceedings of the fourth ACM international workshop on security and privacy analytics, pp. 31–39. [13] Croft, R., Xie, Y., Babar, M.A., 2022. Data preparation for software vulnerability prediction: A systematic literature review. IEEE Transactions on Software Engineering 49, 1044–1063. [14] Cui, H., Zhang, C., Cai, F., 2025. Vulgtda: A software vulnerability detection method via graph transformer and domain adaptation, in: 2025 5th International Conference on Neural Networks, Information and Communication Engineering (NNICE), IEEE. pp. 1053–1056. [15] Curto, C., Giordano, D., Palazzo, S., Indelicato, D., 2024. Multivd: A transformer-based multitask approach for software vulnerability detection, in: Proceedings of the 21st International Conference on Security and Cryptography, pp. 416–423. [16] CWE, . Cwe vulnerabilities. https://cwe.mitre.org/. Accessed: 202511-18. [17] Devlin, J., Chang, M.W., Lee, K., Toutanova, K., 2019. Bert: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp. 4171–4186. [18] Do, C.X., Luu, N.T., Nguyen, P.T.L., 2024. Optimizing software vulnerability detection using roberta and machine learning. Automated Software Engineering 31, 40. [19] Do Xuan, C., Quang, D.B., Quang, V.D., 2026. A novel approach for software vulnerability detection based on ensemble learning model. Computers and Electrical Engineering 130, 110848. [20] Eberendu, A.C., Udegbe, V.I., Ezennorom, E.O., Ibegbulam, A.C., Chinebu, T.I., et al., 2022. A systematic literature review of software vulnerability detection. European Journal of Computer Science and Information Technology 10, 23–37. [21] Ehrenberg, M., Sarkani, S., Mazzuchi, T.A., 2024. Python source code vulnerability detection with named entity recognition. Computers & Security 140, 103802. [22] Feng, Z., 2020. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155 . [23] Ferrag, M.A., Battah, A., Tihanyi, N., Jain, R., Maimuţ, D., Alwahedi, F., Lestable, T., Thandi, N.S., Mechri, A., Debbah, M., et al., 2025a. Securefalcon: Are we there yet in automated software vulnerability detection with llms? IEEE Transactions on Software Engineering . [24] Ferrag, M.A., Battah, A., Tihanyi, N., Jain, R., Maimuţ, D., Alwahedi, F., Lestable, T., Thandi, N.S., Mechri, A., Debbah, M., Cordeiro, L.C., 2025b. Securefalcon: Are we there yet in automated software vulnerability detection with llms? IEEE Transactions on Software Engineering 51, 1248–1265. doi:10.1109/TSE.2025. 3548168. [25] Ferretti, S., D’Angelo, G., Ghini, V., Tomasone, M.B., 2025. Detecting smart contract vulnerabilities using transformers and llms, in: 2025 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops), IEEE. pp. 7–12. [26] Fu, M., Tantithamthavorn, C., 2022. Linevul: A transformer-based line-level vulnerability prediction, in: Proceedings of the 19th International Conference on Mining Software Repositories, pp. 608–620. [27] Gao, C., Yang, W., Ye, J., Xue, Y., Sun, J., 2024. sguard+: Machine learning guided rule-based automated vulnerability repair on smart contracts. ACM Transactions on Software Engineering and Methodology 33, 1–55. [28] Ghaffarian, S.M., Shahriari, H.R., 2017. Software vulnerability analysis and discovery using machine-learning and data-mining techniques: A survey. ACM computing surveys (CSUR) 50, 1–36. [29] Gong, P., Yang, W., Wang, L., Wei, F., HaiLaTi, K., Liao, Y., 2023. Gratdet: Smart contract vulnerability detector based on graph

Naseer et al.: Preprint submitted to Elsevier

representation and transformer. Computers, Materials & Continua 76. [30] Gujar, S.S., 2024. Detectbert: Code vulnerability detection, in: 2024 Global Conference on Communications and Information Technologies (GCCIT), IEEE. pp. 1–21. [31] Gunda, B.S., Krishna, G.B., Rawat, S.S., 2025. Transformer-based semantic embeddings and hybrid neural networks for robust software vulnerability detection, in: 2025 Innovations in Power and Advanced Computing Technologies (i-PACT), IEEE. pp. 1–9. [32] Gupta, A.R., Tomar, D.S., Shekhar, R., 2024. Dl-vulbert: A deep learning classifier for the identification of software vulnerabilities, in: 2024 15th International Conference on Computing Communication and Networking Technologies (ICCCNT), IEEE. pp. 1–7. [33] Hanif, H., Maffeis, S., 2022. Vulberta: Simplified source code pre-training for vulnerability detection, in: 2022 International joint conference on neural networks (IJCNN), IEEE. pp. 1–8. [34] Harzevili, N.s., Belle, A.b., Wang, J., Wang, S., Jiang, Z.m.j., Nagappan, N., 2025. A systematic literature review on automated software vulnerability detection using machine learning. ACM COMPUTING SURVEYS 57. [35] He, H., Wang, S., Wang, Y., Liu, K., Yu, L., 2025. Vultr: Software vulnerability detection model based on multi-layer key feature enhancement. Computers & Security 148, 104139. [36] Hin, D., Kan, A., Chen, H., Babar, M.A., 2022. Linevd: Statementlevel vulnerability detection using graph neural networks, in: Proceedings of the 19th international conference on mining software repositories, pp. 596–607. [37] Hou, F., Zhou, K., Li, L., Tian, Y., Li, J., Li, J., 2022. A vulnerability detection algorithm based on transformer model, in: International Conference on Artificial Intelligence and Security, Springer. pp. 43– 55. [38] Islam, N.T., Parra, G.D.L.T., Manuel, D., Bou-Harb, E., Najafirad, P., 2023. An unbiased transformer source code learning with semantic vulnerability graph, in: 2023 IEEE 8th European Symposium on Security and Privacy (EuroS&P), IEEE. pp. 144–159. [39] Jeon, S., Lee, G., Kim, H., Woo, S.S., 2024. Design and evaluation of highly accurate smart contract code vulnerability detection framework. Data Mining and Knowledge Discovery 38, 888–912. [40] Jiang, X., Wang, S., Gong, Y., Yu, T., Liu, L., Yu, X., 2024. Haformer: Semantic fusion of hex machine code and assembly code for cross-architecture binary vulnerability detection. Computers & Security 145, 104029. [41] JianJie, Y., Le, W., 2023. Code defect detection method based on bert and ensemble, in: 2023 9th International Conference on Computer and Communications (ICCC), IEEE. pp. 2130–2138. [42] Kaanan, E., Karim, T., Shaon, M.S.H., Sultan, M.F., Cuzzocrea, A., Akter, M.S., 2024. Llm-based approach for buffer overflow detection in source code, in: 2024 27th International Conference on Computer and Information Technology (ICCIT), IEEE. pp. 1898–1902. [43] Kalouptsoglou, I., Siavvas, M., Ampatzoglou, A., Kehagias, D., Chatzigeorgiou, A., 2025. Transfer learning for software vulnerability prediction using transformer models. Journal of Systems and Software 227, 112448. [44] Katz, K., Moshtari, S., Mujhid, I., Mirakhorli, M., Garcia, D., 2025. Siexvults: Sensitive information exposure vulnerability detection system using transformer models and static analysis, in: 2025 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), IEEE. pp. 230–241. [45] Khan, A.R., Xu, Y., Li, Y., 2026. Leveraging transformers to discover software vulnerabilities based on source code slices, in: Proceedings of the 2026 Australasian Information Security Conference, pp. 1–9. [46] Kim, J., Lee, S., Kim, H., et al., 2024. Robust vulnerability detection in solidity-based ethereum smart contracts using fine-tuned transformer encoder models. IEEE Access . [47] Kim, S., Choi, J., Ahmed, M.E., Nepal, S., Kim, H., 2022. Vuldebert: A vulnerability detection system using bert, in: 2022 IEEE International Symposium on Software Reliability Engineering Workshops

Page 25 of 27

SLR for transformer-based Software Vulnerability detection (ISSREW), IEEE. pp. 69–74. [48] Kitchenham, B., Charters, S., et al., 2007. Guidelines for performing systematic literature reviews in software engineering. Technical Report. Technical report, ver. 2.3 ebse technical report. ebse. [49] Le, T.H., Chen, H., Babar, M.A., 2022. A survey on data-driven software vulnerability assessment and prioritization. ACM Computing Surveys 55, 1–39. [50] Le, T.H.M., Babar, M.A., Thai, T.H., 2024. Software vulnerability prediction in low-resource languages: An empirical study of codebert and chatgpt, in: Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, pp. 679–685. [51] Li, J., 2025. Macd: Source code vulnerability detection method integrating mamba and attention, in: 2025 7th International Conference on Electronics and Communication, Network and Computer Technology (ECNCT), IEEE. pp. 376–380. [52] Li, S., Chen, D., Zhang, J., Wang, H., Li, L., Qian, Y., Liu, H., 2022. Software vulnerability detection based on anomaly-attention, in: 2022 4th International Conference on Robotics and Computer Vision (ICRCV), IEEE. pp. 261–265. [53] Liang, C., Wei, Q., Jiang, Z., Wang, Y., Du, J., 2024. A source code vulnerability detection method based on adaptive graph neural networks, in: Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering Workshops, pp. 187–196. [54] Lin, G., Wen, S., Han, Q.L., Zhang, J., Xiang, Y., 2020. Software vulnerability detection using deep neural networks: a survey. Proceedings of the IEEE 108, 1825–1848. [55] Liu, C., Chen, X., Li, X., Xue, Y., 2024a. Making vulnerability prediction more practical: Prediction, categorization, and localization. Information and Software Technology 171, 107458. [56] Liu, S., Li, L., Ban, X., Chen, C., Zhang, J., Camtepe, S., Xiang, Y., 2024b. Automatic software vulnerability detection in binary code, in: International Conference on Machine Learning for Cyber Security, Springer. pp. 148–166. [57] Liu, Z., Liao, Q., Gu, W., Gao, C., 2023. Software vulnerability detection with gpt and in-context learning, in: 2023 8th International Conference on Data Science in Cyberspace (DSC), IEEE. pp. 229– 236. [58] Liu, Z., Tang, Z., Zhang, J., Xia, X., Yang, X., 2024c. Pre-training by predicting program dependencies for vulnerability analysis tasks, in: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pp. 1–13. [59] Lu, G., Ju, X., Chen, X., Yang, S., Chen, L., Shen, H., 2023. Assessing the effectiveness of vulnerability detection via prompt tuning: An empirical study, in: 2023 30th Asia-Pacific Software Engineering Conference (APSEC), IEEE. pp. 415–424. [60] Mahyari, A.A., 2024. Harnessing the power of llms in source code vulnerability detection, in: MILCOM 2024-2024 IEEE Military Communications Conference (MILCOM), IEEE. pp. 251–256. [61] Mamede, C., Pinconschi, E., Abreu, R., 2022. A transformerbased ide plugin for vulnerability detection, in: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, pp. 1–4. [62] Matloob, I., Khan, S., Rukaiya, R., Alfraihi, H., Ali Khan, J., 2025. Healthcare fraud detection using adaptive learning and deep learning techniques. Evolving Systems 16, 72. [63] Mechri, A., Ferrag, M.A., Debbah, M., 2025. Secureqwen: Leveraging llms for vulnerability detection in python codebases. Computers & Security 148, 104151. [64] Mylläri, J., Aalto, T., Nurminen, J.K., 2025. Ladle: a method for unsupervised anomaly detection across log types. Automated Software Engineering 32, 34. [65] Nguyen, H.H., Nguyen, N.M., Xie, C., Ahmadi, Z., Kudendo, D., Doan, T.N., Jiang, L., 2023. Mando-hgt: Heterogeneous graph transformers for smart contract vulnerability detection, in: 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR), IEEE. pp. 334–346.

Naseer et al.: Preprint submitted to Elsevier

[66] Ni, C., Yin, X., Li, X., Xu, X., Yu, Z., 2025. Abundant modalities offer more nutrients: Multi-modal-based function-level vulnerability detection. ACM Transactions on Software Engineering and Methodology . [67] Nong, Y., Sharma, R., Hamou-Lhadj, A., Luo, X., Cai, H., 2022. Open science in software engineering: A study on deep learningbased vulnerability detection. IEEE Transactions on Software Engineering 49, 1983–2005. [68] Oladokun, O., Rice, J., 2025. How effective are pretrained programming language-based language models (pllms) in the detection of android vulnerabilities?, in: 2025 IEEE Canadian Conference on Electrical and Computer Engineering (CCECE), IEEE. pp. 150–154. [69] Pavicic, B., . Real-world examples of application security breaches. https://true-positives.com/appsec-blog/cybersecuritybreaches-real-world-examples-lessons-learned. Accessed: 2025-1125. [70] Peng, T., Chen, S., Zhu, F., Tang, J., Liu, J., Hu, X., 2023. Ptlvd: Program slicing and transformer-based line-level vulnerability detection system, in: 2023 IEEE 23rd International Working Conference on Source Code Analysis and Manipulation (SCAM), IEEE. pp. 162– 173. [71] Perera, A., Pillai, B., Tharani, J.S., Rao, A.S., Muthukkumarasamy, V., 2025. Codebert-based embeddings for detecting vulnerable smart contracts, in: 2025 IEEE 50th Conference on Local Computer Networks (LCN), IEEE. pp. 1–9. [72] Perl, H., Dechand, S., Smith, M., Arp, D., Yamaguchi, F., Rieck, K., Fahl, S., Acar, Y., 2015. Vccfinder: Finding potential vulnerabilities in open-source projects to assist code audits, in: Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pp. 426–437. [73] Purba, M.D., Ghosh, A., Radford, B.J., Chu, B., 2023. Software vulnerability detection using large language models, in: 2023 IEEE 34th International Symposium on Software Reliability Engineering Workshops (ISSREW), IEEE. pp. 112–119. [74] Reza, S.I., Moon, I.T., Fahim, M.F.S., Alam, A., Sheikh, M.T., 2026. An empirical analysis of transformer-based models with lime explainability for javascript vulnerability detection, in: 2026 5th International Conference on Electrical, Computer & Telecommunication Engineering (ICECTE), IEEE. pp. 1–6. [75] Rusinova, Z., Chernyshov, Y., Dolganov, A., 2024. Explaining of transformer-based models for vulnerable function detection, in: 2024 IEEE Ural-Siberian Conference on Biomedical Engineering, Radioelectronics and Information Technology (USBEREIT), IEEE. pp. 304–307. [76] Saimbhi, S.S., Akpinar, K.O., 2024. Vulnerai: Gpt based web application vulnerability detection, in: 2024 International Conference on Artificial Intelligence, Metaverse and Cybersecurity (ICAMAC), IEEE. pp. 1–6. [77] Senanayake, J., Kalutarage, H., Al-Kadri, M.O., Petrovski, A., Piras, L., 2023. Android source code vulnerability detection: a systematic literature review. ACM Computing Surveys 55, 1–37. [78] Shang, J., Li, J., Sui, Y., Guo, H., Gao, X., Zhang, D., Guo, Y., Wu, G., 2025. Cegt: Smart contract vulnerability detection via connectivity-enhanced gcn-transformer. Journal of Systems and Software 227, 112454. [79] Shiaeles, M., 2023. Vuldetect: A novel technique for detecting software vulnerabilities using language models. [80] Shir, R., Surve, P., Elovici, Y., Shabtai, A., 2025. Robust vulnerability detection across compilations: Llvm-ir vs. assembly with transformer model. Proceedings of the ACM on Software Engineering 2, 618–639. [81] Shiri Harzevili, N., Boaye Belle, A., Wang, J., Wang, S., Jiang, Z.M., Nagappan, N., 2024. A systematic literature review on automated software vulnerability detection using machine learning. ACM Computing Surveys 57, 1–36.

Page 26 of 27

SLR for transformer-based Software Vulnerability detection [82] Smaili, A., Zhang, Y., Mekkaoui, D.E., Midoun, M.A., Talhaoui, M.Z., Hamidaoui, M., Kong, W., 2025. A transformer-based framework for software vulnerability detection using attention-driven convolutional neural networks. Engineering Applications of Artificial Intelligence 160, 111859. [83] Sohan, M.F., Basalamah, A., 2020. A systematic literature review and quality analysis of javascript malware detection. IEEE Access 8, 190539–190552. [84] Su, H., Xu, Z., Zhang, Y., Tan, Q., 2026. Source code vulnerability detection based on deep learning: a review. Cybersecurity 9, 2. [85] Sultan, M.F., Akter, M.S., Cuzzocrea, A., 2025. Codevul+: A structure-aware framework for cross-repository vulnerability detection, in: 2025 IEEE International Conference on Big Data (BigData), IEEE. pp. 4406–4415. [86] Sun, X., Zhou, M., Cao, S., Wu, X., Bo, L., Wu, D., Li, B., Xiang, Y., 2025. Hgtjit: Just-in-time vulnerability detection based on heterogeneous graph transformer. IEEE Transactions on Dependable and Secure Computing . [87] Sun, Y., Wu, D., Xue, Y., Liu, H., Wang, H., Xu, Z., Xie, X., Liu, Y., 2024a. Gptscan: Detecting logic vulnerabilities in smart contracts by combining gpt with program analysis, in: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pp. 1–13. [88] Sun, Z., Liu, K., Yang, Y., 2024b. Enhancing source code vulnerability detection using flattened code graph structures, in: 2024 6th International Conference on Frontier Technologies of Information and Computer (ICFTIC), IEEE. pp. 209–213. [89] TANKO, M.Y., SULTAN, A.B.M., OSMAN, M.H., ZULZALIL, H., 2025. An approach for vulnerability detection in web applications using graph neural networks and transformers. Journal of Theoretical and Applied Information Technology 103. [90] Tao, W., Su, X., Ke, Y., Han, Y., Zheng, Y., Wei, H., 2025. Transformer-based statement level vulnerability detection by crossmodal fine-grained features capture. Knowledge-Based Systems 316, 113341. [91] Thapa, C., Jang, S.I., Ahmed, M.E., Camtepe, S., Pieprzyk, J., Nepal, S., 2022. Transformer-based language models for software vulnerability detection, in: Proceedings of the 38th Annual Computer Security Applications Conference, pp. 481–496. [92] Tian, L., Zhang, C., 2025. Efvd: A framework of source code vulnerability detection via fusion of enhanced graph representation learning and pre-trained transformer-based model, in: Proceedings of the 2025 5th International Conference on Computer Network Security and Software Engineering, pp. 316–320. [93] Vanam, R.R., Yadavali, V.V., Elumalai, S., et al., 2025. Software vulnerability detection in source code using superb fairy-wren deep transformer guided model for next generation software security, in: 2025 6th International Conference on Inventive Research in Computing Applications (ICIRCA), IEEE. pp. 101–106. [94] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I., 2017. Attention is all you need. Advances in neural information processing systems 30. [95] Wang, R., Xu, S., Ji, X., Tian, Y., Gong, L., Wang, K., 2024a. An extensive study of the effects of different deep learning models on code vulnerability detection in python code. Automated Software Engineering 31, 15. [96] Wang, R., Xu, S., Tian, Y., Ji, X., Sun, X., Jiang, S., 2024b. Scl-cvd: Supervised contrastive learning for code vulnerability detection via graphcodebert. Computers & Security 145, 103994. [97] Wang, Z., Li, G., Li, J., Dong, Y., Xiong, Y., Jin, Z., 2025a. Linelevel semantic structure learning for code vulnerability detection, in: Proceedings of the 16th International Conference on Internetware, pp. 269–280. [98] Wang, Z., Li, G., Li, J., Li, J., Yan, M., Xiong, Y., Jin, Z., 2025b. M2cvd: Enhancing vulnerability understanding through multi-model collaboration for code vulnerability detection. ACM Transactions on Software Engineering and Methodology .

Naseer et al.: Preprint submitted to Elsevier

[99] Wu, B., Zou, F., 2022. Code vulnerability detection based on deep sequence and graph models: A survey. Security and Communication Networks 2022, 1176898. [100] Wu, H., Zhang, Z., Wang, S., Lei, Y., Lin, B., Qin, Y., Zhang, H., Mao, X., 2021a. Peculiar: Smart contract vulnerability detection based on crucial data flow graph and pre-training techniques, in: 2021 IEEE 32nd International Symposium on Software Reliability Engineering (ISSRE), IEEE. pp. 378–389. [101] Wu, T., Chen, L., Du, G., Zhu, C., Shi, G., 2021b. Self-attention based automated vulnerability detection with effective data representation, in: 2021 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking (ISPA/BDCloud/SocialCom/SustainCom), IEEE. pp. 892–899. [102] Wu, X., Li, H., Khomh, F., 2025. What information contributes to log-based anomaly detection? insights from a configurable transformer-based approach. Automated Software Engineering 32, 58. [103] Xuan, C.D., Quang, D.B., Quang, V.D., 2025. Large language models based vulnerability detection: How does it enhance performance? International Journal of Information Security 24, 69. [104] Yang, A.Z., Le Goues, C., Martins, R., Hellendoorn, V., 2024. Large language models for test-free fault localization, in: Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, pp. 1–12. [105] Zahid, M.J.S., 2025. Detectbert: A transformer-based approach for statement-level vulnerability detection in python code. Authorea Preprints . [106] Zhang, S., Fan, D., He, L., Liu, Y., Chen, D., 2024. Twlog: Task workflow-based log anomaly detection, in: Asia-Pacific Web (APWeb) and Web-Age Information Management (WAIM) Joint International Conference on Web and Big Data, Springer. pp. 3–16. [107] Zhang, X., Zhang, F., Zhao, B., Zhou, B., Xiao, B., 2023. Vuldtransformer: source code vulnerability detection via transformer, in: Proceedings of the 14th Asia-Pacific Symposium on Internetware, pp. 185–193. [108] Zhao, J., Liu, W., 2025. Vulnerability code similarity detection method based on transformer, in: 2025 5th International Conference on Sensors and Information Technology, IEEE. pp. 851–854. [109] Zhao, K., Duan, S., Qiu, G., Zhai, J., Li, M., Liu, L., 2024. Python source code vulnerability detection based on codebert language model, in: 2024 7th International Conference on Algorithms, Computing and Artificial Intelligence (ACAI), IEEE. pp. 1–6.

Page 27 of 27

Related documents

• Two's a Crowd: Human and AI-Based Copresence for Developers with ADHD#1006931
• License Compliance in Open Source Cybersecurity Projects#1006932
• Embedding Drift in Code Vulnerability Models Under Intended Behaviour-Preserving Transformations#1006933
• SWE-Proof: Can Language Models Resolve Real-World Issues with Machine-Checked Proofs?#1006934
• Helpful but Fallible: Developer Experiences of AI Tools Under a Coordinated Industrial Roll-out#1006935