Zero-shot Evaluation of Deep Learning for Java Code Clone Detection Thomas S. Heinze1 1 Cooperative University Gera-Eisenach, Gera, Germany
arXiv:2604.13783v1 [cs.SE] 15 Apr 2026
Keywords:
Benchmark, Java, Deep Learning, generalizability, zero-shot evaluation, code clone, code clone detection.
Abstract:
Deep Learning (DL) is becoming more and more widespread in clone detection, motivated by achieving nearperfect performance for this task. In particular in case of semantic code clones, which share only limited syntax but implement the same or similar functionality, Deep Learning appears to outperform conventional tools. In this paper, we want to investigate the generalizability of DL-based clone detectors for Java. We therefore replicate and evaluate the performance of five state-of-the-art DL-based clone detectors, including Transformers like CodeBERT and single-task models like FA-AST+GMN, in a zero-shot evaluation scenario, where we train/fine-tune and evaluate on different datasets and functionalities. Our experiments demonstrate that the models’ generalizability to unseen code is limited. Further analysis reveals that the conventional clone detector NiCad even outperforms the DL-based clone detectors in such a zero-shot evaluation scenario.
1
INTRODUCTION
Code clones, ranging from duplicated code to semantic code clones, i.e., syntactically different fragments of code with same functionality, are frequently found in today’s code bases and can influence aspects like maintainability, code quality, and defect/vulnerability proneness (Roy et al., 2009). While detection of identical or near-miss code clones is considered a largely solved problem, identifying semantic clones remains challenging. In particular, Deep Learning (DL) approaches address this gap and show nearperfect results, as is usually demonstrated on BigCloneBench (Wang et al., 2020; Lu et al., 2021; Guo et al., 2022). As such evaluations are often conducted within this single benchmark, generalizability of the DL models to unseen functionalities, i.e., code implementing behaviour absent in the models’ training data, remains disputed. We therefore want to investigage the models under zero-shot evaluation referring to a scenario where a DL-based clone detector is tested or used on code it was never trained on. Our main contributions are: We present replication and evaluation experiments on the performance of Deep Learning clone detectors for Java, including CodeBERT (Feng et al., 2020), GraphCodeBERT (Guo et al., 2021), UniXcoder (Guo et al., 2022), CodeT5 (Wang et al., 2021), and FAAST+GMN (Wang et al., 2020), as well as the conventional tools NiCad (Roy and Cordy, 2008), NIL (Nak-
agawa et al., 2021), and StoneDetector (Amme et al., 2021). We in particular provide comprehensive analysis of their generalizability using four different evaluation benchmarks from varying application scenarios and domains, besides BigCloneBench, and demonstrate a general drop of on average approx. 41% in the models’ F1 scores under zero-shot evaluation. We believe our paper to be interesting to both, researchers and practitioners, as it shows: (1) clone detectors’ performance evaluations require careful analysis and should not be solely based on BigCloneBench, (2) the threshold value of conventional tools like NiCad is a beneficial configuration parameter when it comes to semantic code clones, and (3) there is no “free lunch”, i.e., there is no clone detector which outperforms all the others in every domain.
2
BENCHMARKS
A number of datasets exists with code clone samples for the Java programming language which can be used for benchmarking clone detectors. While BigCloneBench is the de-facto standard (Svajlenko and Roy, 2015), its usage for training/fine-tuning and evaluating Deep Learning approaches to code clone detection is rather controversial (Krinke and Ragkhitwetsagul, 2022; Schäfer et al., 2022). We therefore use a number of benchmarks beside BigCloneBench in our evaluation experiments, covering various application
Table 1: Code clone detection benchmarks in this paper. Benchmark BigCloneBench SemanticCloneBench FEMPD SeSaMe ProjectCodeNet
#Code Fragments 9,126 1,000 4,388 1,217 2,919
#Positive Samples 56,820 1,000 1,342 66 1,000
#Negative Samples 358,596 1,000 852 546 1,000
contexts and domains, including open-source production code, submissions to programming contests, and code snippets from Q&A fora (cf. Table 1). To foster further research and for replication, we provide all five benchmark datasets online1 . BigCloneBench: Svajlenko et al. introduce the BigCloneBench dataset for evaluating performance, in particular recall, of clone detectors for the Java programming language and provide the de-facto standard dataset in code clone research (Svajlenko and Roy, 2015). BigCloneBench has been derived from the inter-project source code dataset IJADataset 2.0, comprising approx. 365 million lines of code in more than 2.3 million Java source code files from 25,000 open-source projects. The authors of BigCloneBench use a multi-step approach to create the dataset which is centered around 43 functionalities, e.g., copy a file or web download. Using heuristics, candidate methods have been mined from the IJADataset 2.0 and manually classified according to the 43 functionalities. Code clones are generated from methods which have been assigned the same functionality. Eventually, BigCloneBench contains more than 8 million known code clones of different syntactical similarity. In recent years, various research relied on this dataset for Deep Learning approaches to clone detection (cf. (Krinke and Ragkhitwetsagul, 2022; Krinke and Ragkhitwetsagul, 2025)). Instead of the whole dataset, oftentimes, a subset of BigCloneBench is used (Wei and Li, 2017; Wang et al., 2020; Lu et al., 2021). Starting with the authors of the Deep Learning clone detector CDLH (Wei and Li, 2017), this subset is constructed by discarding those “... code fragments without any tagged true and false clone pairs”, yielding approx. 9,100 Java methods. While the positive samples of Java code clones can then be simply derived from BigCloneBench, the construction of negative samples is though kept quite opaque. In our experiments, we use the CodeXGLUE variant (Lu et al., 2021)2 of this unbalanced subset as baseline, which comprises 1,731,860 methods pairs over 9,126 Java methods from BigCloneBench, including 561,521 positive and 1,170,339 negative samples of Java code 1 https://doi.org/10.5281/zenodo.19581107 2 https://github.com/microsoft/CodeXGLUE
clones split across training, validation, and evaluation datasets (Table 1 refers to CodeXGLUE’s evaluation dataset). We note that the usage of BigCloneBench for training/fine-tuning and evaluating Deep Learning approaches is disputed within the literature (Krinke and Ragkhitwetsagul, 2022) (cf. Sect. 5). After all, we hope to contribute a further clarification to this dispute with our work. SemanticCloneBench: Stack Overflow3 is a community web platform, which allows users to ask and answer questions related to various programming topics. Al-Omari et al. (Al-Omari et al., 2020) used Stack Overflow as a source for their SemanticCloneBench dataset4 of (semantic) code clones based on the idea that code snippets in correct answers to the same question are functionally similar and thus constitute a code clone. They apply additional steps, e.g., filtering out syntactical clones and manually validating the identified clones by two judges, which results in overall 4,000 clone pairs, including 1,000 code clones for the Java programming language. While the dataset itself only contains positive samples of code clones, Arshad et al. later proposed a simple approach for generating negative samples in (Arshad et al., 2022): Their idea is to combine each first element of a clone pair in the dataset’s first half with a first element of the clone pairs in the dataset’s second half and to do so similarly for the clone pairs’ second elements. We use the same approach and are able to construct a balanced dataset containing 1,000 positive and 1,000 negative samples for Java code clones. FEMPD: The benchmark FEMPD (Higo, 2024)5 defines a dataset of in particular semantic code clones. The dataset has been generated on the inter-project source code repository IJADataset, following a rigorous approach using various steps, including grouping Java methods according to their static signatures, generating and running test cases for identifying functionally equivalent methods, and manually validating the thus determined semantic clone pairs. This approach results in an unbalanced dataset which contains 1,342 positive samples and 852 negative samples for Java code clones. Note that FEMPD originates from the IJADataset, much like BigCloneBench, though is based on a more strict and rigorous definition of code clone, i.e., functionally equivalent code.
3 https://stackoverflow.com 4 https://drive.google.com/open?id= 1KicfslV02p6GDPPBjZHNlmiXk-9IoGWl 5 https://github.com/YoshikiHigo/FEMPDataset
SeSaMe: Another dataset containing examples of real-world Java code clones is SeSaMe (Kamp et al., 2019)6 . Its authors focus on semantically similar code fragments from real-world production code and therefore mined large open-source Java projects, including, e.g., Eclipse’s Java Development Tools (JDT), Google’s Guava library, and the Open Java Development Toolkit. As a starting point, they consider the API documentations and analyze methods’ documentation comments for textual similarity. In a subsequent step, they manually assess the resulting method pairs according to three similarity dimensions, i.e., methods’ goals, operations and effects. In contrast to the other datasets included in our experiments, SeSaMe does not define a binary classification of method pairs into positive and negative samples of code clones, but rather contains 857 method pairs conjoined with respective manual judgements of their similarity scores. Accordingly, we derive an unbalanced dataset set of 66 positive samples and 546 negative samples for Java code clones by only considering those method pairs which feature a majority rate as similar and a majority rate as dissimilar in all three similarity dimensions, respectively. ProjectCodeNet: The archives of online programming contests like Google Code Jam7 or AtCoder8 provide a rich source of semantic code clones and have therefore been utilized for the definition of clone datasets and benchmarks. We include ProjectCodeNet (Puri et al., 2021)9 in our experiments as a representative example due to its large size and inclusion of Java code. The dataset originates from the AIZU and AtCoder programming contests and contains in its Java benchmark subset 750,000 submissions to 250 programming tasks. Prior cleansing filters out identical problems and near-duplicate submissions. Two accepted submissions to the same task then constitute a positive sample of code clones whereas two accepted submissions to different tasks constitute a negative sample. As each submission consists of a single Java source file with potentially more than one Java method, we additionally filter for submissions comprising a single method. In this way, we were able to construct a balanced dataset with 1,000 positive and 1,000 negative samples of Java code clones.
Table 2: Deep Learning-based clone detectors in this paper. Model CodeBERT GraphCodeBERT UniXcoder CodeT5-base FA-AST+GMN
3
Model Type pre-trained, masked language model (encoder-only) pre-trained, masked language model (encoder-only) pre-trained, unified multimode transformer model pre-trained, seq-to-seq model graph-based neural network
#Parameters 125 million 125 million 125 million 220 million n/a
CLONE DETECTORS
Clone detection is a well-studied research subject and numerous clone detectors have been proposed in the literature (Roy et al., 2009). While most clone detectors are conventional, with the advent of Deep Learning, more and more tools employ this approach for finding code clones. In this section, we will shortly introduce the selected conventional and Deep Learningbased clone detectors used in our experiments.
3.1
DL-based Clone Detectors
The Transformer architecture and pre-trained generalpurpose code models have been shown to achieve promising results in various programming language tasks, including clone detection. We select four different Transformer models to provide for a comprehensive picture of their capabilities (cf. Table 2). CodeBERT (Feng et al., 2020), GraphCodeBERT (Guo et al., 2021), and UniXcoder (Guo et al., 2022)10 are pre-trained models for code, i.e., they have been pre-trained on large code corpora and can be fine-tuned for a specific downstream task like clone detection. While CodeBERT and GraphCodeBERT are masked language models, i.e., are pretrained for predicting masked code from surrounding context, CodeT5 (Wang et al., 2021)11 is another pre-trained Transformer model, which though is pre-trained as sequence-to-sequence model for autoregressively translating an input (code) sequence into an output (code) sequence. UniXcoder provides a uniform multi-mode model, which has been pre-trained for multiple training objectives. In addition, we consider with FAAST+GMN (Wang et al., 2020)12 a representative of a single-task neural network model, which falls into the same family like CDLH (Wei and Li, 2017), ASTNN (Zhang et al., 2019), in contrast to above’s pre-trained general-purpose language models to complete the picture. FA-AST+GMN finds code
6 https://github.com/FAU-Inf2/sesame 7 https://zibada.guru/gcj/
10 https://github.com/microsoft/CodeBERT
8 https://atcoder.jp
11 https://github.com/salesforce/CodeT5
9 https://github.com/IBM/Project CodeNet
12 https://github.com/jacobwwh/graphmatch clone
Table 3: Conventional code clone detectors in this paper conjoined with their used configuration parameters. NiCad v7.0.1 70% sim. threshold (τ=0.3), blind renaming, literal abstraction
NIL v2.0.0 10% filtr. threshold, 70% ver. threshold 5-grams
StoneDetector 70% sim. threshold (τ=0.3), LCS metric, 8-byte hashing
clones by representing two code fragments by data flow-augmented abstract syntax trees and then using graph-matching neural networks to embed and match the two graphs based on their cosine similarity. Note that the five models are usually trained/finetuned and evaluated on subsets of BigCloneBench and then regularly achieve F1 scores of ≥ 0.94 (Wang et al., 2020; Lu et al., 2021; Guo et al., 2022).
3.2
Conventional Tools
For sake of comparison, we include three conventional code clone detectors for Java in our experiments (cf. Table 3): NiCad (Roy and Cordy, 2008) is a hybrid clone detector employing normalization techniques ahead of analyzing code similarity based on the normalized code fragments’ longest common subsequence (LCS). NiCad is in particular good at finding near-miss code clones at very high precision. We use the tool’s most recent free and open version 7.0.1 as available online13 . More recent tools like NIL (Nakagawa et al., 2021)14 focus on largegap code clones with many consecutive code edits or modifications scattered around the code. NIL represents code by N-grams derived from normalized token sequences and thereon measures similarity again using LCS. StoneDetector (Amme et al., 2021; Heinze et al., 2026)15 is another recent clone detector for Java which has been shown to in particular excel at finding code clones with larger syntactical variance (Heinze et al., 2026) and for that purpose employs string metrics like LCS on fingerprints as derived from code fragments’ dominator trees.
4
EVALUATION
In our experiments, we want to investigate the generalizability of DL-based clone detectors trained on BigCloneBench. We therefore first conduct a replication experiment, where the five DL models introduced in Sect. 3 are trained/fine-tuned and evaluated on the CodeXGLUE subset of BigCloneBench, which provides us with a baseline (cf. Table 4). We then use the 13 https://github.com/CordyJ/Open-NiCad 14 https://github.com/kusumotolab/NIL 15 https://github.com/StoneDetector/StoneDetector
thus trained models and evaluate them in a zero-shot evaluation approach for the four other benchmarks introduced in Sect. 2 (cf. Table 5). We in addition include the three conventional tools for comparison. All experiments were conducted on a Ubuntu 24.04 LTS system running in a virtual machine with assigned 8 CPU cores 2.3 GHz, 48 GB RAM, and NVIDIA RTX 6000 Ada GPU (CUDA v13.0).
4.1
Evaluation Metrics
Clone detection can be seen as a binary classification problem, where a pair of code fragments is assgined one of two categories: code clone (positive assignment) or non-clone (negative assignment). Consequently, for a certain clone detector and code clone dataset, we can differentiate between the clone detector’s correct positive assignments, i.e., true positives, correct negative assignments, i.e., true negatives, incorrect positive assignments, i.e., false positives, and incorrect negative assignments, i.e., false negatives. Recall and precision are then standard evaluation metrics for assessing the probability of detecting a true clone and the propability of a correct positive classification using the clone detector, respectively: Recall
=
Precision
=
#True Positives #True Positives + #False Negatives #True Positives #True Positives + #False Positives
Note that a trivial clone detector, which assigns each pair of code fragments as code clone, can achieve perfect recall and – vice versa – a clone detector, which assigns each pair as non-clone, can achieve perfect precision. Thus, assessing a clone detectors’ performance requires to analyze both metrics. In addition, the fall-out or false-positive rate may be used as measure for the probability of false alarms: Fall − out
=
#False Positives #False Positives + #True Negatives
Averaging recall and precision into a single evaluation metric can be done using their harmonic mean, i.e., F1 score, as follows: F1 Score
= 2×
Precision × Recall Precision + Recall
Note though that the F1 score assumes equal importance of recall and precision and may be less informative when compared to using the other two metrics.
Table 4: Experimental results for BigCloneBench (R – recall, P – precision, GPU memory and fine-tuning/evaluation time is given if applicable and available). Clone Detector
R
P
F1
CodeBERT
0.96
0.92
0.94
GraphCode BERT
0.95
0.94
0.95
UniXcoder
0.95
0.93
0.94
CodeT5 FA-AST+GMN NiCad v7.0.1 NIL StoneDetector
0.94 0.94 0.01 0.01 0.01
0.96 0.93 0.92 0.91 0.90
0.95 0.93 0.01 0.02 0.02
Runtime 77 min/ 64 min 634 min/ 101 min 444 min/ 73 min 1,368 min 2,662 min 1 min 1 min 1 min
GPU 10,549 MiB 15,941 MiB 12,833 MiB 31,973 MiB 5,377 MiB -
Table 5: Experimental results for benchmarks SeSaMe, SemanticCloneBench, FEMPD, and ProjectCodeNet (R – recall, P – precision). Clone Detector CodeBERT GraphCode BERT UniXcoder CodeT5 FA-AST+GMN NiCad v7.0.1 NIL StoneDetector Clone Detector
Furthermore, certain clone detectors support a threshold value which allows to define the clone detector’s permissiveness of false positives. In such cases, the performance of the clone detector can be illustrated in terms of its receiver operating characteristic (ROC) curve. The ROC curve plots recall and fall-out, i.e., true-positive and false-positive rate, respectively, at varying threshold values. In this plot, a random classification, which assigns a code clone by flipping a coin, results in a point on the diagonal line, i.e., true positive rate equals false positive rate. The better a clone detector, the farer is the clone detector’s characteristic function from this diagonal line. As the ROC curve allows for evaluating a clone detector for different threshold values, it is apparently more informative when compared to using precision, recall, and F1 score for a single configuration alone.
4.2
Experimental Results
As a first step, we replicate the CodeXGLUE benchmark (cf. Sect. 2), training/fine-tuning and evaluating DL-based clone detectors and just evaluating the conventional clone detectors on the same subset of BigCloneBench. As shown in Table 4, we can reproduce the results as reported in the literature, i.e., all five DL models achieve precision, recall, and F1 scores above 0.9. As expected, the conventional clone detectors in comparison only detect a small fraction of the code clones, resulting in a very low recall, while achieving similar precision scores above 0.9. In the table, we also provide the tools’ runtimes and the DL models’ used GPU memory. As expected, CodeT5 is the largest model and fine-tuning the pre-trained models takes considerably less time than full-training of FA-AST+GMN. Note that evaluating the samples of CodeXGLUE’s evaluation subset takes at least one hour in case of the DL models while lasting approx. one minute in case of the conventional tools.
CodeBERT GraphCode BERT UniXcoder CodeT5 FA-AST+GMN NiCad v7.0.1 NIL StoneDetector
SemanticCloneBench R P F1 0.74 0.54 0.62
R 0.94
SeSaMe P F1 0.15 0.26
0.45
0.72
0.56
0.39
0.33
0.36
0.53 0.28 0.38 0.02 0.14 0.05
0.78 0.78 0.80 1.0 0.99 1.0
0.63 0.41 0.52 0.04 0.25 0.09
0.55 0.11 0.53 0.45 0.53 0.30
0.55 0.29 0.61 1.0 1.0 1.0
0.55 0.16 0.57 0.63 0.69 0.47
R 0.97
FEMPD P F1 0.62 0.76
ProjectCodeNet R P F1 0.81 0.53 0.64
0.65
0.64
0.65
0.53
0.61
0.57
0.68 0.40 0.54 0.18 0.34 0.41
0.67 0.67 0.70 0.62 0.66 0.70
0.67 0.50 0.61 0.28 0.45 0.52
0.78 0.73 0.17 0.02 0.19 0.10
0.51 0.56 0.75 1.0 0.87 0.98
0.62 0.63 0.28 0.04 0.31 0.18
Training or fine-tuning the Deep Learning models on the BigCloneBench subset and evaluating them on one of the other benchmarks, i.e., FEMPD, SeSaMe, SemanticCloneBench, or ProjectCodeNet yet paints a different picture. Like mentioned above, we want to investigate on the models’ generalizability using this zero-shot evaluation scenario. As can be seen in Table 5, performance deteriorates for all five DL models on all four benchmarks with F1 scores dropping on average by approx. 41%. For instance, while recall remains on the same level or drops at most to 0.74 for CodeBERT, its precision shrinks on average to 0.46. Note that a precision score 0.5 equals flipping a coin for deciding whether an identified code clone is indeed a code clone or not. The other Transformer models achieve better precision but lower recall compared to CodeBERT and there is apparently not a single model which outperforms the other ones on all benchmarks. Furthermore, we observe that the Transformer models’ performance in particular degrades for benchmark SeSaMe, which is striking considering the benchmark’s origin in opensource production projects and its ground-truth quality (Sect. 2). Also remarkable, the single-task model FA-AST+GMN achieves the best precision among the five DL clone detectors over the four benchmarks. In contrast, we observe in general a slightly better recall for the three conventional clone detectors NiCad, NIL, and StoneDetector, as well as very good or acceptable precision scores with exclusion of benchmark FEMPD. Obviously, conventional tools
Figure 1: ROC curve for DL-based clone detectors and conventional clone detector NiCad on benchmark SeSaMe.
Figure 3: ROC curve for DL-based clone detectors and conventional tool NiCad on benchmark FEMPD.
Figure 2: ROC curve for DL-based clone detectors and conventional tool NiCad on benchmark SemanticCloneBench.
Figure 4: ROC curve for DL-based clone detectors and conventional tool NiCad on benchmark Project Codenet.
do not suffer from the same generalizability problem as the Deep Learning-based tools. Interestingly, they achieve their best overall performance on SeSaMe, where NiCad and NIL with F1 scores of 0.63, 0.69, respectively, even outperform the DL-based models. In the beginning of the section, we argued that just using precision, recall, and F1 score does not provide for a sufficient discussion of clone detectors’ performance in cases where they support a threshold value and that the ROC curve may then be used. We extend our experiments to track the five DL models recall and fall-out for different threshold values (ranging from 0.0 to 1.0 with the exception of FA-AST+GMN, where it ranges from -1.0 to 1.0). We additionally provide the ROC curve of the conventional tool NiCad for comparison (τ ∈ [0.0, 1.0], c.f. Sect 3). The resulting ROC curves are given in Fig. 1 to Fig. 4. Most apparent, FEMPD yields similar curves for all tools, indicating poor performance, which is attributed to its focus on functionally equivalent code, which seems harder to identify (cf. Sect. 2). Second, each bench-
mark has its own characteristic curves and we again do not find one DL model which outperforms the others for all benchmarks, while UniXcoder and FAAST+GMN show better curves on average (the closer to the upper left the better). Eventually, and maybe unexpectedly, NiCad shows superior sensitivity and specificity in all benchmarks beside FEMPD.
5
RELATED WORK
While BigCloneBench has been widely used for training and evaluating Deep Learning clone detectors for Java, BigCloneBench’s suitability for this purpose became disputed in recent years. Krinke et al. focus on the general benchmark’s ground truth quality and its widespread usage as training dataset in (Krinke and Ragkhitwetsagul, 2022; Krinke and Ragkhitwetsagul, 2025). They specifically identify the benchmark’s overlapping functionalities, invalid positive samples of code clones, bias and imbalance with respect to
Table 6: Related work on Java code clone detection with CodeBERT (R – Recall, P – Precision, BCB – BigCloneBench, SCB – SemanticCloneBench, superscripts *,†,¶,§ indicate different subsets of BigCloneBench). (Lu et al., 2021) (Sonnekalb et al., 2022) (Sonnekalb et al., 2022) (Pinku et al., 2024) (Kitsios et al., 2025) (Kitsios et al., 2025) (Arshad et al., 2022) (Arshad et al., 2022) our paper
Train. BCB BCB BCB BCB§ BCB† BCB† BCB¶ BCB¶ BCB
Eval. BCB BCB* BCB** SCB BCB‡ SCB SCB Android SCB
R n/a 0.52 0.33 0.47 0.84 0.50 0.73 0.64 0.74
P n/a 0.98 0.98 0.70 0.91 0.96 0.53 0.87 0.54
F1 0.94 0.68 0.49 0.56 0.86 0.66 0.61 0.74 0.62
functionalities and semantic code clones as issues impairing its usage. Note that Krinke et al. do not provide experimental analysis of these issues besides a manual investigation of a random sample of BigCloneBench’ code clones. We hope to provide, with our work, comprehensive experimental evidence to enrich this ongoing discussion. Some research experimentally analyze the suitability of BigCloneBench and in particular subsets thereof for Deep Learning of clone detectors: In (Sonnekalb et al., 2022), the authors evaluate a CodeBERT model fine-tuned for clone detection using the CodeXGLUE subset. However, they use their own evaluation datasets, which are derived from BigCloneBench, making sure to rule out code duplicates or the same functionalities. As a result, they report a drop of CodeBERT’s recall from 0.96 to 0.52 and 0.33, respectively, while precision even improves. Note that in our experiments, we rather observe a degradation of CodeBERT’s precision and not recall when using unseen data for evaluation. In the same vein, Schäfer et al. investigate the impact of using a more rigorous segregation of training and evaluation data for FA-AST+GMN on BigCloneBench in (Schäfer et al., 2022). Using samples of different functionalities in training and evaluation data results in a drop of both, recall and precision, and deteriorates the model’s F1 score from 0.95 to 0.72. Note that they apply FA-AST+GMN on a register-based intermediate representation of Java Bytecode instead of Java’s source code and also use the whole BigCloneBench and not a subset thereof. An analysis of the DLbased clone detectors ASTNN (Zhang et al., 2019) and TBCCD for the C programming language on the OJClone benchmark revealed similar effects (Liu et al., 2021). Its authors also investigate possible mitigations, e.g., increasing training data diversity, addressing the out-of-vocabulary problem, and integrating a human-in-the-loop mechanism. In (Arshad et al., 2022), the authors use a CodeBERT model fine-tuned on BigCloneBench for zero-
shot evaluation, much in the same way as we do, and report on observed significant drops of recall and precision by 15%-44%, a finding similar to our results. They however, in contrast to our research, only consider CodeBERT and two evaluation datasets, i.e., SemanticCloneBench and an Android benchmark. They though demonstrate that additionally fine-tuning CodeBERT on the evaluation datasets helps to restore much of the model’s prior performance. Also similarly to our approach, Pinku et al. investigate the usage of Deep Learning to code clone detection (Pinku et al., 2024). They again only consider two of the models included in our experiments, i.e. CodeBERT and FA-AST+GMN. They do not include conventional clone detectors for comparison and do not examine varying threshold values. They though address with ASTNN (Zhang et al., 2019) another graph-based model, and with GPTCloneBench (Alam et al., 2023) another benchmark besides SemanticCloneBench, as well as cross-language approaches. Overall, they report similar results for training on BigCloneBench and zero-shot evaluation on SemanticCloneBench and note a deterioration in the F1 score (0.68 and 0.56 for FA-AST+GMN and CodeBERT, respectively). Interestingly, they observe higher recall and lower precision for FA-AST+GMN and higher precision and lower recall for CodeBERT than we do. Kitsios et al. as well look into the problem of unseen functionalities for code clone detection with models CodeBERT, ASTNN, and CodeGrid in (Kitsios et al., 2025). They also train the models on BigCloneBench and evaluate them on a functionalitydistinct subset of BigCloneBench and on SemanticCloneBench. A deterioration in the models F1 score is observed, while in particular CodeBERT’s recall and ASTNN’s precision is impaired. The authors also consider large language models, i.e., GPT-4o, Llama 3.3 and DeepSeek, with an in general worse performance when compared to CodeBERT or ASTNN and report on contrastive learning for partially mitigating the problem of unseen functionalities. In (Alam et al., 2025), the authors present their findings when comparing performance of conventional clone detectors and two Deep Learning models, i.e., CodeBERT and ASTNN on benchmarks BigCloneBench, GPTCloneBench, and SemanticCloneBench. Similar to us, they observe higher recall but a much degraded precision (0.51-0.54) of Deep learning models in comparison with conventional clone detectors like StoneDetector (Heinze et al., 2026). However, we again include a larger number of Deep Learning models and benchmarks in our experiments and provide further elaborations on this insight by considering varying threshold values and
the ROC metric, which is not included in (Alam et al., 2025). Like us, they also note the better execution times and scalability of conventional clone detectors.
6
CONCLUSION
In this paper, we present our replication and evaluation experiments on generalizability of Deep Learning approaches (DL) to Java code clone detection. In the experiments, we analyze the detection performance of five state-of-the-art DL models, i.e., CodeBERT, GraphCodeBERT, UniXcoder, CodeT5, and FA-AST+GMN as trained/fine-tuned on BigCloneBench under a zero-shot evaluation scenario using the four benchmarks FEMPD, SemanticCloneBench, SeSaMe, and ProjectCodeNet. We also provide an in-depth analysis of the models’ performance in comparison with the conventional tools NiCad, NIL, and StoneDetector. Our experiments demonstrate a significant drop of the DL models performance under zero-shot evaluation (approx. 41% in their F1 scores), that clone detectors’ performance is coupled to the characteristics of the used evaluation benchmark, and that the conventional clone detector NiCad in general outperforms the DL models under zero-shot evaluation. With our work, we hope to contribute insights for further research and provide all datasets used in our experiments online16 . In future work, we want to extend our experiments with respect to more Deep Learning models and benchmarks for Java code clone detection, e.g., ASTNN (Zhang et al., 2019), CDLH (Wei and Li, 2017), and GPTCloneBench (Alam et al., 2023). We also want to integrate our experiments into the CloReCo platform (Burock et al., 2025), in order to facilitate reproducibility of clone detector performance analysis. Eventually, we are interested in analyzing ways to improve the performance of DL-based clone detectors on unseen code, e.g., using techniques of domain adaptation (Gruner et al., 2023).
ACKNOWLEDGEMENTS The author would like to thank Daniel Barié for providing the GPU resources for the experiments.
16 https://doi.org/10.5281/zenodo.19581107
REFERENCES Al-Omari, F., Roy, C. K., and Chen, T. (2020). Semanticclonebench: A semantic code clone benchmark using crowd-source knowledge. In IEEE 14th International Workshop on Software Clones, IWSC 2020, London, ON, Canada, February 18, 2020, pages 57–63. IEEE. Alam, A. I., Roy, P. R., Al-Omari, F., Roy, C. K., Roy, B., and Schneider, K. A. (2023). Gptclonebench: A comprehensive benchmark of semantic clones and cross-language clones using GPT-3 model and semanticclonebench. In IEEE International Conference on Software Maintenance and Evolution, ICSME 2023, Bogotá, Colombia, October 1-6, 2023, pages 1–13. IEEE. Alam, A. I., Roy, P. R., Al-Omari, F., Roy, C. K., Roy, B., and Schneider, K. A. (2025). Are classical clone detectors good enough for the AI era? In IEEE International Conference on Software Maintenance and Evolution, ICSME 2025, Auckland, New Zealand, September 7-12, 2025, pages 295–307. IEEE. Amme, W., Heinze, T. S., and Schäfer, A. (2021). You look so different: Finding structural clones and subclones in java source code. In IEEE International Conference on Software Maintenance and Evolution, ICSME 2021, Luxembourg, September 27 - October 1, 2021, pages 70–80. IEEE. Arshad, S., Abid, S., and Shamail, S. (2022). Codebert for code clone detection: A replication study. In 16th IEEE International Workshop on Software Clones, IWSC 2022, Limassol, Cyprus, October 2, 2022, pages 39–45. IEEE. Burock, F., Amme, W., Heinze, T. S., and Ostryanin, E. (2025). Cloreco: Benchmarking platform for code clone detection. In Mecella, M., Rensink, A., and Maciaszek, L. A., editors, Proceedings of the 20th International Conference on Software Technologies, ICSOFT 2025, Bilbao, Spain, June 10-12, 2025, pages 394–399. SCITEPRESS. Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., and Zhou, M. (2020). Codebert: A pre-trained model for programming and natural languages. In Cohn, T., He, Y., and Liu, Y., editors, Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, Findings of ACL, pages 1536–1547. Association for Computational Linguistics. Gruner, B., Sonnekalb, T., Heinze, T. S., and Brust, C. (2023). Cross-domain evaluation of a deep
learning-based type inference system. In 20th IEEE/ACM International Conference on Mining Software Repositories, MSR 2023, Melbourne, Australia, May 15-16, 2023, pages 158–169. IEEE. Guo, D., Lu, S., Duan, N., Wang, Y., Zhou, M., and Yin, J. (2022). Unixcoder: Unified cross-modal pre-training for code representation. In Muresan, S., Nakov, P., and Villavicencio, A., editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7212–7225. Association for Computational Linguistics. Guo, D., Ren, S., Lu, S., Feng, Z., Tang, D., Liu, S., Zhou, L., Duan, N., Svyatkovskiy, A., Fu, S., Tufano, M., Deng, S. K., Clement, C. B., Drain, D., Sundaresan, N., Yin, J., Jiang, D., and Zhou, M. (2021). Graphcodebert: Pre-training code representations with data flow. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. Heinze, T. S., Schäfer, A., and Amme, W. (2026). Stonedetector : Conventional and versatile code clone detection for java. J. Syst. Softw., 236:112799. Higo, Y. (2024). Dataset of functionally equivalent java methods and its application to evaluating clone detection tools. IEICE Trans. Inf. Syst., 107(6):751–760. Kamp, M., Kreutzer, P., and Philippsen, M. (2019). Sesame: a data set of semantically similar java methods. In Storey, M. D., Adams, B., and Haiduc, S., editors, Proceedings of the 16th International Conference on Mining Software Repositories, MSR 2019, 26-27 May 2019, Montreal, Canada, pages 529–533. IEEE / ACM. Kitsios, K., Sovrano, F., Barr, E. T., and Bacchelli, A. (2025). Detecting semantic clones of unseen functionality. In 40th IEEE/ACM International Conference on Automated Software Engineering, ASE 2025, Seoul, Korea, Republic of, November 16-20, 2025, pages 1312–1324. IEEE. Krinke, J. and Ragkhitwetsagul, C. (2022). Bigclonebench considered harmful for machine learning. In 16th IEEE International Workshop on Software Clones, IWSC 2022, Limassol, Cyprus, October 2, 2022, pages 1–7. IEEE. Krinke, J. and Ragkhitwetsagul, C. (2025). How the misuse of a dataset harmed semantic clone detection. CoRR, abs/2505.04311.
Liu, C., Lin, Z., Lou, J., Wen, L., and Zhang, D. (2021). Can neural clone detection generalize to unseen functionalities? In 36th IEEE/ACM International Conference on Automated Software Engineering, ASE 2021, Melbourne, Australia, November 15-19, 2021, pages 617–629. IEEE. Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C. B., Drain, D., Jiang, D., Tang, D., Li, G., Zhou, L., Shou, L., Zhou, L., Tufano, M., Gong, M., Zhou, M., Duan, N., Sundaresan, N., Deng, S. K., Fu, S., and Liu, S. (2021). Codexglue: A machine learning benchmark dataset for code understanding and generation. In Vanschoren, J. and Yeung, S., editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual. Nakagawa, T., Higo, Y., and Kusumoto, S. (2021). NIL: large-scale detection of large-variance clones. In Spinellis, D., Gousios, G., Chechik, M., and Penta, M. D., editors, ESEC/FSE ’21: 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Athens, Greece, August 23-28, 2021, pages 830–841. ACM. Pinku, S. N., Mondal, D., and Roy, C. K. (2024). On the use of deep learning models for semantic clone detection. In IEEE International Conference on Software Maintenance and Evolution, ICSME 2024, Flagstaff, AZ, USA, October 6-11, 2024, pages 512–524. IEEE. Puri, R., Kung, D. S., Janssen, G., Zhang, W., Domeniconi, G., Zolotov, V., Dolby, J., Chen, J., Choudhury, M. R., Decker, L., Thost, V., Buratti, L., Pujar, S., Ramji, S., Finkler, U., Malaika, S., and Reiss, F. (2021). Codenet: A large-scale AI for code dataset for learning a diversity of coding tasks. In Vanschoren, J. and Yeung, S., editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual. Roy, C. K. and Cordy, J. R. (2008). NICAD: accurate detection of near-miss intentional clones using flexible pretty-printing and code normalization. In Krikhaar, R. L., Lämmel, R., and Verhoef, C., editors, The 16th IEEE International Conference on Program Comprehension, ICPC 2008, Amsterdam, The Netherlands, June 10-13, 2008, pages 172–181. IEEE. Roy, C. K., Cordy, J. R., and Koschke, R. (2009). Comparison and evaluation of code clone de-
tection techniques and tools: A qualitative approach. Sci. Comput. Program., 74(7):470–495. Schäfer, A., Amme, W., and Heinze, T. S. (2022). Experiments on code clone detection and machine learning. In 16th IEEE International Workshop on Software Clones, IWSC 2022, Limassol, Cyprus, October 2, 2022, pages 46–52. IEEE. Sonnekalb, T., Gruner, B., Brust, C., and Mäder, P. (2022). Generalizability of code clone detection on codebert. In 37th IEEE/ACM International Conference on Automated Software Engineering, ASE 2022, Rochester, MI, USA, October 10-14, 2022, pages 143:1–143:3. ACM. Svajlenko, J. and Roy, C. K. (2015). Evaluating clone detection tools with bigclonebench. In Koschke, R., Krinke, J., and Robillard, M. P., editors, 2015 IEEE International Conference on Software Maintenance and Evolution, ICSME 2015, Bremen, Germany, September 29 - October 1, 2015, pages 131–140. IEEE. Wang, W., Li, G., Ma, B., Xia, X., and Jin, Z. (2020). Detecting code clones with graph neural network and flow-augmented abstract syntax tree. In Kontogiannis, K., Khomh, F., Chatzigeorgiou, A., Fokaefs, M., and Zhou, M., editors, 27th IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2020, London, ON, Canada, February 18-21, 2020, pages 261–271. IEEE. Wang, Y., Wang, W., Joty, S. R., and Hoi, S. C. H. (2021). Codet5: Identifier-aware unified pretrained encoder-decoder models for code understanding and generation. In Moens, M., Huang, X., Specia, L., and Yih, S. W., editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 8696– 8708. Association for Computational Linguistics. Wei, H. and Li, M. (2017). Supervised deep features for software functional clone detection by exploiting lexical and syntactical information in source code. In Sierra, C., editor, Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pages 3034–3040. ijcai.org. Zhang, J., Wang, X., Zhang, H., Sun, H., Wang, K., and Liu, X. (2019). A novel neural source code representation based on abstract syntax tree. In Atlee, J. M., Bultan, T., and Whittle, J., editors, Proceedings of the 41st International Con-
ference on Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019, pages 783–794. IEEE / ACM.