Can Code Evaluation Metrics Detect Code Plagiarism? Fahad Ebrahim
Mike Joy
University of Warwick Coventry, United Kingdom [email protected]
University of Warwick Coventry, United Kingdom [email protected]
arXiv:2604.25778v1 [cs.SE] 28 Apr 2026
Abstract Source Code Plagiarism Detection (SCPD) plays an important role in maintaining fairness and academic integrity in software engineering education. Code Evaluation Metrics (CEMs) are developed for assessing code generation tasks. However, it remains unclear whether such metrics can reliably detect plagiarism across different levels of modification (L1-L6), increasing in complexity. In this paper, we perform a comparative empirical study using two open-source labelled datasets, ConPlag (raw and template-free versions) and IRPlag. We evaluate five CEMs, namely CodeBLEU, CrystalBLEU, RUBY, Tree Structured Edit Distance (TSED), and CodeBERTScore. The performance is evaluated using thresholdfree ranking-based measures to assess overall, per dataset, and per-level plagiarism performance. The results are compared against state-of-the-art (SOTA) Source Code Plagiarism Detection Tools (SCPDTs), JPlag and Dolos. Our findings show that without preprocessing, Dolos achieves the highest overall ranking performance, while among the individual metrics, CrystalBLEU, CodeBLEU, and RUBY outperform JPlag. Performance is strongest at L1 and drops from L4 onward, while CrystalBLEU remains competitive on L6. With preprocessing, CrystalBLEU surpasses Dolos overall. Per dataset, Dolos achieved the best ranking on the ConPlag raw dataset, while CrystalBLEU was the best-performing metric on the remaining datasets. At the plagiarism levels, Dolos remains strongest on L4, while CrystalBLEU leads most of the remaining difficult levels. These results indicate that CEMs are comparable to dedicated tools in terms of ranking metrics.
CCS Concepts • Software and its engineering → Software verification and validation; • Computing methodologies → Artificial intelligence; • Applied computing → Education.
Keywords Source Code Plagiarism Detection, Code Evaluation Metrics, Code Similarity, Software Engineering Education
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. Conference’17, Washington, DC, USA © 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-XXXX-X/2018/06 https://doi.org/XXXXXXX.XXXXXXX
ACM Reference Format: Fahad Ebrahim and Mike Joy. 2018. Can Code Evaluation Metrics Detect Code Plagiarism?. In . ACM, New York, NY, USA, 10 pages. https://doi.org/ XXXXXXX.XXXXXXX
1
Introduction
Source code similarity refers to the degree to which two programs are alike [29]. This similarity can be measured in various ways, ranging from simple text matching to more comprehensive structural and semantic comparisons. These different forms of similarity are used in software engineering applications such as malware detection and vulnerability analysis. Another important application is SCPD, which aims to determine whether a program has been copied or reused without proper acknowledgement. In software engineering education, plagiarism can affect fairness and academic integrity [2]. Therefore, reliable SCPD methods are essential to ensure fair assessment. Plagiarism in source code can occur at different levels of modification. Faidhi and Robinson proposed a six-level taxonomy (L1-L6) that can be grouped into three categories [9]. The lower levels (L1-L3) involve syntactic changes such as formatting, identifier renaming, and changing the order of statements. The intermediate levels L4 and L5 include structural modifications related to methods, loops, and conditions. The higher level, L6, involves larger changes in semantics, structure, and decision logic. Different plagiarism levels require similarity measures that can capture lexical, structural, and semantic aspects of code. CEMs were primarily developed for code generation tasks, in which a generated program is compared against a reference implementation for tasks such as code generation, translation, or summarisation. Although they were not developed for plagiarism detection, these metrics can produce numerical similarity scores that can capture different characteristics of source code, such as syntax, structure and semantics. This raises a key question: can CEMs detect source code plagiarism across different levels of modifications? To answer this question, we perform a comparative empirical study using two open-source plagiarism datasets, ConPlag [25], and IRPlag [12], both labelled with plagiarism levels. We evaluate five CEMs: CodeBLEU [22], CrystalBLEU [8], RUBY [27], TSED [26], and CodeBERTScore [31]. These metrics measure similarity based on token overlap, syntactic structure, tree distance, and embedding representations. We assess their performance using threshold-free ranking measures, which evaluate how well plagiarised pairs are ranked above non-plagiarised pairs without fixing a decision threshold. We use threshold-free evaluation, as plagiarism detection is highly sensitive to the choice of threshold. Small changes in the threshold can significantly affect the evaluation metrics. The analysis is conducted overall, per dataset, and by plagiarism level to examine how
Conference’17, July 2017, Washington, DC, USA
performance changes as modifications become more complex. Finally, we compare the results with SOTA SCPDTs, JPlag [21] and Dolos [16], to determine how well CEMs compare against established plagiarism detection tools. Both JPlag and Dolos preprocess the code by default as part of their algorithms, therefore, we also compare the impact of preprocessing on the performance of CEMs. This paper makes three main contributions: • We present a systematic evaluation of CEMs for source code plagiarism detection. We study multiple datasets and all plagiarism levels (L1-L6). • We show that CEMs achieve performance comparable to SCPDTs under ranking-based evaluation. We also show that preprocessing is critical. It consistently improves performance and can change the ranking of methods. • We show that performance drops from L4 onwards for all methods. This highlights the difficulty of detecting structural and semantic plagiarism. We also show that combining metrics improves ranking accuracy and is a promising direction. The paper is organised as follows. Section 2 presents the background and related work. Section 3 describes the methodology, including the datasets, the selected CEMs, and the evaluation measures. Section 4 presents the results. Section 5 discusses the findings, limitations, threats to validity, and implications. The paper concludes with a summary and an outline of future work.
Fahad Ebrahim and Mike Joy
semantic methods that analyse representations such as Abstract Syntax Trees (ASTs) [1, 29]. More recent work often combines multiple signals to capture both surface-level similarity and deeper semantic similarity. Building on these methodological approaches, several systems and tools have been developed to support practical plagiarism detection. Several SCPDTs have been reviewed and compared in the literature [1, 19, 29]. In this study, we focus on two widely used open-source and actively maintained tools. JPlag [21] is a tool that supports languages such as Java, C++, C, and Python. It is based on the Greedy String Tiling (GST) algorithm [28], which finds maximal matching substrings between tokenised programs. By comparing token sequences instead of raw text, JPlag reduces the effect of simple formatting changes. However, since it relies on substring matching, it mainly captures lexical and surfacelevel similarities. JPlag is currently maintained and under active research [23, 24]. Dolos [16] follows a hybrid approach. It combines tokenisation, AST parsing, fingerprinting with rolling hash functions, and indexing. By using ASTs, Dolos captures structural information in addition to token-level similarity. This allows it to detect both lexical and structural similarities and makes it more robust to complex code modifications. Dolos is currently maintained and under active research [13–15].
2.3 2
Background and Related Work
In this section, we discuss plagiarism levels, SCPDTs, and empirical studies involving CEMs applied to code similarity tasks.
2.1
Plagiarism Levels
As described by Faidhi and Robinson [9], plagiarism in programming assignments can take place at different levels, ranging from surface edits to heavy logic changes. L1) Changing only the appearance of the code, such as editing comments, indentation, or spacing. L2) Renaming variables, functions, or classes without changing how the program works. L3) Adding, removing, or reordering constants, variables, or functions in the declarations. L4) Reorganising functions, for example, by changing their parameters, combining several functions into one, or splitting one function into smaller ones. L5) Replacing control structures with equivalent ones, such as using a while loop instead of a for loop, or rewriting conditions differently. L6) Changing the internal decision logic by rewriting expressions or conditional statements while keeping the overall behaviour the same. As the plagiarism level increases, the changes become more complex and more difficult to detect [1].
2.2
Source Code Plagiarism Detection Tools
In general, SCPD methods use different strategies to measure similarity between programs. These approaches range from lexical techniques based on token sequence comparison to structural and
Code Evaluation Metrics for Code Similarity Tasks
There have been only a few studies that inspect CEMs on code similarity tasks. Nikiema et al. systematically examined the robustness of code semantic similarity metrics under controlled transformations for both text and source code [18]. The authors evaluated lexical, embeddingbased, structure-aware, and Large Language Model (LLM)-based metrics and showed that several metrics assign high similarity to semantically different code. In particular, embedding-based measures such as CodeBERT [10] and BERTScore [30], which are based on the BERT model [3], were shown to struggle when small surface changes alter program behaviour. The study also concluded that the choice of distance function, such as cosine versus Euclidean distance, significantly affects semantic interpretation. Dristi and Dwyer [6] analysed surface bias in reference-based CEMs, including CodeBLEU, CrystalBLEU, CodeBERTScore, and CodeScore [5]. They introduced LoCaL, a benchmark dataset of 3117 Python code pairs, labelled with functional similarity using differential fuzzing. The study defined two challenging scenarios, Similar Form Different Semantics and Different Form Similar Semantics, and showed that all four metrics degraded significantly in these cases. CodeBLEU and CodeBERTScore exhibited a strong positive correlation with surface similarity, and CrystalBLEU failed to differentiate when non-equivalent pairs were surface-similar. These two works focused on functional and semantic similarity. The evaluation was based on whether two code snippets behave the same or are semantically equivalent. In contrast, plagiarism detection aims to identify copied or derived code, even when modifications change the structure or behaviour. Therefore, this work focuses on plagiarism detection and evaluates how these metrics perform across different levels of plagiarism.
Can Code Evaluation Metrics Detect Code Plagiarism?
Conference’17, July 2017, Washington, DC, USA
Table 1: Datasets details: counts of non-plagiarised vs. plagiarised pairs, number of plagiarised pairs per level (L1-L6), and total number of instances. Dataset
Figure 1: High-Level Methodology.
3
Code Evaluation Metrics as Plagiarism Detectors
This section presents the overall methodology, datasets, selected CEMs and their details, and performance evaluation metrics.
3.1
Overall Methodology
The high-level methodology used in this paper is presented in Figure 1. The process begins with the code pairs extracted from the datasets, which form the common input to all similarity estimation steps. We estimate similarity values using the CEMs and the tools (JPlag and Dolos). We then evaluate the ranking performance of both the metrics and the tools and conduct a comparative analysis. We also apply preprocessing to the CEMs and examine its impact on ranking performance. This study aims to answer the following research questions. RQ1: How well do CEMs detect source code plagiarism overall and across different plagiarism levels? RQ2: What is the impact of preprocessing on the performance of CEMs, overall and per plagiarism level? RQ3: How do CEMs compare with the current state-of-the-art plagiarism detection tools in detecting source code plagiarism, overall and per plagiarism level? To address RQ3, the main question of the paper, we formulate the following hypothesis H, namely that CEMs achieve performance comparable to SCPDTs under ranking-based evaluation.
3.2
Datasets
There is a scarcity of plagiarism datasets due to their private and sensitive nature. We use two main open-source datasets. The details of both datasets are available in Table 1, in terms of the count of instances, the count of plagiarised and non-plagiarised pairs, along with the count of plagiarised files per level. The ConPlag dataset [25] is a manually constructed dataset for source code plagiarism in competitive programming contests. It consists of 911 labelled Java solution pairs collected from 21 CodeForces problems, including 251 plagiarised pairs and 660 nonplagiarised pairs. The dataset was built from 4,695 accepted Java submissions and filtered using existing detection tools before final manual labelling. ConPlag is provided in two versions: ConPlag1, a raw version containing the original submissions, and ConPlag2, a template-free version where common contest template code has been manually removed by the authors of the dataset. We label the plagiarism levels (L1-L6) manually on the ConPlag dataset. IRPlag [12] is a publicly available dataset designed for evaluating SCPD. It consists of 467 Java source code files covering seven introductory programming tasks, including one original solution per task, multiple intentionally plagiarised versions, and independently written non-plagiarised solutions. The plagiarised files were created
Non-plag.
Plag.
L1
L2
L3
L4
L5
L6
Total
ConPlag1 ConPlag2 IRPlag
660 660 95
251 251 365
71 71 61
4 4 57
11 11 65
63 63 60
67 67 59
35 35 63
911 911 460
All (Pooled)
1415
867
203
65
87
186
193
133
2282
using different plagiarism levels based on the Faidhi and Robinson taxonomy [9]. The plagiarism levels (L1-L6) are already labelled as part of initial annotations. The instances in the datasets are relatively long, especially in ConPlag1. Based on the unique code snippets, the average number of tokens per fragment is 1085, 589, and 172 for ConPlag1, ConPlag2, and IRPlag, respectively. This motivates reducing the average token length through preprocessing. The preprocessing steps include removing comments, extra whitespace and new lines, and removing Java import and package statements. After preprocessing, the average number of tokens drops to 958, 483, and 137, corresponding to reductions of 11.72%, 17.97%, and 20.34% for ConPlag1, ConPlag2, and IRPlag, respectively. For the same unique code snippets, the maximum token counts are 11,909 in ConPlag1, 10,382 in ConPlag2, and 338 in IRPlag for the raw code, compared with 11,677, 10,186, and 283 after preprocessing. For CodeBERTScore, which uses CodeBERT, the maximum token length is 512. The number of unique code fragments exceeding 512 tokens is 743 in ConPlag1 and 443 in ConPlag2, which is reduced to 654 and 268 after preprocessing. None of the IRPlag code fragments exceeds 512 tokens.
3.3
Selected Code Evaluation Metrics
This section covers the technical details of the selected CEMs. 3.3.1 BLEU. Before detailing the CEMs, we start with one fundamental metric used for Natural Language Generation (NLG) evaluation and motivate the code metrics. It is the BLEU metric [20], which is an n-gram-based similarity metric originally proposed for machine translation. It measures the overlap between a candidate and a reference text using clipped n-gram precision combined with a brevity penalty. Let 𝐶 be the candidate program and 𝑅 the reference program. The clipped precision for n-grams of order 𝑛 is defined as Í 𝑝𝑛 =
n-gram∈𝐶 min(count𝐶 , count𝑅 )
Í
.
n-gram∈𝐶 count𝐶
This term measures how many n-grams in the candidate also appear in the reference. The clipping using min(count𝐶 , count𝑅 ) ensures that repeated n-grams in the candidate are not over-counted. The geometric mean over 𝑁 n-gram orders is
Prec = exp
𝑁 ∑︁ 𝑛=1
! 𝑤𝑛 log 𝑝𝑛 .
Conference’17, July 2017, Washington, DC, USA
Fahad Ebrahim and Mike Joy
This combines precision values for different n-gram lengths (e.g., unigrams, bigrams, and trigrams), so that both short and longer token sequences contribute to the final score. The brevity penalty is ( BP =
1 exp(1 − |𝑅|/|𝐶 |)
if |𝐶 | > |𝑅|, if |𝐶 | ≤ |𝑅|.
The brevity penalty reduces the score if the candidate is much shorter than the reference, preventing very short programs from receiving artificially high similarity values. The final BLEU score is BLEU = BP · Prec. In summary, BLEU ranges from 0 to 1, where higher values indicate greater lexical overlap. It assigns higher values when there is consistent lexical overlap between the candidate and reference across multiple n-gram lengths, while penalising excessively short outputs. 3.3.2 CodeBLEU. CodeBLEU extends BLEU by incorporating codespecific information. It combines four components: standard n-gram match, weighted n-gram match, AST match, and data-flow match. The final score is a weighted sum of these components. By including structural and data-flow information, CodeBLEU aims to better capture program behaviour compared to pure lexical overlap. 𝑆 CodeBLEU = 𝛼𝑆 n-gram + 𝛽𝑆 weighted + 𝛾𝑆 syntax + 𝛿𝑆 dataflow, where 𝛼 + 𝛽 + 𝛾 + 𝛿 = 1. Here, 𝑆 n-gram corresponds to standard BLEU, 𝑆 weighted assigns higher weights to keywords, 𝑆 syntax measures AST similarity, and 𝑆 dataflow measures overlap in data dependencies. 3.3.3 CrystalBLEU. CrystalBLEU modifies BLEU by removing the most frequent n-grams from the corpus before computing similarity. These frequent n-grams often represent common programming patterns or boilerplate code. By filtering them out, CrystalBLEU reduces inflated similarity caused by shared templates. This makes it more robust to trivial similarities. Let 𝐺𝑛 (𝐶) denote the set of n-grams of order 𝑛 in program 𝐶, and let 𝐹𝑛 denote the set of frequent n-grams in the corpus. Filtered n-grams are defined as 𝐺𝑛∗ (𝐶) = 𝐺𝑛 (𝐶) \ 𝐹𝑛 . BLEU is then computed using the filtered sets: CrystalBLEU = BLEU(𝐺𝑛∗ (𝐶), 𝐺𝑛∗ (𝑅)). 3.3.4 RUBY. RUBY incorporates structural similarity as it first compares Program Dependency Graphs (PDGs) when they can be constructed. If PDGs are not available, it falls back to comparing ASTs. If ASTs also cannot be constructed, it falls back to a weighted string edit distance between tokenised code. RUBY is defined as
GRS(𝐺, 𝑅) (PDG-based comparison), RUBY(𝐺, 𝑅) = TRS(𝐺, 𝑅) (AST-based comparison), STS(𝐺, 𝑅) (token-based fallback). where 𝐺𝑅𝑆 (𝐺, 𝑅) measures similarity between the PDGs of 𝐺 and 𝑅, 𝑇 𝑅𝑆 (𝐺, 𝑅) measures similarity between the ASTs of 𝐺 and 𝑅, and 𝑆𝑇 𝑆 (𝐺, 𝑅) measures weighted string edit distance between tokenised 𝐺 and 𝑅. 3.3.5 TSED. TSED measures structural similarity by computing the minimum cost required to transform one AST into another. Let 𝐺 1 and 𝐺 2 denote the ASTs of programs 𝑃𝑖 and 𝑃 𝑗 , respectively. The tree edit distance is defined as Δ(𝐺 1, 𝐺 2 ) = min
𝑛 ∑︁
ops
𝑤 (𝑜𝑝𝑖 ),
𝑖=1
where ops is a sequence of edit operations transforming 𝐺 1 into 𝐺 2 , and 𝑤 (𝑜𝑝𝑖 ) denotes the cost of the 𝑖th operation, such as insert, delete, or rename. To obtain a similarity score, the distance is normalised with respect to the maximum number of nodes between the two trees. The final similarity is computed as 𝑆 TSED = max 1 −
Δ(𝐺 1, 𝐺 2 ) ,0 . max(Nodes(𝐺 1 ), Nodes(𝐺 2 ))
3.3.6 CodeBERTScore. This metric extends BERTScore by using contextual representations from a pre-trained model, such as CodeBERT, to align tokens between the candidate and reference texts based on their cosine similarity.
3.4
Performance Evaluation
We evaluate each CEM using threshold-independent measures. In particular, we use the Area Under the Receiver Operating Characteristic Curve (AUROC) and Area Under the Precision-Recall Curve (AUPRC) [11]. These measures assess how well a metric ranks plagiarised pairs above non-plagiarised pairs without fixing a specific threshold. Let 𝑆 (𝑃𝑖 , 𝑃 𝑗 ) ∈ [0, 1] denote the similarity score between two programs. By varying a decision threshold 𝜏, we obtain different classification outcomes. Rather than selecting a single threshold, AUROC and AUPRC summarise performance across all possible thresholds. The Receiver Operating Characteristic (ROC) curve plots the True Positive Rate (TPR) against the False Positive Rate (FPR) at different threshold values. The TPR and FPR are defined as TP FP = Recall , FPR = TP + FN FP + TN AUROC measures the probability that a randomly selected plagiarised pair receives a higher similarity score than a randomly selected non-plagiarised pair. A value of 0.5 indicates random ranking, while a value of 1.0 indicates perfect separation. AUROC reflects the overall ranking ability of a metric. It is suitable when we are interested in separability between plagiarised and non-plagiarised pairs across all thresholds. TPR =
Can Code Evaluation Metrics Detect Code Plagiarism?
Conference’17, July 2017, Washington, DC, USA
The Precision-Recall Curve (PRC) plots Precision against Recall (TPR) at different thresholds, with Precision defined as TP TP + FP The area under the curve of Precision-Recall (PR) corresponds to the Average Precision (AP), which is equal to the following Precision =
AP =
𝑁 ∑︁
(𝑅𝑛 − 𝑅𝑛−1 ) · 𝑃𝑛 ,
𝑛=1
where 𝑃𝑛 is the precision at the 𝑛-th threshold, 𝑅𝑛 is the recall at the 𝑛-th threshold, and 𝑅0 = 0. AUPRC focuses on the trade-off between detecting plagiarised pairs and avoiding false positives. In many plagiarism-detection datasets, non-plagiarised pairs constitute the majority of instances. Therefore, AUPRC provides additional insight into practical performance, particularly under class imbalance and is more suitable for top-k ranking [17]. A higher AUPRC indicates that a metric maintains precision as recall increases. For this reason, both AUROC and AUPRC are reported to provide a complementary evaluation of ranking performance. In addition to overall AUROC and AUPRC values, we compute these measures separately for each plagiarism level 𝐿𝑘 . This allows us to examine how metric performance changes with structural and semantic modifications.
4
Results
In this section, we present the results of all CEMs in addition to JPlag and Dolos. The results are reported for the pooled dataset, for each dataset, and for each plagiarism level (L1–L6). The section also presents the impact of preprocessing on CEM. For all results, we use bootstrapping [4] with 10,000 resamples to estimate 95% confidence intervals for the performance evaluation metrics, following the recommendation of the authors of the ConPlag dataset [25]. Bootstrapped 95% confidence intervals for the pooled raw and preprocessed results are reported in Appendix A.1. We use the default settings for all metrics and tools. For readability, the tables report only the mean, while the values cited in the text include their 95% confidence intervals, and the full interval tables remain in the appendices and the GitHub repository. To test the hypothesis H, we perform paired bootstrap testing as recommended by Dror et al. [7], as the datasets are small in size and imbalanced.
4.1
Raw Results
We start with the results across the pooled datasets on the raw code snippets. The ROC curves presented in Figure 2 show that among the individual CEMs, CrystalBLEU is the best metric with an AUROC of 0.850 (95% CI [0.832, 0.866]), followed by CodeBLEU at 0.837 (95% CI [0.821, 0.854]) and RUBY at 0.822 (95% CI [0.804, 0.842]). We also report FusionTop3 as the unweighted average of the three metrics that showed consistently strong performance across datasets (CrystalBLEU, CodeBLEU, and RUBY). These three metrics were in the same range and achieved aligned scores, with a clear gap separating them from the other metrics. This aggregation is exploratory and does not involve parameter tuning or cross-validation. The fusion aims to assess whether combining
Figure 2: ROC curve across the pooled full datasets.
complementary similarity metrics improves ranking performance without additional tuning. FusionTop3 reaches an AUROC of 0.861 (95% CI [0.845, 0.878]), exceeding all individual metrics and nearly matching Dolos, which attains the highest pooled raw AUROC at 0.864 (95% CI [0.848, 0.879]). JPlag follows at 0.777 (95% CI [0.753, 0.798]), ahead of TSED and CodeBERTScore. The ROC curves show how well each method separates plagiarised and non-plagiarised pairs across different thresholds. For stronger methods, the curves rise quickly toward the top-left corner, which means they rank most plagiarised pairs above non-plagiarised ones. As plagiarism levels become more complex, the curves move closer to the diagonal line, indicating weaker separation. This visual pattern matches the decline observed from L4 onwards in the level-wise results. In addition to the overall CodeBLEU score, we report the performance of its individual components: n-gram match (CB-Ngram), weighted n-gram match (CB-Wngram), syntax match (CB-Syntax), and data-flow match (CB-Dataflow). Across datasets, the syntax component consistently achieves higher ranking values than the other components. For example, in ConPlag2 (raw), CB-Syntax reaches an AUROC of 0.893 (95% CI [0.870, 0.918]), compared to 0.765 (95% CI [0.728, 0.801]) for CB-Ngram and 0.702 (95% CI [0.665, 0.746]) for CB-Dataflow. Similar patterns are observed in other datasets. This indicates that structural similarity contributes more strongly than lexical overlap alone in separating plagiarised and non-plagiarised pairs in this setting. The PR curves for the pooled datasets are available in Figure 3. Here, FusionTop3 has the best AP value of 0.845 (95% CI [0.824, 0.861]), followed by Dolos at 0.842 (95% CI [0.823, 0.858]). Among the individual metrics, CrystalBLEU remains the strongest at 0.834 (95% CI [0.813, 0.854]), while JPlag follows at 0.762 (95% CI [0.736, 0.787]). The PR curves provide a different view of performance. They show how precision changes as recall increases. In some datasets, precision drops more quickly as recall grows, meaning that detecting more plagiarised pairs also increases false positives.
Conference’17, July 2017, Washington, DC, USA
Figure 3: PR curve across the pooled full datasets.
4.2
Figure 4: ROC curve across the pooled full datasets with preprocessing.
Impact of Preprocessing on Code Evaluation Metrics
With preprocessing, the ROC curve is available at Figure 4. Overall, applying preprocessing improves the performance of most CEMs. This leads FusionTop3 and CrystalBLEU to overtake Dolos, with AUROC values of 0.882 (95% CI [0.866, 0.897]) and 0.879 (95% CI [0.862, 0.895]), respectively, compared with 0.864 (95% CI [0.848, 0.879]) for Dolos. The remaining metrics also improve, while JPlag stays unchanged at 0.777 (95% CI [0.753, 0.798]). For CodeBLEU, the relative ordering of the components remains largely unchanged, with the syntax component maintaining higher values than the lexical and data-flow components. In terms of the PR curve, shown in Figure 5, preprocessing also increases most of the CEMs performance values. CrystalBLEU is the best method by AP with a value of 0.865 (95% CI [0.848, 0.885]), followed by FusionTop3 at 0.862 (95% CI [0.841, 0.878]) and then Dolos at 0.842 (95% CI [0.823, 0.858]). So, overall preprocessing increased the performance metrics of most CEMs, leading FusionTop3 and CrystalBLEU to surpass the SOTA SCPDTs, JPlag and Dolos, on the pooled results. Paired bootstrap comparisons for the pooled results are reported in Appendix A.2. In the raw setting, the top metrics all outperform JPlag, while only FusionTop3 remains statistically indistinguishable from Dolos. After preprocessing, FusionTop3 and CrystalBLEU achieve higher AUROC values than Dolos, whereas CodeBLEU and RUBY remain below Dolos but still well above JPlag. These results support the hypothesis, showing that CEMs achieve performance comparable to SCPDTs under certain settings.
4.3
Fahad Ebrahim and Mike Joy
Results Per Dataset
We proceed with the results per dataset, which are available in Table 2 for both AUROC and AP performance values for raw and preprocessed code pairs. For the raw code fragments, Dolos tops ConPlag1 with AUROC/AP 0.840/0.770 (95% CIs [0.808, 0.868] / [0.728, 0.819]), JPlag tops ConPlag2 with 0.929/0.883 (95% CIs [0.907, 0.947] / [0.855, 0.915]), and Dolos also tops IRPlag with 0.717/0.913 (95% CIs [0.661,
Figure 5: PR curve across the pooled full datasets with preprocessing.
0.764] / [0.889, 0.937]). FusionTop3 is competitive on the two ConPlag datasets, while CrystalBLEU remains the strongest individual metric on ConPlag1 and ConPlag2. With preprocessing, Dolos remains the best method on ConPlag1 at 0.840/0.770 (95% CIs [0.808, 0.868] / [0.728, 0.819]), CrystalBLEU becomes the best method on ConPlag2 at 0.942/0.897 (95% CIs [0.925, 0.957] / [0.866, 0.921]), and on IRPlag CrystalBLEU attains the highest AUROC at 0.722 (95% CI [0.665, 0.773]) while CodeBLEU n-gram attains the highest AP at 0.914 (95% CI [0.886, 0.937]). An important observation here is that the class distribution had an impact on the metrics. IRPlag has more plagiarised pairs (365) than non-plagiarised pairs (95), while ConPlag1 and ConPlag2 have more non-plagiarised pairs (660) than plagiarised pairs (251). This led to having high AP and lower AUROC in IRPlag compared to ConPlag.
Can Code Evaluation Metrics Detect Code Plagiarism?
Conference’17, July 2017, Washington, DC, USA
Table 2: AUROC and AP comparison across datasets: Raw (left) vs. Preprocessed (right). Bold indicates improvement after preprocessing. Underline indicates the highest value per column within each block.
Raw ConPlag1
Preprocessed
ConPlag2
IRPlag
ConPlag1
ConPlag2
IRPlag
Metric
AUROC
AP
AUROC
AP
AUROC
AP
AUROC
AP
AUROC
AP
AUROC
AP
CodeBLEU CB-Dataflow CB-Ngram CB-Syntax CB-Wngram
0.752 0.693 0.689 0.801 0.719
0.679 0.597 0.635 0.746 0.647
0.839 0.702 0.765 0.893 0.782
0.764 0.611 0.694 0.832 0.708
0.705 0.660 0.658 0.705 0.651
0.908 0.877 0.897 0.905 0.892
0.750 0.685 0.684 0.802 0.712
0.674 0.598 0.629 0.747 0.641
0.834 0.692 0.756 0.895 0.773
0.764 0.606 0.698 0.837 0.709
0.692 0.628 0.718 0.706 0.655
0.906 0.869 0.914 0.906 0.897
CrystalBLEU RUBY TSED CodeBERTScore
0.770 0.758 0.715 0.722
0.725 0.710 0.606 0.651
0.887 0.848 0.723 0.749
0.845 0.812 0.590 0.673
0.659 0.668 0.564 0.594
0.882 0.888 0.866 0.859
0.779 0.764 0.716 0.732
0.731 0.715 0.639 0.663
0.942 0.890 0.746 0.754
0.897 0.858 0.660 0.677
0.722 0.646 0.531 0.657
0.910 0.890 0.835 0.887
FusionTop3
0.780
0.729
0.894
0.844
0.687
0.895
0.786
0.729
0.935
0.885
0.700
0.903
JPlag Dolos
0.778 0.840
0.716 0.770
0.929 0.916
0.883 0.860
0.557 0.717
0.836 0.913
0.778 0.840
0.716 0.770
0.929 0.916
0.883 0.860
0.557 0.717
0.836 0.913
4.4
Results Per Plagiarism Level
Moving to the results per plagiarism level as shown in Table 3, with both raw and preprocessed code pairs, and both AUROC and AP values across plagiarism levels (L1-L6). Without preprocessing, CodeBLEU is best on L1, Dolos attains the highest AUROC on L2 while CodeBLEU-syntax attains the highest AP, FusionTop3 leads L3 in both AUROC and AP, Dolos is best on L4 and L5, and on L6 FusionTop3 gives the highest AUROC while CrystalBLEU gives the highest AP. With preprocessing, FusionTop3 leads AUROC on L2, L3, and L6, CrystalBLEU leads both AUROC and AP on L5 and the AP on L3 and L6, while Dolos remains the strongest method on L4. Overall, L1 and L2 are the easiest levels, whereas performance drops from L4 onward, particularly in AP, leading to more false positives.
5
Discussion
This section covers the findings and their interpretation and implications, discusses the limitations, and mentions the threats to validity.
5.1
Findings
The results presented in the previous section show consistent patterns across datasets, plagiarism levels, and evaluation metrics. In this section, we interpret these findings and examine what they imply about the effectiveness of CEMs compared to established plagiarism detection tools. We focus on three main aspects: overall detection performance, behaviour across plagiarism levels, and the impact of preprocessing. The discussion connects the quantitative results to practical implications for SCPD and highlights where performance remains strong and where limitations appear. The results show that CEMs can achieve comparable ranking performance to current established plagiarism detection tools. Several
metrics (CodeBLEU, CrystalBLEU, and RUBY) had higher performance than JPlag. On the raw pooled setting, Dolos achieved the highest AUROC, while CrystalBLEU was the strongest individual metric and FusionTop3 achieved the highest AP. After preprocessing, FusionTop3 and CrystalBLEU surpassed Dolos on the pooled results. CrystalBLEU removes the common code and potentially the template, keeping the important part of the code to be compared, which makes it suitable for plagiarism detection and keeps it the strongest individual metric across the evaluated CEMs. TSED did not perform well on the task because it focuses only on structural similarity. Therefore, heavy lexical changes, such as reordering statements and adding wrappers, were not well-captured by this metric. CodeBERTScore failed to separate between plagiarised and non-plagiarised pairs because all pairs produced high similarity values. This aligns with the findings of previous work [6], [18]. Preprocessing consistently improved the performance of several code metrics. Through investigation, we observe that preprocessing impacts the syntactic metrics and the embedding-based metric, CodeBERTScore. The n-grams are affected as they process the comments, and this introduces irrelevant or unnecessary tokens. So, removing comments improved metrics such as CrystalBLEU. In the components of CodeBLEU, we observe that the lexical n-gram components had a greater performance increase than the structurebased components with AST and data-flow. The increase is the highest in CodeBERTScore, as preprocessing helps with the limited token length of that metric, as it reduces the overall token length. Detection performance depends on the plagiarism level. Both tools and most CEMs effectively detected plagiarism across levels L1-L3. From L4 onwards, performance decreases across all methods. We observe that the scenarios that both metrics and tools frequently fail to detect are related to multiple code transformations and the creation of additional functions or classes. Therefore, they fail to detect more complex plagiarism scenarios.
Conference’17, July 2017, Washington, DC, USA
Fahad Ebrahim and Mike Joy
Table 3: Pooled overall performance per plagiarism level (L1-L6): Raw (left) vs. Preprocessed (right). Metrics reported as AUROC and AP. Bold indicates improvement after preprocessing. Underline indicates the highest value per level within each block.
Raw
Preprocessed
Metric
Score
L1
L2
L3
L4
L5
L6
L1
L2
L3
L4
L5
L6
CodeBLEU
AUROC AP AUROC AP AUROC AP AUROC AP AUROC AP
1.000 0.998 0.989 0.950 0.992 0.968 0.999 0.991 0.995 0.977
0.974 0.792 0.954 0.672 0.911 0.533 0.983 0.851 0.912 0.469
0.948 0.630 0.870 0.378 0.912 0.534 0.931 0.632 0.899 0.461
0.783 0.379 0.662 0.224 0.709 0.376 0.787 0.419 0.742 0.332
0.699 0.268 0.611 0.191 0.630 0.262 0.711 0.317 0.639 0.234
0.724 0.209 0.673 0.179 0.708 0.224 0.626 0.154 0.714 0.189
1.000 1.000 0.992 0.958 1.000 0.998 0.999 0.991 1.000 0.998
0.983 0.832 0.948 0.658 0.969 0.720 0.986 0.874 0.963 0.643
0.958 0.660 0.900 0.393 0.927 0.638 0.934 0.658 0.911 0.578
0.775 0.361 0.640 0.213 0.708 0.366 0.790 0.421 0.733 0.336
0.713 0.269 0.606 0.181 0.642 0.264 0.723 0.326 0.651 0.243
0.743 0.203 0.635 0.142 0.747 0.231 0.630 0.152 0.742 0.203
AUROC AP AUROC AP AUROC AP AUROC AP
0.994 0.973 0.995 0.978 0.977 0.896 0.835 0.772
0.930 0.583 0.955 0.776 0.952 0.506 0.453 0.088
0.955 0.679 0.941 0.703 0.879 0.304 0.508 0.128
0.811 0.483 0.766 0.407 0.672 0.180 0.546 0.212
0.745 0.373 0.711 0.338 0.607 0.155 0.521 0.197
0.727 0.233 0.651 0.171 0.599 0.120 0.423 0.085
1.000 0.997 0.999 0.995 0.995 0.943 0.916 0.811
0.986 0.774 0.987 0.873 0.972 0.703 0.678 0.135
0.976 0.779 0.946 0.654 0.895 0.340 0.673 0.179
0.834 0.485 0.776 0.366 0.684 0.209 0.592 0.206
0.794 0.404 0.745 0.333 0.624 0.173 0.600 0.226
0.767 0.254 0.677 0.169 0.597 0.110 0.504 0.098
FusionTop3
AUROC AP
0.999 0.990
0.959 0.764
0.967 0.719
0.825 0.463
0.756 0.351
0.739 0.216
1.000 0.999
0.991 0.849
0.981 0.726
0.838 0.439
0.793 0.372
0.772 0.227
JPlag
AUROC AP AUROC AP
0.993 0.934 0.998 0.991
0.963 0.732 0.983 0.842
0.887 0.408 0.938 0.709
0.739 0.301 0.842 0.544
0.650 0.244 0.790 0.402
0.514 0.108 0.690 0.207
0.993 0.934 0.998 0.991
0.963 0.732 0.983 0.842
0.887 0.408 0.938 0.709
0.739 0.301 0.842 0.544
0.650 0.244 0.790 0.402
0.514 0.108 0.690 0.207
CB-DF CB-Ngram CB-Syntax CB-WN CrystalBLEU RUBY TSED CodeBERTScore
Dolos
The exploratory fusion approach shows more consistent performance. It is the combination of three metrics, each of which captures different characteristics of source code. We observe that, on average, the fusion improves over the constituent individual metrics. This suggests that CEMs can be combined with different similarity algorithms in the future to improve performance. Dataset characteristics also influence metric behaviour. IRPlag differs from ConPlag1 and ConPlag2 in its class distribution, containing 365 plagiarised and 95 non-plagiarised pairs, whereas the ConPlag datasets contain more non-plagiarised than plagiarised pairs. This imbalance corresponds to differences observed between ROC and PR behaviour. The results indicate that evaluation outcomes are influenced by dataset composition and class distribution. Overall, the findings show that CEMs, particularly when combined and supported by preprocessing, provide comparable performance to established plagiarism detection tools.
5.2
Answers to Research Questions
We arrive at the following answers to the research questions: RQ1: Without preprocessing, the CEMs lag behind Dolos, but have better ranking performance than JPlag. CrystalBLEU is the
best-performing individual metric, as it removes the template or repetitive code. CodeBLEU and RUBY follow CrystalBLEU, with TSED and CodeBERTScore performing worst among the evaluated metrics. FusionTop3 further narrows the gap to Dolos. RQ2: Preprocessing improved the performance measures for most CEMs. This led FusionTop3 and CrystalBLEU to outperform Dolos on the pooled results and on ConPlag2, while Dolos remained best on ConPlag1. On IRPlag, CrystalBLEU achieved the highest AUROC and CodeBLEU n-gram achieved the highest AP. At the level-wise view, Dolos remained best on L4, whereas FusionTop3 and CrystalBLEU led most of the other challenging levels. RQ3: Without preprocessing and over the pooled dataset, the individual CEMs remain slightly below Dolos in terms of AUROC, although FusionTop3 nearly matches Dolos and achieves the highest AP, and the strongest individual metrics remain above JPlag. With preprocessing, FusionTop3 and CrystalBLEU achieve better pooled AUROC than both tools. Dolos still performs best on ConPlag1 and on L4.
Can Code Evaluation Metrics Detect Code Plagiarism?
5.3
Code Evaluation Metrics Limitations
CodeBERTScore has two main limitations, which are generating high similarity scores (>0.99) over plagiarised and non-plagiarised pairs, which makes it difficult to differentiate between them. The second limitation is related to the limited token length (<512), and code after that gets truncated. This aligns with the findings of [18]. TSED focuses more on ASTs and structural similarity and struggles with heavy syntax modification and further logical modifications. CodeBLEU, CrystalBLEU, and RUBY, along with JPlag and Dolos, still find it hard to detect higher levels of similarity, such as L5 and L6. They lead to a high number of false positives, leading to lower AP values.
5.4
Threats to Validity
This work is subject to several threats to validity. Firstly, the plagiarism levels L1 to L6 in ConPlag were manually labelled, which may slightly affect the per-level details and analysis. Secondly, all CEMs and plagiarism detection tools were used with default settings. No hyperparameter tuning or threshold optimisation was performed. Default configurations were used to reflect realistic usage scenarios. FusionTop3 was defined based on observed performance. Different parameters can lead to different results. Thirdly, the evaluation is conducted only on monolingual Java datasets. Therefore, this work does not consider other programming languages or cross-language plagiarism. Finally, JPlag was used initially during the annotation of the ConPlag datasets, which may introduce bias towards it during the evaluation of these datasets. Regarding the experimental work, we focused the comparative study on the ranking metrics. In future work, we should focus on retrieval and classification metrics.
5.5
Implications
In educational settings, CEMs are better suited for screening or filtering than for automatic judgment. Top CEMs can rank potentially plagiarised pairs higher than non-plagiarised ones. However, false positives and weaker performance on complex cases (L4-L6) mean that final decisions should remain with the instructor. These methods can support academic integrity efforts, but they should not be used in isolation. CEMs do not replace dedicated plagiarism detection tools, but they can complement them in specific cases. Based on the results, a simple workflow is recommended. First, apply preprocessing. Then, use a strong ranking method, such as CrystalBLEU, to flag suspicious pairs. Finally, confirm these cases using a dedicated plagiarism tool and manual inspection. From a research perspective, the results suggest that future work should focus on combining complementary similarity metrics rather than relying on a single method. The improved performance of FusionTop3 indicates that different representations capture distinct aspects of code similarity. The consistent performance drop from L4 onwards highlights a key limitation of current approaches. Methods that better capture program behaviour and semantic equivalence are needed to address higher-level plagiarism.
Conference’17, July 2017, Washington, DC, USA
6
Conclusion
This paper investigated whether CEMs can be adapted for SCPD in software engineering education. Using two open-source levellabelled datasets, ConPlag and IRPlag, we evaluated five representative metrics and compared their performance against established plagiarism detection tools, JPlag and Dolos, using threshold-free ranking-based measures. The results showed that CrystalBLEU, CodeBLEU, and RUBY achieve competitive overall performance and outperform JPlag on the pooled raw setting, while Dolos retains the highest raw AUROC. FusionTop3 nearly matches Dolos on AUROC and achieves the highest raw AP. The metrics performed strongly at lower plagiarism levels and remained competitive on L6, but performance dropped from L4 onward. Preprocessing improved most of the metrics, allowing FusionTop3 and CrystalBLEU to surpass Dolos on the pooled results, while Dolos remained strongest on ConPlag1 and on L4. As the metrics and tools struggle with larger and heavier changes in structure or logic, we will aim to develop a new tool or system that combines CEMs with complementary algorithms or approaches that focus more on semantics, leading to better detection on L5-L6 plagiarism levels. Future work will consider additional evaluation metrics for retrieval and ranking, hyperparameter tuning per tool and CEM, additional programming languages, and cross-language plagiarism scenarios. Also, the area of AI-obfuscated or AI-generated code is another possible direction for future work. We provide the code at a public GitHub repository https://github. com/FahadEbrahim/CEM-SCPD, as well as some supplementary results and materials.
References [1] Cîmpeanu Alexandra-Cristina and Alexandru-Corneliu Olteanu. 2022. Material survey on source code plagiarism detection in programming courses. In 2022 International Conference on Advanced Learning Technologies (ICALT). IEEE, 387– 389. [2] Georgina Cosma and Mike Joy. 2008. Towards a definition of source-code plagiarism. IEEE Transactions on Education 51, 2 (2008), 195–200. [3] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). 4171–4186. [4] Thomas J DiCiccio and Bradley Efron. 1996. Bootstrap confidence intervals. Statistical science 11, 3 (1996), 189–228. [5] Yihong Dong, Jiazheng Ding, Xue Jiang, Ge Li, Zhuo Li, and Zhi Jin. 2025. CodeScore: Evaluating code generation by learning code execution. ACM Transactions on Software Engineering and Methodology 34, 3 (2025), 1–22. [6] Simantika Bhattacharjee Dristi and Matthew B Dwyer. 2025. Analyzing and Mitigating Surface Bias in Code Evaluation Metrics. arXiv preprint arXiv:2509.15397 (2025). [7] Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker’s guide to testing statistical significance in natural language processing. In Proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: Long papers). 1383–1392. [8] Aryaz Eghbali and Michael Pradel. 2022. CrystalBLEU: precisely and efficiently measuring the similarity of code. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering. 1–12. [9] Jinan AW Faidhi and Stuart K Robinson. 1987. An empirical approach for detecting program similarity and plagiarism within a university programming environment. Computers & Education 11, 1 (1987), 11–19. [10] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020. CodeBERT: A pre-trained model for programming and natural languages. In Findings of the association for computational linguistics: EMNLP 2020. 1536–1547. [11] Ameet V Joshi. 2020. Machine learning and artificial intelligence. Vol. 261. Springer.
Conference’17, July 2017, Washington, DC, USA
[12] Oscar Karnalim, Setia Budi, Hapnes Toba, and Mike Joy. 2019. Source Code Plagiarism Detection in Academia with Information Retrieval: Dataset and the Observation. Informatics in Education 18, 2 (2019), 321–344. doi:10.15388/infedu. 2019.15 [13] Rien Maertens, Peter Dawyndt, and Bart Mesuere. 2023. Dolos 2.0: Towards seamless source code plagiarism detection in online learning environments. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 2. 632–632. [14] Rien Maertens, Peter Dawyndt, and Bart Mesuere. 2025. Source Code Plagiarism Detection as a Service with Dolos. In Proceedings of the 30th ACM Conference on Innovation and Technology in Computer Science Education V. 2. 729–730. [15] Rien Maertens, Maarten Van Neyghem, Maxiem Geldhof, Charlotte Van Petegem, Niko Strijbol, Peter Dawyndt, and Bart Mesuere. 2024. Discovering and exploring cases of educational source code plagiarism with Dolos. SoftwareX 26 (2024), 101755. [16] Rien Maertens, Charlotte Van Petegem, Niko Strijbol, Toon Baeyens, Arne Carla Jacobs, Peter Dawyndt, and Bart Mesuere. 2022. Dolos: Language-agnostic plagiarism detection in source code. Journal of Computer Assisted Learning 38, 4 (2022), 1046–1061. [17] Matthew B McDermott, Haoran Zhang, Lasse H Hansen, Giovanni Angelotti, and Jack Gallifant. 2024. A closer look at AUROC and AUPRC under class imbalance. Advances in Neural Information Processing Systems 37 (2024), 44102–44163. [18] Serge Lionel Nikiema, Albérick Euraste Djire, Abdoul Aziz Bonkoungou, Micheline Bénédicte Moumoula, Jordan Samhi, Abdoul Kader Kabore, Jacques Klein, and Tegawendé F Bissyande. 2025. How Small Transformation Expose the Weakness of Semantic Similarity Measures. arXiv preprint arXiv:2509.09714 (2025). [19] Matija Novak, Mike Joy, and Dragutin Kermek. 2019. Source-code similarity detection and detection tools used in academia: a systematic review. ACM Transactions on Computing Education (TOCE) 19, 3 (2019), 1–37. [20] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 311–318. [21] Lutz Prechelt, Guido Malpohl, Michael Philippsen, et al. 2002. Finding plagiarisms among a set of programs with JPlag. J. Univers. Comput. Sci. 8, 11 (2002), 1016. [22] Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. CodeBLEU: a method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009.10297 (2020). [23] Timur Sağlam, Sebastian Hahner, Larissa Schmid, and Erik Burger. 2024. Obfuscation-resilient software plagiarism detection with jplag. In Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings. 264–265. [24] Timur Sağlam, Nils Niehues, Sebastian Hahner, and Larissa Schmid. 2025. Mitigating Obfuscation Attacks on Software Plagiarism Detectors via Subsequence Merging. In 2025 IEEE/ACM 37th International Conference on Software Engineering Education and Training (CSEE&T). IEEE, 217–228. [25] Evgeniy Slobodkin and Alexander Sadovnikov. 2023. Towards a Dataset of Programming Contest Plagiarism in Java. In Proceeding of the 33rd Conference of FRUCT Association. 386–390. [26] Yewei Song, Cedric Lothritz, Xunzhu Tang, Tegawendé Bissyandé, and Jacques Klein. 2024. Revisiting code similarity evaluation with abstract syntax tree edit distance. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 38–46. [27] Ngoc Tran, Hieu Tran, Son Nguyen, Hoan Nguyen, and Tien Nguyen. 2019. Does BLEU score work for code migration?. In 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC). IEEE, 165–176. [28] Michael J Wise. 1993. String similarity via greedy string tiling and running Karp-Rabin matching. Online Preprint, Dec 119, 1 (1993), 1–17. [29] Morteza Zakeri-Nasrabadi, Saeed Parsa, Mohammad Ramezani, Chanchal Roy, and Masoud Ekhtiarzadeh. 2023. A systematic literature review on source code similarity measurement and clone detection: Techniques, applications, and challenges. Journal of Systems and Software 204 (2023), 111796. [30] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. BERTScore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019). [31] Shuyan Zhou, Uri Alon, Sumit Agarwal, and Graham Neubig. 2023. CodeBERTScore: Evaluating code generation with pretrained models of code. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 13921–13937.
Fahad Ebrahim and Mike Joy
A
Bootstrapped Confidence Intervals and Paired Comparisons A.1 Pooled Confidence Intervals Tables 4 and 5 report bootstrapped 95% confidence intervals for AUROC and AP in the pooled setting, for both raw and preprocessed code pairs. Table 4: Bootstrapped 95% confidence intervals for pooled raw results. Method
AUROC
95% CI
AP
95% CI
FusionTop3 CrystalBLEU Dolos CodeBLEU RUBY JPlag
0.861 0.850 0.864 0.837 0.822 0.777
[0.845, 0.878] [0.832, 0.866] [0.848, 0.879] [0.821, 0.854] [0.804, 0.842] [0.753, 0.798]
0.845 0.834 0.842 0.817 0.811 0.762
[0.824, 0.861] [0.813, 0.854] [0.823, 0.858] [0.795, 0.838] [0.789, 0.832] [0.736, 0.787]
Table 5: Bootstrapped 95% confidence intervals for pooled preprocessed results. Method
AUROC
95% CI
AP
95% CI
FusionTop3 CrystalBLEU Dolos CodeBLEU RUBY JPlag
0.882 0.879 0.864 0.843 0.839 0.777
[0.866, 0.897] [0.862, 0.895] [0.848, 0.879] [0.825, 0.860] [0.821, 0.859] [0.753, 0.798]
0.862 0.865 0.842 0.822 0.826 0.762
[0.841, 0.878] [0.848, 0.885] [0.823, 0.858] [0.801, 0.842] [0.804, 0.846] [0.736, 0.787]
A.2
Paired Bootstrap Differences
We further computed paired bootstrap differences for AUROC in the pooled setting, focusing on the top-performing metrics against Dolos. For comparisons against JPlag, we report AUROC differences computed from the rounded values in Tables 4 and 5. In the raw setting, FusionTop3, CrystalBLEU, CodeBLEU, and RUBY all exceed JPlag, with ΔAUROC values of +0.084, +0.073, +0.060, and +0.045, respectively. Relative to Dolos, FusionTop3 is statistically indistinguishable (ΔAUROC = -0.0023, 95% CI [-0.0155, 0.0104]), and CrystalBLEU is also not clearly separated from Dolos because its interval includes zero (ΔAUROC = -0.0142, 95% CI [-0.0285, 0.0001]). CodeBLEU and RUBY are lower than Dolos (ΔAUROC = -0.0269, 95% CI [-0.0433, -0.0106] and ΔAUROC = 0.0421, 95% CI [-0.0579, -0.0269]). In the preprocessed setting, the same four metrics remain above JPlag: FusionTop3 (ΔAUROC = +0.105), CrystalBLEU (ΔAUROC = +0.102), CodeBLEU (ΔAUROC = +0.066), and RUBY (ΔAUROC = +0.062). Relative to Dolos, FusionTop3 and CrystalBLEU are higher (ΔAUROC = +0.0179, 95% CI [0.0052, 0.0308] and ΔAUROC = +0.0154, 95% CI [0.0028, 0.0287]), whereas CodeBLEU and RUBY remain lower (ΔAUROC = -0.0209, 95% CI [-0.0380, -0.0035] and ΔAUROC = -0.0247, 95% CI [-0.0409, -0.0087]).