arXiv:2604.20742v1 [cs.SE] 22 Apr 2026
Evaluating Software Defect Prediction Models via the Area Under the ROC Curve Can Be Misleading Luigi Lavazza
Gabriele Rotoloni
Sandro Morasca
[email protected] Università degli Studi dell’Insubria Varese, Italy
[email protected] Università degli Studi dell’Insubria Varese, Italy
[email protected] Università degli Studi dell’Insubria Varese, Italy
Abstract
1
Background: Receiver Operating Characteristic (ROC) curves are widely used to evaluate the performance of Software Defect Prediction (SDP) models that estimate module fault-proneness, i.e., the probability that a module is faulty. A ROC curve maps a model’s performance in terms of True Positive Rate and False Positive Rate for any possible threshold set on fault-proneness. The Area Under the ROC Curve (AUC) summarizes the performance of a model across all possible thresholds. Traditionally, ROC curves completely above the bisector of the ROC space are considered better than random, and high AUC values are associated with good performance. Aim: We investigate whether these beliefs are correct, hence if SDP model evaluation based on ROC curves and AUC is reliable. Method: We decorate ROC curves by highlighting the points corresponding to threshold values. We also represent True Positive Rate and False Positive Rate as functions of the threshold. Thus, we can evaluate whether a model classifies both faulty and non-faulty modules better than the random model. Results: We show that commonly used evaluation criteria may lead to wrong conclusions. Conclusions: A high value of AUC does not guarantee that both the True Positive Rate and the False Positive Rate of a model are better than the random model’s for all possible thresholds. Either decorated ROC curves or alternative representations are needed to appreciate all the relevant aspects of SDP models.
Module fault-proneness, i.e., the probability that a software module is faulty, can be estimated via Software Defect Prediction (SDP) models. To classify a module as faulty or not faulty, a threshold 𝑡 ∈ [0, 1] is set on fault-proneness: a module is classified as faulty (positive) if its fault-proneness is above 𝑡 and non-faulty (negative) otherwise. A Receiver Operating Characteristic (ROC) curve plots the values of TPR (the True Positive Rate) against FPR (the False Positive Rate) obtained from a dataset by using all possible threshold values 𝑡 on a fault-proneness model [8]. The Area Under the ROC Curve (AUC) is widely used to evaluate SDP models [18]. Evaluations rely on the following well-established assumptions: i) a ROC curve that is entirely above the bisector of the ROC space performs better than the random classifier, since the bisector is the ROC curve of the random classifier; ii) high values of AUC are associated with good performance; hence, given two models, the one with the higher AUC is considered preferable. Although ROC curves are used to represent the performance obtained by varying the cut-off threshold, ROC curves do not use or represent the value of the threshold itself. Once the modules have been ranked by decreasing fault-proneness, it is possible to draw the ROC curve or to compute AUC [9] based only on module ranking and the actual faultiness of modules. One could wonder if ignoring thresholds is a correct practice. In this paper, we reconsider ROC curves by accounting for the threshold value that generated each one of the curve points. In this way, it becomes apparent that many models, whose performance is considered good when evaluated via traditional criteria, actually do not perform really well. Specifically, it is often the case that either TPR or FPR is good, i.e., many models are either good at classifying defective modules or not defective ones, but not both. The contribution of this paper is of a methodological nature. The problems with ROC curves and AUC derive from the very nature of ROC curves. An empirical study shows that misleading indications are produced frequently. To overcome these problems, we suggest some methodological guidelines. The paper is organized as follows. Section 2 provides some background concerning ROC curves, AUC and the random classifier. Section 3 presents the conceptual observations that explain why the traditional interpretation of ROC curves and AUC can be misleading. Section 4 shows examples of real-life SDP models that are considered good according to traditional criteria, but actually do not perform well. Section 5 discusses the effect of imbalance on ROC curves. Section 6 illustrates an empirical study that evaluates how frequently traditional criteria can be misleading. Section 7 suggests how to evaluate SDP models using “threshold-aware” ROC curves.
CCS Concepts • Software and its engineering → Software defect analysis.
Keywords ROC curves, Area under the curve (AUC), performance metrics, software defect prediction (SDP), software vulnerability estimation ACM Reference Format: Luigi Lavazza, Gabriele Rotoloni, and Sandro Morasca. 2026. Evaluating Software Defect Prediction Models via the Area Under the ROC Curve Can Be Misleading. In Proceedings of The 30th International Conference on Evaluation and Assessment in Software Engineering (EASE 2026). ACM, New York, NY, USA, 11 pages. https://doi.org/XXXXXXX.XXXXXXX Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. EASE 2026, Glasgow, Scotland, United Kingdom © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-XXXX-X/2018/06 https://doi.org/XXXXXXX.XXXXXXX
Introduction
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
Section 8 discusses the related work. Finally, Section 9 draws some conclusions.
2
Background
In this section, we summarize the basics of performance metrics, ROC curves, and AUC, and the related assumptions.
2.1
The Confusion Matrix
The performance of a binary classifier on a set of n instances is usually assessed based on a 2×2 “confusion matrix” that shows how many of those n items are correctly and incorrectly classified. For consistency with the literature, we also say that a faulty module is positive and a non-faulty one is negative. As Table 1 shows, the cells of a confusion matrix contain the numbers of modules that are: correctly estimated negative (True Negatives TN ); incorrectly estimated negative (False Negatives FN ); incorrectly estimated positive (False Positives FP); and correctly estimated positive (True Positives TP). The numbers of actually negative AN and positive modules AP depend exclusively on the dataset and not on the specific binary classifier. An especially important characteristic of a dataset is the prevalence of the positive class, i.e., 𝜌 = AP/𝑛.
Lavazza et al.
The ROC curve goes through point (1,1) when 𝑡=0. Setting the threshold 𝑡=0 implies that all modules are estimated positive, hence TPR=1 and FPR=1. When progressively increasing 𝑡, some module will be estimated negative, thus either decreasing FPR (if the module is negative), or decreasing TPR (if the module is positive). When 𝑡=1, all modules are estimated negatives, hence TPR=0 and FPR=0. Summarizing, a ROC curve as a function of 𝑡 always starts at point (1,1) and monotonically decreases until it reaches point (0,0).
2.3
The ROC Curve of the Random Model
The random prediction model assigns a random score to each of the 𝑛 given modules. When threshold t is used to classify modules as positives or negatives, it is expected that (1−𝑡) ·𝑛 models will be estimated positives and 𝑡 ·𝑛 will be estimated negative. On average, of the AP positives, (1−𝑡)·𝐴𝑃 are correctly estimated positives, etc. Table 2 shows the random model’s expected confusion matrix. Table 2: Expected confusion matrix for the random model.
Estimated Negative Estimated Positive
Actual Negative TN𝑟𝑛𝑑 = 𝑡 ·AN FP𝑟𝑛𝑑 = (1−𝑡) ·AN
Actual Positive FN𝑟𝑛𝑑 = 𝑡 ·AP TP𝑟𝑛𝑑 = (1−𝑡) ·AP
Table 1: Confusion matrix TP
Estimated
Actual Negative Positive
Negative TN (True Negatives) FP (False Positives) AN = TN + FP (Actual Negatives)
Positive FN (False Negatives) TP (True Positives) AP = FN + TP (Actual Positives)
𝑟𝑛𝑑 Accordingly, TPR𝑟𝑛𝑑 = AP = (1−𝑡AP)·AP = 1 −𝑡, and TNR𝑟𝑛𝑑 = 𝑡·AN = AN = 𝑡, hence FPR𝑟𝑛𝑑 = 1−TNR𝑟𝑛𝑑 = 1−𝑡 =TPR𝑟𝑛𝑑 . Thus, the performance of the random model when the threshold is 𝑡 is represented in the ROC space by point (1−𝑡,1−𝑡): the ROC curve of the random model is the bisector of the ROC space, i.e., the segment joining points (0,0) and (1,1), as shown in Figure 1.
TN𝑟𝑛𝑑 AN
Based on a confusion matrix, several performance metric can be computed: TPR=TP/AP (the True Positive Rate, alias Recall), FPR= FP/AN (the False Positive Rate), PPV =TP/(TP+FP) (the Positive Predicted Value, alias Precision), etc.
2.2
The Receiver Operating Characteristic (ROC) Curve
In this paper, we consider SDP models that estimate module faultproneness, i.e., the probability that a software module is faulty (i.e., probabilistic scoring classifiers [8]). To classify a module as faulty or not, a threshold 𝑡 ∈ [0, 1] is set on fault-proneness: a module is classified as faulty (positive) if its fault-proneness is above 𝑡 and non-faulty (negative) otherwise. Many different binary classifiers can be obtained by varying 𝑡, with different classification performance. Each of them is characterized by a different value for the pair ⟨FPR, TPR⟩. A ROC curve [8] illustrates the overall diagnostic ability of a fault-proneness model. A ROC curve plots the values of 𝑦 =TPR against the values of 𝑥 =FPR obtained on a dataset by using all possible threshold values 𝑡 on the fault-proneness model. A ROC curve is shown in Figure 2. The possible set of values of pairs ⟨FPR, TPR⟩, i.e., the Cartesian product [0, 1] × [0, 1], is called the ROC space.
Figure 1: The ROC curve of the random model. In the figures, points ◦, △, +, ×, ♦, ▽, ⊠, ∗, ⊞ are associated with thresholds 0.1, 0.2, ..., 0.9, respectively. The random model provides an easy-to-implement and inexpensive way to make predictions, which does not consider any feature of a module. This characteristic of the random model qualifies it as
Evaluating Software Defect Prediction Models via the Area Under the ROC Curve Can Be Misleading
a perfect baseline: whatever model is proposed must provide better than random performance. There would be no point in collecting modules’ features (including the ground truth) and training a model, if one can get similar or better results by just tossing a coin. An SDP model with TPR > TPR𝑟𝑛𝑑 and FPR < FPR𝑟𝑛𝑑 definitely performs better than the random model, being better at classifying both positive and negative modules. We say that these classifiers perform better than random. Since TPR𝑟𝑛𝑑 = FPR𝑟𝑛𝑑 = 1−𝑡, in the ROC space, all points that are located above and to the left of point (1−𝑡, 1−𝑡) correspond to classifiers that perform better than random when threshold 𝑡 is used to classify modules. Points that are above and to the right of point (1−𝑡, 1−𝑡) represent better than random estimates of positive modules, but worse than random estimates of negative modules. Similarly, points below and to the left of point (1−𝑡, 1−𝑡) represent better than random estimates of negative modules, but worse than random estimates of positive modules. That is, these points represent neither better nor worse performance with respect to the random model: they just represent different trade-offs in the performances concerning positive and negative modules. Clearly, the points that are located below and to the right of point (1−𝑡, 1−𝑡) have worse than random performance when the threshold 𝑡 is used to classify modules. These points are below the bisector and represent classifiers that perform worse than the random classifier for both positive and negative modules.
3
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
Fundamental Observations
The ROC curve does not show any information concerning thresholds [9]. For instance, there may be a part of the curve where many points (corresponding to many threshold values) are concentrated, and other parts where points are more dispersed: the distribution of ROC points with respect to threshold values is not visible, by just looking at the ROC curve. Similarly, given a point of the curve, we cannot in general say what threshold value originated it: the best we can do is to assume that the points close to point (1,1) are associated with low threshold values, and the points close to point (0,0) are associated with high threshold values. With the random model, the situation is quite different, since we know exactly that point (1−𝑡,1−𝑡) is associated with threshold 𝑡. For a generic threshold 𝑡, we do not know the relative position of the corresponding point of the model’s ROC curve with respect to the random performance, represented by point (1−𝑡, 1−𝑡). The consequence of this observation, along with the considerations given in Section 2.3, is that AUC > 0.5 and the ROC curve entirely above the bisector are not sufficient conditions to conclude that the considered model is better than random for all threshold values. As already mentioned, a model is better than random if it classifies both positive and negative instances better than the random model, i.e., if it has both better TPR and FPR. Since the random model features TPR = FPR =1−𝑡, a model is better than random if the following condition holds:
The Area Under the ROC Curve (AUC)
∀𝑡 ∈ (0,1), TPR(𝑡) ≥ 1−𝑡 ∧ FPR(𝑡) ≤ 1−𝑡 ∧
The AUC is the area of the region delimited by the ROC curve and the segments that join point (0,0) with (1,0) and (1,0) with (1,1). Researchers and practitioners use AUC as a performance metric that provides an overall evaluation of the model. AUC is used in Empirical Software Engineering, to evaluate fault-proneness models [1–4, 21], as well as for other purposes, and in other domains. AUC is considered representative of the performance of the model because the longer the ROC curve lingers close to point (0,1), which represent the perfect prediction, the better. The closer the ROC gets to point (0,1), the greater the AUC, in general; therefore, a large value of AUC is generally associated with good performance.
∃𝑡 ∈ (0,1), TPR(𝑡) > 1−𝑡 ∨ FPR(𝑡) < 1−𝑡
2.4
Table 3: Interpretation of AUC AUCrange AUC = 0.5 0.5 < AUC < 0.7 0.7 ≤ AUC < 0.8 0.8 ≤ AUC < 0.9 0.9 ≤ AUC
Evaluation totally random, as good as tossing a coin poor, not much better than a coin toss acceptable excellent outstanding
Hosmer at al. [12] propose the intervals in Table 3 as guidelines to interpret the values of AUC as a measure for the discriminating power of a model for all values of threshold 𝑡. In addition to being used to evaluate a model, AUC is used to compare the performance of two models: the model with the higher AUC is preferred, even though it may not be sufficient evidence when the involved ROC curves cross each other [22].
(1)
where TPR(𝑡) and FPR(𝑡) indicate the values of TPR and FPR obtained with threshold 𝑡. Section 4 shows that condition (1) is often not satisfied (i.e., for some 𝑡, it is TPR(𝑡) < 1−𝑡 or FPR(𝑡) > 1−𝑡), even if the ROC curve is entirely above the bisector and AUC is largely greater than 0.5.
4
Model Evaluation and Comparison in Practice
In this section, we show examples of actual SDP modes—built via state-of-the-art techniques and evaluated against real-life datasets— in which the conditions described in Section 3 do occur.
4.1
Single Model Evaluation
To illustrate the concepts described in Section 3, we use as an example the leave-one-out cross-validation of Binary Logistic Regression (BLR) defect prediction models obtained from the poi 3.0 project dataset [13]. The resulting ROC curve is shown in Figure 2. According to the usual criteria, the model’s performance is not only largely better than random, since the ROC curve is well above the bisector, but actually good, since AUC= 0.803. Figure 3 represents the same ROC curve shown in Figure 2, with the points corresponding to thresholds 𝑡 that are multiple of 0.1 (i.e., 0.1, 0.2, ... , 0.9) highlighted. Points with the same shape correspond to the same threshold value 𝑡. For instance, the points denoted by × are associated with 𝑡=0.4 on the ROC curve and on the bisector. For some threshold values, the model’s performances are definitely better than random: for instance, the model’s point corresponding to 𝑡=0.6 (the blue ▽) is above and to the left of the corresponding random model’s point (green point ▽): when 𝑡=0.6,
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
Figure 2: The ROC curve of the BLR model for the poi 3.0 project (AUC=0.803).
Lavazza et al.
Figure 4: TPR and FPR as functions of the threshold, for the BLR model for the poi 3.0 project. BLR model features better than random TPR and FPR only when the threshold is in the (0.464, 0.969) range.
Figure 3: The ROC curve of the BLR model for the poi 3.0 project. Points representing specific values of the threshold are highlighted.
the BLR model has both greater TPR and smaller FPR than the random model. However, for values of 𝑡 ≤ 0.4 (e.g., points ◦, △, + and ×) the BLR model has worse FPR than the random model: it performs very well in classifying positive (faulty) modules, at the expense of a poor classification of negative (non-faulty) modules. To explicitly show the performance associated with a threshold, we can plot TPR and FPR vs. threshold values. These plots for our BLR model are in Figure 4, where the green line represents both TPR𝑟𝑛𝑑 and FPR𝑟𝑛𝑑 . FPR is better than random (i.e., below the FPR𝑟𝑛𝑑 line) only when the threshold is in the (0.464, 0.969) range, while TPR is always better than random (i.e., for all values of the threshold). Thus, the
Figure 5: The ROC curve for the BLR model for the poi 3.0 project: the section of the curve where performance is better than random is highlighted. Figure 5 highlights in red the section of the ROC curve where the performance of the model is better than random with respect to both TPR and FPR. Though this section is only a part of the curve, AUC is computed using the entire curve: a practice that can provide misleading indications. To illustrate how much AUC can be misleading, take the Random Forest (RF) model for project ant 1.5, whose ROC curve (having AUC=0.8) is shown in Figure 6. Although the value of AUC could induce to consider the model as fairly good, by highlighting the points corresponding to specific values of the threshold (as in Figure 7), it
Evaluating Software Defect Prediction Models via the Area Under the ROC Curve Can Be Misleading
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
Figure 6: The ROC curve of the RF model for the ant 1.5 project (AUC=0.8).
Figure 8: TPR and FPR as functions of the threshold, the RF model for the ant 1.5 project.
is possible to see that for all the highlighted values of the threshold the model yields worse than random TPR. Figure 8 shows that for all values of the threshold—not just those highlighted in Figure 7—the model yields worse than random TPR. Figure 8 also shows that the model has very good FPR: it is this very good FPR that “pulls” the ROC curve towards the y axis of the ROC space, causing the ROC curve to be above the bisector and the AUC to be high.
4.2
Perfect Models?
A model whose ROC curve goes through point (0,1) has AUC=1. Because of the monotonicity property of the ROC curves, this ROC curve is necessarily made of the segment from (1,1) to (0,1) and the segment from (0,1) to (0,0). Such a model is considered perfect [22].
Figure 9: ROC curve of SVM model for ckjm. Figure 7: The ROC curve of the RF model for the ant 1.5 project (AUC=0.8). Points representing specific values of the threshold are highlighted. Figure 8 makes it explicit that the model performs worse (respectively better) then random in classifying positive (respectively negative) modules, for any value of the threshold. This fact cannot be seen by just looking at the ROC curve (Figure 6), which is well above the bisector and has a fairly high AUC.
However, even such a model could provide sub-optimal performance, for some threshold values. As an example, take the SVM model obtained for project ckjm, whose ROC curve is in Figure 9. The model provides perfect predictions for 𝑡 = 0.4 (and for other values of 𝑡 close to 0.4, not highlighted in Figure 9), but several values of 𝑡 do not yield perfect predictions: e.g., with 𝑡 = 0.3, FPR > 0.8 (i.e., worse than random), while TPR=0 for 𝑡 ≥ 0.6. This is possible because AUC measures only the ranking. Unfortunately, even a perfect ranking (i.e., all defective modules preceding non-defective
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
Lavazza et al.
ones) does not exclude that some thresholds yield poor performance. AUC is a property of the ROC curve, which considers all the possible thresholds. So, when AUC=1, one could be induced to think that the classifier yields perfect predictions with any value of threshold 𝑡. This may very well not be the case: AUC=1 if there is some range [𝑡𝑙 , 𝑡ℎ ] such that 𝑡𝑙 <𝑡 ≤𝑡ℎ yields perfect predictions, so that the ROC curve goes through point (0,1). When 𝑡 is out of the [𝑡𝑙 , 𝑡ℎ ] range, the prediction is not perfect, and possibly not even better than random. In the case illustrated by Figure 9, the perfect prediction range is rather small, being a subset of (0.3, 0.5). For other models, 𝑡𝑙 could be very close to zero and 𝑡ℎ could be very close to one, so that any sensible value of 𝑡 (i.e., any value not extremely close to zero or one) would yield perfect predictions. Unfortunately, AUC does not tell us if we are in this lucky situation or in a situation like the one described above, when we obtain perfect predictions only if we carefully pick a suitable value of 𝑡.
4.3
Model Comparison
Model A is considered preferable to model B when model A has greater AUC than model B [8]. However, when two ROC curves cross each other, it is recognized that in some conditions model B could yield better performance than model A, even though AUC𝐴 > AUC𝐵 [8, 22]. Instead, it is universally accepted that if ROC curve A dominates ROC curve B (i.e., ROC curve A is never below ROC curve B and is above it in at least one point), model A is better than model B. If ROC curve A dominates ROC curve B, then AUC𝐴 > AUC𝐵 . However, AUC𝐴 > AUC𝐵 does not imply that ROC curve A dominates ROC curve B; in fact, the two curves can cross each-other. In this section, we show that even when ROC curve A dominates ROC curve B, it is possible that model B is preferable, for some values of the threshold. As a consequence, it is even more possible that model B is preferable, for some values of the threshold, when AUC𝐴 > AUC𝐵 and curve A does not dominate curve B (i.e., the curve cross each other). As an example, let us consider the fault-proneness models for project poi 2.5, obtained via RF and BLR, whose ROC curves are shown in Figure 10. The ROC curve of the RF model (AUC=0.87) dominates the ROC curve of the BLR model (AUC=0.73): based on the traditional interpretation of ROC curves and AUC, we should conclude that the RF model is better than the BLR model. However, for each point 𝑃 of the RF model’s curve, there is at least one point 𝑃 ′ of the BLR model’s curve that achieves either better TPR or better FPR (but not both). Noticeably, by just looking at the ROC curves, we do not know whether 𝑃 and 𝑃 ′ are obtained with the same or different fault-proneness threshold values. Comparing ROC curves without considering the threshold does not seem to provide a really good insight into the relative models’ performance. In analogy to condition (1), to state that model A is better than model B, we can require that for all thresholds, the corresponding point of model A has neither TPR nor FPR worse than the corresponding point of model B, and, for at least one threshold, it has better TPR or better FPR: ∀𝑡 ∈ (0,1), TPR𝐴 (𝑡) ≥ TPR𝐵 (𝑡) ∧ FPR𝐴 (𝑡) ≤ FPR𝐵 (𝑡) ∧ ∃𝑡 ∈ (0,1), TPR𝐴 (𝑡) > TPR𝐵 (𝑡) ∨ FPR𝐴 (𝑡) < FPR𝐵 (𝑡)
(2)
Figure 10: ROC curves of RF and BLR models for poi 2.5. Note that, when condition (2) holds, ROC curve A dominates ROC curve B, while the reverse is not true in general. Many published papers adopted dominance as the criterion used to conclude that the model proposed in the paper outperforms previously published models, both in Software Engineering and in other fields. However, it may be the case that dominance is satisfied, while condition (2) is not, as shown in Figure 11, which represents the same ROC curves given in Figure 10, with the points corresponding to specific threshold values highlighted.
Figure 11: Comparison of RF and BLR models for poi 2.5. When 𝑡=0.6 (point ▽) the RF model is better than the BLR model with respect to both TPR and FPR. However, for multiple threshold values, it is not so: when 𝑡=0.5 (point ♦) the BLR model achieves better TPR and worse FPR, while with 𝑡=0.8 (point ∗) the BLR model achieves slightly better FPR and worse TPR. For several threshold values, the two models provide different trade-offs concerning the
Evaluating Software Defect Prediction Models via the Area Under the ROC Curve Can Be Misleading
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
classification of positive and negative modules, neither of them being better at classifying both positives and negatives.
Figure 14: TPR and FPR of RF and BLR models for jedit 4.2 as functions of the threshold. Figure 12: Comparison of RF and BLR models for poi 2.5. It is interesting to note that, although the ROC curves do not cross each other, both the TPR(𝑡) and FPR(𝑡) curves cross each other (as shown in Figure 12), thus clearly indicating that neither model is better than the other for all threshold values. Let us now consider the RF and BLR models for jedit 4.2. Figure 13 shows that the ROC curve of the BLR model (AUC=0.85) dominates the ROC curve of RF model (AUC=0.8). However, Figure 14 clearly shows that both models yield TPR values that are generally largely worse than random, hence probably neither model would be considered usable by a practitioner.
Figure 13: ROC curves of RF and BLR models for jedit 4.2.
5
The Effect of Imbalance
In imbalanced datasets, probabilities are biased towards the most common event [6, 11]. Thus, a large prevalence of positive modules (𝜌 close to 1) leads to maximizing TPR, at the expense of having poor FPR, for most values of the threshold. Similarly, a very small prevalence (𝜌 close to 0) leads to maximizing FPR, at the expense of having poor TPR, for most values of the threshold. In these conditions, ROC curves tend to remain well above the bisector, thus showing fair or even good performance, characterized by decent values of AUC [15]. Let us consider the RF models obtained for projects log4j 1.2 (𝜌 = 0.92) and e-learning (𝜌 = 0.08). Figure 15 shows the TPR and FPR obtained by the mentioned models, as functions of the threshold. It is apparent that the two models achieve quite different performances: log4j 1.2 has very good TPR for threshold values up to around 0.8, while FPR is always largely worse than random. Instead, e-learning has very good FPR for threshold values above 0.1, while TPR is often largely worse than random. The situation shown in Figure 15 is hardly surprising. When the positive modules are very prevalent, it is easy to identify those modules; the difficult part is identifying them without misclassifying too many negative modules. The same type of reasoning applies for the prevalence of negative elements. Figure 15 shows that the considered models do not perform well: both of them achieve very good performance with the prevalent class of modules, but misclassify the minority class modules. Nonetheless, the ROC curves of both models are reasonably good, as shown in Figure 16: the corresponding AUC values are 0.81 for log4j 1.2 and 0.86 for e-learning. Figure 16 shows the effects of imbalance, which are not visible in regular ROC curves. The largest part of the ROC curve of the model for log4j 1.2 (which has prevalence 𝜌 = 0.917) is made of points obtained with threshold 𝑡 > 0.9. Instead, the largest part of the ROC curve of the model for e-learning (which has prevalence 𝜌 = 0.078) is made of points obtained with threshold 𝑡 < 0.1. This
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
Lavazza et al.
Figure 15: TPR(𝑡) and FPR(𝑡), when prevalence 𝜌 is high (log4j 1.2, left) and low (e-learning, right).
Figure 16: ROC curves with thresholds highlighted, when prevalence 𝜌 is high (log4j 1.2, left) and low (e-learning, right). phenomenon is consistent with the observation given above, that when positive modules are largely prevalent most modules are estimated positive, and vice-versa. In conclusion, in case of severely imbalanced datasets, AUC appears definitely overoptimist, while ROC curves hide worse than random performance concerning either TPR or FPR.
6
We trained models on 65 datasets from the Jureczko&Madeyski (J&M) collection [13] and 11 datasets from the NASA collection [20]. The descriptive statistics of the datasets are given in Table 4. Table 4: Descriptive statistics of the datasets used in the empirical study.
The Empirical Study
One could wonder how often AUC provides unreliable indications: to clarify this issue, we carried out an empirical study. For this study, we employed Binary Logistic Regression (BLR) and Random Forest (RF) models, as these techniques compute a score for each module in the [0,1] range, which is interpreted as a probability of a module to be positive.
NASA J&M
# modules Mean Min Max 857 124 1952 283.1 10 965
% faulty modules Mean Min Max 15.2% 1.8% 35.5% 34.1% 2.2% 98.8%
Of the many metrics provided in both the J&M and NASA datasets, only a subset appear relevant to estimate module defectiveness: we used only those metrics (8 from the J&M datasets and 9 from the
Evaluating Software Defect Prediction Models via the Area Under the ROC Curve Can Be Misleading
NASA datasets) to build SDP models. We built models using groups of up to three metrics as independent variables [10]. Therefore, we obtained 83 =56 models for each dataset from the J&M collection, and 93 =84 models for each dataset from the NASA collection. The estimates were evaluated via Leave-One-Out Cross Validation. All the computations were carried out using the R environment. Specifically, the RF models were built using the RandomForest library. This library allows for the tuning of the mtry hyperparameter, but this was not needed since we considered no more than three features at a time. We set the number of trees to 1,500, since using more trees did not seem to improve performance. To evaluate to what extent the problems described in Sections 3 and 4 affect SDP models, we computed the following numbers: (1) How many models were obtained. (2) How many models feature a ROC curve completely above the bisector (hence, have AUC >0.5). (3) How many models have AUC ≥ 0.8. Of the models at points (2) and (3) above, we checked how often some points of the curve have worse than random TPR or FPR. Concerning model comparison, we selected the cases where there is a clear winner, according to the traditional interpretation of the ROC curves, i.e., a model’s ROC dominates the other models’ ROC curves. Then, we checked if the winner model is actually better than the other models with respect to both TPR and FPR, for all threshold values.
6.1
Results
As mentioned before, we built 56 models for each dataset from the J&M collection, and 84 from the NASA collection. We used two Machine Learning techniques, so we obtained a total of 2 × ((56 × 65) + (84 × 11)) = 9,128 models. Of these models, 8,651 have AUC > 0.5; 4,934 models generate ROC curves that have no points below the bisector; 2,304 models are associated with ROC curves that are strictly above the bisector (except for points (0,0) and (1,1)). Of the 2,304 models whose ROC curves are completely above the bisector, only 114 (around 5%) have performance that is better than random with respect to both TPR and FPR for all thresholds. 1,504 of the 2,304 models with ROC curves completely above the bisector have AUC ≥ 0.8, but only 100 of them (6.6%) are better than random with respect to both FPR and TPR for all thresholds. We also compared all the pairs of models obtained using the same dataset. For each model and dataset, we generated 56 models for J&M datasets and 84 for NASA datasets using two modeling techniques, thus obtaining 112 and 168 total models per dataset for the two collections. Hence, we performed = 112×111 =6,216 pair2 wise comparisons for J&M datasets and = 168×167 =14,028 compar2 isons for NASA datasets, for a grand total of (6, 216 × 65) + (11 × 14, 028)=558,348 pairwise comparisons. In 82,489 cases, one ROC curve dominates the other. However, only in 901 cases (1.1%) one curve has better TPR and FPR for all threshold values. If we only consider the comparisons in which one curve dominates the other, and the difference in AUC values is greater than 0.2, we still have that only in 823 comparisons out of 29,845 (2.8%) one curve has better TPR and FPR than the other for all thresholds.
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
In conclusion, both when evaluating a single SDP model and when comparing SDP models, the traditional interpretations of the ROC curve and the AUC are very frequently misleading.
6.2
Threats to Validity
External validity. For the empirical study, we employed two widely used dataset collections, and widely used modeling techniques (BLR and RF), whose predictions can be interpreted as probabilities. Other models, datasets, and contexts (such as just-in-time SDP) could have been considered. However, we argue that similar observations could have been made with other probabilistic classifiers and datasets, thus limiting threats to external validity. Internal validity. We trained and tested our models via mature and stable libraries of the R environment (pROC for ROC curves, stats for BLR models, and RandomForest for RF models). Construct validity. We study the construct validity of AUC as a sensible performance metric for SDP models. Thus, we detect and discuss its construct validity issues throughout the article. As we stated in Section 2.2, we consider SDP models that estimate the probability that a software module is faulty. However, several Machine Learning algorithms return a score that may not be reliably taken as a probability. If so, it is recommended that the output of these models be calibrated to represent probabilities, for instance with Platt scaling [19]. Conclusion validity. We do not claim that a model with performance worse than random for some threshold values should be always discarded. We point out, however, that a good ROC curve with a good AUC does not necessarily represent a model that is good (or even perfect, as in Figure 9) in general. Similarly, a model with a higher AUC than another, or even with a ROC curve that dominates another model’s, is not necessarily better. A model should be selected while keeping the preferred threshold range in mind.
7
Guidelines
Here, we provide some suggestions about the evaluation of SDP models in a “threshold-aware” way. We only consider the comparison of models: this is not a limitation, since the evaluation of a single SDP model is in fact a comparison with the random model. Let us consider SDP models A and B; let 𝑅𝑂𝐶𝐴 and 𝑅𝑂𝐶𝐵 be the ROC curves of the two models, and let (FPR𝐴 (𝑡), TPR𝐴 (𝑡)), respectively (FPR𝐵 (𝑡), TPR𝐵 (𝑡)), be the point of 𝑅𝑂𝐶𝐴 , respectively, 𝑅𝑂𝐶𝐵 , when the threshold’s value is 𝑡. If condition (2) holds, there is no doubt that A is preferable. Otherwise, it is necessary to consider which model provides the best trade-off between TPR and FPR, for what threshold values. There are many ways to make such evaluation. Here we illustrate a possible procedure, with the help of Figure 17, where model A and B are the RF and BLR models, respectively, for project poi 2.5. Based on context-specific considerations, one can define the minimum acceptable performance in terms of TPR𝑚𝑖𝑛 and FPR𝑚𝑎𝑥 : in Figure 17, TPR𝑚𝑖𝑛 =0.7 and FPR𝑚𝑎𝑥 =0.3. In the threshold range [𝑡𝐴𝑙 , 𝑡𝐴ℎ ] model A has TPR𝐴 ≥ TPR𝑚𝑖𝑛 and FPR𝐴 ≤ FPR𝑚𝑎𝑥 . Similarly, model B has acceptable performance in [𝑡𝐵𝑙 , 𝑡𝐵ℎ ]. It is now possible to identify the ranges where both models are better than TPR𝑚𝑖𝑛 and FPR𝑚𝑎𝑥 (if any) and compare the performance of A and B in those ranges.
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
Figure 17: Comparison when condition (2) does not hold.
The latter comparison can be carried out according to multiple criteria. A first criterion is that a model that has acceptable performance in a wide range (like that of the RF model A in Figure 17) is preferable to a model that performs acceptably in a narrow range, because the narrower the range, the more likely the chosen threshold will be outside the range, thus yielding poor predictions. Another criterion consists in evaluating the relative performance in the acceptable ranges: e.g., Figure 17 shows that model A (RF) has both better TPR and better FPR in the entire intersection of [𝑡𝐴𝑙 , 𝑡𝐴ℎ ] and [𝑡𝐵𝑙 , 𝑡𝐵ℎ ] (though not in the entire [𝑡𝐴𝑙 , 𝑡𝐴ℎ ] range). An alternative procedure is applicable when the cost of false negatives (e.g., the cost due to releasing a defective module) and false positives (wasted effort) is known. In such case, instead of looking at TPR(𝑡) and FPR(𝑡), as in Figure 17, we can plot the cost as a function of 𝑡: a model that minimizes the cost, especially for a large range of threshold values, is definitely preferable.
8
Related Work
Machine Learning research has criticized ROC curves because they only show how well samples have been ranked, and ignore the distribution of probabilities among samples [9]. In their audit about Machine Learning experimentation in Software Engineering, Destefanis et al. [7] show that AUC is the second most popular performance metric. They note that AUC is “chanceanchored” [5], since the value of AUC of a binary classifier can immediately be compared to the value of the random classifier, which is 0.5. In that, AUC is akin to an effect measure, since its value tells “how far” the classifier is from being totally random [14]. In a medical paper, Calster et al. point out how ROC curves and AUC can be difficult to interpret when trying to measure models’ improvement when adding new markers [23]. They make similar observations to those in our paper, showing how, even when the “improved” ROC curve is completely above the base curve, there are thresholds in which the performance of the new curve are actually lower. The paper criticizes other metrics specifically used
Lavazza et al.
to measure improvement, indicating how making more “thresholdaware” evaluations can be useful in different applications. A recent paper in the Journal of Epidemiology reiterates the issue, arguing that researchers should avoid showing ROC curves without any information on threshold values [24]. They also suggest the use of “classification plots,” i.e., plots of TPR and FPR values over threshold values, like the one depicted in Figure 8. However, some considerations seem questionable, at least for fault-proneness models. Specifically, they argue that ROC curves should not be used because curves with equal AUC (obtained on the same test dataset) can vary a lot, adding “complexity without adding useful information.” We think that, even without explicit information on predicted probabilities, the shape of a ROC curve can be informative, especially when comparing curves with the same AUC. It has been shown that the values of any performance metric (e.g., the Matthews Correlation Coefficient, the F-score, etc.) can be plotted in the ROC space [15–17]: it is thus possible to evaluate the position of every point of a ROC curve both with respect to the corresponding random performance and with any traditional performance metric. It is also possible to plot the value of any performance metric as a function of the threshold, and compare it with the corresponding random model’s plot.
9
Conclusions
We showed that ROC curves and their associated AUC can yield misleading indications concerning the performance of binary predictive models. In fact, although ROC curves that are entirely above the bisector of the ROC space are traditionally considered better than the random classifier, it is often the case that for some threshold values either TPR or FPR are worse than the random model’s. Similarly, when comparing models, a model whose ROC curve is completely above the other model’s curve (hence, has higher AUC) may have worse TPR or FPR, for some thresholds. Many research papers in the Software Engineering area (including the majority of papers dealing with SDP models [18]) use the AUC to evaluate models. However, as we have shown, the AUC is not a reliable indicator of the performance of classifiers. As a consequence, it is possible that many evaluations reported in research papers are not correct. Reconsidering the usage of AUC appears urgent. To this end, we suggest using more reliable indicators, like the graphs that show how a performance metric depends on the threshold value. These graphs can be used to show together TPR(𝑡) and FPR(𝑡) (as in Figure 17), to appreciate the trade-off achieved in correctly classifying positive and negative modules, for any possible (and reasonable) value of 𝑡. In case the relative cost of false positives and false negatives is known, it is also possible to plot the cost function and choose the model accordingly.
Acknowledgments This work has been partly supported by the “Fondo di Ricerca d’Ateneo” funded by the Università degli Studi dell’Insubria.
Online Resources A package with all the ROC curves we generated and the data we collected is available at https://doi.org/10.6084/m9.figshare.31100191.
Evaluating Software Defect Prediction Models via the Area Under the ROC Curve Can Be Misleading
References [1] Erik Arisholm, Lionel C Briand, and Magnus Fuglerud. 2007. Data mining techniques for building fault-proneness models in telecom Java software. In The 18th IEEE International Symposium on Software Reliability (ISSRE’07). IEEE, 215–224. [2] Sarah Beecham, Tracy Hall, David Bowes, David Gray, Steve Counsell, and Sue Black. 2010. A systematic review of fault prediction approaches used in software engineering. Technical Report. Technical Report Lero-TR-2010-04, Lero. [3] Cagatay Catal. 2012. Performance evaluation metrics for software fault prediction studies. Acta Polytechnica Hungarica 9, 4 (2012), 193–206. [4] Cagatay Catal and Banu Diri. 2009. A systematic review of software fault prediction studies. Expert systems with applications 36, 4 (2009), 7346–7354. [5] Davide Chicco, Matthijs J Warrens, and Giuseppe Jurman. 2021. The Matthews correlation coefficient (MCC) is more informative than Cohen’s Kappa and Brier score in binary classification assessment. Ieee Access 9 (2021), 78368–78381. [6] Jan Salomon Cramer. 1999. Predictive performance of the binary logit model in unbalanced samples. Journal of the Royal Statistical Society: Series D (The Statistician) 48, 1 (1999), 85–94. [7] Giuseppe Destefanis, Leila Yousefi, Martin Shepperd, Allan Tucker, Stephen Swift, Steve Counsell, and Mahir Arzoky. 2026. An audit of machine learning experiments on software defect prediction. Empirical Software Engineering 31, 4 (2026), 83. [8] Tom Fawcett. 2006. An Introduction to ROC Analysis. Pattern Recogn. Lett. 27, 8 (June 2006), 861–874. doi:10.1016/j.patrec.2005.10.010 [9] Cesar Ferri, Peter Flach, José Hernández-Orallo, and Athmane Senad. 2005. Modifying ROC curves to incorporate predicted probabilities. In Proceedings of the second workshop on ROC analysis in machine learning, Vol. 4140. International Conference on Machine Learning, 33–40. [10] Wei Fu and Tim Menzies. 2017. Easy over hard: A case study on deep learning. In Proceedings of the 2017 11th joint meeting on foundations of software engineering. 49–60. [11] David W. Hosmer and Stanley Lemeshow. 2000. Applied logistic regression (Wiley Series in probability and statistics) (2 ed.). Wiley-Interscience Publication, Hoboken, NJ. [12] David W Hosmer Jr, Stanley Lemeshow, and Rodney X Sturdivant. 2013. Applied logistic regression. John Wiley & Sons. [13] Marian Jureczko and Lech Madeyski. 2010. Towards identifying software project clusters with regard to defect prediction. In Proceedings of the 6th International
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
Conference on Predictive Models in Software Engineering. 1–10. [14] Luigi Lavazza and Sandro Morasca. 2022. Comparing 𝜙 and the F-measure as Performance Metrics for Software-related Classifications. EMSE 27, 7 (2022). [15] Luigi Lavazza, Sandro Morasca, and Gabriele Rotoloni. 2023. On the Reliability of the Area Under the ROC Curve in Empirical Software Engineering. In Proceedings of the 24th International Conference on Evaluation and Assessment in Software Engineering (EASE). Association for Computing Machinery (ACM). [16] Luigi Lavazza, Sandro Morasca, and Gabriele Rotoloni. 2025. Software Defect Prediction evaluation: New metrics based on the ROC curve. Information and Software Technology (2025), 107865. [17] Sandro Morasca and Luigi Lavazza. 2020. On the assessment of software defect prediction models via ROC curves. Empirical Software Engineering 25, 5 (2020), 3977–4019. [18] Rebecca Moussa and Federica Sarro. 2022. On the Use of Evaluation Measures for Defect Prediction Studies. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). ACM. [19] John Platt et al. 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers 10, 3 (1999), 61–74. [20] Martin Shepperd, Qinbao Song, Zhongbin Sun, and Carolyn Mair. 2013. Data quality: Some comments on the NASA software defect datasets. IEEE Transactions on software engineering 39, 9 (2013), 1208–1215. [21] Yogesh Singh, Arvinder Kaur, and Ruchika Malhotra. 2010. Empirical validation of object-oriented metrics for predicting fault proneness models. Software quality journal 18, 1 (2010), 3. [22] David L Streiner and John Cairney. 2007. What’s under the ROC? An introduction to receiver operating characteristics curves. The Canadian Journal of Psychiatry 52, 2 (2007), 121–128. [23] Ben Van Calster, Ewout W Steyerberg, Ralph B D’Agostino Sr, and Michael J Pencina. 2014. Sensitivity and specificity can change in opposite directions when new predictive markers are added to risk models. Medical Decision Making 34, 4 (2014), 513–522. [24] Jan Y Verbakel, Ewout W Steyerberg, Hajime Uno, Bavo De Cock, Laure Wynants, Gary S Collins, and Ben Van Calster. 2020. ROC curves for clinical prediction models part 1. ROC plots showed no added value above the AUC when evaluating the performance of clinical prediction models. Journal of Clinical Epidemiology 126 (2020), 207–216.