Concept Drift Detection and Adaptive Retraining of Malware Classification Models

Concept Drift Detection and Adaptive Retraining of Malware Classification Models Christofer Washington Berruz Chungata∗ Martin Jureček† Katerina Potika∗ William B. Andreopoulos∗ Mark Stamp∗‡

arXiv:2608.13465v1 [cs.LG] 13 Aug 2026

August 14, 2026 Abstract Concept drift refers to changes over time in the statistical properties of data, as compared to the data that was used to train a learning model. Machine learning models for malware detection or classification are particularly susceptible to performance degradation caused by concept drift, as attackers constantly modify existing malware. In this chapter, we analyze two machine learning-based approaches to automated concept drift detection—a novel approach based on One-Class Support Vector Machines (OCSVM) and a previously-studied technique based on Minibatch 𝐾-Means (MK-Means). For comparison we also consider Maximum Mean Discrepancy (MMD), a statistical technique for detecting changes in multidimensional data. We conduct an extensive series of experiments comparing the effectiveness of four learning models, namely, Multilayer Perceptron (MLP), Random Forest (RF), Support Vector Machines (SVM), and eXtreme Gradient Boosting (XGB). For each of these models, we consider three distinct scenarios: A static scenario where no model retraining occurs, a periodic scenario where models are constantly retrained irrespective of concept drift, and a drift-aware scenario where models are only retrained when concept drift is detected. Under the drift-aware scenario, we analyze the tradeoff between accuracy and training efficiency using Pareto Front analysis. We find that all three concept drift detection techniques achieve classification accuracy comparable to periodic retraining, while offering substantially greater efficiency in terms of the number of models that must be retrained. In addition, drift-aware retraining based on our OCSVM technique generally outperforms the MK-Means and MMD approaches. Overall, these results provide strong evidence that we are able to accurately detect concept drift in malware classification models. Furthermore, our concept drift detection techniques are efficient and practical, and the process of updating learning models can easily be fully automated. ∗

Department of Computer Science, San Jose State University Faculty of Information Technology, Czech Technical University in Prague ‡ mark.stamp@sjsu.edu †

1

Keywords: Concept Drift · Malware · One-Class Support Vector Machines · OCSVM · Maximum Mean Discrepancy · MMD · MK-Means · Pareto Front

1

Introduction

Change over time in the statistical properties of data on which learning models have been trained is known as concept drift. Given that machine learning models learn from data, concept drift can degrade the performance of such models during inference. Concept drift is a particularly important problem in malware detection as malware is constantly evolving. According to [3, 50], attackers modify malware for two purposes—to add or modify functionality and to evade detection. As a result, developers and researchers using machine learning models for malware detection or classification must take concept drift into consideration when deploying these models in real-world systems. The number of cyberattacks due to malware, such as ransomware attacks, has increased significantly in recent years [6]. A single malware sample that is incorrectly classified as benign can wreak havoc on a system by bringing down machines or holding data hostage for ransom. One solution to mitigate the effects of concept drift is to constantly retrain the models used for detection. However, the resource demands of this approach may be substantial, in terms of computational resources and energy usage. In an era where significant research efforts focus on minimizing the resource consumption of complex machine learning models, such as Deep Neural Networks (DNN) and Large-Language Models (LLM) [15, 34], constant retraining would seem to be a step backwards. In this research, we conduct experiments involving two machine learning-based approaches for concept drift detection in malware classification models—one based on Minibatch 𝐾-Means (MK-Means) and another based on One-Class Support Vector Machines (OCSVM). For comparison, we also consider concept drift detection using the well-known statistical technique of Maximum Mean Discrepancy (MMD), which can be applied to detect changes in multidimensional data. To demonstrate the effectiveness of these concept drift detection approaches, we consider the following three scenarios, which are discussed in more detail in Section 3.2. i) A classification model is trained on the initial temporal segment of the training data, and no retraining occurs. This represents a scenario where concept drift is not considered. We refer to this as the static scenario. ii) As in i), a classification model is trained on the initial temporal segment of the training data. Then the model is retrained at regular periodic intervals. This represents a scenario where concept drift is considered, but no effort is made to minimize the number of models that must be retrained. We refer to this as the periodic retraining scenario. iii) As in ii), a classification model is trained on the initial temporal segment of the training data and concept drift is considered. However, instead of

2

periodic retraining, we only retrain when concept drift is detected via one of the methods mentioned above. This represents a scenario where we account for concept drift, but we also attempt to minimize model retraining. We refer to this as the drift-aware retraining scenario. For each of these three scenarios, we consider four classification models, namely, Multilayer Perceptron (MLP), Random Forest (RF), eXtreme Gradient Boosting (XGB), and Support Vector Machines (SVM). Furthermore, for each of these modelscenario combinations, we consider 20 distinct combinations of malware families. Thus, we conduct 80 static experiments, 80 periodic retraining experiments, and 240 drift-aware retraining experiments (80 using each of the OCSVM, MK-Means, and MMD drift detectors), giving us a total of 400 distinct experiments. The remainder of this chapter is organized as follows. Section 2 provides relevant background information on concept drift, MK-Means, OCSVM, MMD, and the learning models used in our classification experiments. In Section 3, we describe the dataset used in this research, and the development environment for our experiments. In addition, this section describes the static, periodic retraining, and drift-aware retraining scenarios in detail. Section 4 presents results and analysis for all of our experiments. We demonstrate that using drift-aware retraining yields a large increase in accuracy, as compared to the static case, and only a slight decrease in accuracy, as compared to periodic retraining. We further analyze the drift-aware retraining case, and provide detailed information on the tradeoff between accuracy and efficiency. We conclude the chapter in Section 5, where we also provide suggestions for future work. Note that this chapter expands on and extends the research that appeared in the paper [12]. Specifically, the MMD technique presented in this chapter is analyzed in [12], which in the present chapter, merely serves as a baseline for comparison of the MK-Means and OCSVM techniques.

2

Background

In this section, we first discuss concept drift and malware detection. Then we provide details on the OCSVM, MK-Means, and MMD techniques, which form the basis for our automated concept drift detection experiments. Finally, we briefly introduce the learning models that are used in our experiments.

2.1

Concept Drift

Concept drift is defined as changes in the statistical properties of data over time, which can affect the performance of machine learning models [5]. Machine learning models deployed in real-world systems may be adversely affected by concept drift, as the data used for training the model can age out or become irrelevant. This is certainly the case for machine learning-based malware detection systems, as malware families generally tend to evolve over time.

3

Concept drift can occur abruptly, gradually, or it can be recurring [5]. Abrupt concept drift is the easiest to detect, while gradual and recurring concept drift require more sophisticated methods that incorporate previous knowledge of the data and the model [11, 43, 49, 59]. According to [5], any concept drift detection technique can be classified into one of four categories, namely, data distribution-based, performance-based, hybrid, and context-based. Performance-based methods are considered more reliable than data distribution-based methods, as they focus on instances where the model is performing poorly [5]. However, such methods rely on the ability to measure welldefined performance metrics (e.g., accuracy, precision, recall, F1-score), and when using machine learning models to classify malware, such information is generally lacking. Given the limitations of performance-based methods in the malware domain, we focus our attention on data distribution-based methods. That is, we require drift detection techniques that can be performed by directly analyzing the data, without relying on measures of model performance. We validate that our drift detection approaches succeed by comparing the performance of classification models under three distinct scenarios—details on these scenarios are given in Section 3.2.

2.2

Malware Detection

Malware detection is the process of identifying whether a given piece of software is malicious (malware) or benign. Malware detection is a vitally important area of research for at least three reasons. First, false negatives (malware misclassified as benign) can lead to significant damage to a system, such as data loss and system downtime. Second, false positives (benign misclassified as malware) can lead to significant inconvenience to users. Even a relatively small false positive rate can result in a “boy who cried wolf” syndrome, where users lose trust in the system and ignore warnings. Third, malware is constantly evolving, making it challenging to detect new malware variants [3, 50]. Over the past 20 years, malware research has shifted dramatically, to where the focus today is dominated by machine learning (ML) and deep learning (DL) techniques. From classical ML models, such as Support Vector Machines (SVM) and Random Forests (RF), to DL models, such as Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN), learning models are now routinely applied to the malware detection and classification problems [4, 35, 42]. In recent years, the impact of concept drift on malware detection has gained some attention from researchers—although certainly not as much attention as the importance of the topic warrants. For example, the authors of [31] proposed a novel technique for detecting and classifying concept drift in malware by learning supposedly drift-invariant features from control flow graphs. Another example of such research can be found in [1], which introduced MORPH, a pseudo-label-based concept drift adaptation method for neural networks as used for malware detection. Additionally, the paper [25] considers DREAM, a semi-supervised system that enhances drift detection in deep learning-based malware classifiers. Note that the

4

approaches considered in our research differ from [1, 25, 31], as we use unsupervised machine learning techniques that are faster, simpler, and more efficient to implement. Our methodology builds on and extends the work in [36]. We note in passing that research into malware evolution is related to concept drift, in the sense that such research deals with changes to malware over time. However, in concept drift research, the focus is on how these changes affect the performance of machine learning models, while malware evolution research deals with the changes themselves. Hidden Markov Models (HMM) [53], Word2Vec [39], SVMs [55], and hierarchical tree models [60] have all been used to study malware evolution. Research into concept drift detection for malware requires data that contains not only a significant number of malware samples over an extended period of time, but also include reliable timestamps. This is challenging because the time when a malware sample was originally created is not generally available. Therefore, modification or collection dates are used as proxies for the creation date. Prior to the KronoDroid dataset [23], other malware datasets used for research involving temporal aspects of malware analysis included the Malicia dataset [37] and the malware collection in [28].

2.3

Concept Drift Detection Techniques

In this section, we introduce the two unsupervised learning techniques that we employ for concept drift detection, namely, Minibatch 𝐾-Means (MK-Means) and One-Class Support Vector Machines (OCSVM). Furthermore, we introduce Maximum Mean Discrepancy (MMD), an established technique to detect data drift in multidimensional data. Additional details on precisely how we employ these techniques for concept drift detection are provided in Section 3.2.

2.3.1

Minibatch 𝐾-Means

Minibatch 𝐾-Means (MK-Means) [2] is a more scalable variant of the well-known 𝐾-Means clustering algorithm [24]. While many 𝐾-means variants exist, such as 𝐾Means++ [27], MK-Means has been optimized for large datasets by using stochastic optimization principles. The main difference between 𝐾-Means and MK-Means is that the latter uses small random subsets of the data (i.e., minibatches) to update the centroids, instead of using the entire dataset at each iteration. As with 𝐾-means, MK-Means is an unsupervised clustering algorithm that uses a hill climb approach to find improved centroids of clusters. Formally, the 𝐾-Means algorithm aims to minimize Within-Cluster Sum of Squares (WCSS), which is defined as 𝐾 ∑︁ ∑︁ arg min ‖𝑥 − 𝜇𝑖 ‖2 {𝑆1 ,...,𝑆𝐾 } 𝑖=1 𝑥∈𝑆

𝑖

where 𝜇𝑖 is the mean, or centroid, of the data points in cluster 𝑆𝑖 , and the nonempty subsets 𝑆1 , 𝑆2 , . . . , 𝑆𝐾 of 𝑆 form a partition of the set 𝑆. Note that a centroid 𝜇𝑖 need not be an actual data point in its cluster 𝑆𝑖 .

5

Given that it is a hill climb algorithm, MK-Means is sensitive to the initial placement of the centroids. Hence, MK-Means can produce different clustering results, even for a fixed number of clusters 𝐾. Intuitively, a good clustering result should produce clusters that are well separated from each other, while any individual cluster should be relatively compact. The silhouette coefficient provides a single value that incorporates both of these desirable aspects of cluster quality. For a given clustering, of the 𝑛 data points {𝑥1 , 𝑥2 , . . . , 𝑥𝑛 }, the silhouette coefficient of a specific data point 𝑥𝑖 is defined as 𝑠(𝑖) =

𝑏(𝑖) − 𝑎(𝑖) (︀ )︀ max 𝑎(𝑖), 𝑏(𝑖)

where 𝑎(𝑖) is the average distance between 𝑥𝑖 and all other points in its same cluster, and 𝑏(𝑖) is the minimum of the average distances from 𝑥𝑖 to points in each of the other clusters. For a reasonable clustering, we expect 𝑏(𝑖) > 𝑎(𝑖), in which case 𝑠(𝑖) = 1 −

𝑎(𝑖) 𝑏(𝑖)

In this form, we see that a silhouette coefficient value 𝑠(𝑖) closer to 1 represents a better clustering result for 𝑥𝑖 than values closer to 0. Thus, the larger the average silhouette coefficient 𝑛 1 ∑︁ 𝑠= 𝑠(𝑖) 𝑛 𝑖=1

the better the overall quality of a given clustering. For concept drift detection, we use MK-Means to cluster consecutive (overlapping) pairs of temporal batches of samples, and we compute the average silhouette coefficient for each such clustering. If the change in average silhouette coefficient from one pair of batches to the next exceeds a specified threshold, this implies that the features representing the underlying samples have changed, and hence concept drift has occurred. More details on MK-Means for concept drift detection are provided in Section 3.2.2. We note that our use of MK-Means for concept drift detection in malware closely follows the approach in [36].

2.3.2

One-Class Support Vector Machines

Support Vector Machines (SVM) are supervised machine learning models used for classification and regression tasks. One-Class SVMs (OCSVM) are a variant of SVMs that are trained in an unsupervised mode, and hence labeled data is not required. OCSVMs were originally introduced in [45]. OCSVMs are used widely for outlier detection [16, 33]. Outlier detection is a generic problem where, for a given dataset, there exists two regions defined as inliers and outliers. The outliers differ significantly from the inliers, while the inliers form the bulk of the data. Many techniques exist for outlier detection, including Robust Covariance [44], Isolation Forest [32], and Local Outlier Factor [9]; refer to [46] for a visual comparison of the results of these techniques on a synthetic dataset.

6

OCSVM tries to find the region where the training samples are representative of the dominant distribution of the data. OCSVMs include a regularization parameter 𝜈 ∈ (0, 1), which sets an upper limit on the fraction of training data points that can be considered outliers. Larger values of 𝜈 can lead to a more complex, but looser, decision boundary, while smaller values of 𝜈 yield a simpler, but tighter, decision boundary. For a dataset with a high number of outliers, too small of values of 𝜈 can result in long training times and models that are overly sensitive to noise. For a given OCSVM model, let 𝑛outliers be the number of outliers and 𝑛inliers the number of inliers. We define 𝑛outliers 𝑟= (1) 𝑛inliers If this ratio 𝑟 changes over time, it is reasonable to assume that the underlying distribution of the data has changed. Therefore, we can use OCSVM to detect concept drift by monitoring the ratio of the number of outliers to the number of inliers. Additional details on how we employ OCSVM for concept drift detection are given in Section 3.2.2. As far as the authors are aware, this is the first time that OCSVM has been used for concept drift detection, although OCSVMs have previously been used for malware anomaly detection [47].

2.3.3

Maximum Mean Discrepancy

Concept drift can cause model performance degradation due to changes in the underlying data distribution. The authors of [41] provide an empirical study of various distribution shift detection techniques, with the goal of better understanding the relationship between distribution shift and model performance degradation. Given two sets of samples, 𝑥 = {𝑥1 , 𝑥2 , . . . , 𝑥𝑛 } and 𝑥′ = {𝑥′1 , 𝑥′2 , . . . , 𝑥′𝑚 }, the goal in distribution shift detection is to determine whether the probability distributions 𝑝(𝑥) and 𝑞(𝑥′ ) are similar. To detect whether two distributions are similar, we can use hypothesis testing with a two-sample test. Formally, we want to test the null hypothesis 𝐻0 : 𝑝(𝑥) = 𝑞(𝑥′ ) against the alternative 𝐻𝐴 : 𝑝(𝑥) ̸= 𝑞(𝑥′ ) at a specified significance level 𝛼. We note in passing that distribution shift depends on the chosen representation of the samples, which we refer to as the latent space. Depending on the latent space, two-sample tests can give different results. Maximum Mean Discrepancy (MMD) [21] is a popular kernel-based technique for multivariate two sample testing. MMD can distinguish between two distributions 𝑝 and 𝑞 based on the mean embeddings 𝜇𝑝 and 𝜇𝑞 in a special latent space (i.e., Hilbert space) that we denote as ℋ. MMD uses a kernel 𝑘 to map samples onto ℋ. The MMD is computed as MMD(ℋ, 𝑝, 𝑞) = ‖𝜇𝑝 − 𝜇𝑞 ‖ℋ .

7

Given samples drawn from 𝑝 and 𝑞 we compute the unbiased estimator MMD2 =

𝑛

𝑛

𝑚

𝑚

𝑖̸=𝑗

𝑖=1 𝑗=1

𝑖̸=𝑗

1 ∑︁ 2 ∑︁ ∑︁ 1 ∑︁ ′ 𝑘(𝑥 , 𝑥 ) − 𝑘(𝑥 , 𝑥 ) + 𝑘(𝑥′𝑖 , 𝑥′𝑗 ) 𝑖 𝑗 𝑖 𝑗 𝑛2 𝑛𝑚 𝑚2 2

where 𝑘 is the Gaussian exponential kernel, 𝑘(𝑥, 𝑥 ˜) = 𝑒−‖𝑥−˜𝑥‖ /𝜎 , and 𝜎 is the me′ dian distance between all pair of points (𝑥, 𝑥 ) [21]. The authors of [41] note that we can compute a 𝑝-value using permutation tests on the resulting kernel matrix to perform hypothesis testing. Once a 𝑝-value is computed, we accept or reject the null hypothesis 𝐻0 at a chosen significance level 𝛼 depending on whether 𝑝-value > 𝛼 (accept 𝐻0 ) or 𝑝-value ≤ 𝛼 (reject 𝐻0 ). The Python package Alibi Detect [54] contains implementations of all techniques described in [41]; we use these implementations for our experiments.

2.4

Learning Models

In this section, we introduce the four learning models that are consider in our experiments. These are standard learning models that serve to test our concept drift detection techniques.

2.4.1

Multilayer Perceptron

A MultiLayer Perceptron (MLP) is a classic type of feedforward Artificial Neural Network (ANN). An MLP consists of an input layer, an output layer, and one or more hidden layers. Each hidden layer is fully connected to the layers above and below. MLPs are usually trained using backpropagation. Training MLPs is done in batches and epochs—one epoch represents a forward and backward pass through the entire training set, which is usually divided into minibatches. The main hyperparameters of an MLP architecture are the number of hidden layers, the number of neurons in each layer, and the choice of activation functions. For classification tasks, cross-entropy is widely used, and it is available in scikit-learn [48]. For binary classification, Binary Cross Entropy (BCE), also known as Log Loss, is used [51]. Assuming a classifier is a probabilistic model, BCE measures the distance between two Bernoulli distributions—the true distribution and the predicted distribution given by the classifier [51]. Thus, for 𝑛 samples, the BCE is computed as BCE = −

𝑛 )︀ 1 ∑︁ (︀ 𝑦𝑖 log(𝑝𝑖 ) + (1 − 𝑦𝑖 ) log(1 − 𝑝𝑖 ) 𝑛 𝑖=1

where 𝑦𝑖 is the actual label (0 or 1) of sample 𝑖, and 𝑝𝑖 is the predicted probability of the 𝑖th sample being in class 1. Note that due to the log term, BCE assigns a higher penalty to more confident, but incorrect, predictions.

8

2.4.2

Support Vector Machine

A Support Vector Machine (SVM) is a supervised machine learning model. The goal of an SVM is to find the optimal hyperplane that separates the data into classes. To achieve separability, SVMs can map the data into a higher dimensional space using a kernel function. The kernel function is selected such that computing the dot product in a higher dimensional space does not incur any significant computational cost in terms of the input data—this is known as the “kernel trick.” The most popular kernel functions are linear, polynomial, and radial basis function (rbf) [57]. For additional details on SVMs, see, for example [14].

2.4.3

Random Forest

Random Forest (RF) is an ensemble machine learning technique that combines decision trees to make a prediction. Random Forest uses “bagging” to create uncorrelated decision trees [26]. Random Forest is a classical machine learning model that in practice can often rival more complex neural network models. An important hyperparameter of RF is the depth of the trees. A deeper depth means that more decisions are included in each tree, which can possibly lead to a higher accuracy, at the cost of a higher training time, as well as an increased chance of overfitting. RF models tend to generalize well given the uncorrelated nature of the component decision trees. For more details on RFs, refer to [8].

2.4.4

Extreme Gradient Boosting

Similar to RF, eXtreme Gradient Boosting (XGB) is an ensemble machine learning technique based on decision trees. It is important to note that XGBoost is an optimized version of a general gradient boosting algorithm, with the goal of maximizing performance and efficiency [10]. XGBoost uses boosting instead of bagging, which distinguishes it from RF. In practice, this means that while RF builds independent decision trees by random bagging of the features and samples, XGBoost builds decision trees sequentially. In XGBoost, each new (possibly weak) classifiers aims to predict the errors of the previous classifier [30]. The intuition behind XGBoost is that each new classifier will generally be able to improve on the errors of the previous classifiers.

3

Implementation

In this section, we first discuss the dataset that we use in our experiments. Then we present a detailed view of our experimental setup.

3.1

Dataset

The study of concept drift in malware detection requires a dataset in which samples can be placed on a relative timeline. We use the KronoDroid dataset, which was

9

introduced in [23]; we retrieved the dataset from [22]. The KronoDroid dataset contains 41,382 Android malware samples belonging to 240 distinct malware families. Note that the KronoDroid dataset contains both real and emulated samples—we only use real samples in this research. Each sample in the dataset includes 200 static features (permissions, intents, hardware/software requirements, etc.) and 289 dynamic features (e.g., system calls). According to the paper introducing KronoDroid [23], each sample contains four distinct timestamps: Earliest Modification, Last Modification, First Seen VT (on Virus Total) and First Seen ITW (in the wild). However, in the dataset available from Github [22], malware samples contain only two timestamps: EarliestModDate and HighestModDate. In this research, we use the HighestModDate timestamp as it is the best available timestamp in the dataset with the least number of invalid samples, according to [23].

3.1.1

Malware Families

Although there are 240 malware families in the KronoDroid dataset, we focus our attention on the five malware families with the largest number of samples. By doing so, we can study concept drift and evaluate model performance for a significant number of samples over meaningful periods of time. In this respect, our approach is analogous to that followed in [36]. The five malware families under consideration are the following. Airpush/StopSMS is adware that displays unwanted ads and may silently collect and forward user data [17]. Subsequently, we refer this family simply as Airpush. SMSReg is riskware that, for example, may include a fake “Battery Improve” application [18]. Malap is spyware that collects sensitive information from a device [23]. Boxer is a Trojan that pretends to be a legitimate installer but actually sends premium-rate SMS messages, unbeknownst to the user [19]. Agent is a Trojan that downloads and installs adware or malware onto a victim device [20].

3.1.2

Data Preprocessing

As mentioned above, each malware sample in our dataset includes 489 features, 200 of which are static and 289 of which are dynamic. However, some features are not numerical, but are instead text. As a preprocessing step, we remove all non-numerical features using the Polars library [40]. The dimension of each feature vector after this preprocessing is 470. Additionally, we discard samples whose HighestModDate timestamp seems to be incorrect. In concrete terms, only samples with a timestamp in the format “MM/DD/YYYY” and years in the range of 2000 to 2025 are considered valid. Table 1

10

summarizes the extracted samples from the KronoDroid dataset; we explain the meaning of the “batches” column in Section 3.2.1. Table 1: KronoDroid dataset after preprocessing

3.2

Family

Samples

Batches

Earliest date

Latest date

Agent Airpush Boxer Malap SMSreg

2,826 7,760 3,597 4,018 4,989

56 155 71 80 99

2008-02-28 2008-02-29 2005-01-01 2008-02-29 2008-02-29

2020-07-17 2018-06-15 2018-06-15 2020-11-11 2020-11-09

Total

23,190

461

2005

2020

Experiment Setup

As discussed above, in this research we consider three scenarios to evaluate the benefits of employing concept drift detection in the malware domain. We refer to these three scenarios as static training, periodic retraining, and drift-aware retraining. In each scenario, we train each of the four machine learning classifiers discussed above (MLP, SVM, RF, and XGB) on each of 20 distinct classification tasks. Intuitively, we expect that periodic retraining will yield the highest accuracy while the static training accuracy will be the lowest. If we are able to accurately detect concept drift, then the drift-aware retraining accuracies should be comparable to the periodic retraining accuracies, while reducing the work required to retrain models.

3.2.1

Definitions

Let ℛ = {ocsvm, mk-means, mmd} denote the set of drift detection techniques considered. We let ℒ = {MLP, SVM, RF, XGB} be the set of learning models under consideration. Let 𝐹 denote the samples belonging to a specific malware family. For each experiment, let 𝐹𝑑 be the samples of the malware family for which we want to detect where concept drift has occurred, and let 𝐹𝑐 be the “control” samples, where 𝐹𝑐 is a different malware family than 𝐹𝑑 . For our binary classification experiments, 𝐹𝑑 and 𝐹𝑐 are the classes that we train models to distinguish. We employ the notation ℱ = {(𝐹𝑑 , 𝐹𝑐 ) | 𝐹𝑑 ̸= 𝐹𝑐 }, and since we consider five families, |ℱ| = 20. For each malware family 𝐹 , we order the samples from oldest to most recent, based on the HighestModDate timestamp. Let 𝑛 be the number of samples in 𝐹 , and denote the ordered samples of 𝐹 as (𝑓0 , 𝑓1 , . . . , 𝑓𝑛−1 ), where 𝑓𝑖 precedes 𝑓𝑗 in the temporal ordering whenever 𝑖 < 𝑗. For a malware family 𝐹 , we partition the samples into consecutive temporal batches, truncating so that all temporal batches are of the same size. Note that the smaller the batch size, the sooner we can detect concept drift, but smaller batch sizes also tend to increase the variance, adding noise to the process. In all of our

11

experiments, we define the batch size as 𝑏 = 50. Small-scale tests showed that the results did not improve significantly for larger values of 𝑏, while adverse variance effects were observed for smaller batch sizes. Assuming that |𝐹 | = 𝑛, the number of batches in 𝐹 is ⌊𝑛/50⌋. Letting 𝐵𝑖 represent the 𝑖th batch, we have 𝐵𝑖 = {𝑓𝑖·𝑏 , 𝑓𝑖·𝑏+1 , . . . , 𝑓(𝑖+1)·𝑏−1 } Thus, 𝐵0 = {𝑓0 , 𝑓1 , . . . , 𝑓49 }, 𝐵1 = {𝑓50 , 𝑓51 , . . . , 𝑓99 }, and so on. Let 𝐵𝑖train be the training subset and 𝐵𝑖test be the testing subset of 𝐵𝑖 . We define 𝐵𝑖train to be the first 𝑡 samples in 𝐵𝑖 and 𝐵𝑖test to be the remaining 𝑏 − 𝑡 samples in 𝐵𝑖 . We use 𝑡 = 30 samples for training and hence the number of samples used for testing per batch is 𝑏 − 𝑡 = 50 − 30 = 20. For example, for the first batch 𝐵0 , we have 𝐵0train = {𝑓0 , 𝑓1 , . . . , 𝑓29 } and 𝐵0test = {𝑓30 , 𝑓31 , . . . , 𝑓49 }. The “batches” column in Table 1 provides the number of batches per family.

3.2.2

Concept Drift Detection

As mentioned above, we consider two machine learning-based approaches for concept drift detection—one based on MK-Means, another based on OCSVM—as well as the statistical-based MMD technique. In this section, we describe these three approaches in more detail. MK-Means — We employ the silhouette coefficient, based on MK-Means clustering, to detect concept drift as follows. Using the notation above, we let ℬ0 = {𝐵0 , 𝐵1 } and ℬ1 = {𝐵1 , 𝐵2 }. Then we perform MK-Means clustering on the samples in ℬ0 and separately perform MK-Means clustering on the samples in ℬ1 . Next, we compute the average silhouette coefficient 𝑠0 for the clustering of ℬ0 and compute the average silhouette coefficient 𝑠1 for the clustering of ℬ1 . Then if |𝑠1 − 𝑠0 | > 𝒯mk-means , where 𝒯mk-means is a specified threshold, we assume that concept drift has occurred at batch 𝐵1 . We repeat this process for subsequent consecutive pairs of batches. OCSVM — We employ OCSVM to detect concept drift as follows. We train an OCSVM on 𝐵0 and compute the ratio of outliers to inliers—as per equation (1)—which we denote as 𝑟0 . The ratio 𝑟0 represents characteristics of the data in 𝐵0 . We then reuse this same OCSVM model to compute 𝑟1 , the ratio of outliers to inliers over 𝐵1 . If |𝑟1 − 𝑟0 | > 𝒯ocsvm , where 𝒯ocsvm is a specified threshold, then we assume that the samples in 𝐵1 differ from those in 𝐵0 , and hence concept drift has occurred. If concept drift is detected, we then retrain the OCSVM on 𝐵1 and recompute 𝑟1 on 𝐵1 using this updated OCSVM model at the next step. On the other hand, if concept drift is not detected in 𝐵1 , we do not retrain the OCSVM. In either case, we then move on to consider 𝐵2 , and the process repeats. MMD — We employ MMD to detect concept drift as follows. To identify whether concept drift has occurred at batch 𝐵𝑖 , we let 𝐵ref = 𝐵𝑖−1 and 𝐵new = 𝐵𝑖 . We then perform a two-sample test using MMD with a Gaussian kernel and

12

compute a 𝑝-value for hypothesis testing as described in Section 2.3.3. Given a confidence level 𝛼, we let 𝒯mmd = 𝛼 to keep notation homogenous across all drift detectors. If 𝑝-value ≤ 𝒯mmd , we conclude that drift has occurred at 𝐵𝑖 ; otherwise, no drift is detected.

3.2.3

Training a model

Consider a given pair of malware families which, as above, we denoted as (𝐹𝑑 , 𝐹𝑐 ), with 𝑁 batches 𝐵𝑖 over 𝐹𝑑 . Let 𝑋 train and 𝑋 test be two disjoint subsets of 𝐹𝑐 . Let 𝐷𝑖train = (𝐵𝑖train ∪ 𝑋 train ) and 𝐷𝑖test = (𝐵𝑖test ∪ 𝑋 test ). This yields a labeled dataset, where the samples in 𝐵𝑖 are one class, and the selected samples from 𝐹𝑐 , namely, 𝑋 train ∪ 𝑋 test , are the other class. A classification model 𝑀𝑖 is trained on 𝐷𝑖train and evaluated on 𝐷𝑖test. From Section 3.2.1, we recall that |𝐵𝑖train | = 𝑡 and |𝐵𝑖test | = 𝑏 − 𝑡. Ideally, we want a balanced dataset with |𝑋 train | = 𝑡 and |𝑋 test | = 𝑏 − 𝑡. Therefore, in all experiments we let 𝑐 } 𝑋 train = {𝑓0𝑐 , 𝑓1𝑐 , . . . , 𝑓𝑡−1 and 𝑐 𝑐 𝑋 test = {𝑓𝑡𝑐 , 𝑓𝑡+1 , . . . , 𝑓𝑏−1 }

As mentioned above, in all of our experiments, we let 𝑡 = 30 and 𝑏 = 50, and hence 𝑏 − 𝑡 = 20.

3.2.4

Static Training

For a given drift family 𝐹𝑑 , control family 𝐹𝑐 , and learning model 𝐿 ∈ ℒ, we train the model on 𝐷0train . We denote this trained model as 𝑀0 . Then the accuracy 𝐴static (𝐹𝑑 , 𝐹𝑐 , 𝐿) under the static training scenario is computed as 𝐴static (𝐹𝑑 , 𝐹𝑐 , 𝐿) =

𝑁 −1 ∑︁

𝑀0 (𝐷𝑖test )

⧸︂ 𝑁∑︁ −1

𝑖=0

|𝐷𝑖test |

(2)

𝑖=0

where 𝑀0 (𝐷𝑖test ) is the number of correct predictions made by model 𝑀0 over 𝐷𝑖test, and the drift family consists of 𝑁 batches numbered 0 through 𝑁 − 1. This experiment represents the scenario where we train one model on the initial temporal batch, and use this model to classify the test samples in all subsequent batches.

3.2.5

Periodic Retraining

For a given drift family 𝐹𝑑 , control family 𝐹𝑐 , and learning model 𝐿, we train a model 𝑀𝑖 on 𝐷𝑖train for each batch in the drift family 𝐹𝑑 . The overall accuracy for this periodic retraining experiment is given by 𝐴periodic (𝐹𝑑 , 𝐹𝑐 , 𝐿) =

𝑁 −1 ∑︁ 𝑖=0

13

𝑀𝑖 (𝐷𝑖test )

⧸︂ 𝑁∑︁ −1 𝑖=0

|𝐷𝑖test |

(3)

where 𝑀𝑖 (𝐷𝑖test ) is the number of correct predictions made by 𝑀𝑖 over 𝐷𝑖test, and the drift family consists of 𝑁 batches numbered 0 through 𝑁 − 1. The periodic retraining experiment simulates the scenario where we train a new instance of the same model type on every batch of samples. Note that it is not necessary that the hyperparameters of the model remain the same across batches, and hence we recompute the hyperparameters for each model 𝑀𝑖 .

3.2.6

Drift-Aware Retraining

The drift-aware retraining scenario is slightly more involved. For a given drift family 𝐹𝑑 , control family 𝐹𝑐 , and learning model 𝐿, suppose that we detect concept drift at batches 𝐵𝑖1 , 𝐵𝑖2 , . . ., 𝐵𝑖ℓ , where 0 < 𝑖1 < 𝑖2 < · · · < 𝑖ℓ < 𝑁 − 1 with 𝑁 being the number of batches in the drift family 𝐹𝑑 . Then defining 𝑖0 = −1 and 𝑖ℓ+1 = 𝑁 − 1, we have 𝑖𝑗+1 ℓ ∑︁ ∑︁

⧸︂ 𝑁∑︁ −1

𝑀𝑖𝑗 +1 (𝐷𝑘test ) 𝐴drift (𝐹𝑑 , 𝐹𝑐 , 𝐿, 𝑅) = 𝑗=0 𝑘=𝑖𝑗 +1

3.2.7

|𝐷𝑖test |

(4)

𝑖=0

Efficiency

The models trained in the drift-aware scenario are a subset of the models trained in the periodic retraining scenario. This follows since the batches where concept drift is detected are a subset of all of the batches in 𝐹𝑑 , and in the periodic retraining scenario, we retrain a model at every batch of 𝐹𝑑 . Consequently, one measure of efficiency for the drift-aware scenario is the percentage of batches where concept drift is detected—the more drift points detected, the more models that must be trained, resulting in a higher overall cost for training. In this sense, the periodic retraining scenario has the worst possible efficiency, and the static scenario has the best possible efficiency, since at least one model must be trained. For specified families (𝐹𝑑 , 𝐹𝑐 ) and drift detection technique 𝑅, let ℓ be the number of drift points detected. Then we define the drift-aware training efficiency as ⎧ ℓ ⎪ ⎨ 1− if 𝑅 = mk-means 𝑁 −2 ℰ(𝐹𝑑 , 𝐹𝑐 , 𝑅) = (5) ⎪ ⎩ 1− ℓ otherwise 𝑁 −1 where 𝑁 is the number of batches in 𝐹𝑑 . Since our clustering approach uses pairs of batches, for the MK-Means drift detection technique we have 𝑁 − 2 possible drift points, whereas for the other two drift detection techniques, we have 𝑁 − 1 possible drift points. Note that by the efficiency measures in equation (5), the static scenario will always achieve the maximum efficiency of 1. However, in any case where concept drift occurs, we expect the static scenario to have the lowest accuracy of our three

14

experimental scenarios. Hence, the tradeoff between efficiency and accuracy must be considered. We provide a detailed discussion of this inherent tradeoff in Section 3.4.

3.2.8

Workflow

The experiment workflow consists of preprocessing the dataset and then conducting the experiments. The preprocessing step includes removing non-numerical features and discarding samples with obviously incorrect timestamps. Details on the workflow for the experiments is given in Algorithm 1.

Algorithm 1 Experiment workflow 1: // Step 0: Preliminaries 2: Input: Select drift and control malware families (𝐹𝑑 , 𝐹𝑐 ) ∈ ℱ 3: Let 𝑁 = number of batches in 𝐹𝑑 4: Select learning model 𝐿 ∈ ℒ = {MLP, SVM, RF, XGB} 5: Select drift detector 𝑅 ∈ ℛ = {ocsvm, mk-means, mmd} 6: Specify drift detector threshold 𝒯𝑅 7: Output: Accuracy for static, periodic, and drift-aware scenarios 8: Efficiency for drift-aware scenario 9: // Step 1: Train models (including hyperparameter tuning) 10: for 𝑖 = 1, 2, . . . , 𝑁 do 11: Train model 𝑀𝑖 (of type 𝐿) on batch 𝐵𝑖 12: // Step 2: Static scenario 13: Use trained 𝑀0 to compute 𝐴static (𝐹𝑑 , 𝐹𝑐 , 𝐿) // equation (2) 14: // Step 3: Periodic retraining 15: Use all models 𝑀𝑖 to compute 𝐴periodic (𝐹𝑑 , 𝐹𝑐 , 𝐿) // equation (3) 16: // Step 4: Drift-aware retraining 17: Detect drift points 𝐷 using 𝑅 and threshold 𝒯𝑅 // Section 3.2.2 18: Use 𝐷 to compute accuracy 𝐴drift (𝐹𝑑 , 𝐹𝑐 , 𝐿, 𝑅) // equation (4) 19: Compute efficiency ℰ(𝐹𝑑 , 𝐹𝑐 , 𝑅) // equation (5) 20: // Step 5: Return results 21: Return: 𝐴static (𝐹𝑑 , 𝐹𝑐 , 𝐿), 𝐴periodic (𝐹𝑑 , 𝐹𝑐 , 𝐿), 𝐴drift (𝐹𝑑 , 𝐹𝑐 , 𝐿, 𝑅), 22: and ℰ(𝐹𝑑 , 𝐹𝑐 , 𝑅)

3.3

Hyperparameters

Before presenting our experimental results, we need to consider hyperparameters. In this section, we first discuss hyperparameter tuning of the learning models, which affect the classification accuracy of each model. After discussing hyperparameter tuning for our learning models, we consider the hyperparameters for our drift detection techniques, which affect the tradeoff between accuracy and efficiency under the drift-aware scenario.

15

3.3.1

Hyperparameter Tuning of Learning Models

An important step in our classification experiments is to perform hyperparameter tuning to obtain optimized learning models. We use Optuna [38], which performs a Tree-Structured Parzen Estimator (TPE) [56] search for the best hyperparameters. TPE is a Bayesian optimization algorithm. Given an objective function 𝑓𝜃 where 𝜃 is a set of parameters, TPE performs multiple trials over 𝜃, iteratively searching for optimal values. This approach is different from a grid search because TPE does not try every value defined in the space—it uses Bayesian estimates to find better regions. We use a total of 100 trials per model. The set of hyperparameters considered for each ML model are given in Table 2. Here, we employ the notation (𝑎, 𝑏) to indicate that the open interval from 𝑎 to 𝑏 was specified for Optuna, while the notation {𝑥, 𝑦} denotes a discrete set of values.

Table 2: Hyperparameters for learning models Model

Hyperparameters

Search space

SVM

𝑐 (regularization) kernel

(𝑒−4 , 𝑒4 ) {rbf}

RF

n estimators max depth

(10, 500) (5, 100)

XGB

n estimators max depth learning rate

(10, 300) (5, 15) {0.01, 0.05, 0.1, 0.2, 0.3}

MLP

n units activation solver

(8, 256) {relu, tanh, logistic} {adam, sgd}

Given (𝐹𝑑 , 𝐹𝑐 , 𝐿), where 𝐿 is the selected type of learning model, the objective function of each model 𝑀𝑖 is defined as 𝑓𝑖𝜃 =

𝑀𝑖𝜃 (𝐷𝑖test ) |𝐷𝑖test |

(6)

where 𝑀𝑖𝜃 is initialized using parameters 𝜃, with the model trained on 𝐷𝑖train , and 𝑀𝑖𝜃 (𝐷𝑖test ) denotes the number of correct predictions of 𝑀𝑖𝜃 on 𝐷𝑖test , that is, we are optimizing each model based on the test accuracy. All other parameters needed for a model 𝑀𝑖 not mentioned in the table are set to their default values in the respective libraries. Since we tune the hyperparameters of one model 𝑀𝑖 per (𝐹𝑑 , 𝐹𝑐 , 𝐿) combination and per temporal batch, the selected hyperparameters need not be the same. For all models, we use early stopping—if a set of hyperparameters gives an objective function with ideal separation, we stop searching the space. For each drift family, there are four (𝐹𝑑 , 𝐹𝑐 ) pairs, we have 461 total temporal batches (accounting for all drift families), and there are four machine learning models. Therefore, we have 4 · 4 · 461 = 7,376 combinations. Furthermore, for each of

16

these combinations, we run 100 trials, which gives us a total number of (𝐹𝑑 , 𝐹𝑐 , 𝐿) combinations for which hyperparameters are determined of 100 · 7,376 = 737,600. We note in passing that the standard approach would be to tune the hyperparameters using a separate validation set. However, because of the small number of samples, such a validation set is not available. Tuning on the training set would also not be ideal because the reported metrics are computed on the test set, and overfitting is likely, especially given the limited sample size. Thus, the use of testset tuning is intentional and can be interpreted as an oracle-style upper bound on performance rather than as a realistic deployment procedure. The goal of our experimental design is to initially establish the two baseline scenarios (static and periodic retraining), and then compare these two to various concept drift detection strategies under the drift-aware scenario. Under all three of these scenarios, the hyperparameter tuning is conducted in a similar fashion, and hence any change in accuracy in the drift-aware scenario should be attributable to the drift detection mechanism.

3.3.2

Hyperparameters for Drift Detectors

It is clear from our experimental design that if all batches are detected as drift points, then the drift-aware scenario is identical to the periodic retraining scenario. Furthermore, in equation (5) we define efficiency so that the larger the number of detected drift points, the lower the efficiency in the drift-aware scenario. In practice, we would want to balance accuracy and efficiency. Since the hyperparameters of the drift detectors determine their respective detection processes, the tradeoff between accuracy and efficiency can be analyzed as a function of these hyperparameters. We can model this problem as a multi-objective optimization of conflicting objectives. This means that an increase in one objective produces a decrease in the other objective. Given a drift detector 𝑅 ∈ ℛ, we denote its hyperparameter space as Ω𝑅 . The drift-aware accuracy 𝐴drift and retraining efficiency ℰ are the objective functions. Note that hyperparameter tuning of the models 𝑀𝑖 is orthogonal to the hyperparameter tuning of the drift detector, and hence we do not need to retrain the 𝑀𝑖 models. Therefore, we can simply rerun Step 4 (lines 16–19) of Algorithm 1 using a grid search over Ω𝑅 to obtain multiple values for 𝐴drift and ℰ. Table 3 shows the hyperparameter space that we use in a grid search for our OCSVM, MKMeans, and MMD concept drift detection techniques. We test all learning models and family combinations, and consequently, we run Step 4 of Algorithm 1 a total of 4 · 20 · (540 + 240 + 100) = 70,400 times.

3.4

Pareto Front

According to Algorithm 1, in the drift-aware scenario, the accuracy 𝐴drift depends on 𝐹𝑑 , 𝐹𝑐 , 𝐿, and 𝑅, while the efficiency ℰ depends only on 𝐹𝑑 , 𝐹𝑐 , and 𝑅. However,

17

Table 3: Hyperparameters for each drift detector Hyperparameters

Ω𝑅 Values tested

MK-Means

clusters 𝒯mk-means

{2, 4, 6, 8} {0.01, 0.02, . . . , 0.6}

4 · 60 = 240

OCSVM

𝜈 𝒯ocsvm

{0.1, 0.2, . . . , 0.9} {0.01, 0.02, . . . , 0.6}

9 · 60 = 540

MMD

𝒯mmd = 𝛼

{0.001, 0.002, . . . , 0.1}

100

Drift detector 𝑅

Total values

this is not entirely correct, as both accuracy and efficiency depend on the selected hyperparameters 𝜔 ∈ Ω𝑅 where, as above, Ω𝑅 is a set of hyperparameters corresponding to drift detector 𝑅. Furthermore, there is an inherent tradeoff between accuracy and efficiency. One technique for selecting the hyperparameters 𝜔 is to find Pareto efficient solutions, also known as a Pareto Front [7]. Pareto Fronts are widely used in engineering and economics where tradeoffs have to be made between two conflicting objectives [13], such as the tradeoff between fuel efficiency and engine power. By using a Pareto Front, it is possible to find a set of points where optimal tradeoffs can be made—in a sense that is made clear below. The Pareto Front is also referred to as the set of non-dominated solutions. Formally, we consider a compact parameter space R𝑛 and an objective space R𝑚 with a mapping function 𝑓 such that 𝑓 : 𝑋 → R𝑚 where 𝑋 ⊆ R𝑛 . Then, the objective values can be represented as a set of vectors 𝑌 = { 𝑦 ∈ R𝑚 | 𝑦 = 𝑓 (𝑥), 𝑥 ∈ 𝑋}. Let ≻ be a preference operator that dictates whether we want to maximize or minimize the objective. A vector 𝑦 ′′ ∈ R𝑚 is said to strictly dominate another vector 𝑦 ′ ∈ R𝑚 if 𝑦 ′′ ≻ 𝑦 ′ . Then the Pareto Front can be expressed as 𝑃 (𝑌 ) = { 𝑦 ′ ∈ 𝑌 | { 𝑦 ′′ ∈ 𝑌 | 𝑦 ′′ ≻ 𝑦 ′ , 𝑦 ′′ ̸= 𝑦 ′ } = ∅}. Intuitively, the points belonging to the Pareto Front create a boundary that surrounds all other points of the objective. Points forming the boundary have the best possible tradeoff, in the sense that moving inside the boundary produces a negative effect on at least one of the objectives. For a drift detector 𝑅 and its parameter space Ω𝑅 , we perform a Pareto Front analysis per (𝐹𝑑 , 𝐹𝑐 , 𝐿) combination, that is, for each combination of drift family, control family, and learning model. Suppressing the dependence on (𝐹𝑑 , 𝐹𝑐 , 𝐿), as the objectives, we use (Δdrift , ℰ), where Δdrift = 𝐴drift − 𝐴static .

18

Note that we use Δdrift instead of 𝐴drift in the objective because it better conveys the desired optimization, which is to maximize the difference from the static baseline. Of course, given Δdrift and the static baseline 𝐴static , we can trivially determine 𝐴drift . We use the Paretoset Python package, which implements the Skyline Query operator [29, 52] to determine the Pareto Front for a set of points in the objective space. We note in passing that finding the Pareto Front without a set of points in the objective space can be achieved through evolutionary methods [13, 58] which fall under multi-objective optimization algorithms. We do not require a multi-objective optimization algorithm since a grid search generates the set of points from which we can directly determine the Pareto Front.

3.5

Consolidating Drift-Aware Experiments

We now discuss how to use Pareto Front analysis to obtain meaningful results for comparing our drift detection techniques to the static and periodic scenarios. Again, this additional analysis is required since it is necessary to tune various hyperparameters of each drift detection technique, and when doing so, there is an inherent tradeoff between accuracy and efficiency. Recall that 𝐿 ∈ ℒ is the learning model, 𝑅 ∈ ℛ is the drift detection technique, Ω𝑅 set of all hyperparameters tested for 𝑅, while 𝜔 represents a specific selection of hyperparameters from Ω𝑅 . Also, 𝐹𝑑 is the drift family, and 𝐹𝑐 is the control family, with (𝐹𝑑 , 𝐹𝑐 ) ∈ ℱ. We require the additional notation Ω*𝑅 = set of hyperparameters tested corresponding to Pareto Front 𝜔 * = specific selection of hyperparameters from Ω*𝑅 Since we have five malware families, |ℱ| = 20, and the values of |Ω𝑅 | are given in Table 3.

3.5.1

Mean Static and Periodic Accuracies

In the static scenario, the accuracy depends only on 𝐹𝑑 , 𝐹𝑐 , and 𝐿, and hence we denote each of these accuracy values as 𝐴static (𝐹𝑑 , 𝐹𝑐 , 𝐿). Then the average accuracy per learning model 𝐿 is given by ∑︁ 1 𝒜static (𝐿) = 𝐴static (𝐹𝑑 , 𝐹𝑐 , 𝐿) |ℱ| (𝐹𝑑 ,𝐹𝑐 )∈ℱ

Similarly, in the periodic scenario, the average accuracy per learning model is ∑︁ 1 𝒜periodic (𝐿) = 𝐴periodic (𝐹𝑑 , 𝐹𝑐 , 𝐿) |ℱ| (𝐹𝑑 ,𝐹𝑐 )∈ℱ

3.5.2

Mean Drift-Aware Accuracy

The drift-aware scenario is more complex. As in the static and periodic scenarios, the accuracy depends on 𝐹𝑑 , 𝐹𝑐 , and 𝐿. However, in the drift-aware scenario, the

19

accuracy also depends on the drift detection technique 𝑅 and, more specifically, the selected hyperparameters 𝜔 ∈ Ω𝑅 . Hence, in this scenario, we denote the accuracy as 𝐴drift (𝐹𝑑 , 𝐹𝑐 , 𝐿, 𝑅, 𝜔). For a given 𝜔 ∈ Ω𝑅 , we can average over all (𝐹𝑑 , 𝐹𝑐 ) ∈ ℱ to obtain ∑︁ 1 𝐴drift (𝐹𝑑 , 𝐹𝑐 , 𝐿, 𝑅, 𝜔) (7) 𝐴ℱ drift (𝐿, 𝑅, 𝜔) = |ℱ| (𝐹𝑑 ,𝐹𝑐 )∈ℱ

On the other hand, for a given (𝐹𝑑 , 𝐹𝑐 ) ∈ ℱ , we can average over all 𝜔 ∈ Ω𝑅 to obtain 1 ∑︁ 𝑅 𝐴Ω (𝐹 , 𝐹 , 𝐿, 𝑅) = 𝐴drift (𝐹𝑑 , 𝐹𝑐 , 𝐿, 𝑅, 𝜔) (8) 𝑐 𝑑 drift |Ω𝑅 | 𝜔∈Ω𝑅

From equation (7), the mean accuracy per (𝐿, 𝑅) pair can be computed as 1 ∑︁ ℱ 𝐴drift (𝐿, 𝑅, 𝜔) 𝒜𝑥drift (𝐿, 𝑅) = |Ω𝑅 |

(9)

𝜔∈Ω𝑅

Alternatively, based on equation (8), we can compute the mean accuracy per (𝐿, 𝑅) pair as ∑︁ 1 𝑅 𝒜𝑦drift (𝐿, 𝑅) = 𝐴Ω (10) drift (𝐹𝑑 , 𝐹𝑐 , 𝐿, 𝑅) |ℱ| (𝐹𝑑 ,𝐹𝑐 )∈ℱ

It is clear that 𝒜𝑥drift (𝐿, 𝑅) = 𝒜𝑦drift (𝐿, 𝑅) and we denote this common value as 𝒜drift (𝐿, 𝑅). The calculations of 𝒜drift (𝐿, 𝑅), using either equation (9) or (10), is illustrated in Table 4. Table 4: Compute 𝒜drift as average of final row or final column (𝑁 = |ℱ|) 𝐴(𝐹𝑑0 , 𝐹𝑐0 , 𝜔0 ) 𝐴(𝐹𝑑1 , 𝐹𝑐1 , 𝜔0 ) 𝐴(𝐹𝑑2 , 𝐹𝑐2 , 𝜔0 ) .. .

𝐴(𝐹𝑑0 , 𝐹𝑐0 , 𝜔1 ) 𝐴(𝐹𝑑1 , 𝐹𝑐1 , 𝜔1 ) 𝐴(𝐹𝑑2 , 𝐹𝑐2 , 𝜔1 ) .. .

𝐴(𝐹𝑑0 , 𝐹𝑐0 , 𝜔2 ) 𝐴(𝐹𝑑1 , 𝐹𝑐1 , 𝜔2 ) 𝐴(𝐹𝑑2 , 𝐹𝑐2 , 𝜔2 ) .. .

··· ··· ··· .. .

𝐴(𝐹𝑑0 , 𝐹𝑐0 , 𝜔|Ω|−1 ) 𝐴(𝐹𝑑1 , 𝐹𝑐1 , 𝜔|Ω|−1 ) 𝐴(𝐹𝑑2 , 𝐹𝑐2 , 𝜔|Ω|−1 ) .. .

𝐴Ω drift (𝐹𝑑0 , 𝐹𝑐0 ) 𝐴Ω drift (𝐹𝑑1 , 𝐹𝑐1 ) 𝐴Ω drift (𝐹𝑑2 , 𝐹𝑐2 ) .. .

𝐴(𝐹𝑑𝑁 −1 , 𝐹𝑐𝑁 −1 , 𝜔0 ) 𝐴(𝐹𝑑𝑁 −1 , 𝐹𝑐𝑁 −1 , 𝜔1 ) 𝐴(𝐹𝑑𝑁 −1 , 𝐹𝑐𝑁 −1 , 𝜔2 ) · · · 𝐴(𝐹𝑑𝑁 −1 , 𝐹𝑐𝑁 −1 , 𝜔|Ω|−1 ) 𝐴Ω drift (𝐹𝑑𝑁 −1 , 𝐹𝑐𝑁 −1 ) 𝐴ℱ drift (𝜔0 )

3.5.3

𝐴ℱ drift (𝜔1 )

𝐴ℱ drift (𝜔2 )

···

𝐴ℱ drift (𝜔|Ω|−1 )

𝒜drift

Mean Drift-Aware Accuracy over Pareto Front

Computing of 𝒜drift (𝐿, 𝑅) using either equation (9) or (10) treats all hyperpameter combinations 𝜔 ∈ Ω𝑅 the same. This is clearly not ideal, as many hyperparameter values will yield suboptimal results. To mitigate this issue, we now consider an approach analogous to that in Section 3.5.2, but restricted to points on the relevant Pareto Front. For each 𝜔 ∈ Ω𝑅 , we first determine 𝐴ℱ drift (𝐿, 𝑅, 𝜔) using equation (7). Then we compute ∑︁ 1 𝒜𝑥* 𝐴ℱ (𝐿, 𝑅, 𝜔 * ) (11) drift (𝐿, 𝑅) = |Ω*𝑅 | * * drift 𝜔 ∈Ω𝑅

20

where Ω*𝑅 is the set of hyperparameter values on the Pareto Front that correspond 𝑥* to 𝐴ℱ drift (𝐿, 𝑅, 𝜔). Note that this computation of 𝒜drift (𝐿, 𝑅) is analogous to that of 𝒜𝑥drift (𝐿, 𝑅) in equation (9), but restricted to the Pareto Front of 𝐴ℱ drift (𝐿, 𝑅, 𝜔). The calculation of 𝒜𝑥* (𝐿, 𝑅) is illustrated in Table 5. drift ℱ * Table 5: Compute 𝒜𝑥* drift as average of 𝐴drift (𝜔𝑖 )

𝐴(𝐹𝑑0 , 𝐹𝑐0 , 𝜔0 ) 𝐴(𝐹𝑑1 , 𝐹𝑐1 , 𝜔0 ) 𝐴(𝐹𝑑2 , 𝐹𝑐2 , 𝜔0 ) .. .

𝐴(𝐹𝑑0 , 𝐹𝑐0 , 𝜔1 ) 𝐴(𝐹𝑑1 , 𝐹𝑐1 , 𝜔1 ) 𝐴(𝐹𝑑2 , 𝐹𝑐2 , 𝜔1 ) .. .

𝐴(𝐹𝑑0 , 𝐹𝑐0 , 𝜔2 ) 𝐴(𝐹𝑑1 , 𝐹𝑐1 , 𝜔2 ) 𝐴(𝐹𝑑2 , 𝐹𝑐2 , 𝜔2 ) .. .

··· ··· ··· .. .

𝐴(𝐹𝑑0 , 𝐹𝑐0 , 𝜔|Ω|−1 ) 𝐴(𝐹𝑑1 , 𝐹𝑐1 , 𝜔|Ω|−1 ) 𝐴(𝐹𝑑2 , 𝐹𝑐2 , 𝜔|Ω|−1 ) .. .

𝐴(𝐹𝑑𝑁 −1 , 𝐹𝑐𝑁 −1 , 𝜔0 ) 𝐴(𝐹𝑑𝑁 −1 , 𝐹𝑐𝑁 −1 , 𝜔1 ) 𝐴(𝐹𝑑𝑁 −1 , 𝐹𝑐𝑁 −1 , 𝜔2 ) · · · 𝐴(𝐹𝑑𝑁 −1 , 𝐹𝑐𝑁 −1 , 𝜔|Ω|−1 ) 𝐴ℱ drift (𝜔0 ) ⏟

𝐴ℱ drift (𝜔1 )

𝐴ℱ drift (𝜔2 )

···

𝐴ℱ drift (𝜔|Ω|−1 ) 𝒜𝑥* drift

⏞ *

Determine Pareto Front Ω

|Ω|−1 from {𝐴ℱ drift (𝜔𝑖 )}𝑖=0

Alternatively, we can first sum over the Pareto Front points for each combination of (𝐹𝑑 , 𝐹𝑐 , 𝐿, 𝑅). Analogous to equation (8), in this case we compute Ω*

𝑅 𝐴drift (𝐹𝑑 , 𝐹𝑐 , 𝐿, 𝑅) =

∑︁ 1 𝐴drift (𝐹𝑑 , 𝐹𝑐 , 𝐿, 𝑅, 𝜔 * ) * |Ω𝑅 | * * 𝜔 ∈Ω𝑅

where Ω*𝑅 is the set of hyperameter values that correspond to Pareto Front points Ω*𝑅 (𝐹𝑑 , 𝐹𝑐 , 𝐿, 𝑅) over all (𝐹𝑑 , 𝐹𝑐 ) ∈ ℱ, we obtain of (𝐹𝑑 , 𝐹𝑐 , 𝐿, 𝑅). Then averaging 𝐴drift 𝒜𝑦* drift (𝐿, 𝑅) =

1 |ℱ|

∑︁

Ω*

𝑅 (𝐹𝑑 , 𝐹𝑐 , 𝐿, 𝑅) 𝐴drift

(12)

(𝐹𝑑 ,𝐹𝑐 )∈ℱ

Note that this computation of 𝒜𝑦* drift (𝐿, 𝑅) is analogous to that in equation (10), but restricted to the relevant Pareto Front values. The calculation of 𝒜𝑦* drift (𝐿, 𝑅) is illustrated in Table 6. Table 6: Compute 𝒜𝑦* drift as average of final column (𝑁 = |ℱ |) 𝐴(𝐹𝑑0 , 𝐹𝑐0 , 𝜔0* ) 𝐴(𝐹𝑑1 , 𝐹𝑐1 , 𝜔0* ) 𝐴(𝐹𝑑2 , 𝐹𝑐2 , 𝜔0* ) .. .

𝐴(𝐹𝑑0 , 𝐹𝑐0 , 𝜔1* ) 𝐴(𝐹𝑑1 , 𝐹𝑐1 , 𝜔1* ) 𝐴(𝐹𝑑2 , 𝐹𝑐2 , 𝜔1* ) .. .

𝐴(𝐹𝑑0 , 𝐹𝑐0 , 𝜔2* ) 𝐴(𝐹𝑑1 , 𝐹𝑐1 , 𝜔2* ) 𝐴(𝐹𝑑2 , 𝐹𝑐2 , 𝜔2* ) .. .

··· ··· ··· .. .

* 𝐴(𝐹𝑑0 , 𝐹𝑐0 , 𝜔|Ω * |−1 ) * 𝐴(𝐹𝑑1 , 𝐹𝑐1 , 𝜔|Ω* |−1 ) * 𝐴(𝐹𝑑2 , 𝐹𝑐2 , 𝜔|Ω * |−1 ) .. .

*

𝐴Ω drift (𝐹𝑑0 , 𝐹𝑐0 ) * 𝐴Ω drift (𝐹𝑑1 , 𝐹𝑐1 ) Ω* 𝐴drift (𝐹𝑑2 , 𝐹𝑐2 ) .. . *

* 𝐴(𝐹𝑑𝑁 −1 , 𝐹𝑐𝑁 −1 , 𝜔0* ) 𝐴(𝐹𝑑𝑁 −1 , 𝐹𝑐𝑁 −1 , 𝜔1* ) 𝐴(𝐹𝑑𝑁 −1 , 𝐹𝑐𝑁 −1 , 𝜔2* ) · · · 𝐴(𝐹𝑑𝑁 −1 , 𝐹𝑐𝑁 −1 , 𝜔|Ω 𝐴Ω * |−1 ) drift (𝐹𝑑𝑁 −1 , 𝐹𝑐𝑁 −1 )

𝒜𝑦* drift

When considering all of the hyperparameter combinations 𝜔 ∈ Ω𝑅 , equations (9) and (10) show that the order of summation can be interchanged. However, when restricting our attention to hyperparameter values of 𝑅 corresponding to Pareto Front points, this is no longer the case—in general, there is no reason to expect 𝑦* that 𝒜𝑥* drift (𝐿, 𝑅) will yield the same result as 𝒜drift (𝐿, 𝑅).

21

3.5.4

Representative Accuracy over Pareto Front

In practice, we would need to select a specific value for the hyperparameters of the drift detection technique 𝑅, and we would almost certainly want to select such a value from the Pareto Front. In this section we define such a representative value that is, in a sense, near the “middle” and, therefore, we refer to it as the median. For a give pair of malware families (𝐹𝑑 , 𝐹𝑐 ), learning model 𝐿, and drift detection technique 𝑅, let (𝑥min , 𝑦min ) be the point on the Pareto Front that gives the minimum ΔAccuracy and let (𝑥max , 𝑦max ) be the point on the Pareto Front where the maximum ΔAccuracy is attained. Then let 𝑥𝑎 =

𝑥min + 𝑥max 𝑦min + 𝑦max and 𝑦𝑎 = 2 2

The equation of the line from the origin through (𝑥𝑎 , 𝑦𝑎 ) is given by 𝑦=

𝑦𝑎 𝑥 𝑥𝑎

(13)

* ∈ Ω* to be the hyperparameter values corresponding to the point on Let 𝜔𝑚 𝑅 the Pareto Front that is closest to the line in equation (13). Then we define the median accuracy for a given (𝐿, 𝑅) pair as

𝒜𝑚* drift (𝐿, 𝑅) =

1 |ℱ|

∑︁

* 𝐴drift (𝐹𝑑 , 𝐹𝑐 , 𝐿, 𝑅, 𝜔𝑚 )

(𝐹𝑑 ,𝐹𝑐 )∈ℱ

For each (𝐿, 𝑅) pair, we now have four distinct measures of the accuracy, namely, 𝑦* 𝑚* 𝒜drift (𝐿, 𝑅), 𝒜𝑥* drift (𝐿, 𝑅), 𝒜drift (𝐿, 𝑅), and 𝒜drift (𝐿, 𝑅)

(14)

Of course, for each of the accuracy measures in equation (14) there is a corresponding efficiency that we denote as 𝑦* 𝑚* 𝑥* (𝐿, 𝑅) (𝐿, 𝑅), and ℰdrift (𝐿, 𝑅), ℰdrift ℰdrift (𝐿, 𝑅), ℰdrift

(15)

respectively.

3.6

Software and Development Environment

Table 7 summarizes the libraries and software used in this research. Note that we primarily use Python for the experiments. In addition, we use Docker to run a PostgreSQL container and a volume to store the hyperparameter tuning trial information generated by Optuna [38]. The best models 𝑀𝑖 are stored on disk as .pkl files. We export hyperparameter tuning information and experiment results as .csv files for later analysis using Matplotlib. All experiments were run on a CPU machine; no GPUs were employed. To allow for reproducibility, any machine learning model, function, or library used in this research that requires a random seed as input has been set to a fixed value of 420.

22

Table 7: Software summary Software Python3.10

Main programming language used

Poetry

Python dependency management and packaging tool

Polars

Loading and preprocess KronoDroid dataset Type inference and data manipulation

Matplotlib

Build and visualize graphs and plots

NumPy

Numerical computations and array manipulations

Alibi-Detect

4

Purpose

Implementation for MMD

Pytorch

Backend needed to run MMD

Paretoset

Find the Pareto Front

Optuna

Automatic hyperparameter optimization framework

scikit-learn

OCSVM and MK-Means implementations Implementations of MLP, SVM, and RF classifiers

XGBoost

XGB implementation

Docker

Postgres container for storing hyperparameters

Psycopg2

PostgreSQL database adapter for Python

Experimental Results and Analysis

In this section, we present the results for the concept drift detection experiments discussed above. The primary goal of these experiments is to determine the accuracy and efficiency of each drift detector (MK-Means, OCSVM, and MMD), as compared to the static and periodic retraining scenarios.

4.1

Static and Periodic Results

Recall that our static and periodic experiments are independent of any drift detection technique. In subsequent sections, we consider accuracy and efficiency in the drift-aware scenario based on each of the three drift detectors under consideration. Figures 14 through 17 in the Appendix provides bar graphs of the static accuracies 𝐴static and periodic accuracies 𝐴periodic . These results are provided for all four learning models 𝐿 and all 20 family combinations for (𝐹𝑑 , 𝐹𝑐 ). From Figures 14 through 17 we observe that MLP, RF, and MLP all perform well, with XGB lagging. We also generally observe qualitative consistency as, for example, across all models the periodic scenario yields a large improvement in accuracy for (Malap, Agent) and (SMSreg, Malap), while (Airpush, Boxer) and (Boxer, SMSreg) only improve marginally. Furthermore, SVM achieves the highest accuracy in the static scenario for 40% of the family combinations (8 out of 20), followed by MLP and RF at 25% each, with XGB performing best in the remaining 10% of

23

0.80

0.9334

0.7578

0.9500

0.8416

0.9529

0.8152

0.9666

1.00

0.8248

Average accuracy

the combinations. However, in the periodic retraining scenario, MLP performs best for 65% of the family combinations, followed by SVM and RF at 25% and 10%, respectively, with XGB being best for only 1 such combination. In Figure 1 we provide the average 𝐴static and 𝐴periodic for each learning model across all family combinations. This graph clearly demonstrates the value of accounting for concept drift, as the best average accuracy in the static scenario is 0.8416 (SVM model), while the worst average accuracy in the periodic retraining scenario is 0.9334 (XGB model). Furthermore, using the best model in the periodic scenario (MLP), we obtain an average accuracy of 0.9666.

0.60 0.40 Static Periodic

0.20 0.00 MLP

RF

SVM

XGB

Figure 1: Average static and periodic accuracy per learning model Another aspect to consider is the effect of hyperparameter tunning of each learning model instance. If hyperparameter tunning was generally successful for learning model 𝐿, we expect the distribution of its periodic accuracies to be more compact and have a higher median, as compared to the distribution of the static accuracies. This is indeed the case, as illustrated in Figure 2, which validates our use of TPE for hyperparameter tuning. Overall, these static and periodic results confirm the success of our experimental design and hyperparameter tuning. It is also clear that retraining models is necessary to counter concept drift. However, given the computational expense of training learning models, we would like to achieve comparable results to the periodic scenario with less work. To this end, we now consider drift-aware retraining for each of the three drift detectors introduced in Section 2.3. Our primary goal is to determine whether we can achieve results comparable to periodic retraining, but at a significantly lower cost, in terms of the number of models trained.

4.2

MK-Means Drift-Aware Results

Recall that MK-Means has two hyperparameters, (𝒯mk-means , clusters). Figure 3 contains a plot of the Pareto Front for the specific case (𝐹𝑑 , 𝐹𝑐 , 𝐿, 𝑅) = (Agent, Airpush, MLP, mk-means)

24

Figure 2: Box plot of static and periodic accuracy As previously noted, from the Pareto Front coordinates, it is trivial to determine the corresponding hyperparameters (𝒯mk-means , clusters). (𝐹𝑑 , 𝐹𝑐 , 𝐿, 𝑅) = (Agent, Airpush, MLP, MK-Means) 1.0

Efficiency

0.8

0.6

0.4

0.2 Non-Pareto Front Pareto Front 0.0

−0.050−0.025 0.000 0.025 0.050 0.075 0.100 0.125 ΔAccuracy

Figure 3: MK-Means Pareto Front example From Figure 3 we can see the advantage of choosing hyperparameter values based on the Pareto Front. For example, if we require an efficiency in the range of, say, 60% to 65%, choosing hyperparameter values from the Pareto Front could provide an improvement in accuracy of more than 3%, as compared to values that are not on the Pareto Front.

25

0.80

0.7578 0.8185 0.8529 0.8871 0.9023 0.9334

0.8416 0.8812 0.9052 0.9266 0.9378 0.9500

0.8152 0.8686 0.8941 0.9235 0.9358 0.9529

Accuracy

1.00

0.8248 0.8862 0.9135 0.9377 0.9480 0.9666

In Figure 4, we provide a bar graph comparing the average accuracies for each learning model under the static, drift-aware, and periodic scenarios, where the drift detection is based on MK-Means. As discussed in Section 3.5, for the drift-aware scenario, we have four distinct methods for computing the average accuracy, which 𝑦* 𝑚* we denote as 𝒜drift , 𝒜𝑥* drift , 𝒜drift , and 𝒜drift . Recall that 𝒜drift is the average accuracy over all (𝒯mk-means , clusters) pairs tested, while the latter three averages are based on Pareto Front analysis.

0.60 0.40 Static 𝒜𝑦* drift

0.20

𝒜drift 𝒜𝑚* drift

𝒜𝑥* drift Periodic

0.00 MLP

RF

SVM

XGB

Figure 4: MK-Means accuracies From Figure 4, we observe that for all learning models tested, using MKMeans for concept drift detection produces a gain in accuracy, as compared to the static scenario. Also, by most of the measures considered, drift-aware retraining based on MK-Means results in only a relatively small loss in accuracy as compared to the more costly periodic scenario. For example, considering the median 𝒜𝑚* drift for the MLP model, we see that the drift-aware scenario improves on the static scenario by nearly 15%, since (0.9480 − 0.8248)/0.8248 = 0.1494, while being within 2% of the accuracy achieved in the more costly periodic scenario, since (0.9666 − 0.9480)/0.9666 = 0.0192. Figure 5 shows the analogous efficiency results. The improvement in efficiency for the drift-aware scenario averaged over all hyperparameters (𝒯mk-means , clusters) tested, as indicated by ℰdrift , is over 80%, although the improvement when restricted to the better choices of hyperparameters, as represented by the Pareto Front-based 𝑥* , ℰ 𝑦* , and ℰ 𝑚* is not quite as impressive. Nevertheless, for the values of ℰdrift drift drift 𝑚* we achieve a savings—in terms of the number of models that must be median ℰdrift trained—of more than 58%, as compared to the periodic retraining scenario.

4.3

OCSVM Drift-Aware Results

Our analysis of concept drift detection based on OCSVM is similar to that for MKMeans, above. In Figure 6, we give the Pareto Front for the same specific case that we considered for MK-Means, namely, (𝐹𝑑 , 𝐹𝑐 , 𝐿, 𝑅) = (Agent, Airpush, MLP, ocsvm)

26

0.60

0.8210 0.7046 0.6679 0.6964

0.8210 0.6989 0.6707 0.6899

0.8210 0.7499 0.7045 0.7485

0.80

0.8210 0.7093 0.5949 0.5867

Efficiency

1.00

0.40 ℰdrift 𝑦* ℰdrift

0.20

𝑥* ℰdrift 𝑚* ℰdrift

0.00 MLP

RF

SVM

XGB

Figure 5: MK-Means efficiencies As in the MK-Means case, we again see the value of Pareto Front analysis for selecting hyperparameters to meet a desired balance between accuracy and efficiency. (𝐹𝑑 , 𝐹𝑐 , 𝐿, 𝑅) = (Agent, Airpush, MLP, OCSVM) 1.0

Efficiency

0.8

0.6

0.4

0.2 Non-Pareto Front Pareto Front 0.0

−0.050−0.025 0.000 0.025 0.050 0.075 0.100 0.125 ΔAccuracy

Figure 6: OCSVM Pareto Front example In Figures 7 and 8, respecitvely, we provide bar graphs of our various accuracy and efficiency metrics. We note that, similar to MK-Means, OCSVM achieves a substantial accuracy gain over the static baseline at a significant lower retraining cost than the periodic retraining scenario. For example, if we consider the median 𝒜𝑚* drift for the MLP model, the drift-aware scenario improves on the static scenario by 15% ((0.9486 − 0.8248)/0.8248 = 0.1501), while being within 1.9% of the accuracy

27

0.80

0.7578 0.8335 0.8785 0.8888 0.9047 0.9334

0.8416 0.8816 0.9176 0.9273 0.9368 0.9500

0.8152 0.8801 0.9187 0.9309 0.9434 0.9529

Accuracy

1.00

0.8248 0.8954 0.9325 0.9391 0.9486 0.9666

achieved in the more costly periodic scenario ((0.9666 − 0.9486)/0.9666 = 0.0186), a marginal improvement over MK-Means.

0.60 0.40 𝒜drift 𝒜𝑚* drift

Static 𝒜𝑦* drift

0.20

𝒜𝑥* drift Periodic

0.00 MLP

RF

SVM

XGB

0.7785 0.5895 0.7256 0.7095

0.60

0.7785 0.6085 0.6873 0.7283

0.80

0.7785 0.5826 0.6820 0.6951

Efficiency

1.00

0.7785 0.5861 0.6463 0.6586

Figure 7: OCSMV accuracies

0.40 ℰdrift 𝑦* ℰdrift

0.20

𝑥* ℰdrift 𝑚* ℰdrift

0.00 MLP

RF

SVM

XGB

Figure 8: OCSVM efficiencies With respect to efficiency, the improvement for the OCSVM-based drift-aware scenario averaged over all hyperparameters tested is more than 77%. The improve𝑚* and ment in efficiency when restricted to the median Pareto Front-based value ℰdrift MLP model is more than 65%. Recall that the corresponding median value MKMeans is about 58%. Thus, by this measure, OCSVM offers improved efficiency, as compared to MK-Means.

4.4

MMD Drift-Aware Results

Our MMD-based concept drift detection technique differs from MK-Means and OCSVM in that for MMD we only consider one hyperparameter. This hyperparameter, which we denote as 𝒯mmd , is a confidence value for hypothesis testing. In Figure 9, we give the Pareto Front for the same specific case that we considered for both MK-Means and OCSVM, that is, (𝐹𝑑 , 𝐹𝑐 , 𝐿, 𝑅) = (Agent, Airpush, MLP, mmd)

28

Compared to MK-Means and OCSVM, we observe that for this MMD example, there is more consistency in the sense that points tend to be very close to the Pareto Front, modulo a small number of outliers. (𝐹drift , 𝐹control , 𝐿, 𝑅) = (Agent, Airpush, MLP, MMD)

Efficiency

0.85

0.80

0.75 Non-Pareto Front Pareto Front 0.00

0.01

0.02 ΔAccuracy

0.03

Figure 9: MMD Pareto Front example In Figure 10 we provide bar graphs of the various accuracy metrics and in Figure 11 we give the corresponding efficiency results. As with both MK-Means and OCSVM, MMD achieves a substantial gain over the static baseline at a significant lower retraining cost, as compared with the periodic retraining scenario. For example, if we consider the median 𝒜𝑚* drift for the MLP model, the drift-aware scenario improves on the static scenario by 13.72%, since (0.9380 − 0.8248)/0.8248 = 0.1372, while being within 3% of the accuracy achieved in the much more costly periodic retraining scenario, since (0.9666 − 0.9380)/0.9666 = 0.0296. These numbers are slightly worse than the results we obtained using MK-Means and OCSVM. With respect to efficiency, the improvement over the periodic scenario, averaged over all hyperparameters tested, is more than 64%. On the other hand, the efficiency 𝑚* for when restricted to Pareto Front is slightly better—for example, the value ℰdrift the MLP model is more than 69%. Recall that the corresponding median value MK-Means is about 58%, and for OCSVM it is about 65%. For MK-Means and OCSVM, the average efficiency declined significantly when we restricted the hyperparameters to the Pareto Front. This indicates that for MK-Means and OCSVM, many of the hyperparameter choices fail to adequately detect concept drift, while for the better hyperparameter choices (i.e., those on the Pareto Front), we obtain strong results. This is not the case for MMD, where hyperparameter choices yield more consistent efficiency results, regardless of whether

29

0.80

0.7578 0.8801 0.8796 0.8831 0.8843 0.9334

0.8416 0.9154 0.9148 0.9178 0.9166 0.9500

0.8248 0.9348 0.9335 0.9364 0.9380 0.9666

0.8152 0.9213 0.9199 0.9230 0.9240 0.9529

Accuracy

1.00

0.60 0.40 𝒜drift 𝒜𝑚* drift

Static 𝒜𝑦* drift

0.20

𝒜𝑥* drift Periodic

0.00 MLP

RF

SVM

XGB

Figure 10: MMD accuracies

0.6437 0.6617 0.6996 0.6985

0.6434 0.6684 0.6901 0.7017

0.6437 0.6714 0.6941 0.6990

0.80

0.6434 0.6738 0.6948 0.6912

Efficiency

1.00

0.60 0.40 ℰdrift 𝑦* ℰdrift

0.20

𝑥* ℰdrift 𝑚* ℰdrift

0.00 MLP

RF

SVM

XGB

Figure 11: MMD efficiencies we restrict to the Pareto Front. The violin graphs in Figures 18 and 19 serve to further emphasize these points. These results strongly suggest that we could reduce the hyperparameter search space for MK-Means and OCSVM without affecting the accuracy or training efficiency results.

4.5

Comparison of Drift Detection Techniques

In this section we provide a comparative analysis of the three drift detection techniques discussed above, with the emphasis on aggregate results. We pay particular 𝑚* attention to 𝒜𝑚* drift and ℰdrift , since since these values correspond to typical selections of drift-detector hyperparameters In Figure 12(a), we give accuracy results, averaged over all (𝐹𝑑 , 𝐹𝑐 , 𝐿), that is, all selections of drift family, control family, and learning model. Figure 12(b) gives the corresponding efficiency averages. From Figure 12 we see that the best 𝒜𝑚* drift is obtained with the OCSVM drift𝑚* , detector, while MK-Means performs only marginally worse. With respect to ℰdrift OCSVM and MMD are virtually identical. Recall that we define efficiency in terms of the number of times that concept

30

0.70

mk-means

ocsvm

mmd

mk-means

0.50

ocsvm

0.6804 0.6979 0.6976

0.6595 0.6853 0.6946

0.7157

0.60

0.60

0.6688

0.70

0.5917

0.80

0.6435

0.80

0.8210 0.7785

0.9310 0.9334 0.9157

0.9187 0.9215 0.9151

0.90

Efficiency

mmd

(a) Accuracy

) ℰd 𝑚 *

rif ( t 𝐿

,𝑅

) ℰ 𝑦*

dr ift (𝐿

,𝑅

,𝑅 rif ( t 𝐿

ℰd

dr

𝒜 𝑚*

rif t (𝐿

ift (𝐿

,𝑅

)

)

) ,𝑅

) ift (𝐿

dr

𝒜 𝑦*

,𝑅

,𝑅 dr ift (𝐿

𝒜 𝑥*

𝒜

dr

ift ( 𝐿

,𝑅

)

)

0.50

ℰd 𝑥 *

Accuracy

0.90

0.8914 0.9118 0.9120

0.8636 0.8726 0.9129

1.00

(b) Efficiency

Figure 12: Drift detector comparison averaged over all (𝐹𝑑 , 𝐹 𝑐, 𝐿) drift is detected, which necessitates the retraining of the learning models. Since the same learning models are trained for all three drift detectors, the time required to train any individual learning model has no bearing on this measure of efficiency. However, the time required to train each of our three drift detection techniques varies significantly, and this is relevant when selecting between these techniques. A comparison of the average time per (𝐹𝑑 , 𝐹𝑐 ) pair that is required for each of the three drift-detection techniques is given in the form of a bar graph in Figure 13. By this measure, OCSVM requires the least time, by a wide margin, with MKMeans requiring the most time, also by a wide margin. The primary reason for these large discrepancies is that the OCSVM model is only retrained when concept drift is detected, whereas MK-Means and MMD need to be updated for each block, regardless of whether drift is detected. It is also worth noting that OCSVM has the largest number of hyperparameters to test and, as discussed in Section 4.4, this number could likely be significantly reduced, which would further increase its timing advantage. According to a majority of the metrics considered, our OCSVM concept drift detector outperforms the MK-Means and MMD approaches. Thus, we conclude that OCSVM is the optimal choice from among the three concept drift detection techniques analyzed in this chapter.

5

Conclusion and Future Work

In this chapter, we considered the problem of concept drift detection when using machine learning models for malware classification. We analyzed three distinct concept drift detection approaches, two of which use machine learning techniques, namely, Minibatch K-Means (MK-Means) and One-Class Support Vector Machine (OCSVM), and one of which uses Maximum Mean Discrepancy (MMD), a statistical

31

7.71

8

Min Max Mean Standard deviation

0.39

1.85 0.62

0.15

0.07

0.46

1.06

3.17

0.07

2

1.34

4

1.07

Seconds

6

0 MK-Means

OCSVM

MMD

Figure 13: Time required to detect drift points per (𝐹𝑑 , 𝐹𝑐 ) pair analysis technique. We compared drift-aware detection strategies based on each of these three techniques, and in each case, we compared the results to a static scenario (no model retraining occurs), and to periodic retraining (models are constantly retrained, regardless of whether concept drift has occurred). We also performed an extensive investigation of the tradeoff between accuracy and training efficiency, using Pareto Front analysis. In addition, we analyzed the time required to train each of the concept drift detectors. Our OCSVM-based concept drift detection technique generally performed best, with respect to accuracy and training time, and with respect to training efficiency, OCSVM was equivalent to MMD. In terms of accuracy, efficiency, and training time, MK-Means was generally the least effective of the three drift detection techniques. Over the set of hyperparameters tested, we found that MMD was the most consistent, but for the best selections of hyperparameters, OCSVM performed somewhat better than MMD. It is interesting—and perhaps somewhat surprising—that a machine learning based approach (OCSVM) outperformed a statistical based approach (MMD). One plausible explanation is that while MMD is more sensitive to changes in the underlying statistics, OCSVM is better at detecting only those changes that are likely to actually matter, from a machine learning model perspective. This is a topic that is worthy of further investigation. An additional avenue for future research is refining the window size used for concept drift detection. While we only considered a fixed window of size 50, the smallest practical window size would be preferred, since we want to retrain our models as soon as malware has evolved. Additionally, a deeper analysis of the parameters of MK-Means and OCSVM could yield a better understanding of the underlying data distribution and thereby improve the accuracy and efficiency of these drift detectors. Another promising topic for further exploration is the use of Long-Short Term Memory (LSTM) models as drift detectors. LSTM models can account for temporal dependencies, which might enable such models to dynamically learn the optimal

32

window size as part of the training process. Finally, evaluating the performance of proposed drift detection techniques in real-time malware detection systems would provide valuable insights into their practical applicability and scalability.

References [1] Md Tanvirul Alam, Romy Fieblinger, Ashim Mahara, and Nidhi Rastogi. MORPH: Towards automated concept drift adaptation for malware detection. In Proceedings 2024 Network and Distributed System Security Symposium, NDSS, 2024. https://arxiv.org/abs/2401.12790. [2] Javier B. Alonso. K-means vs Mini Batch K-means: A comparison. http: //hdl.handle.net/2117/23414, 2013. [3] John Aycock. Computer Viruses and Malware. Springer, 2006. [4] Muhammad Azeem, Danish Khan, Saman Iftikhar, Shaikhan Bawazeer, and Mohammed Alzahrani. Analyzing and comparing the effectiveness of malware detection: A study of machine learning approaches. Heliyon, 10(1):e23574, 2024. [5] Firas Bayram, Bestoun S. Ahmed, and Andreas Kassler. From concept drift to model degradation: An overview on performance-aware drift detectors. Knowledge-Based Systems, 245:108632, 2022. [6] A. Bertia, Basil Xavier Simon, G. Jaspher W. Kathrine, and G. Matthew Palmer. A study about detecting ransomware by using different algorithms. In 2022 International Conference on Applied Artificial Intelligence and Computing, ICAAIC, pages 1293–1300, 2022. [7] Anna Bogomolnaia, Hervé Moulin, and Richard Stong. Collective choice under dichotomous preferences. Journal of Economic Theory, 122(2):165–184, 2005. [8] Leo Breiman. Random Forests. Machine Learning, 45:5–32, 2001. [9] Markus M. Breunig, Hans-Peter Kriegel, Raymond T. Ng, and Jörg Sander. LOF: identifying density-based local outliers. In Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data, SIGMOD ’00, pages 93–104, 2000. [10] Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, pages 785–794, 2016. [11] Rohgi Toshio Meneses Chikushi, Roberto Souto Maior de Barros, Marilu Gomes N. Monte da Silva, and Bruno Iran Ferreira Maciel. Using spectral entropy and Bernoulli map to handle concept drift. Expert Systems with Applications, 167:114114, 2021. [12] Christofer Washington Berruz Chungata, Martin Jureček, Katerina Potika, and Mark Stamp. Maximum mean discrepancy for concept drift detection in malware classification models. In Proceedings of the 12th IEEE International

33

Conference on Big Data Computing Service and Machine Learning Applications, IEEE BigDataServices, 2026. [13] Carlos A. Coello Coello, B. Gary Lamont, and David A. Van Veldhuizen. Evolutionary Algorithms for Solving Multi-Objective Problems. Springer, 2nd edition, 2007. [14] Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine Learning, 20:273–297, 1995. [15] Radosvet Desislavov, Fernando Martı́nez-Plumed, and José Hernández-Orallo. Trends in ai inference energy consumption: Beyond the performance-vsparameter laws of deep learning. Sustainable Computing: Informatics and Systems, 38:100857, 2023. [16] Stephan Dreiseitl, Melanie Osl, Christian Scheibböck, and Michael Binder. Outlier detection with One-Class SVMs: An application to melanoma prognosis. In AMIA Annual Symposium Proceedings, pages 172–176, 2010. [17] F-Secure Labs: Adware:Android/Airpush. https://www.f-secure.com/swdesc/adware-android-airpush.shtml, 2025. [18] F-Secure Labs: Riskware:Android/SmsReg. https://www.f-secure.com/swdesc/riskware-android-smsreg.shtml, 2025. [19] F-Secure Labs: Trojan:Android/Boxer. https://www.f-secure.com/vdescs/trojan-android-boxer.shtml, 2025. [20] F-Secure Labs: Trojan:W32/Agent. https://www.f-secure.com/v-descs/ agent.shtml, 2025. [21] Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13:723–773, 2012. [22] Alejandro Guerra-Manzanares. GitHub - aleguma/kronodroid: KronoDroid dataset. https://github.com/aleguma/kronodroid, 2021. [23] Alejandro Guerra-Manzanares, Hayretdin Bahsi, and Sven Nõmm. KronoDroid: Time-based hybrid-featured dataset for effective android malware detection and characterization. Computers & Security, 110:102399, 2021. [24] J. A. Hartigan and M. A. Wong. Algorithm AS 136: A K-means clustering algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics), 28(1):100–108, 1979. [25] Yiling He, Junchi Lei, Zhan Qin, Kui Ren, and Chun Chen. Combating concept drift with explanatory detection and adaptation for android malware classification. In Proceedings of the 2025 ACM Conference on Computer and Communications Security, CCS, pages 1–15, 2025. [26] IBM: Random Forest. https://www.ibm.com/think/topics/random-forest, 2021.

34

[27] Abiodun M. Ikotun, Absalom E. Ezugwu, Laith Abualigah, Belal Abuhaija, and Jia Heming. K-means clustering algorithms: A comprehensive review, variants analysis, and advances in the era of big data. Information Sciences, 622:178–210, 2023. [28] Samuel Kim. PE header analysis for malware detection. Master’s thesis, San Jose State University, California, 2018. https://scholarworks.sjsu.edu/ etd_projects/624/. [29] H. T. Kung, F. Luccio, and F. P. Preparata. On finding the maxima of a set of vectors. Journal of the ACM, 22(4):469–476, 1975. [30] Alon Lev. XGBoost versus Random Forest. https://www.qwak.com/post/ xgboost-versus-random-forest, 2022. [31] Adrian Shuai Li, Arun Iyengar, Ashish Kundu, and Elisa Bertino. Revisiting concept drift in windows malware detection: Adaptation to real drifted malware with minimal samples. In Proceedings 2025 Network and Distributed System Security Symposium, NDSS, 2025. https://arxiv.org/abs/2407.13918. [32] Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In 2008 Eighth IEEE International Conference on Data Mining, pages 413–422, 2008. [33] Larry M. Manevitz and Malik Yousef. One-Class SVMs for document classification. Journal of Machine Learning Research, 2:139–154, 2002. [34] Leo Mei and Mark Stamp. Energy considerations for large pretrained neural networks. https://arxiv.org/abs/2506.01311, 2025. [35] Hoda El Merabet and Abderrahmane Hajraoui. A survey of malware detection techniques based on machine learning. International Journal of Advanced Computer Science and Applications, 10(1), 2019. [36] Aniket Mishra and Mark Stamp. Cluster analysis and concept drift detection in malware. Journal of Computer Virology and Hacking Techniques, 21, 2025. [37] Antonio Nappa, M. Zubair Rafique, and Juan Caballero. The Malicia dataset: Identification and analysis of drive-by download operations. International Journal of Information Security, 14(1):15–33, 2014. [38] Optuna: Efficient optimization algorithms — Optuna 3.5.0 documentation. https://optuna.readthedocs.io/en/stable/tutorial/10_key_ features/003_efficient_optimization_algorithms.html, 2025. [39] Sunhera Paul and Mark Stamp. Word embedding techniques for malware evolution detection. In Mark Stamp, Mamoun Alazab, and Andrii Shalaginov, editors, Malware Analysis Using Artificial Intelligence and Deep Learning, pages 321–343. Springer, 2021. [40] Polars: Data types and structures. https://docs.pola.rs/user-guide/ concepts/data-types-and-structures/, 2025. [41] Stephan Rabanser, Stephan Günnemann, and Zachary C. Lipton. Failing loudly: an empirical study of methods for detecting dataset shift. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, 2019.

35

[42] Ananya Redhu, P. Choudhary, K. Srinivasan, and Tapan Kumar Das. Deep learning-powered malware detection in cyberspace: a contemporary review. Frontiers in Physics, 12, 2024. [43] Gordon J. Ross, Niall M. Adams, Dimitris K. Tasoulis, and David J. Hand. Exponentially weighted moving average charts for detecting concept drift. Pattern Recognition Letters, 33(2):191–198, 2012. [44] Peter J. Rousseeuw and Katrien Van Driessen. A fast algorithm for the minimum covariance determinant estimator. Technometrics, 41(3):212–223, 1999. [45] Bernhard Schölkopf, John C. Platt, John Shawe-Taylor, Alex J. Smola, and Robert C. Williamson. Estimating the support of a high-dimensional distribution. Neural Computation, 13(7):1443–1471, 2001. [46] scikit-learn: 2.7. novelty and outlier detection. https://scikit-learn.org/ stable/modules/outlier_detection.html. [47] Tongxin Shi, Roy A. McCann, Ying Huang, Wei Wang, and Jun Kong. Malware detection for Internet of Things using one-class classification. Sensors, 24(13):4122, 2024. [48] sklearn.neural network.mlpclassifier — scikit-learn 0.20.3 documentation. https://scikit-learn.org/stable/modules/generated/sklearn. neural_network.MLPClassifier.html, 2010. [49] Yiliao Song, Guangquan Zhang, Haiyan Lu, and Jie Lu. A fuzzy drift correlation matrix for multiple data stream regression. In 2020 IEEE International Conference on Fuzzy Systems, FUZZ-IEEE, pages 1–6, 2020. [50] Mark Stamp. Introduction to Machine Learning with Applications in Information Security. Chapman and Hall/CRC, Boca Raton, second edition, 2022. [51] Juan Terven, Diana-Margarita Cordova-Esparza, Julio-Alejandro RomeroGonzález, Alfonso Ramı́rez-Pedraza, and E. A. Chávez-Urbiola. A comprehensive survey of loss functions and metrics in deep learning. Artificial Intelligence Review, 58:195, 2025. [52] Eleftherios Tiakas, Apostolos N. Papadopoulos, and Yannis Manolopoulos. Skyline queries: An introduction. In 2015 6th International Conference on Information, Intelligence, Systems and Applications, IISA, pages 1–6, 2015. [53] Lolitha Sresta Tupadha and Mark Stamp. Machine learning for malware evolution detection. In Mark Stamp, Corrado Aaron Visaggio, Francesco Mercaldo, and Fabio Di Troia, editors, Artificial Intelligence for Cybersecurity, pages 183– 213. Springer, 2022. [54] Arnaud Van Looveren, Janis Klaise, Giovanni Vacanti, Oliver Cobb, Ashley Scillitoe, Robert Samoilescu, and Alex Athorne. Alibi detect: Algorithms for outlier, adversarial and drift detection, 2019. [55] Mayuri Wadkar, Fabio Di Troia, and Mark Stamp. Detecting malware evolution using support vector machines. Expert Systems with Applications, 143:113022, 2020.

36

[56] Shuhei Watanabe. Tree-structured parzen estimator: Understanding its algorithm components and their roles for better empirical performance. https: //arxiv.org/abs/2304.11127, 2023. [57] Kinza Yasar and Fred Tabsharani. What is a support vector machine (SVM)? https://www.techtarget.com/whatis/definition/supportvector-machine-SVM, 2023. [58] Qingfu Zhang and Hui Li. MOEA/D: A multiobjective evolutionary algorithm based on decomposition. IEEE Transactions on Evolutionary Computation, 11(6):712–731, 2007. [59] Si-si Zhang, Jian-wei Liu, and Xin Zuo. Adaptive online incremental learning for evolving data streams. Applied Soft Computing, 105:107255, 2021. [60] Yu Zhang, Qingzhong Liu, and Yuanquan Shi. MANAGE: A novel malware evolution model based on digital genes. In 2022 7th IEEE International Conference on Data Science in Cyberspace, DSC, pages 64–70, 2022.

Appendix In this Appendix, we provide additional relevant graphs. Figures 14 through 17 provide bar graphs for each learning model considered, where static and periodic accuracies are given for each combination of malware families (𝐹𝑑 , 𝐹𝑐 ). Figure 18 contains violin plots related to accuracy and efficiency over all hyperparameters tested, while Figure 19 has the analogous plots restricted to the corresponding Pareto Front.

0.80 0.60

0.5964 0.8335 0.9375 0.9790 0.7800 0.9276 0.9919 0.9923 0.8223 0.9068 0.9248 0.9915 0.7641 0.9827 0.8933 0.9877 0.8014 0.9764 0.9595 0.9972 0.5494 0.9534 0.7531 0.9806 0.9841 0.9947 0.8237 0.9875 0.5889 0.9737 0.8247 0.9886 0.9828 0.9960 0.6831 0.9215

Accuracy

1.00

0.8442 0.9629 0.9906 0.9982

Model: MLP

0.40 Static Periodic

0.20 0.00

Figure 14: Static and periodic accuracies for MLP model

37

en t, A (Ag irp us en h t (Ag , Bo ) x en e (Ag t, M r) al en (Ai t, S ap) MS rp re u g) (Ai sh, A rp us gent h (Ai ) rp , Box u e s (Ai r rp h, Ma ) us l a h, p (Bo SMS ) re xe g) r (Bo xe , Age r, n t A (Bo irp ) us xe (Bo r, M h) al xe ap r, S (Ma MS ) re l g (Ma ap, Ag ) la en p, t A (Ma irp ) us la (Ma p, B h) ox la e p (SM , SM r) Sr Sr eg e (SM Sr g, Ag ) eg e n , (SM Air t) pu Sr s e (SM g, B h) ox Sr e eg , M r) al ap )

(Ag

Accuracy 0.60

1.00

0.80

0.60

1.00

0.80

0.60

0.7661 0.9379 0.9241 0.9960 0.5938 0.7683 0.9643 0.9692 0.7742 0.9002 0.9463 0.9844 0.9258 0.8832 0.9748 0.9755 0.8750 0.9803 0.9074 0.9919 0.8923 0.9694 0.9708 0.9979 0.4275 0.9347 0.6375 0.9744 0.9559 0.9916 0.7956 0.9712 0.6710 0.9556 0.7313 0.9861 0.9813 0.9939 0.5899 0.8955

0.80

0.7469 0.8942 0.9335 0.9621 0.5835 0.8848 0.8920 0.9192 0.8668 0.9174 0.9450 0.9835 0.9402 0.9331 0.8860 0.9155 0.8683 0.9856 0.9345 0.9835 0.8384 0.9835 0.9606 0.9852 0.6894 0.9281 0.6478 0.9478 0.9547 0.9891 0.8578 0.9487 0.7851 0.9540 0.8111 0.9558 0.9881 0.9960 0.7015 0.9326

Accuracy 1.00

0.7362 0.9388 0.9737 0.9897 0.5746 0.7879 0.8246 0.9420 0.6426 0.7926 0.9984 0.9795 0.6863 0.9655 0.8256 0.9684 0.6518 0.8754 0.7077 0.9634 0.8062 0.9215 0.9430 0.9771 0.5425 0.9053 0.5172 0.9634 0.8359 0.9803 0.8566 0.9694 0.6306 0.9197 0.8982 0.9702 0.8177 0.9803 0.6328 0.8778

Accuracy

Model: RF

0.40

0.20 Static Periodic

0.00

Figure 15: Static and periodic accuracies for RF model

Model: SVM

0.40

0.20 Static Periodic

0.00

Figure 16: Static and periodic accuracies for SVM model

Model: XGB

0.40

0.20 Static Periodic

0.00

Figure 17: Static and periodic accuracies for XGB model

38

Figure 18: Violin plots for accuracy and efficiency over all hyperparameters

Figure 19: Violin plots for accuracy and efficiency on Pareto Front

39

Related documents

• Concept Drift Adaptation Using Self-Supervised and Reinforcement Learning In Android Malware Detection#224441
• SEED: Semi-supervised Continual MalwarE Detection for Tackling ConcEpt Drift on a BuDget#224422
• Detecting Concept Drift in Evolving Malware Families Using Rule-Based Classifier Representations#134497
• A Shape-Based Method for Concept Drift Detection and Signal Denoising#779261
• Maintaining IoT Device Identification under Concept Drift via Budget-Aware Traffic Labeling#459577