arXiv CSopen access

EXE-Bench: Ranking the Tradeoffs of AI-based Windows Malware Detectors for Real-World Usability

1

EXE-Bench: Ranking the Tradeoffs of AI-based Windows Malware Detectors for Real-World Usability

arXiv:2607.24177v1 [cs.CR] 27 Jul 2026

Andrea Ponte† , Daniel Gibert† , Matous Kozak, Dmitrijs Trizna, Maura Pintor Member, IEEE, Battista Biggio Fellow, IEEE, Fabio Roli Fellow, IEEE, Luca Demetrio Member, IEEE

Abstract—Due to the lack of systematic evaluations, we are not yet able to determine which AI-based Windows malware detector to deploy in production, since existing evaluations (i) differ in terms of data used for both training and testing; (ii) do not consider temporal analysis to showcase whether models withstand the passage of time; (iii) avoid security evaluations with adversarial attacks that could highlight their brittleness against content-injection attacks; and (iv) neglect the computational requirements for deployment, risking slow inference on endpoints. For these reasons, we develop EXE-Bench, a comprehensive benchmark of AI-based Windows malware detectors. EXE-Bench assesses performance, temporal and adversarial robustness, and computational overhead, aggregating them into a single score for direct and fair model comparison. Through EXE-Bench, we highlight how evaluations conducted only after deployment are suboptimal and unable to provide a complete picture of their performance. In particular, through our analysis, we remark how much domain knowledge instilled through feature engineering is still extremely useful in this domain, resisting both time and adversarial attacks, in stark contrast with most of the deep networks that only excel right after deployment. Index Terms—Malware Detection, Benchmark, Machine Learning, Deep Learning, Robustness, Drift.

I. I NTRODUCTION Deciding which machine learning model to develop and deploy as a Windows malware detector is not an easy task, exacerbated by the fact that, during the last decade, we witnessed a surge in research proposals all claiming to achieve the best results. In particular, in the context of static analysis, i.e., inferring maliciousness only by looking at the metadata and representation of programs without executing them, the predominant approaches can be divided into three distinct families: (i) feature-based [1]–[3], that leverage domain-knowledge to extract an abstract, compact representation of programs used to train models; (ii) end-to-end [4]–[9], that train deep neural network directly on bytes, without pre-processing them; and (iii) certifiable [10]–[15], which leverage current findings in the image domain [16], [17] to produce models robust against minimal manipulations of the input, thus providing theoretical bounds on their security. Andrea Ponte, Luca Demetrio, Dmitrijs Trizna, and Fabio Roli are with the Department of Informatics, Bioengineering, Robotics and Systems Engineering, University of Genova, Italy (e-mail: [email protected]). Daniel Gibert is with the Artificial Intelligence Research Institute (IIIACSIC), Spain (e-mail:[email protected]). Matous Kozak is with the Department of Information Security at Faculty of Information Technology, CTU in Prague, Czech Republic. Dmitrijs Trizna is also with Department of Computer, Control and Management Engineering, Sapienza University, and AISLE. Maura Pintor, Battista Biggio and also Fabio Roli are with the Department of Electrical and Electronic Engineering, University of Cagliari, Italy. † These authors contributed equally to this work.

However, while all these models exhibit excellent performance in terms of accuracy and extremely-low false alarms, their evaluations cannot be directly compared due to completely different experimental settings that each of them applies. In particular, all these models are trained on possibly different data sources, using different splits of the same datasets, or entirely different samples used for training and testing models. Hence, while the results of those evaluations are consistent and valid within their scope, the comparison between techniques is not practical, leading to unfair settings. Also, the main goal of these evaluations is solely confined to assessing accuracy on a single test set, neglecting that Windows malware detectors are deployed in an alwaysevolving environment subject to concept drift [18], i.e. the continuous change in the data distribution which violates the i.i.d. assumption considered when training models. Measuring accuracy alone also hides fallacies of Windows malware detectors against adversarial EXEmples [19]–[22], i.e. Windows malware minimally-perturbed to exploit the blind spots of machine learning techniques, thus evading detection. Since most of the models were proposed before the advent of these attacks, they rightly did not consider these issues. However, follow-up investigations incorporated adversarial EXEmples in their evaluations, but fell short in ways similar to those described earlier, even when proposing defenses [10]– [15], [23]. Attacks are compared with different settings, thus providing partial results that cannot be directly compared with parallel studies. Also, these evaluations neglect the fact that the security of machine learning models should be characterized against increasingly strong attacks, rather than testing only one configuration. In particular, models that break with a few bytes alterations [22] highlight a stark sensitivity to input changes, making those techniques unreliable. Lastly, all evaluations neglect the fact that these Windows malware detectors could be deployed in practice on endpoints, which are not likely to be equipped with the necessary hardware to support hundreds of predictions at once, or that they cannot redirect all their computational capabilities only to compute inference on a few downloaded programs. Hence, while evaluations report the training time and resources needed to develop models, there is no clear way to compare those on the most frequent operation they are tasked with, which is prediction. In practice, some models might be incredibly proficient in detecting malware, but computations take seconds or minutes, making them not usable in practice. For these reasons, we overcome the assessment of accuracy alone, and we develop EXE-Bench, a systematic benchmark to score models w.r.t. four main performance metrics (Sect. III),

2

namely: (i) performance right after deployment, to assess their functionality in the present; (ii) performance over time, to assess whether their predictive capabilities remain stable even when facing the passage of time; (iii) performance against adversarial EXEmples, to assess whether they can resist attacks with increasing strength; and (iv) computational complexity, by assessing how fast they are when deployed. All these metrics are combined into a single score providing a global ranking of each model, showcasing the ones achieving the best trade-off between these aspects for real-world usability. All evaluations are conducted on the same identical settings (Sect. IV), by training models on exactly the same data, the same training-test splits, with inference time computed on the same hardware. To mimic the passage of time, we leverage a dataset collected 5 years later than the one used at training time, and models are evaluated on sequential 4-month windows, thus enabling a characterization of the degradation of performance in the presence of concept drift. In this respect, while the other metrics are known from scattered previous work, the temporal metric we propose is novel, also pointing out possible fallacies of previous work [18] since they did not consider the number of samples contained in each quarter, wrongly influencing the temporal analysis. Lastly, attacks are computed by selecting state-of-the-art techniques [21], [22] with increasingly stronger settings to characterize their robustness from negligible manipulations to heavier attacks injecting entire megabytes (MB) into malware programs, as done for other domains as well [24], [25]. With EXE-Bench we compare 30 different models, and we rank all of them according to our methodology (Sect. V). Our findings suggest that this domain still needs the use of domain-knowledge, instilled in the form of features extracted from Windows programs, as we highlight the superiority of those in almost all metrics. While the rise of deep learning techniques rivals feature engineering on accuracy, most of them fall short in all the other metrics, being unable either to withstand the passage of time or remain robust against minimal perturbations. Also, we are able to discourage certain technologies (like image classifiers retrofitted to perform malware detection) being completely suboptimal on most metrics, rising awareness of which are the reliable models to deploy. As far as we know, this is the first fair and comprehensive benchmark in the domain of Windows malware detection with machine learning, and it can potentially stand out as a defacto standard similar to previous approaches in the image domain [25], [26], paving the road towards safer, more reliable and more capable AI-based antivirus programs. We release all the code for computing the benchmark on GitHub,1 along with an interactive dashboard hosting the collected results.2 II. BACKGROUND AND R ELATED W ORK Windows PE File Format. All AI-based antivirus programs need to digest Windows programs, stored on disk according to the Windows Portable Executable (PE) file format3 as depicted

MZ

PE

DOS Header and Stub

PE Header

Optional Header

Section Table

A

B

C

D

.text

.data

E

…

Overlay

F

Fig. 1: PE File Format.

in Fig. 1. Starting with the DOS Header and Stub (A) kept for retro-compatibility, each program holds its metadata inside the PE and Optional Headers (B,C), needed by the OS to correctly initialize the loading. These are followed by the Section Table (D), which instructs the OS where the main content of the program is physically located inside the file, i.e., the Sections (E). Each section contains a different aspect of the program, like the compiled code, the resources, initialized values, etc. Lastly, a program can have appended bytes at the end, i.e., the overlay (F), which is not considered by the loader. AI-based Windows Malware Detectors. Feature-based models. This type of model relies on the manual extraction of hand-crafted features from PE files, used to train a machine learning classifier. Features can be extracted from the raw binary program [1], [2] or from its disassembled counterpart [3]. However, disassembling a PE file is not always straightforward as it requires the use of tools such as Radare2,4 IDA Pro,5 or Ghidra,6 , which add an additional pre-processing layer to the pipeline, increasing the processing time. Therefore, in this work we focus exclusively on the EMBER features [1], a well-known set of features extracted directly from the raw binary, including general file information, header metadata, section characteristics, imported and exported functions, and statistical features derived from bytes and strings. End-to-end models. These architectures operate directly on raw byte sequences from executable files, allowing the models to learn hierarchical representations of low-level byte patterns without requiring feature engineering. These are based on deep neural networks with different architectures, by either feeding bytes to an embedding layer [4]–[7], i.e., a space learned at training time that imposes a distance metric over discrete values (bytes in this case), or processing each program as an image later fed to convolutions [8], [9]. Certifiable models. This family of techniques [16], [17] has been proposed in the image domain to provide theoretical guarantees on robustness against adversarial attacks carried on with adversarial examples — minimally-perturbed test-time samples inducing classification errors. Hence, these techniques have been also adapted to the Windows malware detection domain in the form of: (1) randomized smoothing [10], [11] and (2) de-randomized smoothing [12]–[15]. Both are designed to provide theoretical guarantees on the robustness to small perturbations, differing in how the input is handled and how the certification is computed. Randomized smoothing approaches transform a base non-robust classifier into a smoothed classifier that is probabilistically robust, by

1 https://github.com/zangobot/exebenchmark

4 https://www.radare.org/n/

2 https://exebench.github.io/

5 https://hex-rays.com/ida-pro

3 https://learn.microsoft.com/en-us/windows/win32/debug/pe-format

6 https://github.com/NationalSecurityAgency/ghidra

3

P

Performance

T

Robustness

R

Inference Time

I

𝒎𝟓

2

1

𝒎𝟑𝟎

3

…

… 𝒎𝟏

Temporal Analysis

𝒎𝟏

𝒎𝒏

1 Training

𝒎𝟑𝟎

2 Testing

𝟎. 𝟖𝟔

𝟎. 𝟕𝟑 𝟎. 𝟗𝟐

3 Ranking

Fig. 2: Design of EXE-Bench. Models are trained on the same data (1), evaluated through the lenses of test-time performance, stability over time, robustness against attacks, and inference time requirements (2). These results are aggregated into a single metric, used to rank models (3).

adding random noise to the input binary file multiple times and evaluating the classifier on each noisy instance. The final prediction is determined through majority voting, and it is possible to guarantee that small perturbations to the input will not alter the model’s predictions. De-randomized smoothing [12]–[15] approaches operate by partitioning an executable into fixed-sized chunks, which are then independently evaluated by the classifier. Inference is then calculated through majority voting over the predictions of all chunks. Unlike randomized smoothing, these approaches can derive deterministic robustness certificates under different types of attacks if the number of predictions for the majority class exceeds that of the other class by a large enough margin [15]. Adversarial EXEmples. Recent work highlighted the brittleness of AI-based Windows malware detectors when exposed to adversarial EXEmples [19]–[21]— minimally-perturbed evasive malware samples. Instead of being crafted by adding noise to each feature (i.e., computing attacks in the so-called feature space), these attacks leverage functionality-preserving manipulations in the problem space [27], i.e., directly changing the programs without corrupting them. Related Work. Current evaluations of AI-based Windows malware detectors fall into two primary categories: (i) algorithm performance benchmarks, which compare various machine learning approaches including traditional methods (e.g., SVM, Random Forest) and modern neural networks (e.g., MalConv [4]) across different public or private datasets [28], [29]; and (ii) adversarial robustness evaluations, which assess model resilience against different evasion attacks [21], [30]– [32]. These studies evaluate both white-box (i.e., worst-case scenario, attackers own the models) and black-box (i.e., attackers can only access scores) attack scenarios, analyzing evasion rates, attack transferability across different target models, and perturbation characteristics. Research in this area ranges from small-scale controlled experiments to comprehensive multiyear evaluations, often incorporating structured comparison frameworks with specific attack constraints [21], [30]–[32]. Limitations of Existing Evaluations. Current evaluations suffer from four critical issues that prevent reliable assessments and deployment decisions for AI-based Windows malware detectors. First, dataset heterogeneity creates incomparable

results across studies. Previous evaluations use different data sources and scales, from small-scale assessments with 104 samples [21] to larger datasets including thousands of samples [29], [30], thus making any comparison between findings impossible. Thus, evaluations are conducted with different experimental setups leveraging different train-test splits, inconsistent choices of the detection thresholds, and mixed use of pretrained versus freshly trained models. Studies often evaluate different model architectures with hyperparameters picked with different policies, making performance comparisons meaningless and hindering reproducibility. Second, temporal drift analysis is largely absent from existing evaluations. Most works ignore the temporal nature of data rather than evaluating model degradation over time, despite malware and goodware evolution being a critical factor in production deployment. Studies using multi-year datasets [30] typically aggregate results over periods without analyzing temporal stability patterns. Third, robustness against attacks is only partially evaluated, without constructing security evaluations [24] that assess the robustness against several adversarial attacks, from those that alter a few bytes to ones that inject portions of other programs into malicious samples. As a result, models are only evaluated against specific strategies, rather than characterized as a whole. Finally, current benchmarks focus on isolated metrics, neglecting the fact that AI-based antivirus should achieve production-ready performance on many aspects such as computational requirements, deployment feasibility, and not only accuracy alone. III. EXE-B ENCH : S YSTEMATIC E VALUATION OF AI- BASED W INDOWS M ALWARE D ETECTORS We now detail our methodology to rank machine learningbased Windows malware detectors, thus compiling the leaderboard of EXE-Bench as shown in Fig. 2. All models are trained on the same data (step 1), and we proceed by computing four metrics that characterize the development and deployment of detectors (step 2): (i) the Performance Metric, which quantifies the performance at deployment time; (ii) the Temporal Metric, which quantifies the performance over time, after deployment; (iii) the Robustness Metric, which quantifies the resistance against adversarial EXEmples; and (iv) the Inference Metric,

4

which quantifies the computational requirements when computing inference. We then combine these into a single metric used to rank models (step 3), providing a general and modelagnostic leaderboard of all the considered models. Performance Metric We quantify the performance at deployment time through the application of the well-known F1 Score on the test set, depicted in this paper as P (f ) = F 1(f ). In general, models are evaluated in terms of accuracy by fixing a detection threshold at 1% False Positive Rate (FPR), thus being fair in terms of false alarms. Also, since models are trained and tested on the same data, this metric avoids biases caused by dataset heterogeneity (as described in Sect. II. However, certifiable models cannot be tuned accordingly, since changing their output might interfere with their certification schema, and thus they are deployed to leverage majority voting only. Since it is possible that also future extensions of the benchmark will pose a similar issue, we then rely on (i) calibrating models with a probability output at 1%FPR, and (ii) keeping the certifiable models as is, to remain compliant with their development. Hence, the F1 score is able to weight this slight advantage given to the certifiable models in our benchmark by also considering their non-fixed FPR. Temporal Metric. We propose a metric to quantify the performance that quantifies the performance in time, when the model is subject to concept drift [18] (i.e., the evolution of the distribution of programs induced by alterations of malware families, frameworks, class distributions on the collected data, and more). To do so, we first need to define how the flow of time can be formalized in terms of future samples belonging to disjointed splits, i.e., sets containing samples attributed to a specific month of a specific year. Let D = {S1 , S2 , . . . , Sk } be a dataset consisting of multiple splits, where each split Si might contain a different number of samples. Each split is i defined as Si = {(xij , yij )}N j=0 , where xij is an input PE file, yij its associated label (0 if benign, 1 if malicious), and Ni the number of samples for split i. Thus, the temporal metric can be computed as: X |Si | F 1i (f ) (1) T (f ) = |D| i where Si is a set of samples from a selected period of time (e.g., samples between January and March 2021), with cardinality |Si | used to weight the F1 score over the sum of all the samples in the dataset |D|.We note here that the threshold used to compute the F1 score is the same that is used for the Performance metric. The metric has values between 0 and 1, with 1 meaning that the F1 score of the model was perfect in all temporal splits, while 0 means that the detector is failing on all splits. We chose this metric instead of the AUC (Area Under the Curve), which was previously used for temporal evaluations [18] since the latter could provide incorrect results, as it does not take into account the number of samples used to compute each point of the curve (an intuitive example to support this choice is presented in Sect. A) Robustness Metric. We devise a metric to express the robustness against adversarial EXEmples, inspired by previous work [25]. Similar to what is done for the image domain, we build a security evaluation curve, i.e., how the accuracy of the

evaluated model changes when increasing the strength of the attacker, measured through a manipulation budget [24], [33]. In this domain, the budget of the attack can be expressed as the number of bytes that are either replaced or injected during the attack [19]–[21]. Hence, we define the Detection Rate at ϵ (DRϵ ) on a set of adversarial EXEmples DEXE as: X 1 (2) 1f (x′i )=1∧d(xi ,x′i )≤ϵ DRϵ (f ) = |DEXE | i which counts how many adversarial EXEmples x′ ∈ DEXE computed with a budget ϵ (w.r.t their original point xi ) are still correctly classified as malicious. The budget is bound to the Levenshtein distance (since we need to take into account both insertions and replacements) computed between xi and x′i . This formulation does not depend on a specific manipulation or a specific optimization algorithm, thus it can be computed by leveraging different strategies all at once. Hence, this metric moves away from the robustness analysis performed in isolated settings, differently from previous work (as described in Sect. II). Thus, we can quantify the degradation of robustness as the AUC, i.e., the integral of this curve: Z R(f ) = DRϵ (f )dϵ (3) where, in theory, we sample all possible perturbation budgets. Not feasible in practice, we approximate the integral by executing selected attacks, where ϵ depends on the used strategy (i.e., can be either fixed a priori, or estimate after the execution of the attack as later described in Sect. IV). Lastly, this metric can be normalized between 0 and 1 by dividing it by the area defined by the maximum ϵ estimated empirically (i.e., the maximum Levenshtein distance computed on all pairs of original and adversarial sample among all the selected attacks against all the models considered in the benchmark). Thus, when this metric has value 1, the evaluated model is perceived as robust, while 0 means the contrary. Inference Metric. The most frequent operation computed by a Windows malware detector is inference, hence it is necessary to rely not only on accurate but also fast models. Also, it is very likely that these detectors could be deployed on generalpurpose laptops with no GPU support. Hence, we track the average inference time ti (f ) of the model f on a set of training samples, and we quantify the computational requirements as: I(f ) = e−ti (f )

(4)

which penalizes models depending on how slow they are when computing predictions. Thanks to the usage of the exponential with negative power, this metric has a theoretical maximum of 1 (with an impossible-to-achieve 0 seconds of inference time) and minimum of 0 (reached with incredibly slow models).7 EXE-Bench leaderboards. Given these metrics, we can produce a summary score for each model, by combining the collected information in one single comprehensive metric: S(f ) =

P (f ) + T (f ) + R(f ) + I(f ) 4

(5)

also tried a linear metric such as 1+t1 (f ) , but it provided no i meaningful changes in our results. We keep all the computations of this version of the metric on the public repository for transparency. 7 We

5

on PE timestamps is sound and aligned with the proposed best practices [18]. We report the quantities of malware and goodware for each bin in Fig. 3.

IV. E XPERIMENTAL S ETUP

8 https://github.com/zangobot/maltorch

1366

2346 2022-Q1

451

2512 3029 2021-Q3

3160

1256 2238

1577

968

1038

621

363

1989

2021-Q2

2021-Q1

2020-Q3

2020-Q2

2020-Q1

192 2019-Q3

2019-Q2

102

251

366

103

511

1644 851

2074

11391

16518

69209

Goodware Malware

2019-Q1

We leverage the EMBER 2017 dataset [1] for training all models. Originally consisting of 400,000 benign, 400,000 malicious programs and 300,000 unlabeled programs, dated 2017 or earlier, we were able to retrieve 349,994/400,000 (87%) benign and 399,992/400,000 (99%) malicious programs using VirusTotal, by excluding the 300,000 unlabeled samples from our study. The resulting dataset, consisting of 749,986 executable files, has been randomly divided into training (80%), validation (10%) and test (10%) sets. Temporal Analysis. Since EMBER collects samples up to 2017, we use the Speakeasy dataset [34] to assess the capability of models over time. The training set was collected in January 2022, while the test set was collected in April 2022. Since this dataset is only used to compute inference in this work, we merge the training and test sets, resulting in a total of 125,921 samples divided into 7 malware families. To divide it into disjoint temporal bins, we use the timestamp contained in PE files, composing 10 bins from January 2019 (1 year after the most recent training data) to April 2022 (the latest period of gathering of Speakeasy), grouped in four-month periods. Also, since timestamps could be easily manipulated, we discard from these datasets all values that are potentially invalid, namely past outliers (containing distant past data and manipulated 0-epoch timestamps), and future outliers, containing samples unrealistically dated in the future data. Similarly to the Android domain, where samples can be downloaded from AndroZoo or marketplaces reporting compilation timestamps [18], in the Windows domain the ground truth on the first appearance can only be established through services like VirusTotal, since datasets like EMBER are only shared in this way. However, such information is not available for the Speakeasy dataset: at the time of writing, only 33% of the programs used for the temporal analysis are also present on VirusTotal. Moreover, only 7% of those have a non-null “first seen” date (which should be the information to use as indicated by previous work [18]), forcing us to inspect the “first submission date” as an alternative (present in all reports). However, 45% of the considered samples have been submitted to VirusTotal after the collection date mentioned by the author of the dataset (January / April 2022 for training and test, respectively), thus making the acquired information unreliable (as shown in Tab. IV). Hence, our choice of relying

104

Past Out.

A. Datasets and Temporal Analysis Setup

105

Number of Samples

We now describe the datasets used during training and testing (Sect. IV-A), the considered machine learning models (Sect. IV-B), which adversarial attacks we use (Sect. IV-C), and how we quantify the computational requirements (Sect. IV-D). All the experiments have been conducted using the maltorch library,8 which provided APIs for testing and computing adversarial attacks.

Future Out.

This score is used to produce the leaderboard of EXE-Bench, by ordering models w.r.t. this metric.

Fig. 3: Temporal bins of Speakeasy Dataset in quarters, each with the quantities of malware and goodware for each. B. Models Selected for EXE-Bench Our benchmark comprises a total of 30 models, divided into feature-based, end-to-end, and certifiable models. Feature-based Models. We consider EMBER GBDT, developed by Anderson et al. [1]. This model is implemented as a Gradient Boosting Decision Tree (GBDT) [35] provided by the LightGBM library [36] and trained on the EMBER features. End-to-end Models. We list and briefly describe here the endto-end models we consider in our work. MalConv. Developed by Raff et al. [4], this model leverages a shallow convolutional neural network that takes as input sequences of 2,000,000 bytes (∼ 2 MB) and consists of an 8-dimensional embedding layer, a gated convolutional layer, followed by a global max-pooling and a feed-forward layer, for a total of 1,067,529 parameters. AvastConv. Developed by Krčál et al. [5], this model takes as input sequences of 512,000 bytes (∼ 500 KB) and it is implemented through a convolutional neural network that consists of an 8-dimensional embedding layer, two convolutional layers, followed by a max-pooling and two convolutional layers, a global average pooling, and four feed-forward layers, with an overall parameter count of 904,697. BBDNN. Developed by Coull et al. [6], this model takes sequences of 102,400 bytes (∼ 100 KB) as input, and it is implemented as a convolutional neural network consisting of a 10-dimensional embedding layer, and five blocks of convolutional and a max-pooling layers. Their output is passed to global max-pooling and global average-pooling layers concatenated along the feature dimension, culminating in a last feed-forward layer, for a total of 895,275 parameters. NGramConv. Developed by Gibert et al. [7], this model takes as input sequences of 512,000 (∼ 500 KB), approximating the extraction of n-grams through convolutions. This model leverages an embedding layer, a single convolutional layer with a small kernel size followed by a global max-pooling layer and a feed-forward layer, resulting in 17,957 parameters. ResNet18. Originally proposed to deal with images [37], we consider a ResNet18 (11,177,025 parameters) tailored to detect

6

malware [8], [9], which takes in input programs as 256x256 grayscale images. We consider this model as end-to-end since no domain knowledge is used to extract information. Certifiable Models. We list here the certifiable models we consider in our work, divided into randomized and derandomized smoothing as previously detailed in Sect. II. Randomized Smoothing. This family of models considers randomly-perturbed variations of input samples before computing predictions through majority voting. Hence, they can be grouped into two categories based on the randomization strategy: (i) Byte Deletion [10], which generates noisy instances by randomly deleting bytes; and (ii) Byte Ablation [10], [11], which generates noisy instances by randomly ablating bytes. We thus produce one variant of the end-to-end models for each ablation technique, identified by the name of the model followed by RsDel for byte deletion or RS for byte ablation (e.g., MalConvRsDel and MalConvRS). De-Randomized Smoothing (DRS). This family of models computes predictions by dividing samples into chunks, aggregating the decisions through majority voting. They can be grouped into four categories: (i) Fixed-size DRS (FDRS) [12], [15], which splits the input into chunks of a predetermined size (e.g., 32768); (ii) K-Partitions DRS (KDRS) [13], which partitions the input into K partitions (e.g., K=12); (iii) Random DRS (RDRS) [14], which divides a file into chunks based on a configurable parameter (e.g., p=10%) that determines the size of each chunk relative to the total file size, and it extracts multiple chunks from random locations; and (iv) Sequential DRS (SDRS) [14], determines the size of the chunks in the same way as RDRS but it extracts the chunks sequentially from the beginning of the file toward the end in a deterministic manner, without random sampling. We thus produce one variant of the end-to-end models for each chunking technique, identified by the name of the model followed by the abbreviation of the respective chunking strategy (e.g., MalConvFDRS, MalConvKDRS, MalConvRDRS and MalConvSDRS). General Training Settings. The GBDT model has been implemented with LightGBM. All neural networks (coded in PyTorch [38]) are trained with a batch size of 64, Adam optimizer with 10−3 as learning rate, the Binary Cross Entropy as loss, and an early stopping condition on the loss computed on a validation set, halting training whether such metric has not improved in the last 5 epochs (extended to 10 epochs for the certifiable models needed to take into account the effect of randomization and chunking, thus providing more stable results in terms of optimization). We take into account the slight class imbalance by weighting them w.r.t. their proportion in our dataset. All models have been trained and tested on a machine equipped with an Intel i9 14900KF (24 cores) CPU, 64 GB of RAM, and an Nvidia 4090 RTX GPU with 24GB of memory. C. Setup of Adversarial Attacks Data. To avoid possible corruptions due to obfuscated or packed samples, we selected 5000 unpacked programs from the EMBER test set for the robustness evaluation. In this way, we are favoring the models in the benchmark to be tested

against samples extracted from the time period in which they are more confident. These samples have been selected by analyzing them with Detect it Easy,9 a well-known tool used for detecting packed samples, and keeping them whether the tool responded negatively. Adversarial Attacks. To provide insights on the robustness against adversarial attacks, we leverage three different techniques proposed in the literature [21], [22]. To expand the applicability of our benchmark, we solely rely on black-box evaluations, i.e., strategies that only require the answers from the target to optimize attacks iteratively. Thus, each of the attacks will leverage a genetic algorithm as an optimization algorithm, aligned with previous work [22]. The latter works by generating a population of variants of the sample to optimize, and scores them according to the response of the target. The process is repeated until the budget (i.e., the number of queries) is consumed. All attacks hence use, at maximum, 500 queries, and produce 10 variants at each iteration of the attack. Regarding the manipulation, these attacks leverage the following functionality-preserving manipulations that either re-write or inject new content into Windows programs. FullDOS. This technique replaces the entire DOS header (A in Fig. 1) with adversarial content [21], by keeping only the magic number MZ and the offset to the PE header (B in Fig. 1). Thus, the size of the manipulation is exactly the amount of bytes that are replaced by the attack. Content-shift. This technique injects content between the end of headers and the first section of the program (E in Fig. 1) [21]. To avoid corruptions, this content must preserve the file alignment, i.e., a field in the Optional Header (C in Fig. 1) specifying to the loader that the relevant content will be located at multiples of its value. Hence, since different programs can have different alignment, we extract the maximum file alignment inside the dataset we sliced from our test samples, amounting to 4096 bytes. GAMMA. This technique harvests byte strings from benign programs and injects them into non-executable sections (D and E in Fig. 1) of input samples [22], thus relying on a pool of benign applications to use during its initialization step. We setup two versions of this attack, by considering samples collected (i) from a fresh installation of Windows 11, to inject 5, 10, 20, 30, 50 sections of initialized data (.rdata section) from programs contained in sysWOW64; and (ii) from the Speakeasy dataset, to inject 5, 10 sections of initialized data (.rdata section). Since not all samples from Windows 11 contain the .rdata section, some attacks end up injecting empty sections with their section entry intact. While this could be seen as a suboptimal setting, we include them into the analysis as they provide a valid perturbation that can be used to compute the metric (which, we remind, can contain all possible attacks regardless of their nature). Lastly, GAMMA also uses a regularization parameter to provide a penalty term on the size of the perturbation during the optimization. We set this parameter to 0 to inject as many bytes as possible. Transfer Evaluations. We also characterize robustness through transfer attacks, i.e., adversarial EXEmples optimized 9 https://github.com/horsicq/Detect-It-Easy

7

Model

S

P

T

R

I

FR

EmberGBDT BBDnn BBDnnFDRS BBDnnKDRS AvastStyleConvFDRS AvastStyleConvKDRS NGramConv BBDnnRsDel MalConv NGramConvFDRS MalConvKDRS BBDnnRDRS BBDnnSDRS MalConvSDRS MalConvRsDel MalConvRDRS AvastStyleConvRDRS AvastStyleConvSDRS ResNet18 NGramConvKDRS MalConvFDRS NGramConvRsDel NGramConvSDRS AvastStyleConvRsDel BBDnnRS NGramConvRDRS AvastStyleConv AvastStyleConvRS MalConvRS NGramConvRS

0.86 0.85 0.80 0.75 0.67 0.67 0.66 0.56 0.67 0.67 0.65 0.68 0.69 0.61 0.46 0.61 0.61 0.61 0.61 0.64 0.62 0.45 0.56 0.42 0.47 0.56 0.45 0.32 0.39 0.19

0.99 0.97 0.96 0.94 0.95 0.93 0.98 0.98 0.99 0.93 0.94 0.91 0.91 0.94 0.96 0.94 0.94 0.94 0.97 0.93 0.94 0.93 0.93 0.97 0.67 0.93 0.78 0.63 0.92 0.27

0.78 0.67 0.66 0.56 0.54 0.57 0.70 0.58 0.52 0.63 0.56 0.57 0.57 0.55 0.53 0.56 0.50 0.49 0.52 0.54 0.45 0.63 0.54 0.53 0.52 0.54 0.03 0.40 0.50 0.43

0.80 0.79 0.58 0.52 0.19 0.19 0.06 0.61 0.18 0.15 0.11 0.45 0.50 0.12 0.14 0.11 0.16 0.18 0.01 0.13 0.11 0.11 0.11 0.04 0.62 0.11 0.01 0.12 0.05 0.04

0.87 0.98 0.99 0.99 0.99 0.99 0.92 0.05 0.98 0.96 0.99 0.80 0.79 0.84 0.19 0.84 0.82 0.83 0.95 0.96 0.99 0.14 0.66 0.15 0.09 0.65 0.98 0.12 0.09 0.01

4.25 4.75 6.00 8.25 9.25 10.00 10.75 11.00 11.00 12.75 13.00 15.25 15.25 15.50 16.25 16.25 16.50 16.75 17.00 17.00 17.25 17.75 19.00 19.25 20.25 21.25 23.50 25.25 25.75 29.00

TABLE I: EXE-Bench leaderboard, reporting values of all metrics. We mark in bold the best values and ties.

against one model, and later tested on the real target [39]. All the adversarial EXEmples that we compute, whether or not they evade the model used to optimize them, are saved and tested against all the other detectors. This is especially useful since we were unable to compute attacks against certifiable models, due to their demanding computational requirements. While this might favour certifiable models, we will later show that they do not occupy relevant places in the leaderboard. D. Computational Requirements Since inference time (pre-processing and forward operations) depends on the size of the input, we first sample the average (µ) and standard deviation (σ) of the size of programs in the Speakeasy test set. Then, we randomly sampled 500 malicious and 500 legitimate programs whose sizes fall within the interval µ ± kσ, fixing k = 0.5 to focus on inputs that are close to the center of their distribution, and we measured the average inference time on all models, including their preprocessing phase if any. This measurement has been done on CPU, since models might be deployed on endpoints unlikely equipped with powerful GPUs to support frequent inference. V. E XPERIMENTAL R ESULTS A. EXE-Bench Leaderboard We report in Tab. I the results of our benchmark, displaying the values of our metrics for each model (also visually represented in Fig. 4). Through the lenses of the aggregated score (S), we order models from the top-performing to the least

effective, allowing us to draw conclusions from the resulting leaderboard. This ranking is supported by a statistically significant level of agreement among the four evaluation metrics. Specifically, Kendall’s coefficient of concordance yields (W = 0.48, p = 0.0021), indicating moderate agreement between the metric-specific rankings. This result suggests that, while the four metrics capture complementary aspects of model behavior, they still produce sufficiently consistent orderings to justify the use of the aggregated score (S) as a meaningful summary of overall performance. For completeness, we also report the Friedman rank (FR) of each model, computed as the sum of its ranks across the individual metric leaderboards (P, T, R, and I), with lower values indicating better overall performance. Thus, we can proclaim the GBDT model trained on EMBER features [1] as the winner of our benchmark, followed by BBDnn [6]. This implies that the extraction of information through domain-knowledge has an impact on all the relevant aspects of Windows malware detection, ranging from performance when deployed and in the future, to increased robustness to adversarial attacks. This also confirms that, in this domain and contrary to image classification or detection and with this dataset size (∼ 600k training samples), learning an abstract representation purely based on raw bytes cannot guarantee the preservation of all those aspects at once. Interestingly, all certifiable models suffer from the aggregation of many metrics at once, showing the inevitable trade-off between accuracy on clean samples and robustness, exacerbated by the presence of drift in data distributions. Among those, the de-randomized models that take into account chunk views on input programs (FDRS and KDRS) are the ones that balance the mentioned trade-off, while most of the other ones fail to reach the top of the leaderboard. On the contrary, models trained with randomized smoothing techniques (RS and RsDel) are ranked the worst in our analysis. While not peaking in accuracy-related metrics, these methods require each program to be sliced hundreds of times, expanding the computational cost to unbearable amounts, losing positions. Both MalConv and BBDnn do not gain much from the introduction of certifiable techniques, thus performing better overall than their de-randomized and randomized smoothing versions. On the other hand, both NGramConv and AvastStyleConv benefit from being trained with those techniques, as the certifiable variants appear higher in the leaderboard than the original version. Thus, our benchmark also points out that applying certifiable approaches might not be a winning strategy by default, since there are many aspects that are impacted rather than the improvement of robustness alone. Sensitivity Analysis of Leaderboard. We now assess the stability of the leaderboard by performing a Leave-One-Out (LOO) sensitivity analysis, by recomputing the aggregated ranking after removing one metric at the time, measuring its correlation with the original ranking. We report in Tab. V the Pearson’s (r) and Spearman’s (ρ) coefficients with their corresponding p-values. High LOO correlations indicate that the ranking obtained without a given metric remains largely consistent with the original leaderboard, hence the removed dimension captures information already reflected by the remaining metrics. On the contrary, low correlations indicate that the

8

EmberGBDT Score = 0.86

BBDnn Score = 0.85

BBDnnFDRS Score = 0.80

BBDnnKDRS Score = 0.75

BBDnnSDRS Score = 0.69

BBDnnRDRS Score = 0.68

NGramConv Score = 0.66

MalConvKDRS Score = 0.65

NGramConvKDRS Score = 0.64

MalConvFDRS Score = 0.62

ResNet18 Score = 0.61

MalConvSDRS Score = 0.61

MalConvRDRS Score = 0.61

NGramConvRDRS Score = 0.56

BBDnnRsDel Score = 0.56

BBDnnRS Score = 0.47

MalConvRsDel Score = 0.46

NGramConvRsDel Score = 0.45

AvastStyleConv Score = 0.45

AvastStyleConvRsDel Score = 0.42

Inference

Temporal

Robustness

Performance

AvastStyleConvKDRS AvastStyleConvFDRS Score = 0.67 Score = 0.67

NGramConvFDRS Score = 0.67

AvastStyleConvSDRS AvastStyleConvRDRS Score = 0.61 Score = 0.61

MalConvRS Score = 0.39

AvastStyleConvRS Score = 0.32

MalConv Score = 0.67

NGramConvSDRS Score = 0.56

NGramConvRS Score = 0.19

Fig. 4: Radar plots summarizing the results of EXE-Bench, ordering models by their score (S). Rank 1 2 3 4 5 6

Model

P

Model

T

Model

R

Model

I

EmberGBDT MalConv NGramConv BBDnnRsDel ResNet18 AvastStyleConvRsDel

0.99 0.99 0.98 0.98 0.97 0.97

EmberGBDT NGramConv BBDnn BBDnnFDRS NGramConvFDRS NGramConvRsDel

0.78 0.70 0.67 0.66 0.63 0.63

EmberGBDT BBDnn BBDnnRS BBDnnRsDel BBDnnFDRS BBDnnKDRS

0.80 0.79 0.62 0.61 0.58 0.52

AvastStyleConvKDRS BBDnnKDRS AvastStyleConvFDRS MalConvKDRS MalConvFDRS BBDnnFDRS

0.99 0.99 0.99 0.99 0.99 0.99

omitted metric provides unique information to the benchmark, having thus a stronger influence on the final ranking. Overall, the Inference metric yields the weakest LOO correlations (ρ = 0.2538, r = 0.2623), indicating that it has the largest impact on the leaderboard. In contrast, the Performance metric exhibits the highest agreement with the remaining metrics, suggesting a higher degree of redundancy with the other evaluation metrics. This effect is also reflected in the magnitude of the ranking changes observed after removing each metric. The exclusion of the Inference metric produces the largest positional shifts across models: BBDnnRsDel moves from the 22nd to the 4th position, while removing the Temporal metric results in a maximum displacement of six positions. These findings suggest that the Inference metric captures aspects of model behavior that are not fully represented by the other evaluations, and therefore plays a key role in shaping the final leaderboard. To facilitate further exploration of these effects, we developed and released an interactive dashboard10 that allows users to inspect the leaderboard and experiment with alternative metric weightings. The latter can help practitioners in picking the best model that suit their need (e.g., limited hardware capacities, infrequent re-training, etc.) instead of using the average default weighting. B. Performance Metric Leaderboard We report here the ranking based only on the Performance metric P, by listing the best six models in Tab. II (second column), and also visualizing their ROC curve on the test set in Fig. 5. When considering the performance on the test set alone, most of the regular versions of models are present 10 https://exebench.github.io/

True Positive Rate

TABLE II: Top six models according to each metric of EXE-Bench, displayed column-wise.

1.0 0.9 0.8 0.7 0.6 0.5 0.4

10 3

EmberGBDT NGramConv BBDnnRsDel MalConv ResNet18 AvastStyleConvRsDel 2 10 10 1 100 False Positive Rate

Fig. 5: ROC curves of the top-six models according to the Performance metric.

among the top-performing ones, missing only AvastStyleConv (which achieved a suboptimal F1 score of 0.76 on the test set). Interestingly, the RsDel certification schema (which generally decreases the ranking of models to which it is applied, as highlighted in Sect. V-A) slightly improves the F1 of BBDnn, being an incredible boost for AvastStyleConv (i.e., it improves F1 by ∼ 0.2). By looking only at this leaderboard, the positions are different w.r.t. the complete one, as MalConv is ranked higher and also surpasses the second-best model BBDnn. This is not surprising, since these models have been developed solely to excel in terms of accuracy, thus possibly neglecting the other metrics that we observe. For instance, MalConv is not able to keep up with both the passage of time and robustness, falling down in the overall ranking as it is likely overfitting the distribution of the present data available during deployment. On the contrary, as highlighted in Fig. 5, no model is a match for the EmberGBDT, which achieves almost perfect results already at 10−3 FPR. To summarize,

1.0

1.0

0.8

0.9

0.6

0.8

0.4 0.2

Detection Rate

F1-Score

9

EmberGBDT NGramConv BBDnn BBDnnFDRS NGramConvFDRS NGramConvRsDel

0.7 0.6

10000

2022-Q1

2021-Q3

2021-Q2

2021-Q1

2020-Q3

2020-Q2

2020-Q1

2019-Q3

0

2019-Q2

5000 2019-Q1

Samples

0.5 15000

Fig. 6: Fluctuations of the F1 score of the top six models, ranked by the temporal metric, and the number of samples contained in each temporal bin.

this reduced leaderboard represents a breakdown of the regular comparisons between models, by just picking the ones with the best score in terms of detection rate and low false alarms. However, this metric alone does not provide a full picture of the behavior of the analyzed models.

C. Temporal Metric Leaderboard We now report the ranking based on the Temporal metric T, by listing the best six models in Tab. II (third column), while also depicting the fluctuations of the F1 score on the various bins in Fig. 6. We notice the presence of certifiable approaches among the best models, with de-randomized smoothing leading the way. This could be explained by the fact that, while portions of code and data might change over time (i.e., being completely different in terms of bytes), the information contained inside headers (including the Import Address Table, that explains to the operating system which API to import) is likely to be less prone to such a change. Hence, the FDRS methods, which segment each input program into chunks of fixed size, combined with the majority voting, enable these models to weigh more the decision on headers than the bytes contained in sections (which might be noisier from one bin to the other). Interestingly (and differently from the analysis on the Android domain [18]), the top six models are not really strictly losing performance over time, fluctuating bin by bin. These fluctuations can be caused by a mixture of factors, ranging from the inaccurate attribution due to timestamp manipulation, or by the scarcity of the number of samples contained in those bins. However, in the context of a benchmark, the evaluation is fair for all models, since they are evaluated on the same data, by addressing the size of bins through the weighted average. To summarize, this reduced leaderboard gives clear insights on which model can be more resistant to the passage of time, providing guidance on what to deploy in production.

EmberGBDT BBDnn BBDnnFDRS BBDnnKDRS BBDnnSDRS BBDnnRDRS 101 102

103 104 105 Perturbation (bytes)

106

107

Fig. 7: Security evaluations of the top-six models in terms of robustness. The x axis is the number of perturbed bytes, while the y axis is the Detection Rate (DR).

D. Robustness Metric Leaderboard We now report the ranking based only on the Robustness metric R, by listing the best six models in Tab. II (fourth column), while also depicting their security evaluation curves in Fig. 7. GBDT and BBDnn obtain almost the same score, with GBDT being slightly better: as shown in Fig. 7, the neural network exhibits better robustness only after having injected a few MB into samples (far right of the figure). However, except for BBDnn and its variants, all the other endto-end models perform poorly. In particular, upon inspection, we highlight that ResNet18, the image-based classification method, is broken against the weakest attacks of our benchmark, i.e., FullDOS and Content-shift. This can be easily explained by the fact that folding programs into images (i.e., breaking byte strings into rows and columns) does not provide any spatial advantage, since different rows in the produced image are not necessarily semantically close. Also, adversarial content is likely to form contiguous blocks that disrupt the convolution operations, easily misleading all the predictions. Interestingly, the regular BBDnn model is more robust than its randomized and de-randomized smoothing versions with a striking gap in terms of the computed metric. This could be caused by GAMMA injecting portions of legitimate programs into malware, used during the slicing provided by the randomization. In particular, each of these methods computes inference only on a smaller view of the sample, which can now also contain bytes harvested from goodware programs. Thus, many of the blocks are patterns of bytes considered legitimate, altering the decision of the majority voting schema. Lastly, the robustness of BBDnn might also be an effect of its reduced input window (100KB), differently from all the other networks. Hence, content-injection attacks like GAMMA might create plenty of sections whose content falls outside such input window, thus enlarging the perturbation is not bringing an advantage to the attacker. To summarize, extracting features improves robustness against adversarial attacks, followed by neural networks that compute predictions by looking at a reduced portion of the entire input executable. On the contrary, methods claimed as robust are not really effective against content-injection attacks, with randomization and majority voting system being their limit.

10

101

Inference Time (s)

100 10 1 10 2

MalConv AvastStyleConv BBDnn NGramConv EmberGBDT ResNet18 MalConvRsDel AvastStyleConvRsDel BBDnnRsDel NGramConvRsDel MalConvRS AvastStyleConvRS BBDnnRS NGramConvRS MalConvFDRS AvastStyleConvFDRS BBDnnFDRS NGramConvFDRS MalConvKDRS AvastStyleConvKDRS BBDnnKDRS NGramConvKDRS MalConvRDRS AvastStyleConvRDRS NGramConvRDRS BBDnnRDRS MalConvSDRS AvastStyleConvSDRS BBDnnSDRS NGramConvSDRS

10 3

Fig. 8: Boxplot with average (dot), median (line), and standard deviation (blue area) of inference time of all the models in the benchmark.

E. Inference Metric Leaderboard We report the ranking based only on inference time by listing the six best models in the last column of Tab. II, while also providing a comprehensive view on inference time expressed in seconds in Fig. 8. Lastly, even if omitted from the benchmark, we report in Tab. III the hours needed for training all models for completion. Counter-intuitively, the fastest models are the de-randomized ones, surpassing all others in the reduced leaderboard. Even if inference is computed multiple times due to the certification schema (e.g., 12 for KDRS), these models receive in input samples negligible in size, allowing very-fast predictions. In particular, these are even faster than computing a prediction on the whole sample, as globally shown in Fig. 8. On the contrary, the slowest models are the ones using randomized smoothing, since inference is repeated 100 times on truncated samples, providing computational overhead required by the sampling. To summarize, this reduced leaderboard provides a snapshot of which model can produce fast predictions. While this metric alone cannot provide a solid ranking, it might be useful to prioritize speed w.r.t. other metrics, i.e., in contexts in which quick filters are needed before more rigorous analyses. F. Concluding Summary Accuracy alone is not enough. Our results show that being accurate on a test set is not the only aspect at deployment time. When complemented with efficiency, temporal and adversarial robustness, practitioners can collect a complete snapshot of the capabilities of models before deployment. Feature engineering makes the difference. EMBER GBDT provides top rankings on all metrics except inference time, due to the inevitable feature extraction phase. However, this model is still capable of computing inference in less than one second on a compact set of features, without relying on deploying a large convolutional neural network with less accurate results (but slightly faster). Such a top-tier result underlines the idea that we still indeed need feature engineering in the domain of Windows malware detection, as we have not yet reached the

stage at which deep neural networks surpass the application of domain knowledge. Also, the application of domain knowledge also improves stability over time and robustness against attacks, making them a perfect fit for production environments. These models can be rapidly trained (4 minutes) once all features have been processed, and updates can be shipped fast since all the already-collected samples are already processed, needing to rely on feature extraction of only the fresh ones (reasonably less than the pre-owned ones). Efficiency of De-randomized smoothing. Even if this family of methods is not scoring a top rank in the leaderboard, it is interesting to notice that they are fast to train and deploy. With more research, it is possible that they will conquer higher places in the ranking, while being more deployable-friendly. Unreliable image-based detection. While classifying malware as an image might speed up the process, using images as a representation for Windows programs is neither robust to adversarial attacks nor robust to the passage of time, being among the worst w.r.t. these two aspects. Hence, we deeply discourage the usage of these types of detectors in production. VI. L IMITATIONS While our methodology is able to provide a clear winner by analyzing different axes, we acknowledge the presence of different limitations of EXE-Bench, by also showing how these can be addressed and tackled with ease. Static analysis only. All the models we consider only perform static analysis on samples, ignoring models trained on the output of dynamic analysis. While the literature on the matter is vast [40], these methods are characterized by huge computational and deployment costs due to the instantiation of emulators or sandboxes to safely detonate malware and track their behavior. While our current analysis does not cover this aspect, EXE-Bench is general enough to compute these metrics for any Windows detector. This is achieved thanks to the generality of the metrics, since they all characterize a relevant aspect of models, regardless of their nature. EMBER dataset. While the dataset we used is relevant, more recent data sources have been released, including SOREL [41], and EMBER2024 version [42]. While timely, their usage does not change the proposed methodology, and can be easily included in EXE-Bench either as training data or future time splits without changing the overall methodology. Missing models. While we evaluated state-of-the-art techniques, we have left behind some techniques leveraging other pre-processing or hardening approaches. We excluded transformer-based detectors due to the GPU memory required for training: prior work [43] reports comparable detection performance to our CNN-based models, but at substantially higher computational cost, with inference times 570.57% higher than MalConv. We also did not include methods based on the assembly representation of programs [3], [44], such as GNN-based approaches that operate on Control Flow Graph or Function Call Graph representations of programs, because many programs are packed or obfuscated. In such cases, disassemblers may produce incomplete, misleading, or unreadable output, and the quality of their output highly

11

depends on the source of these technologies. Unfortunately, the best disassemblers are commercial products with expensive licenses, like IDA Pro,11 which also introduce latency when computing inference times. On the same note, we did not include adversarial training [23], [45], [46] since it is extremely time-consuming and computationally expensive. For example, the fastest method reported in previous work [23], [45] required 5.7 days of training for just 3 epochs on fewer than 300k samples, while the slowest approach took months. In contrast, our training dataset is twice their size, thus using this technique on more than 3 epochs would have required prohibitively long runtimes. Furthermore, the attacks used to compute adversarial EXEmples while training [23], [45] are either not publicly released or are missing components, making exact reproduction impractical. Also, even if missing, the benchmark is still general enough to tackle the future inclusion of all AI-based Windows malware detectors without loss of generality. The quantities we measure are model agnostic, and they solely depend on the performance and computational requirements of those models. Limitations of randomized-smoothing approaches. While randomized smoothing provides bounds on the accuracy when manipulating up to a specific number of bytes [10] (i.e. the certified accuracy), we were unable to include such metric in our benchmark due to computational requirements. For instance, RsDel methods would need ∼ 4000 randomizations of each sample to estimate the certified accuracy. Moreover, during inference, the original implementations of RsDel and RS do not fit the VRAM of our GPUs, forcing us to cut samples according to the input window of each model. Nevertheless, these issues do not pose a problem to the benchmark itself, but rather highlight the need for more computational power to handle certifiable models. In fact, without these simplifications, certifiable models would have sunk even lower in the benchmark due to their infeasible computational time. The same can be said for their adversarial evaluation: directly attacking them would have required much more computational capacity. Even so, the collected results highlight the necessity for more research in the context of certification. Biases of the temporal analysis. While we proved that timestamps are the only available information we can leverage, we acknowledge that the temporal evaluation might be biased by other factors, like the presence of timestomping techniques (i.e., misleading the temporal attribution by altering the original compilation timestamp12 ), by the specific era we have considered (2019-2022), or also by the vendor that originally acquired the dataset. Nevertheless, we have followed the best practices streamlined by previous work [18], and, whether it would be possible to retrieve the ground truth of the first seen dates, our leaderboard can be re-evaluated accordingly, as the way EXE-Bench operates remains unchanged. Missing attacks. While we computed a massive amount of adversarial EXEmples, the literature contains other techniques which we do not have included into EXE-Bench [20], [47]. In particular, some models might be weaker to some techniques 11 https://hex-rays.com/ida-pro 12 https://attack.mitre.org/techniques/T1070/006/

and not others, or their input windows might be too small and thus cut all the injected adversarial content (e.g., BBDnn), thus providing an advantage in terms of robustness. However, while these techniques require commercial tools like IDA Pro [20], or are known to corrupt samples [47], our methodology remains sound, and those attacks can be added at any time after having provided a correct implementation of those techniques. Absence of metrics for Malware Classification. We did not take into account malware classification, which requires methods that determine the family of malware rather than only stating its maliciousness. However, while missing, this inclusion would require the adaptation of metrics to this domain, taking into account attribution mistakes (i.e., determine the wrong malware family) on the present and future data, while also re-defining robustness. The latter would require rethinking the threat model, which now focuses only on evasion rather than wrong attribution. Hence, including such a task in our benchmark would have required a different approach, also regarding the models proposed in the state of the art, which are mostly tailored to detection rather than classification. VII. F UTURE W ORK AND C ONCLUSIONS Future Work. We highlight possible extensions of our methodology that could also inspire further research activities. Classification over time. While not only we want to expand our methodology to address malware classification as well, we are also willing to investigate how to quantify this task over time, taking into account the change in the family distributions on different bins. This would also fairly characterize how models perform in the presence of new families appearing at some point in time, while others disappear, thus being more difficult to observe in the future. Benchmarking more detectors. We are working to expand the results of the benchmark towards more models, also including dynamic analysis detectors, i.e., models trained on reports obtained by executing or emulating programs into isolated environments. While this could require investigating whether other metrics can be used, the current version of EXE-Bench can already include all models under the same leaderboard, even if different pre-processing is applied to the programs. Diversify adversarial attacks. While the metric we have built is general enough to encompass the presence of, in theory, all possible attacks, we will expand the results of the benchmark by including a more comprehensive set of previously-proposed strategies [20], [47], [48]. These would grant EXE-Bench a pivotal impact on this research topic, standing as a fair methodology to assess robustness of detectors. Conclusions. We propose EXE-Bench, a novel and systematic evaluation procedure to rank Windows malware detectors w.r.t. their predictive capabilities on the present and the future data, robustness, and computational requirements. This is achieved through the definition of a metric that takes into account these quantities, thus creating a general leaderboard of models. Through our benchmark, we pinpoint the need for evaluations considering multiple axes, since performance on test is, alone, useless to understand the aspects needed in production environments, like stability in time and robustness. From our

12

results, feature engineering seems to be a winning strategy to balance all these requirements, while some methodologies (unsuitable under the evaluated criteria) should be avoided due to their utter inefficacy. We believe that EXE-Bench can provide a starting point for practitioners to pick up a model to deploy depending on the metrics deemed more relevant to them, while also standing as a systematic evaluation procedure to rank all the future models that will be released in the future. ACKNOWLEDGMENT This work has been partially supported by FISA-202300128 funded by the MUR program “Fondo italiano per le scienze applicate”; and by PNRR MUR Project SERICS (PE00000014) and ”Future Artificial Intelligence Research (FAIR)”, funded by the European Union – NextGenerationEU, PE00000013 CUP J33C24000420007. D. Gibert was supported by grant RYC2023-043607-I funded by MICIU/AEI/10.13039/501100011033 and FSE+. M. Kozak was supported by MEYS of the Czech Republic, grant No. SGS26/187/OHK3/3T/18 of the Grant Agency, Czech Technical University in Prague. D. Trizna was enrolled in the Italian National Doctorate on Artificial Intelligence run by Sapienza University of Rome in collaboration with the University of Genova. This research project was made possible through the access granted by the Galician Supercomputing Center (CESGA) to its supercomputing infrastructure. The supercomputer FinisTerrae III and its permanent data storage system have been funded by the NextGeneration EU 2021 Recovery, Transformation and Resilience Plan, ICT2021-006904, and also from the Pluriregional Operational Programme of Spain 2014-2020 of the European Regional Development Fund (ERDF), ICTS-2019-02-CESGA-3, and from the State Programme for the Promotion of Scientific and Technical Research of Excellence of the State Plan for Scientific and Technical Research and Innovation 2013-2016 State subprogramme for scientific and technical infrastructures and equipment of ERDF, CESG15-DE-3114. R EFERENCES [1] H. S. Anderson and P. Roth, “EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models,” ArXiv e-prints, Apr. 2018. [2] J. Saxe and K. Berlin, “Deep neural network based malware detection using two dimensional binary program features,” in 10th Int. Conf. on Malicious and Unwanted Software (MALWARE), 2015, pp. 11–20. [3] M. Ahmadi, D. Ulyanov, S. Semenov, M. Trofimov, and G. Giacinto, “Novel feature extraction, selection and fusion for effective malware family classification,” in Proc. of 6th ACM Conference on Data and Application Security and Privacy, 2016, p. 183–194. [4] E. Raff, J. Barker, J. Sylvester, R. Brandon, B. Catanzaro, and C. K. Nicholas, “Malware detection by eating a whole EXE,” in Workshop of the The 32nd AAAI Conference on Art. Int., 2018. [5] M. Krčál, O. Švec, M. Bálek, and O. Jašek, “Deep convolutional malware classifiers can learn from raw executables and labels only,” in ICLR 2018 Workshop, 2018. [6] S. E. Coull and C. Gardner, “Activation analysis of a byte-based deep neural network for malware classification,” in IEEE Security and Privacy Workshops (SPW), 2019. [7] D. Gibert, C. Mateu, J. Planes, and J. Marques-Silva, “Auditing static machine learning anti-malware tools against metamorphic attacks,” Computers & Security, vol. 102, p. 102159, 2021. [8] L. Nataraj, S. Karthikeyan, G. Jacob, and B. S. Manjunath, “Malware images: visualization and automatic classification,” in 8th Int. Symposium on Visualization for Cyber Security (VizSec). ACM, 2011, p. 4.

[9] D. Gibert, C. Mateu, J. Planes, and R. Vicens, “Using convolutional neural networks for classification of malware represented as images,” J. Comput. Virol. Hacking Tech., vol. 15, no. 1, pp. 15–28, 2019. [10] Z. Huang, N. Marchant, K. Lucas, L. Bauer, O. Ohrimenko, and B. I. P. Rubinstein, “RS-Del: Edit distance robustness certificates for sequence classifiers via randomized deletion,” in Adv. in Neural Information Processing Systems, 2023. [11] D. Gibert, G. Zizzo, and Q. Le, “Towards a practical defense against adversarial attacks on deep learning-based malware detectors via randomized smoothing,” in 28th Eu. Sym. on Research in Computer Security Workshops. Springer, 2023. [12] ——, “Certified robustness of static deep learning-based malware detectors against patch and append attacks,” in Proc. of 16th ACM Workshop on Art. Int. and Security, 2023, p. 173–184. [13] S. Saha, W. Wang, Y. Kaya, S. Feizi, and T. Dumitras, “Drsm: Derandomized smoothing on malware classifier providing certified robustness,” ICLR 2024, 2024. [14] D. Gibert, G. Zizzo, Q. Le, and J. Planes, “Adversarial robustness of deep learning-based malware detectors via (de)randomized smoothing,” IEEE Access, vol. 12, pp. 61 152–61 162, 2024. [15] D. Gibert, L. Demetrio, G. Zizzo, Q. Le, J. Planes, and B. Biggio, “Certified adversarial robustness of machine learning-based malware detectors via (de)randomized smoothing,” 2024. [Online]. Available: https://arxiv.org/abs/2405.00392 [16] J. Cohen, E. Rosenfeld, and J. Z. Kolter, “Certified adversarial robustness via randomized smoothing,” in Proc. of 36th Int. Conf. on Machine Learning, vol. 97, 2019, pp. 1310–1320. [17] A. Levine and S. Feizi, “(de)randomized smoothing for certifiable defense against patch attacks,” in Adv. in Neural Information Processing Systems, vol. 33, 2020, pp. 6465–6475. [18] F. Pendlebury, F. Pierazzi, R. Jordaney, J. Kinder, and L. Cavallaro, “{TESSERACT}: Eliminating experimental bias in malware classification across space and time,” in 28th USENIX sec. sym., 2019, pp. 729–746. [19] B. Kolosnjaji, A. Demontis, B. Biggio, D. Maiorca, G. Giacinto, C. Eckert, and F. Roli, “Adversarial malware binaries: Evading deep learning for malware detection in executables,” in 26th European signal processing conference (EUSIPCO). IEEE, 2018, pp. 533–537. [20] K. Lucas, M. Sharif, L. Bauer, M. K. Reiter, and S. Shintre, “Malware makeover: Breaking ml-based static analysis by modifying executable bytes,” in Proc. of ACM Asia Conference on Computer and Communications Security, 2021, pp. 744–758. [21] L. Demetrio, S. E. Coull, B. Biggio, G. Lagorio, A. Armando, and F. Roli, “Adversarial exemples: A survey and experimental evaluation of practical attacks on machine learning for windows malware detection,” ACM Trans. Priv. Secur., vol. 24, no. 4, Sep. 2021. [22] L. Demetrio, B. Biggio, G. Lagorio, F. Roli, and A. Armando, “Functionality-preserving black-box optimization of adversarial windows malware,” IEEE Tran. on Information Forensics and Security, vol. 16, pp. 3469–3478, 2021. [23] K. Lucas, S. Pai, W. Lin, L. Bauer, M. K. Reiter, and M. Sharif, “Adversarial training for {Raw-Binary} malware classifiers,” in 32nd USENIX sec. sym., 2023, pp. 1163–1180. [24] B. Biggio and F. Roli, “Wild patterns: Ten years after the rise of adversarial machine learning,” Elsevier Pattern Recognition, vol. 84, pp. 317–331, 2018. [25] A. E. Cinà, J. Rony, M. Pintor, L. Demetrio, A. Demontis, B. Biggio, I. B. Ayed, and F. Roli, “Attackbench: Evaluating gradient-based attacks for adversarial examples,” in Proc. of AAAI Conference on Art. Int., vol. 39, no. 3, 2025, pp. 2600–2608. [26] F. Croce, M. Andriushchenko, V. Sehwag, E. Debenedetti, N. Flammarion, M. Chiang, P. Mittal, and M. Hein, “Robustbench: a standardized adversarial robustness benchmark,” in 35th Conf. on Neural Information Processing Systems Datasets and Benchmarks Track, 2021. [27] F. Pierazzi, F. Pendlebury, J. Cortellazzi, and L. Cavallaro, “Intriguing properties of adversarial ml attacks in the problem space,” in IEEE symposium on security and privacy (SP). IEEE, 2020, pp. 1332–1349. [28] R. Vinayakumar, M. Alazab, K. P. Soman, P. Poornachandran, and S. Venkatraman, “Robust intelligent malware detection using deep learning,” IEEE Access, vol. 7, pp. 46 717–46 738, 2019. [29] A. Fahim, S. Dey, M. N. Absur, M. Kamrul Siam, M. T. Huque, and J. Jafor Godhuli, “Optimized approaches to malware detection: A study of machine learning and deep learning techniques,” in IEEE 14th Int. Conf. on Communication Systems and Network Technologies, 2025, pp. 269–275.

13

[30] M. Imran, A. Appice, and D. Malerba, “Evaluating realistic adversarial attacks against machine learning models for windows pe malware detection,” Future Internet, vol. 16, no. 5, 2024. [31] P. Louthánová, M. Kozák, M. Jureček, M. Stamp, and F. Di Troia, “A comparison of adversarial malware generators,” Journal of Computer Virology and Hacking Techniques, vol. 20, no. 4, pp. 623–639, 2024. [32] S. Verwer, A. Nadeem, C. Hammerschmidt, L. Bliek, A. Al-Dujaili, and U.-M. O’Reilly, “The robust malware detection challenge and greedy random accelerated multi-bit search,” in Proc. of 13th ACM Workshop on Art. Int. and Security, 2020, p. 61–70. [33] M. Pintor, F. Roli, W. Brendel, and B. Biggio, “Fast minimum-norm adversarial attacks through adaptive norm constraints,” Adv. in Neural Information Processing Systems, vol. 34, pp. 20 052–20 062, 2021. [34] D. Trizna, “Quo vadis: hybrid machine learning meta-model based on contextual and behavioral malware representations,” in Proc. of 15th ACM Workshop on Art. Int. and Security, 2022, pp. 127–136. [35] J. H. Friedman, “Greedy function approximation: a gradient boosting machine,” Annals of statistics, pp. 1189–1232, 2001. [36] G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu, “Lightgbm: A highly efficient gradient boosting decision tree,” Adv. in Neural Information Processing Systems, vol. 30, 2017. [37] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778. [38] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al., “Pytorch: An imperative style, high-performance deep learning library,” Adv. in Neural Information Processing Systems, vol. 32, 2019. [39] A. Demontis, M. Melis, M. Pintor, M. Jagielski, B. Biggio, A. Oprea, C. Nita-Rotaru, and F. Roli, “Why do adversarial attacks transfer? explaining transferability of evasion and poisoning attacks,” in 28th USENIX sec. sym., 2019, pp. 321–338. [40] D. Gibert, C. Mateu, and J. Planes, “The rise of machine learning for detection and classification of malware: Research developments, trends and challenges,” Elsevier Journal of Network and Computer Applications, vol. 153, p. 102526, 2020. [41] R. Harang and E. M. Rudd, “Sorel-20m: A large scale benchmark dataset for malicious pe detection,” arXiv preprint, 2020. [42] R. J. Joyce, G. Miller, P. Roth, R. Zak, E. Zaresky-Williams, H. Anderson, E. Raff, and J. Holt, “Ember2024-a benchmark dataset for holistic evaluation of malware classifiers,” in Proc. of 31st ACM SIGKDD Conf. on Knowledge Discovery and Data Mining V. 2, 2025, pp. 5516–5526. [43] L. Kurlandski, H. Berger, Y. Pan, and M. Wright, “Beyond raw bytes: Towards large language models,” in Proc. of 33rd Network and Distributed Systems (NDSS) Symposium, 2026. [44] Y.-H. Chen, S.-C. Lin, S.-C. Huang, C.-L. Lei, and C.-Y. Huang, “Guided malware sample analysis based on graph neural networks,” IEEE Transactions on Information Forensics and Security, vol. 18, pp. 4128–4143, 2023. [45] K. Lucas, W. Lin, L. Bauer, M. K. Reiter, and M. Sharif, “Training robust ml-based raw-binary malware detectors in hours, not months,” in Proc. of 2024 on ACM SIGSAC Conference on Computer and Communications Security, 2024, p. 124–138. [46] M. Kozak, L. Demetrio, D. Trizna, and F. Roli, “Updating windows malware detectors: Balancing robustness and regression against adversarial exemples,” Elsevier Computers & Security, vol. 155, p. 104466, 2025. [47] W. Song, X. Li, S. Afroz, D. Garg, D. Kuznetsov, and H. Yin, “Mabmalware: A reinforcement learning framework for blackbox generation of adversarial malware,” in Proc. of ACM on Asia conference on computer and communications security, 2022, pp. 990–1003. [48] M. Kozák, M. Jureček, M. Stamp, and F. D. Troia, “Creating valid adversarial examples of malware,” Springer Journal of Computer Virology and Hacking Techniques, vol. 20, no. 4, pp. 607–621, 2024.

1 ∗ (0 + 0.8)/2 = 0.4. However, the result is penalized by the first bin being under-represented, weighting the same on the average. On the contrary, by weighting this number on the amount of samples contained into the bins, we would compute 2 (0 ∗ 102 ) + 0.8 ∗ 100 102 ) = 0.78. B: Training Time in Hours We report in Tab. III the estimated training time (in hours) of all models. Except for one model (i.e., MalConvRS requires ∼ 146 hours of training time), training does not pose a huge bottleneck, since such operation is not a frequent as inference. On the contrary, while feature extraction seems slow (EMBER GBDT requires roughly 7 hours to process and train 600k programs), we remark that samples are only processed once, thus speeding up next updates (training alone on 600k featurized samples only requires 4 minutes), differently from deep neural networks that must be re-trained from scratch. Model

Time (h)

AvastConvRsDel BBDnnRsDel MalConvSDRS MalConvRDRS NGramConvRsDel BBDnnSDRS BBDnnRDRS AvastConvFDRS NGramConvRDRS NGramConvSDRS MalConvFDRS ResNet18 NGramConvFDRS MalConvRsDel AvastConvRDRS

0.46 0.83 1.04 1.04 1.32 1.47 1.47 2.02 2.05 2.05 2.13 2.19 2.44 2.91 2.95

Time (h)

AvastConvSDRS AvastConvKDRS BBDnnFDRS NGramConvKDRS BBDnnKDRS BBDnn MalConvKDRS EmberGBDT AvastConv AvastConvRS BBDnnRS NGramConvRS MalConv NGramConv MalConvRS

2.95 3.54 4.99 5.18 5.29 6.00 6.44 7.18 7.62 7.99 10.51 18.12 19.54 23.96 146.47

TABLE III: Training time (hours) of all the considered models. C: Metadata gathered from VirusTotal Bin

First Submission

Tot. Total

2019-Q1 2,440 2019-Q2 1,895 2019-Q3 1,043 2020-Q1 2,500 2020-Q2 1,401 2020-Q3 2,198 2021-Q1 4,128 2021-Q2 3,494 2021-Q3 5,541 2022-Q1 18,864

Train

First Seen Test

Total

Train

Test

1846 (75.7%) 1035 (56.1%) 811 (43.9%) 105 (4.3%) 53 (50.5%) 52 (49.5%) 1618 (85.4%) 1356 (83.8%) 262 (16.2%) 11 (0.6%) 11 (100.0%) 0 (0.0%) 262 (25.1%) 223 (85.1%) 39 (14.9%) 47 (4.5%) 40 (85.1%) 7 (14.9%) 508 (20.3%) 336 (66.1%) 172 (33.9%) 37 (1.5%) 26 (70.3%) 11 (29.7%) 707 (50.5%) 502 (71.0%) 205 (29.0%) 95 (6.8%) 56 (58.9%) 39 (41.1%) 1052 (47.9%) 850 (80.8%) 202 (19.2%) 211 (9.6%) 113 (53.6%) 98 (46.4%) 1249 (30.3%) 996 (79.7%) 253 (20.3%) 221 (5.4%) 96 (43.4%) 125 (56.6%) 1528 (43.7%) 1116 (73.0%) 412 (27.0%) 287 (8.2%) 144 (50.2%) 143 (49.8%) 1796 (32.4%) 1169 (65.1%) 627 (34.9%) 214 (3.9%) 115 (53.7%) 99 (46.3%) 3571 (18.9%) 1848 (51.8%) 1723 (48.2%) 1078 (5.7%) 648 (60.1%) 430 (39.9%)

≥ Collection Date 6,344 (44.9%)

5,936

408

-

-

-

TABLE IV: Summary of metadata collected from VirusTotal. The last row considers only samples first submitted after the collection date (Train ≥ 2022-02, Test ≥ 2022-05). D: Rank Correlation Analysis

A PPENDIX A: Area Under the Curve fallacy The Area Under the Curve (AUC) quantifies the area below the curve, empirically approximated with a finite number of rectangles or trapezoids covering such region, without addressing the number of samples used to produce each point in the curve. Given 2 splits (considered as one single unit away on the x axis), one with 2 samples and F1 score of 0, and one with 100 samples and F1 score of 0.8, the total AUC would be

Model

Leave-One-Out Metric

Pearson r (p-val.) Spearman ρ (p-val.)

Performance Temporal Robustness Inference

+0.5291 (0.0026) +0.3984 (0.0292) +0.2162 (0.2513) +0.2623 (0.1614)

+0.3419 (0.0644) +0.5622 (0.0012) +0.3673 (0.0459) +0.2538 (0.1759)

TABLE V: Correlation with final Rank (Leave-One-Out).