1
FDM: A Framework for Decision-making to build ML-based Malware detection systems
arXiv:2606.06894v1 [cs.CR] 5 Jun 2026
Tadiwa Vhito, Jakapan Suaboot*, Warodom Werapun, Norrathep Rattanavipanon College of Computing Prince of Songkla University Phuket, Thailand {s6530621001, jakapan.su, wwarodom, norrathep.r}@phuket.psu.ac.th
Abstract—Selecting appropriate machine learning (ML) configurations for malware detection is a complex, multi-criteria problem: model choice, feature engineering strategy, and update mechanism must jointly satisfy operational constraints that vary substantially across deployment contexts. This paper proposes a Framework for Decision-making to build ML-based Malware detection systems (FDM), a structured methodology that formalises this selection process through the Weighted Configuration Compatibility Score (WCCS), a multi-criteria scoring function that maps five quantifiable operational parameters (platform constraint, resource budget, response latency, update frequency, and detection sensitivity) to ranked recommendations across nine configuration dimensions. To validate the framework, four experiments were conducted on three datasets (a private Windows API call dataset covering eight malware families, the public Malimg image benchmark of 9,339 samples from 25 families, and a public Android static API dataset of 15,036 applications). Key results include: (i) XGBoost achieved the best accuracy-to-resource ratio in binary classification (97.46% test accuracy, <70 MB RAM), outperforming LSTM and BiLSTM which consumed up to 2.8 GB; (ii) in multi-class classification, classical models (XGBoost 79.03%) outperformed recurrent deep models (BiLSTM 72.27%), reversing the binary ranking; (iii) class-incremental learning with EfficientNetB0 maintained 99.13% accuracy with only 0.65 percentage-point accuracy degradation across eleven incremental steps; (iv) transfer learning reduced training time by an average of 2.14× on image-based malware data with no significant accuracy cost; and (v) autoencoder pre-processing yielded a 14× training speedup at a cost of only 0.86 pp accuracy. These findings collectively confirm that the optimal ML configuration is contextdependent, validating the FDM’s core premise and demonstrating its practical utility for cybersecurity practitioners. Index Terms—Malware Detection, Transfer learning, ClassIncremental Learning, Decision-making framework, Contextaware deployment.
I. I NTRODUCTION The rapid growth of artificial intelligence (AI) technology has caused an unprecedented surge in malware attacks, presenting a critical challenge in modern cybersecurity. This rapid evolution is evidenced by the discovery of more than 30 million new malware strains and potentially unwanted programs in just the first half of 2026 as reported by the AVTEST Institute1 . According to the World Economic Forum2 , ∗ Corresponding author 1 https://portal.av-atlas.org/malware 2 https://www.weforum.org/stories/2023/06/asia-pacific-region-the-newground-zero-cybercrime/
the average annual cost of cybercrime is expected to increase to more than $23 trillion by 2027. Therefore, an urgent advancement in defensive capabilities is required. Malware detection presents significant challenges due to the sophisticated evasion techniques employed by malware creators. These evasion techniques can generally be categorized into three groups [1]: (i) transformation-based methods (such as metamorphism, polymorphism [2], dead code insertion and instruction replacement or reordering [3]), (ii) concealmentbased tactics (including delay execution and code obfuscation), and (iii) attack-based techniques (such as exploitation attacks [4]). The use of anti-sandboxing techniques further complicates dynamic analysis-based detection methods, as malware can be designed to postpone or evade malicious actions when it identifies a sandbox environment [5]. Certain evasion methods, notably polymorphism and instruction replacement or reordering, allow attackers to generate malware variants that maintain the same functionalities while displaying different signatures, thus successfully evading detection, particularly by signaturebased techniques. Furthermore, zero-day attacks, which exploit undiscovered vulnerabilities in software or operating systems, render prevention efforts almost futile [6]. Consequently, there is a continuous demand for researchers to develop innovative malware detection methods (e.g. [7], [8], [9]). Malware detection has evolved from signature-based methods using predefined patterns, e.g., YARA rules [10], to identify known threats. Although effective for simple malware and offering low false alarms, signature-based approaches can easily be circumvented through the evasion techniques mentioned above to modify malware signatures and thwart the detection system. Hence, machine learning (ML) has revolutionized malware detection, becoming a primary tool to identify malicious behavior instead of a static signature. Deep learning, in particular, excels at revealing complex patterns that classical (non-deep) learning methods cannot automatically capture. This indeed improves the accuracy of malware classification while reducing the complexity of requiring human experts to design features when using the classical methods (e.g., [11]). Much existing ML-based malware detection research follows a fragmented approach in which researchers select specific dataset configurations, e.g., using public [12], [13] or private datasets [14], [15] using static [8], [16], [17] or dynamic [18], [19] analysis methods, or using the pre-analysis
2
features from existing papers [20]. The researchers then develop models claiming superior accuracy within narrow constraints, i.e., their chosen dataset type, analysis methodology, and evaluation metrics, without demonstrating generalizability across different operational environments. This has produced numerous isolated solutions while leaving significant research gaps in systematic guidance for optimal selection of MLtechniques across diverse contexts and constraints. We argue that building an effective machine learningbased malware detection system requires navigating a complex landscape of technical choices, including feature extraction, feature selection, and deep learning hyperparameter tuning. In practice, these variables are highly interdependent: the choice of deployment platform constrains feasible model architectures, which in turn dictate the appropriate data preprocessing pipelines and viable training strategies. Without a structured methodology, practitioners must manage these cascading decisions in an ad-hoc manner, often resulting in suboptimal configurations that fail to balance detection performance, resource consumption, and long-term maintainability. This systemic lack of guidance raises a fundamental research question: what constitutes the most suitable machine learning methodology for malware detection given varying organizational contexts, data characteristics, and operational constraints? To answer this, we propose a Framework for Decisionmaking to build ML-based Malware detection systems (FDM). The framework provides systematic guidance for malware prevention teams in constructing detection systems suited to diverse organizational constraints and requirements, offering a principled approach to methodology selection rather than another isolated solution. The main contributions of this paper are as follows: 1) A quantifiable decision-making framework (FDM) that encodes five operational input dimensions (Q1–Q5) into a Weighted Configuration Compatibility Score (WCCS), producing ranked ML configuration recommendations across nine deployment dimensions, spanning data acquisition and feature extraction to model selection and update strategy. 2) Empirical benchmarks of classical and deep learning models across five experimental tasks: binary API-based malware classification, multi-class API-based classification, class-incremental learning, transfer learning across three heterogeneous datasets, and autoencoder-based feature extraction. Measurements cover test accuracy, AUC, training time, inference speed, peak RAM, and model size (parameter count), enabling direct comparison of model suitability under different operational constraints. 3) A unified FDM validation table mapping each experimental finding to specific FDM input dimensions and recommendation codes, demonstrating that no universal “onesize-fits-all” machine learning setup exists for detecting malware, which is precisely the condition the FDM is designed to address. 4) Three illustrative deployment scenarios that demonstrate how the FDM produces internally consistent, contextappropriate configurations for IoT edge devices, ran-
somware defence systems, and critical-infrastructure endpoints. The remainder of this paper is organised as follows. Sections II and III cover the background concepts and review related work. Section IV introduces the research methodology. Section V details the proposed FDM framework, followed by the experimental evaluation and results in Section VI. Finally, Sections VII and VIII discuss the findings, limitations, and the concluding remarks. II. BACKGROUND In this section, we first provide background knowledge about malware analysis techniques (II-A). Then, we discuss malware detection approaches: signature-based, behaviorbased, and heuristic-based (II-B). After that, we describe the malware signature creation process for traditional antivirus systems (II-C), and the end-point protection maintenance cycle, which is an essential component of the deployment phase to ensure systems can adapt to both known and unknown threats (II-D). A. Malware analysis techniques To examine files and obtain features, software analysis is carried out. The analysis is typically divided into two categories: static and dynamic analysis [14], each with distinct advantages and specific use cases due to the complementary strengths of these approaches, details as follows: 1) Static analysis: When performing a static analysis, one examines the binary structure and metadata of a program without executing it to create a dataset by extracting features such as import/export tables, section headers, API calls, strings, and file metadata from PE headers, and then applies classification methods (such as rules, machine learning models, or signature matching) to these features to determine the potential maliciousness of the file. Alternatively, a software disassembling technique can be used to extract information, such as the opcode and register records, from the assembly code. An example of research that uses static analysis is Gülmez et al. [21], where static analysis is carried out using the Distorm3 disassembler. They extract opcode sequences, convert them into subgraphs, and apply the Random Forest (RF) classification to detect malicious code. 2) Dynamic analysis: When performing dynamic analysis, the program is executed in a virtual environment such as a debugger or a sandbox, and its real-time behavior is analyzed and recorded, such as how it interacts with other programs, what it accesses, network activity, and memory allocation. The record of this can be used to create a dataset that can be used for malware detection. An example of this is a paper by Suaboot et al. [14] where they executed software and malware in a sandbox and monitored the dynamic API calls in real time. The API calls are then exported in order to be used as the dataset to train their Sub-Curve HMM method. B. Malware detection approaches To determine whether a file is malicious or benign, identify malware types (e.g., ransomware, trojan, and spyware),
3
and enable appropriate responses, different malware detection approaches are used. There are three main categories [22]: signature-based, behavior-based, and heuristic detection. Each has distinct strengths and limitations in identifying known and unknown threats as follows. 1) Signature-based method: Signature-based detection identifies malware by matching patterns of programs, or signatures, against a database of known threats, making it effective for real-time detection and generally less resource-intensive compared to other approaches. However, it struggles with variants of known malware and is less effective against unknown threats and zero-day exploits. To address these limitations, YARA [23] provides a more advanced solution by allowing malware researchers to create detailed descriptions of malware based on textual or binary patterns. Each YARA rule includes a set of strings and a boolean expression, offering flexible and enhanced pattern matching that can identify complex variants not captured by traditional signature-based methods. 2) Behavior-based: Behavior-based detection takes a different approach by focusing on the real-time actions of software, allowing it to identify malware based on its behavior rather than relying on known signatures. This method can detect new or previously unseen threats by observing malicious activities, regardless of whether the specific malware variant has been encountered before. Although behavior-based is the most flexible when considering unseen malware, compared to signature and heuristic-based methods, the behavior-based detection can be more resource-intensive and may result in a higher rate of false positives, as legitimate software might exhibit suspicious behaviors. 3) Heuristic-based: In general, heuristic-based detection serves as a more generalized approach compared to signaturebased detection, enabling the identification of variants of known malware, even if these specific versions have not been previously encountered. Heuristic detection uses a plethora of algorithms [24] and rules [25] to evaluate the characteristics and behaviors of applications in order to screen for suspicious activity or sections of an application’s data that would point to malicious intent. Initial Classification
Suspicious program
Signature creation Yara rules
File hash-based Pattern matching
IoCs ML-based
Analysis Static Dynamic Memory forensic
Query
Signature verification failed
VirusTotal Verified Endpoint
Push update patch to endpoint
Adding of signature to db
Verification of the signature
Fig. 1. Processes of creating a signature for traditional signature-based antivirus systems.
C. Process of malware signature creation for traditional antivirus systems When creating malware signatures for updating the endpoint protection, the steps involved are initial classification, analysis, signature creation, signature verification, adding of signature to the database, and pushing an update patch to the endpoint, which is shown in Fig. 1. Step 1: Initial classification: For a program under review or labeled as suspicious, the program’s hash (SHA-1, SHA-256, or MD5) is first checked on platforms such as VirusTotal to determine whether a report already exists. Step 2: Analysis: If a report on the application can not be found on platforms like VirusTotal, then different types of analysis, including Static, Dynamic, and Memory forensics, are conducted on the application. From this analysis, the researcher gathers information on the suspicious activity of the application using tools like a disassembler and a sandbox. Step 3: Signature creation: The information that was gathered is then used to create a signature using differing approaches such as Yara rules, File hash-based, Pattern matching, Indicators of Compromise (IoCs), and Threat Intelligence Feeds. Step 4: Signature verification: The confirmation of the signature’s accuracy in representing the program and that it does not flag the wrong program. If the signature does not accurately represent the program, then in this process, the researcher has to redo the analysis of the program (Step 2). Step 5: Adding of signature to database: Once the signature has been verified then it is added to the database of the systems alongside other signatures. Step 6: Push update patch to endpoint: The signature is then updated on the endpoint device via a patch that is released by the administrator, including future updated signatures from when new variants have emerged. D. Endpoint protection While the previous section detailed the traditional, reactive process of creating individual malware signatures, modern endpoint protection must go further to address rapidly evolving unseen threats. Through an automated maintenance pipeline, modern endpoint protection combines endpoint-based scanning with advanced cloud-based analysis to continuously push security updates to endpoints, ensuring defense against newly detected malware. Fig. 2 illustrates the six-stage deployment and maintenance pipeline. On the protected endpoint, a scanner (i) first checks incoming files against known signatures; on a match, the file is quarantined and the user is alerted. If no signature matches, dynamic behavior detection (ii) scores the sample using a local ML model and, when the score exceeds a threshold, uploads the sample to the vendor’s cloud for deeper analysis. In the cloud, feature extraction (iii) derives static and dynamic feature sets (API calls, PE header fields, memory dumps, network traffic, permissions, behavioral patterns, and signatures)
4
Deployment and Maintenance phase Protected Endpoints (i) Scanner detection 1: If Sig matched
2: quarantine and alert user
3: else
4: analyze behavior
(ii) Dynamic Behavior detection 1: score = Model.detect( behavior )
2: If score exceeds threshold
3: upload sample to Vendor’s cloud
4: else
5: stop
Vendor’s cloud (iv) Identifying and assessing threats
(iii) Feature extraction BEHAVIORAL PATTERN
API CALLS
NETWORK ANALYSIS PE HEADER
PERMISSIONS
SIGNATURE
MEMORY DUMP
+ Explanation
(vi) Endpoint security update
Sig
(v) Database updating
+ Model
Sig
+ Model
Security-update Server
Fig. 2. Modern endpoint protection: automated ML-based deployment and maintenance pipeline.
which feeds into threat identification and assessment End (iv). point from-scratch training. Owoh Cloudet al. [31] propose a hybrid Here, automated analysis and reverse engineering tools help GRU–GAN model for dynamic API call sequences, achieving processing feature withextraction reduced memory overhead compared to analysts identify new threats. After that, the confirmed threats 98.2%Preaccuracy Suspicious program upload Binary filemodels. trigger database updating (v), where new signatures and an standalone deep These studies collectively demonstrate that high detection updated model are committed to the security-update server. Yes Memory dumb Initial Classification diverse configurations. However, Finally, the cycle closes with an endpoint security update (vi) accuracy is achievable across Grayscale image that pushes the updated signature database and model back to each paper optimises within a single combination of feature type, dataset, and model architecture, leaving practitioners all protected endpoints. No Saved Incremental without guidance on which configuration to select when their learning model deployment context differs (in platform, resources, or update III. R ELATED W ORK Result requirements) from the experimental setting. Dataset This section reviews the literature across three areas relevant Model storage to the FDM: (a) ML detection approaches that collectively B. Multi-Criteria Decision Making for ML Configuration Semotivate the need for systematic configuration guidance; lection (b) multi-criteria decision-making (MCDM) methods; and The challenge of selecting among competing ML config(c) context-aware deployment frameworks that inform the urations under multiple, sometimes conflicting criteria has FDM’s design. been addressed in various domains using MCDM methods. Research by Kumar and Kaur [32] applied a fusion of WSM, A. ML Approaches for Malware Detection TOPSIS, and VIKOR to rank algorithms for medical preNumerous papers propose specific ML configurations eval- diction, showing that formalizing evaluation criteria produces uated on narrow benchmark conditions. Bayazit et al. [8] insightful, reproducible recommendations. In the context of evaluate recurrent deep learning models on a static-feature modern deployment, configuration selection must increasAndroid dataset [26] (396 malware, 1,126 benign samples), ingly account for “platform-aware” and “hardware-aware” finding BiLSTM outperforms LSTM, GRU, and RNN. No- constraints [33]. Recent taxonomies in Neural Architecture tably, the paper dismisses classical ML without empirical Search have shifted focus toward multi-objective optimization memory constraints, energy consumption, comparison, illustrating the fragmented nature of the literature. that treats latency, Processes of developing transfer learning model Kalash et al. [27] demonstrate that CNN-based image classi- and accuracy as primary objectives [34], [35]. This is essential fication achieves 98.52% accuracy on Malimg and 99.97% for “Green Machine Learning,” where the environmental cost on the Microsoft dataset, establishing image-based malware of a model is weighed against its classification effectiveness Finish visualisation as a viable detection feature. Zhangjie et al. to Begin ensure efficient deployment [36]. As Menghani [37] high[28] augment LSTM training with GAN-generated malware lights, practitioners must identify “Pareto-optimal” models that samples, reaching 99.94% on augmented data but 86.5% on provide the highest possible accuracy under specific resource novel real-world samples; this gap highlights overfitting risk constraints, such as a maximum RAM limit or execution time. when the training distribution does not match deployment. On the other hand, cross-level optimization surveys [38] highWong et al. [29] combine ShuffleNet and DenseNet-201 for light that achieving resource efficiency in IoT systems requires feature extraction with SVM classification across four im- selecting configurations that align with the limitations of the age datasets, achieving 85.79–99.14% accuracy depending on deployment hardware. Furthermore, the application of multidataset complexity. Sudhakar et al. [30] fine-tune ResNet50’s criteria decision-making frameworks [39] in adjacent domains, final layer on Malimg (99.18% accuracy, 5.14 ms inference), such as precision marketing, has demonstrated the effectivedemonstrating that lightweight transfer learning can match ness of translating qualitative operational requirements into Malare dataset
Obtain pretrained model from repository (e.g. Non-malare)
Setup initial hyper parameters
Trainretrain the ne model
Valid ation
Unsatisfied
Fine tune hyper parameters
Satisfy
Mbased uery Signature verification failed
5 irusotal
quantifiable evaluation scales to systematically guide complex selection processes. The FDM differs from these generic MCDM applications by providing a specialized framework for malware detection, where input dimensions (Q1–Q5) and recommendation codes (P1–P5) are grounded in empirical experiments. Unlike general weighting schemes, the WCCS scoring enables finergrained alignment between a practitioner’s operational context, such as limited hardware or strict update requirements, and the resulting configuration recommendations. C. Context-Aware and Adaptive ML Deployment
RQ4: How effectively does class-incremental learning alerified low models to integrate new malware families without full retraining, and which model ush type best resists catastrophic Adding of erification of update ndpoint signature to the signature patch to forgetting? db endpoint • RQ5: Can autoencoder-based feature extraction reduce inference latency and model complexity with minimal accuracy cost, and under what operational constraints is this trade-off justified? •
The experimental answers to RQ1–RQ5 are synthesised in Section VI-F to produce a validated FDM configuration mapping, demonstrating that the FDM’s WCCS scores correctly predict which configurations dominate under each set of operational constraints. egin
Existing work highlights the requirement of adapting deployed malware detection systems to varying operational contexts. For example, systems like CASANDRA [40] employ online learning to address population drift, while others [41] focus on strict hardware constraints, such as optimizing models for varying RAM capacities or deploying “TinyML” configurations for IoT environments [42]. Recent frameworks extend this to dynamic runtime adaptation, balancing accuracy with end-to-end latency [43], [44] or switching architectures to maintain energy efficiency (e.g., EcoMLS [45]). While adaptive systems effectively manage runtime conditions, they assume that the foundational model architecture, feature type, and training strategy have already been established. The FDM is complementary to these adaptive systems: it targets the configuration selection phase that precedes deployment. By quantitatively mapping operational contexts to specific pre-deployment strategies, the FDM provides the principled baseline choices required by systems like CASANDRA and EcoMLS, addressing the configuration gap they leave open.
Set oal bective of classification method
V. P ROPOSED F RAMEWORK FOR D ECISION - MAKING TO BUILD ML- BASED M ALWARE DETECTION SYSTEMS (FDM) Acquisitio Feature etraction
This section proposes the FDM, which incorporates a quantifiable recommendation algorithm that maps the practitioner’s operational goals (expressed as numeric input parameters) to a ranked set of technical configuration recommendations. The proposed algorithm applies a Weighted Configuration Compatibility Score (WCCS), a mathematically grounded multicriteria scoring method [39], to measure how well each candidate configuration matches the stated operational requirements. Analysi M model hyper parameter tuning
Finish
rocesses of developing incremental learning model
c
c
c
c1
c1
M1
M
M2 c2
c2 c1
c3
A. System Development Workflow In general, to develop an ML-based method for malware detection, researchers need to make a series of decisions that impact the overall efficiency and effectiveness of the method. Fig. 3 shows the basic decision-making process of developing ML-based malware detection systems.
IV. R ESEARCH M ETHODOLOGY This study follows a two-phase design. In the first phase, the FDM is conceptually developed: operational input dimensions (Q1–Q5) are identified from a systematic analysis of the factors that practitioners report as most influential in malware detection system design, and the WCCS scoring mechanism is formalised following established multi-criteria decision analysis principles [39]. In the second phase, four empirical experiments on three labelled datasets are conducted to validate the FDM’s recommendations and provide the evidence base for its configuration codes. The study is guided by five research questions: • RQ1: How do classical (RF, XGBoost, SVM) and deep learning (LSTM, BiLSTM) models compare in binary API-based malware classification when long sequences are standardised by splitting into fixed-length segments? • RQ2: Does the relative ranking of classical and deep models change in the more demanding multi-class setting, and what explains any reversal? • RQ3: What are the training-time and accuracy costs and benefits of transfer learning (ImageNet pre-training) versus training from scratch, and do these vary with input modality?
MaintenanceUtility after initial classification
Deploymen Usecase options
Start
Define the malware detection implementation constraints
1
Data set acquisition, labeling and preprocessing Malware detection systems deployment
2
Assess alternative malware detection methods against constraints
4
Malware detection model selection,development and tunning
Update Malware detection systems maintenance
3
Not satisfied
Model performance evaluation
Satisfied
Finish
Fig. 3. ML-Based Malware Detection Development Lifecycle.
1) Requirement gathering and analysis: Similar to software development, when designing a malware detection system, it is crucial to gather requirements, constraints and define goals the organization wants to achieve. This process is crucial as it affects many of the decisions in the rest of the steps. This step is shown in Fig. 3, 1 .
6
2) Dataset acquisition: In Fig. 3, 2 , after requirement gathering and analysis, the next process is data acquisition, which encompasses feature extraction and engineering, feature scaling and selection, and data processing and shaping to obtain a usable dataset. The process involves: (1) Sample collection: gathering malware samples and benign files from sources such as VirusTotal and malware repositories; (2) Preprocessing: removing irrelevant features, eliminating redundancy, and normalizing data to ensure consistency; (3) Feature extraction: choosing the analysis approach (static, dynamic, or hybrid) based on classification goals; (4) Feature representation: converting extracted features into graph-based, vectorized, numerical, or embedded representations; and (5) Feature selection: using statistical and ML techniques such as correlation analysis, PCA, and Chi-square tests to retain the most discriminative features. 3) Machine learning model development and evaluation: Fig. 3, 3 . Setting up ML models is critical for malware classification because the decisions made during development substantially affect classification performance. The process comprises: data loading and labeling; pre-processing raw data for efficient model ingestion; dataset splitting for training, validation, and testing; algorithm selection aligned with research objectives and dataset characteristics; model training; hyperparameter optimization using automated tools such as Optuna [46]; and model evaluation using accuracy, precision, recall, F1-score, and ROC-AUC. However, if the model’s performance is not satisfactory, the developer could try alternative ML models until the desired evaluation criteria are met. 4) Deployment and maintenance: Once the developer has successfully built the ML model for their specific need, it is deployed in a real environment. Here, deployment strategies differ depending on the target platform, computational resources, and real-time detection requirements. Three primary strategies exist: on-device deployment, where the detection method runs locally on endpoint hardware (e.g., Windows, Android) for real-time classification; cloud-based deployment, where samples are uploaded to cloud servers when on-device resources are insufficient; and hybrid deployment, combining both. Ongoing maintenance involves periodic model retraining on updated datasets. Depending on the organization’s requirements, various malware signature update strategies will be determined to make sure the system is resilient against emerging evasion techniques. B. FDM Components This section describes the components of the proposed FDM. Fig. 4 illustrates the decision flow connecting all components through the FDM algorithms to determine the following five key configuration categories (DS, FT, ML, DP, and MA). 1) Dataset Sources (DS): • DS1 – Secondary data, pre-reports and labels: Sources providing reports of seen payloads, which consist of labels and a behavioral report. This might include raw behaviors such as list of API calls, mutex objects, APK permissions and so on, which are secondary information
Start
Determine DS
DS4
Unseen payload
Signature, IOC and behavioral analysis FT2-FT4
DS1,DS3
Pre-analyzed report
DS1,DS2
Known-payload & label
Automated analysis: Static/Dynamic Determine Feature engineering methods
FT2-FT4
FT1-FT5 Determine ML-models
ML1-2; DP1-2; MA1-2 Training & Testing
Harvest New Malware
Model fine-tuning
Deployment Finish
Fig. 4. Decision-making flow of the FDM, showing the ML-based malware detection system development process. Each node represents a configuration option: DS=Dataset Source, FT=Feature Type, ML=ML Model, DP=Data Preprocessing approach, MA=Model Maintenance technique.
of the malware. VirusTotal [10] is the primary example; it is used for labeling and behavioral pre-analysis. Dadkhah et al. [47] provide multiple layers of malware behaviors including network traffic, system calls, and CPU utilization. This type of data source is safe and could be effective to develop efficient malware detection models as it combines multiple views of malware. • DS2 – Actual payloads and labels: Sources providing a wide variety of pre-labeled file types (e.g., .exe, .pdf, .xlsm) and scripts (e.g., .vbs, .js, .ps1). VirusShare [48] is the primary example, providing a private repository of pre-labeled malware samples. Although this data source is the most dangerous (sophisticated malware could escape from an improperly configured environment), it is the most effective way for a malware analyst to investigate payload behaviors in depth and discover more comprehensive characteristics. • DS3 – Labeled features: This type is the most convenient data source for an AI engineer rather than a specialized malware analyst. It provides features extracted from processed samples (e.g., BODMAS [49], EMBER [50], and other datasets publicly available in Kaggle and Github), enabling researchers to perform additional feature engineering. • DS4 – Unseen payloads: An in-house harvesting of unknown payloads with indicators of compromise (IoC) collected from endpoint devices. These samples require dataset preparation before integration. This data source requires a malware analyst to be involved in building a
7
signature, IoC indicators, and a behavioral report. 2) Feature Types (FT): FT1 – Image-based features: Binary files converted into images, enabling the use of transfer learning from pretrained models, specifically, those that are designed to extract and process visual features, such as CNN and autoencoders. • FT2 – Sequence features : Behavioral sequences extracted via dynamic analysis, capturing API call patterns and temporal correlations. • FT3 – Graph features: A Control Flow Graph (CFG) is a graph-based representation of function-call relationships, suited to graph neural network (GNN) classifiers. CFGs are usually extracted from static analysis, e.g., by decompiling the executable into high-level code from which the CFG is generated. • FT4 – Static features: Various features such as PE headers, byte histograms, and printable strings extracted directly from binaries without execution, which can be obtained from static analysis procedures. • FT5 – Latent features: When using an autoencoder (e.g. [51]) to process malware data (like API call sequences, byte frequencies, or image-converted binaries), the features extracted from the middle layer are latent features or encoded features. The latent feature is the compressed, hidden variables that capture the underlying structure of the malware data. •
3) Machine Learning Models (ML): ML1 – Deep learning: CNN architectures (EfficientNetB0, MobileNetV2) and recurrent models (LSTM, BiLSTM) for image and sequence classification, respectively. • ML2 – Classical and ensemble ML: RF, XGBoost, SVM and voting classifiers; preferred when computational resources are limited. •
4) Data Pre-processing Approaches (DP): DP1 – Sequence standardization: Partitioning variablelength data streams or sequences into fixed-length segments. This establishes a uniform input format required by various sequence-based deep and non-deep ML techniques. • DP2 – Dimensionality reduction: Utilizing feature compression techniques (e.g., autoencoder, PCA and t-SNE) to map high-dimensional inputs into a compact latent representation. This significantly reduces downstream computational complexity and testtime inference latency, making the pipeline highly suitable for deployments with strict resource constraints. •
5) Model Maintenance Techniques (MA): •
MA1 – Transfer learning: Adapts model weights pretrained on large-scale external datasets (e.g., ImageNet) to new malware data distributions. This approach serves as a rapid model initialisation strategy when labelled malware data is scarce, and acts as an efficient periodic updating mechanism to significantly reduce retraining time and computational resource consumption.
•
MA2 – Class-incremental learning: Facilitates the continuous addition of new malware families to an existing model without requiring full retraining. This strategy governs both the initial model training for extensibility and the subsequent updating mechanism, effectively mitigating catastrophic forgetting while retaining prior knowledge of older threats.
C. Input and Output The FDM defines system inputs (i.e., system’s requirements) as five quantifiable parameters (i.e., Q1–Q5), and nine recommendations for configurable aspects of the malware detection system, including development and deployment, i.e., as framework’s outputs. Each requirement of the system maps a practitioner’s operational goal to a numeric or ordinal scale, enabling the WCCS algorithm to compute a mathematically comparable score across all configuration candidates. Tables I and II define the input and output of the FDM, details as follows. Q1 – Platform Constraint Index (PCI): Captures the capability and resource limits of the target deployment platform, scored as 1 = mobile/edge device (e.g., Android gateway), 2 = server or local cloud, and 3 = unrestricted high-performance cluster. It constrains the feasible set of ML models (P2.1) and feature extraction methods (P3.1): edge devices (PCI=1) require lightweight models (RS=1), whereas unrestricted clusters (PCI=3) can accommodate deep CNN architectures (RS=3). Q2 – Resource Budget Index (RBI): Represents the available memory, compute, and GPU resources for training and inference, scored as 1 = constrained (<4 GB RAM, CPU-only), 2 = moderate (4–16 GB RAM, optional GPU), and 3 = high-performance (>16 GB RAM, dedicated GPU). It governs model complexity and feature dimensionality: constrained deployments (RBI=1) favour low-dimensional features and classical ML, whereas high-performance environments (RBI=3) enable deep learning and high-dimensional feature spaces. Q3 – Response Latency Level (RLL): Specifies the required inference speed for the deployment use-case, scored as 1 = offline/batch (hours acceptable), 2 = near-real-time (seconds acceptable), and 3 = strict real-time (<100 ms required). It determines the permissible feature extraction method: strict real-time requirements (RLL=3) mandate static analysis (ET=1) without sandboxing overhead, whereas offline deployments (RLL=1) can accommodate dynamic or hybrid analysis (ET=2 or 3). Q4 – Update Frequency Score (UFS): Defines the required cadence for retraining or updating the model and dataset, scored as 1 = one-time (interval >3 months), 2 = periodic (weekly to monthly), and 3 = continuous (daily or event-triggered). It drives data acquisition (P1) and the model updating strategy (P4): continuousupdate deployments (UFS=3) require in-house or private data collection and class-incremental learning (MA2), whereas one-time deployments (UFS=1) can rely on public datasets.
8
TABLE I F RAMEWORK ’ S I NPUTS : D EFINITION AND S CALE OF THE Q UANTIFIABLE PARAMETERS Param. Name Q1 Platform Constraint Index (PCI)
Description Capability and resource limits of the target deployment platform.
Q2
Resource Budget Index (RBI)
Available memory, compute, and GPU resources for training and inference.
Q3
Response Latency Level (RLL)
Required inference speed for the deployment use-case.
Q4
Update Frequency Score (UFS)
Required cadence for retraining or updating the model and dataset.
Q5
Sensitivity Ratio (SR)
Relative operational cost of a false negative vs. a false positive; SR = Cost(FN)/Cost(FP).
Q5 – Sensitivity Ratio (SR): Quantifies the relative operational cost of a false negative versus a false positive, defined as SR = Cost(FN)/Cost(FP) and graded as low (SR < 1.0, FP-sensitive), medium (1.0 ≤ SR ≤ 3.0, balanced), and high (SR > 3.0, FN-sensitive). High-SR environments, such as critical infrastructure, prioritise recall and trigger full preprocessing pipelines, whereas low-SR environments prioritise precision. The recommendation codes (P1–P5) in Table II, column Quantifiable Scale, suggests the system’s configurations based on the value calculated from the compatibility functions detailed in the following Section V-D. D. Weighted Configuration Compatibility Scoring (WCCS) The WCCS formalizes the mapping from operational goals to ML configuration choices. For each recommendation dimension k (k ∈ {P1, P2.1, P2.2, P2.3, P3.1, P3.2, P3.3, P4, P5}) and each candidate configuration option o ∈ Ωk , the WCCS is defined as: WCCSk (o) =
5 X
wkj · ckj (Qj , o)
(1)
j=1
where WCCSk (o) ∈ [0, 1] is the overall compatibility score that option o achieves for recommendation area k, with higher values indicating a better-suited option; wkj ∈ [0, 1] is the importance weight P of input dimension Qj for recommendation area k (with j wkj = 1), and ckj (Qj , o) ∈ [0, 1] is the compatibility of option o with requirement value Qj . The set Ωk comprises the candidate configuration options considered for area k (for instance, the candidate ML models for P2.1), and o denotes an individual option drawn from this set. Intuitively, Eq. 1 scores each candidate option as a weighted average of its compatibility with the five operational requirements. The compatibility term ckj (Qj , o) ∈ [0, 1] measures
Scale / Levels 1 = Mobile/Edge device (e.g., Android gateway) 2 = Server or local cloud 3 = Unrestricted high-performance cluster 1 = Constrained (<4 GB RAM, CPUonly) 2 = Moderate (4–16 GB RAM, optional GPU) 3 = High-performance (>16 GB RAM, dedicated GPU) 1 = Offline/batch (hours acceptable) 2 = Near-real-time (seconds acceptable) 3 = Strict real-time (<100 ms required) 1 = One-time (interval >3 months) 2 = Periodic (weekly to monthly) 3 = Continuous (daily or eventtriggered) Low: SR < 1.0 (FP-sensitive) Medium: 1.0 ≤ SR ≤ 3.0 (balanced) High: SR > 3.0 (FN-sensitive)
how well option o meets the j-th requirement, ranging from a complete mismatch (0) to a perfect fit (1), while the weight wkj expresses the relative importance of that requirement for decision k. Because the weights are normalised to sum to one, more relevant requirements contribute proportionally more to the total, so an option that aligns with the higherweighted requirements can outrank one that is strong on only a peripheral criterion. The recommended configuration for area k is then: o∗k = argmax WCCSk (o)
(2)
o∈Ωk
where o∗k denotes the recommended option for area k, that is, the candidate in Ωk that maximises the WCCS. Eq. 2 then selects, for each decision k, the option that attains the highest score; the argmax operator returns the maximising option o ∈ Ωk itself rather than the score value. Performing this selection across all decision areas (P1–P5) yields the complete configuration that the FDM recommends for a given requirement vector Q. The compatibility function ckj (Qj , o) encodes domain knowledge: for numeric dimensions Q1–Q4, it uses a piecewise linear scoring function that assigns 1.0 when an option perfectly suits the requirement level and decreases linearly toward 0.0 as the mismatch grows. For Q5 (SR), a monotonic sigmoid-like mapping is applied so that recall-oriented models score highly when SR is large, while precision-oriented configurations are preferred when SR is small. This approach is more robust than simple if-else branching because it naturally handles partial matches: a configuration that is not optimal on every dimension but scores highly overall may be more practical than one optimal on only one dimension. Table III presents the importance weight matrix wkj . The dominant weights reflect the primary driver of each recommendation: Q4 (UFS) dominates P4 (w = 0.70) since update frequency directly determines the updating strategy; Q5 (SR) dominates P5 (w = 0.75) since the sensitivity ratio directly
9
TABLE II F RAMEWORK ’ S O UTPUTS : R ECOMMENDATION C ODES AND Q UANTIFIABLE S CALES
Code P1
Name Data acquisition
Detail Suggest the dataset source based on update cadence and sensitivity requirements.
Options Pre-reported (DS1), Raw malware with labels (DS2), Labelled features (DS3), In-house (DS4) P2: ML model fine-tunings Ranked list of ML models by WCCS2.1 RF, XGBoost, CNN, compatibility score. LSTM, BiLSTM, GNN
P2.1
ML model
P2.2
Hyperparameter auto-tuning
Recommend HPO tool based on model complexity and resource budget.
Optuna (default), Ray Tune, Hyperopt
P2.3
Fixed hyperparameters
Specify hyperparameters to fix to ensure reproducibility.
Epoch count, sampling rate
P3.1
Feature extraction
P3.2
Pre-processing
P3.3
Feature selection
P4
Updating priority
P5
Detection sensitivity
P3: Data Pre-processing Specify static, dynamic, or hybrid exStatic (ET=1), Dynamic traction based on latency and infrastruc(ET=2), Hybrid (ET=3) ture. Recommend pre-processing stages: Cleansing, cleansing, normalization, and EDA. normalization, EDA
Specify features from the dataset most discriminative for the target malware families. Determine model/dataset refresh frequency and updating mechanism.
API calls, opcodes, PE headers, byte histograms, strings Continuous (MA2), Periodic (MA1), One-time
Determine the acceptable FP–FN tradeoff and HPO objective.
Recall-focused, Balanced, Precisionfocused
governs the FP–FN trade-off objective; and Q1 (PCI) is the strongest driver of P2.1 (w = 0.30) because the deployment platform immediately constrains the feasible model set. The remaining weights are calibrated to the empirically validated input–output associations reported in Table XIV: Q1 and Q2 jointly drive model selection (P2.1); Q3 (latency) and Q1 (platform) govern feature extraction (P3.1); Q2 and Q3 govern pre-processing (P3.2); Q2 constrains feature selection (P3.3); and Q4, with Q2 as a secondary factor, drives the update strategy (P4). E. Decision-Making Algorithms The FDM procedure is split into two single-column algorithms that together implement the WCCS formulation (Eq. 1–2). Algorithm 1 covers data and feature configuration (Steps 1–3): it selects the dataset source, feature extraction method, and preprocessing pipeline driven primarily by Q3 , Q4 , and Q5 , and outputs the selected feature type FT for use in Algorithm 2. Algorithm 2 covers model selection and deployment (Steps 4–8): Step 4 applies the full WCCS ranking to select ML models; Steps 5–6 determine the training strategy and hyperparameter search space; Steps 7–8 apply direct threshold mappings to set the update mechanism and the FP– FN trade-off objective. Each step maps to a node cluster in Fig. 4.
Quantifiable Scale Acquisition Level (AL): 1=Public, 2=Private repo, 3=In-house collection. Driven by Q4 (UFS, w=0.45).
Resource Score (RS): 1=Low (RF, XGBoost), 2=Medium (LSTM, BiLSTM), 3=High (CNN). Output: top-k ranked list. Search Space Index (SSI): 1=Small (<10 params), 2=Medium (10–30), 3=Large (>30). Mapped from Q2 and model RS. Epoch count: integer ∈ [10, 200]; sampling rate: float ∈ [0.1, 1.0]. Values determined by Q3 and dataset size. Extraction Type (ET) ∈ {1, 2, 3}. Constrained by Q1 and Q3: Q1=1 (Edge) or Q3=3 → ET=1; otherwise ET ∈ {2, 3}. Pipeline Stage Count (PSC): 1=Minimal, 2=Standard, 3=Full pipeline. Driven by Q3 and Q2 (autoencoder pre-processing under tight latency/budget); high Q5 (SR>3.0) → PSC=3. Feature Dimensionality (FD): Low (<100), Medium (100–1,000), High (>1,000). Constrained by Q2 (RBI, w=0.40). Update Interval (UI) in days: UI≤7 (continuous), 7<UI≤30 (periodic), UI>90 (onetime). Driven by Q4 (w=0.70). Sensitivity Ratio SR=Cost(FN)/Cost(FP): SR>3.0 → recall objective; 1.0 ≤ SR ≤ 3.0 → F1; SR<1.0 → precision. Dominant weight Q5 (w=0.75).
F. Example Deployment Scenarios To demonstrate the practical utility of the WCCS algorithm, Table IV presents the complete FDM recommendation set for three representative real-world deployment scenarios: (i) securing an IoT network, (ii) protecting import-export business endpoints from ransomware, and (iii) safeguarding critical electric power infrastructure. Each scenario yields a distinct input vector Q and a correspondingly different but internally consistent configuration recommendation, illustrating how the framework adapts to diverse operational contexts. The three scenarios demonstrate that the WCCS algorithm produces coherent, contextually appropriate configurations. In Scenario 1, the tight resource and latency constraints (Q1=1, Q2=1, Q3=3) drive the framework toward lightweight, staticanalysis-based classifiers (RF/XGBoost on PE features) with a minimal preprocessing pipeline. In Scenario 2, the combination of high sensitivity (SR≈4.5) and continuous update requirements (Q4=3) selects LSTM with API sequences, a full preprocessing pipeline, daily class-incremental updates, and a recall-maximizing HPO objective. In Scenario 3, both constraints further intensify (SR≈8.0, Q1=3), enabling the most resource-intensive configuration: CNN with image-based features, full pipeline, and continuous CIL updates, all optimized for maximum recall to minimize the risk of catastrophic missed detections.
10
TABLE III WCCS I MPORTANCE W EIGHT M ATRIX (wkj ); EACH RECOMMENDATION ( COLUMN ) SUMS TO 1.00. Input Q1 (PCI) Q2 (RBI) Q3 (RLL) Q4 (UFS) Q5 (SR) Total
P1 Data Acq. 0.10 0.10 0.15 0.45 0.20 1.00
P2.1 ML Model 0.30 0.25 0.15 0.15 0.15 1.00
P2.2 HPO 0.25 0.40 0.10 0.10 0.15 1.00
P2.3 Fixed HP 0.10 0.20 0.40 0.15 0.15 1.00
P3.1 Feat. Ext. 0.30 0.10 0.40 0.10 0.10 1.00
P3.2 Pre-proc. 0.05 0.30 0.35 0.05 0.25 1.00
P3.3 Feat. Sel. 0.10 0.40 0.20 0.10 0.20 1.00
P4 Update 0.05 0.15 0.05 0.70 0.05 1.00
P5 Sensitivity 0.00 0.05 0.10 0.10 0.75 1.00
TABLE IV FDM C ONFIGURATION R ECOMMENDATIONS FOR T HREE E XAMPLE S CENARIOS
Parameter
Scenario 1: IoT Network Security
Deployment context
P3.2 Pre-proc.
Fleet of IoT gateways/sensors on a factory floor; malware on any device disrupts production. 1 – Mobile/Edge (IoT gateways) 1 – Constrained (<4 GB, CPU) 3 – Real-time (<100 ms) 2 – Periodic (weekly) Medium (SR≈2.0) Public + VirusShare (AL=1–2); periodic refresh RF / XGBoost (RS=1); lightweight for edge HW Optuna SSI=1 (small search space) Epoch: 50–100; rate: 0.5 Static (ET=1); FT4 (PE headers, byte hist.) Minimal (PSC=1): cleansing
P3.3 Feature Sel.
FD=Low (<100): PE headers, strings
P4 Update Priority (Model Maintenance) P5 Sensitivity
Q1 (PCI) Q2 (RBI) Q3 (RLL) Q4 (UFS) Q5 (SR) P1 Data Acq. P2.1 ML Model P2.2 HPO P2.3 Fixed HP P3.1 Feature Ext.
Scenario 3: Critical Infrastructure (Electric Power Distribution) SCADA/ICS systems controlling power distribution; a missed detection risks grid outage. 3 – Unrestricted on-premise 3 – High (>16 GB, GPU) 3 – Real-time (<100 ms) 3 – Continuous (daily) High (SR≈8.0) In-house ICS samples + VirusShare (AL=3); classified dataset EfficientNetB0 CNN (RS=3); highest accuracy for binary image analysis Optuna SSI=3 (large search space) Epoch: 150–200; rate: 1.0 Static (ET=1); FT1 (binary image)
MA1 (TL Fine-Tuning); UI=7–30 days (Periodic)
Scenario 2: Endpoint Ransomware Defence (Import-Export Business) Windows workstations and file servers; ransomware encrypts shipment data irreversibly. 2 – Server/cloud 2 – Moderate (4–16 GB, opt. GPU) 2 – Near-real-time (seconds) 3 – Continuous (daily) High (SR≈4.5) VirusShare + in-house captures (AL=2– 3); daily pipeline LSTM (RS=2); captures ransomware API sequences Optuna SSI=2 (medium search space) Epoch: 100–150; rate: 0.3 Dynamic (ET=2); FT2 (API call sequences) Full (PSC=3): cleansing + norm. + EDA FD=Medium (100–1,000): API ngrams, opcodes MA2 (Class-Incremental); UI≤7 days (Continuous)
Balanced (F1 objective); SR≈2.0
Recall-focused (minimise FN); SR≈4.5
Recall-focused (minimise FN); SR≈8.0
VI. E XPERIMENTS AND F RAMEWORK VALIDATION All experiments were conducted on a custom desktop workstation (13th Gen Intel Core i5-13500, 32 GB RAM, GPU NVIDIA GeForce RTX 4060 with VRAM 8GB GDDR6) running Ubuntu 22.04.2 LTS. Pre-processing pipelines were implemented in MATLAB R2022b; model training and evaluation used Python 3 with Pandas, Scikit-learn, and Keras. Three labelled datasets were employed throughout: 1) Dynamic API call dataset [14]: a private collection of Windows API call sequences captured by dynamic execution, covering eight malware families (Hivecoin, Ramnit, FakeAV, Lokibot, Ransom, Rootkit, Keylogger, Zeus) and a benign class. Sequences exceeding 1,000 API calls are segmented into fixed-length subsequences of exactly 1,000 calls. 2) Malimg dataset [7]: a public benchmark of 9,339 malware binary files rendered as 32×32 grayscale images, drawn from 25 distinct families. 3) Android static API dataset [17]: a public Android dataset containing 15,036 applications (5,560 malware, 9,476 benign), represented by permission vectors extracted from each APK’s manifest file. All classical machine-learning models were hyperparameteroptimised via Optuna [46] (30–100 trials depending on the
Full (PSC=3): cleansing + norm. + EDA FD=High (>1,000): binary image pixels, PE histogram MA2 (Class-Incremental); UI≤7 days (Continuous)
experiment). Deep learning models were trained with early stopping (patience = 5 epochs). Hardware and software configurations were held constant across all experiments to ensure reproducible comparisons.
A. Experiment 1: API-Based Malware Classification with Sequence Splitting This experiment evaluates sequence splitting as a preprocessing strategy for Windows API call–based malware classification, benchmarking both classical machine learning (ML) classifiers and deep learning sequence models under identical conditions. Splitting long traces into fixed-length subsequences of 1,000 calls is consistent with prior work showing that segmented sequences yield superior learning signals compared to truncated or zero-padded alternatives [52]. Two classification tasks are examined: (i) binary classification (malware family vs. benign Windows processes) and (ii) multiclass classification (all families simultaneously). 1) Binary Classification: Each of the eight malware families was evaluated independently against benign Windows processes. Three classical models, namely Random Forest (RF), XGBoost, and SVM, were compared against two recurrent models, LSTM and Bidirectional LSTM (BiLSTM).
11
Algorithm 1 FDM Phase 1: Data & Feature Configuration Require: Q1 –Q5 (PCI, RBI, RLL, UFS, SR) Ensure: P1, P3.1, P3.2, P3.3, DP, FT 1: // Step 1: Data Acquisition (P1) 2: if Q4 = 1 then 3: P1 ← DS3 (public dataset; AL = 1) 4: else if Q4 = 2 then 5: P1 ← DS3 or DS2 (AL = 1–2) 6: else 7: P1 ← DS2 or DS4 (in-house; AL = 2–3) 8: end if 9: if Q5 > 3.0 then ▷ high sensitivity override 10: P1 ← DS4 (AL = 3) 11: end if 12: // Step 2: Feature Extraction (P3.1) constrained by platforms 13: if Q1 = 1 or Q3 = 3 then ▷ Edge dev. OR strict real-time 14: FT ← FT1 or FT4; P3.1: ET = 1 (Static) 15: else if Q3 = 2 then 16: FT ← FT1, FT2, or FT4; P3.1: ET = 1–2 17: else 18: FT ← FT2, FT3, or FT5; P3.1: ET = 2–3 19: end if 20: // Step 3: Pre-processing & Feature Selection 21: P3.2: PSC ← Q2 ; if Q5 > 3.0 then PSC ← 3 22: P3.3: FD ← Low(Q2 =1) / Med(Q2 =2) / High(Q2 =3) 23: if F T = F T 2 then 24: DP ← DP1 (Sequence standardization) 25: else if Q2 = 1 then 26: DP ← DP2 (Dimensionality reduction) 27: end if
Algorithm 2 FDM Phase 2: Model development & Deployment Require: Q2 –Q5 ; FT, DP (from Alg. 1) Ensure: P2.1, P2.2, P2.3, P4, P5; full set R 1: // Step 4: ML Model Selection (P2.1) via WCCS 2: for all o ∈ {RF, XGBoost, LSTM, BiLSTM, CNN, GNN} do P5 3: WCCS2.1 (o) ← j=1 w2.1,j · c2.1,j (Qj , o) (Eq. 1) 4: end for 5: P2.1 ← top-k models by WCCS2.1 (descending) 6: // Step 5: Hyperparameter Optimisation (P2.2, P2.3) 7: SSI ← RS of P2.1 model ▷ 1=small, 2=med, 3=large 8: P2.2 ← Optuna (SSI); P2.3: epoch ∈ [10, 200], rate ∈ [0.1, 1.0] 9: // Step 6: Updating Priority (P4) 10: if Q4 = 3 then 11: P4 ← MA2 (Class-Incr.); UI ≤ 7 days 12: else if Q4 = 2 then 13: P4 ← MA1 (TL fine-tune); UI = 7–30 days 14: else 15: P4 ← none; UI > 90 days 16: end if 17: // Step 7: Detection Sensitivity (P5) 18: if Q5 > 3.0 then 19: P5 ← Recall-focused; HPO obj. = Recall 20: else if Q5 ≥ 1.0 then 21: P5 ← Balanced; HPO obj. = F1 22: else 23: P5 ← Precision-focused; HPO obj. = Precision 24: end if 25: return R = {P1, P3.1–P3.3, DP, P2.1, P2.2,} 26: P2.3, P4, P5}
Tables V and VI summarise per-family and aggregated performance. Key metrics are test accuracy, AUC, and peak RAM usage. Analysis. XGBoost achieved the highest mean test accuracy across binary tasks (97.46%) with a near-perfect mean AUC of 99.66%, outperforming all other models including deep learning architectures. Random Forest followed closely (97.06% accuracy, 99.52% AUC) while consuming negligible RAM (<20 MB). SVM, although fast to train and memorylean, produced the lowest mean accuracy (94.88%) and AUC (94.31%), particularly struggling on the Ramnit family where its AUC fell to 87.76%. Among deep learning models, LSTM achieved a slightly higher mean test accuracy (96.55%) than BiLSTM (96.49%), though BiLSTM produced a higher mean AUC (99.25% vs. 99.09%), reflecting better probabilistic calibration. Both recurrent models required 1.7–2.8 GB peak RAM (roughly 40× or more than the classical models), with training times of 6–25 minutes per family, making them unsuitable for resourceconstrained endpoints. These findings directly inform FDM input dimension Q2 (Resource Budget Index): for deployments with limited RAM or CPU, XGBoost provides the best accuracy-to-resource ratio while also attaining the highest AUC overall. Among the deep
models, BiLSTM yields the best AUC (99.25%); however, it does not surpass XGBoost on either accuracy or AUC and is justified only in resource-rich deployments that specifically require recurrent modelling of sequential features. 2) Multi-Class Classification: In this experiment we extend the evaluation to simultaneous classification of all malware families (multi-class setting). The same sequence-splitting pre-processing and Optuna hyperparameter optimisation are applied. This represents the more realistic operational scenario where a deployed system must distinguish among all known families without separate binary detectors. Tables VII and VIII report results on the combined dataset (train: 17,795 / val: 5,932 / test: 5,932 sequences). Analysis. Multi-class classification reveals a marked drop in accuracy for all models compared to binary tasks. XGBoost retains the lead at 79.03% test accuracy, while both LSTM and BiLSTM fall below 74%, contrary to their strong binary performance. This reversal, in which classical models outperform deep sequence models in the multi-class case, is attributed to the greater structural diversity of API call patterns across nine simultaneous classes, which demands feature representations that sequence models have insufficient capacity to discriminate without significantly larger training corpora. For the FDM, this finding informs recommendation
12
TABLE V B INARY MALWARE CLASSIFICATION : CLASSICAL ML MODELS (RF, XGB OOST, SVM). B EST VALUE AMONG RF, XGB OOST, AND SVM PER FAMILY ( AND OVERALL MEAN ) IN BOLD FOR VALIDATION , TEST ACCURACY, AND AUC.
Model
Malware Family
Random Forest
XGBoost
SVM
Val Acc (%)
Test Acc (%)
AUC (%)
Train Time (min)
RAM (MB)
FakeAV Hivecoin Keylogger Lokibot Ramnit Ransom Rootkit Zeus
96.79 98.87 98.46 97.78 98.71 97.31 97.56 97.43
95.91 98.26 97.64 97.03 97.81 96.61 96.30 96.91
99.55 99.91 99.64 99.65 99.42 98.69 99.48 99.80
1.82 1.18 3.34 1.32 1.86 5.28 2.09 3.18
16.56 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Mean
97.86
97.06
99.52
2.51
2.07
FakeAV Hivecoin Keylogger Lokibot Ramnit Ransom Rootkit Zeus
97.27 99.18 98.32 97.86 99.10 97.66 97.82 97.75
96.69 98.56 97.84 96.95 98.71 96.96 96.64 97.36
99.66 99.94 99.69 99.69 99.87 99.04 99.61 99.80
5.73 2.63 7.38 3.64 1.45 9.06 6.88 12.38
68.43 5.31 28.91 0.91 0.47 23.02 0.31 1.70
Mean
98.12
97.46
99.66
6.14
16.13
FakeAV Hivecoin Keylogger Lokibot Ramnit Ransom Rootkit Zeus
95.03 96.00 96.97 95.80 94.47 96.61 95.55 96.33
94.84 94.56 95.96 94.65 92.42 96.38 94.45 95.76
95.14 95.89 95.45 94.97 87.76 95.40 94.41 95.45
2.09 1.84 6.38 2.40 0.85 9.44 2.12 5.81
0.31 0.16 0.00 0.00 0.00 0.00 0.00 0.00
Mean
95.85
94.88
94.31
3.87
0.06
TABLE VI B INARY MALWARE CLASSIFICATION : LSTM AND B I LSTM DEEP LEARNING MODELS .
Model
LSTM
BiLSTM
Malware Family
Train Time (min)
Test Time (s)
Val Acc (%)
Test Acc (%)
AUC (%)
RAM (MB)
Ramnit Hivecoin FakeAV Lokibot Keylogger Zeus Ransom Rootkit
7.1 12.1 13.2 14.7 20.2 20.1 22.3 12.3
0.36 0.35 0.59 0.57 0.57 0.55 0.69 0.60
97.9 97.9 97.1 97.1 98.4 97.7 97.5 97.5
96.3 97.1 96.2 96.5 97.1 97.1 96.8 95.3
99.2 99.7 98.6 99.4 99.2 99.3 98.6 98.7
1841 1723 1836 1885 2056 1861 1880 1869
Mean
15.3
0.54
97.6
96.55
99.09
1869
Ramnit Hivecoin FakeAV Lokibot Keylogger Zeus Ransom Rootkit
6.6 9.4 10.2 12.2 23.6 25.0 16.8 13.7
0.53 0.58 0.54 0.66 0.78 0.89 0.62 0.81
98.2 97.9 97.2 97.2 98.4 97.9 97.7 97.1
97.7 96.8 95.4 96.5 96.4 97.2 97.0 94.9
99.5 99.7 99.2 99.3 99.1 99.5 98.6 99.1
2717 2816 2761 2837 2685 2748 2747 2833
Mean
14.7
0.68
97.7
96.49
99.25
2768
13
TABLE VII M ULTI - CLASS API MALWARE CLASSIFICATION : CLASSICAL ML MODELS . Val Acc (%)
Test Acc (%)
Train Time (min)
RAM (MB)
XGBoost Random Forest SVM
79.74 78.94 61.83
79.03 78.22 60.94
353.9 100.4 90.6
189.4 16.5 101.7
TABLE VIII M ULTI - CLASS API MALWARE CLASSIFICATION : DEEP LEARNING MODELS . Model
Val Acc (%)
Test Acc (%)
Train Time (min)
RAM (MB)
BiLSTM LSTM
75.29 73.60
72.27 73.18
62.1 72.1
2854.7 2545.2
code P2.1 (ML model selection): in the multi-class setting, resource-efficient tree-based ensembles outperform deep sequence models, so the FDM should recommend classical classifiers unless the training corpus per class is substantially larger. B. Experiment 2: Class-Incremental Learning This experiment assesses how well models accommodate new malware families without full retraining, directly evaluating FDM component P4 (Model Updating Techniques) under the incremental update mode. The Malimg dataset [7] was used; images were pre-processed as 64×64 grayscale inputs. Models started with 14 known families and were incrementally updated one class at a time, yielding eleven evaluation points as the number of known classes grew from 14 to 24 (Fig. 5). Catastrophic forgetting is quantified as the absolute accuracy drop between the best-ever accuracy observed before any new class was introduced and the final accuracy after all classes were added. Lower values indicate better knowledge retention. TABLE IX C LASS -I NCREMENTAL L EARNING (CIL) SUMMARY RESULTS ON M ALIMG . C ATASTROPHIC F ORGETTING (CF) IS THE ABSOLUTE ACCURACY DROP FROM THE INITIAL ( PRE - INCREMENT ) ACCURACY TO THE FINAL ACCURACY. Model EfficientNetB0 MobileNetV2 Random Forest XGBoost
Final Test Acc. (%)
Total Runtime (s)
CF (pp)
99.13 98.96 97.81 97.46
345.4 223.6 95.7 913.6
0.65 0.54 1.98 2.33
Analysis. As shown in Table IX and Fig. 5, CNN-based image classifiers (EfficientNetB0 and MobileNetV2) exhibited substantially greater resistance to catastrophic forgetting (CF of 0.65 pp and 0.54 pp, respectively) than tree-based models (XGBoost: 2.33 pp; RF: 1.98 pp). Both CNN models maintained accuracy above 98.9% after accommodating the full set of evaluated classes. Among classical models, Random Forest converged approximately 9.5× faster than XGBoost (95.7 s vs. 913.6 s) while
Test accuracy (%)
Model
100 99 EfficientNetB0 MobileNetV2 Random Forest XGBoost
98 97 14
16
18 20 22 Number of known classes
24
Fig. 5. Test accuracy of four models as new malware classes are introduced incrementally (Malimg dataset, classes 14–24). CNN-based models (EfficientNetB0, MobileNetV2) exhibit substantially less catastrophic forgetting than tree-based models.
suffering only marginally lower forgetting, an important tradeoff for latency-sensitive update pipelines. From the FDM perspective, this experiment validates recommendation code P4 (Model Updating Techniques): for environments that expect frequent addition of new malware families (high Q4 Update Frequency Score), CNN-based architectures on visualised binary representations are preferable, as they degrade gracefully under incremental update cycles. Resource-limited endpoints (low Q2) may instead accept RF’s slightly higher forgetting in exchange for dramatically faster update times. C. Experiment 3: Transfer Learning for Model Initialisation This experiment investigates whether initialising CNN models from ImageNet pre-trained weights provides measurable advantages over training from scratch in terms of accuracy and training efficiency. Three heterogeneous datasets are used, namely (i) the API call dataset (sequences converted to imagelike representations), (ii) the Android manifest permission dataset, and (iii) the Malimg grayscale image dataset, to understand whether the benefit of transfer learning depends on the input modality. These findings directly inform FDM dimension Q4 (Update Frequency Score) and the feature-type conditioning of recommendation code P4 (MA1). Analysis. Tables X–XII report results across all three datasets. Transfer learning offers a consistent training-time advantage on genuine image data (Malimg): pre-trained models complete training in an average of 401.3 s vs. 857.6 s for scratch-trained models (2.14× speed-up), while delivering comparable accuracy (93.84% vs. 93.99%). DenseNet121 and MobileNetV2 are the models whose pre-trained variant surpasses its scratch counterpart on both accuracy and training speed. For non-image modalities the picture is more nuanced. On the Android manifest permission dataset (Table XI), pretrained CNN models generally outperform scratch-trained counterparts (DenseNet121: 69.29% vs. 36.99%), suggesting that low-level visual features learned from ImageNet still provide a useful inductive bias even for tabular–binary feature maps. On the raw API call dataset (Table XII), accuracy is
14
TABLE X T RANSFER LEARNING VS . TRAINING FROM SCRATCH : M ALIMG GRAYSCALE IMAGE DATASET (25- CLASS CLASSIFICATION ). Model DenseNet121 ResNet50 EfficientNetB0 MobileNetV2
Training
Test Acc (%)
Train Time (s)
Test Time (s)
Pretrained Scratch Pretrained Scratch Pretrained Scratch Pretrained Scratch
95.88 94.77 95.45 95.56 91.49 94.03 92.55 91.60
628.5 946.0 317.0 1335.7 347.4 659.2 312.4 489.6
18.4 17.6 17.4 17.8 15.9 18.1 17.2 17.3
93.84 93.99
401.3 857.6
17.2 17.7
Mean (pretrained) Mean (scratch)
Speed-up 1.51× 4.21× 1.90× 1.57× 2.14×
TABLE XI T RANSFER LEARNING VS . TRAINING FROM SCRATCH : A NDROID MANIFEST PERMISSION DATASET ( BINARY CLASSIFICATION ). Model DenseNet121 ResNet50 MobileNetV2 EfficientNetB0
Training
Test Acc (%)
Train Time (s)
Test Time (s)
Pretrained Scratch Pretrained Scratch Pretrained Scratch Pretrained Scratch
69.29 36.99 67.19 67.19 62.75 63.01 68.36 66.07
1535.5 1586.3 — — 934.0 819.2 1135.9 1261.6
38.0 32.9 — — 32.9 39.4 32.8 33.3
TABLE XII T RANSFER LEARNING VS . TRAINING FROM SCRATCH : W INDOWS API CALL DATASET ( BINARY CLASSIFICATION ; SEQUENCES RENDERED AS 2-D FEATURE MAPS ). Model DenseNet121 ResNet50 MobileNetV2 EfficientNetB0
Training
Test Acc (%)
Train Time (s)
Test Time (s)
Pretrained Scratch Pretrained Scratch Pretrained Scratch Pretrained Scratch
69.65 44.28 61.19 40.80 18.41 30.85 72.64 26.37
1092.5 651.2 469.7 436.3 578.3 738.1 456.1 398.9
26.1 26.4 15.6 15.4 16.6 16.4 16.4 16.8
substantially lower for all configurations (18–73%), confirming that 2-D renderings of sequential API data do not form the natural images for which ImageNet representations were optimised. From an FDM standpoint, these results validate the following guidance encoded in recommendation P4: (a) pretraining accelerates convergence on image-based malware detection without sacrificing accuracy; (b) for non-image feature types (e.g., API sequences), transfer learning from ImageNet provides limited benefit, and purpose-built architectures such as LSTM/BiLSTM are preferred; (c) EfficientNetB0 and DenseNet121 are the most reliable pretrained backbones across datasets. D. Experiment 4: Autoencoder Feature Extraction Feature dimensionality reduction is a practical concern for resource-limited endpoints. This experiment compares a twostage pipeline (autoencoder pre-training followed by CNN classification on compressed features) against a single-stage CNN baseline. Both pipelines are evaluated on the Malimg
dataset, with images resized to 64×64 pixels and flattened to 4,096-dimensional vectors (60 % train / 20 % validation / 20 % test, Optuna-optimised hyperparameters). This validates FDM recommendation code P3.2 (Pre-processing). Analysis. As shown in Table XIII, the autoencoder pipeline reduces training time by 14.2× (from 117.7 min to 8.3 min) and test-time inference by 4.3× (8.51 s to 1.97 s) in exchange for a modest accuracy penalty of 0.86 pp on the test set and only 0.11 pp on the validation set. The autoencoder encoder compresses 4,096-dimensional pixel vectors into a compact latent representation, so the supervised classifier head updates only 0.23 M trainable parameters, versus 27.8 M for the single-stage baseline CNN (a ∼119× reduction), which is the principal driver of the 14.2× training speedup. The classifier model totals 3.4 M parameters (an 8.1× reduction in classifier size), and the full two-stage pipeline, including the 10.7 M autoencoder, totals 14.1 M parameters, about half the baseline. Equivalently, the autoencoder pipeline yields roughly 14× more accuracy per training hour (693 vs. 49). These results confirm that autoencoder-based feature ex-
15
TABLE XIII AUTOENCODER FEATURE EXTRACTION VS . CNN BASELINE ON M ALIMG (25- CLASS CLASSIFICATION ). Trainable Params COUNTS THE PARAMETERS UPDATED DURING SUPERVISED TRAINING ; Total Params COUNTS ALL PARAMETERS ( FOR THE AUTOENCODER PIPELINE , THE 10.7 M AUTOENCODER PLUS THE 3.4 M CLASSIFIER ). Pipeline
Test Acc. (%)
Val Acc. (%)
Train Time (min)
Test Time (s)
Trainable Params (M)
Total Params (M)
CNN baseline Autoencoder + CNN
96.21 95.35
96.47 96.36
117.7 8.3
8.51 1.97
27.80 0.23
27.80 14.11
∆ (autoencoder vs. baseline)
−0.86
−0.11
−109.4
−6.54
−27.57
−13.69
traction is viable when training time or inference latency is a binding constraint, and the 0.86 pp accuracy trade-off is operationally acceptable. This aligns with FDM guidance: organisations with a high Response Latency Level Q3 or low Resource Budget Q2 should consider autoencoder preprocessing (P3.2) to reduce classifier complexity without material accuracy loss. E. Framework Validation The cross-validation of the FDM recommendations against the empirical results in this sections is as follows. For a constrained, real-time deployment (Q1=1, Q2=1, Q3=3), the WCCS assigns the highest compatibility score to RF and XGBoost (RS=1). The binary classification experiments confirm this: XGBoost achieved 97.46% mean test accuracy and RF 97.06%, while consuming only 69 MB and near-zero RAM respectively, compared to BiLSTM’s peak of 2.8 GB. For continuous-update deployments (Q4=3), the FDM recommends class-incremental learning (MA2); the CIL experiments confirm that EfficientNetB0 and MobileNetV2 achieve the lowest catastrophic-forgetting rates (0.65 and 0.54 percentage points respectively) and maintain above 98.9% accuracy as the class count grows across the incremental steps (Fig. 5). For moderate-resource, image-feature deployments (Q2=2, FT1), the FDM recommends transfer learning (MA1); experiments confirm that pretrained models achieve equivalent accuracy to scratch training in an average of 2.14× less time across four CNN architectures on the Malimg dataset. F. Key Findings Table XIV synthesises how each experimental outcome maps to the FDM input dimensions (Q1–Q5) and recommendation codes (P1–P5). Taken together, the four experiments demonstrate that no single ML configuration dominates across all operational contexts; this is precisely the condition the FDM is designed to address. 1) Model selection is resource-sensitive (Q1, Q2 → P2): Binary classification results confirm that XGBoost offers the best accuracy-resource balance for endpoints with moderate RAM budgets (<70 MB) while also achieving the highest AUC overall; BiLSTM, the strongest deep model on AUC (99.25%), still trails XGBoost and demands far more memory (∼2.8 GB RAM, >6 min per family). 2) Multi-class complexity reverses model ranking (task complexity → P2.1): The nearly 7 pp accuracy gap
between XGBoost (79.03%) and BiLSTM (72.27%) in multi-class classification shows that deep sequence models require substantially larger corpora to generalise across many families simultaneously, a finding that FDM encodes in the model-selection guidance of P2.1. 3) Incremental update strategy depends on update frequency and resource constraints (Q2, Q4 → P4): EfficientNetB0 achieves the lowest catastrophic forgetting (0.65 pp) but requires GPU-class resources; RF achieves comparable update speed (∼96 s) with higher forgetting (1.98 pp) but negligible RAM, a trade-off explicitly parameterised in FDM via Q2 and Q4. 4) Transfer learning is modality-dependent (Q4, P3 → P4/MA1): Pre-trained ImageNet weights provide a 2.14× training speed-up on genuine malware images (Malimg) but negligible or negative benefit on API call–derived feature maps, confirming that FDM should condition transfer-learning recommendations on the selected feature type P3. 5) Autoencoder pre-processing is preferable under latency or memory constraints (Q2, Q3 → P3.2): A 14× training acceleration with only 0.86 pp accuracy cost positions autoencoder pipelines as the FDM-recommended pre-processing choice whenever Q3 is high (low response latency tolerance) or Q2 is low (tight resource budget). Collectively, these results demonstrate that the FDM’s quantitative scoring mechanism (WCCS) correctly identifies the dominant performance-resource trade-off axes and steers practitioners toward configurations that are both high-performing and operationally feasible within their declared constraints. No individual model or pipeline achieves optimal performance across all five input dimensions simultaneously, underscoring the practical necessity of a structured decision-making framework. VII. D ISCUSSION A. Experimental Results The experimental results across all five tasks validate the FDM’s core hypothesis: the optimal ML configuration is determined by operational constraints, not by accuracy alone. Resource and latency constraints (Q1, Q2, Q3). In binary API-based classification, XGBoost achieved the highest mean test accuracy of 97.46% (AUC: 99.66%) while consuming at most 69 MB of RAM. In contrast, LSTM and BiLSTM achieved 96.55% and 96.49% mean accuracy respectively, but required 1.7–2.8 GB peak RAM, a ∼40× overhead directly
16
TABLE XIV M APPING OF EXPERIMENTAL FINDINGS TO FDM INPUT DIMENSIONS AND RECOMMENDATION CODES . Q1 = P LATFORM C ONSTRAINT, Q2 = R ESOURCE B UDGET, Q3 = R ESPONSE L ATENCY, Q4 = U PDATE F REQUENCY, Q5 = S ENSITIVITY R ATIO .
Experiment / Finding
FDM Inputs
FDM Code
Validated Recommendation
Binary classification: XGBoost best accuracyto-RAM ratio
Q1 (platform constraint), Q2 (low budget)
P2.1
Prefer gradient-boosted tree over DL when RAM <100 MB
Binary classification: BiLSTM best AUC among DL models
Q1 (platform constraint), Q2 (high budget), Q3
P2.1
Use BiLSTM only in resource-rich settings; XGBoost still leads on accuracy and AUC
Multi-class: classical models outperform DL
Task complexity (many classes)
P2.1
Recommend tree ensemble for multi-class when data per class is limited
CIL: CNN models resist catastrophic forgetting
Q4 (high update freq.)
P4
Use CNN-based incremental learner for frequent new-family additions
CIL: RF fastest update cycle
Q2 (low budget), Q4
P4
RF acceptable for low-resource incremental update pipelines
Transfer learning: 2× speedup on image data
Q4 (update freq.)
P4
Pre-train on ImageNet when feature type is malware visualisation
Transfer learning: no benefit on API features
Q4, P3 choice
P4
Avoid ImageNet transfer when feature type is API sequence
Autoencoder: 14× training speed-up, −0.86 pp accuracy
Q2, Q3
P3.2
Deploy autoencoder pre-processing when latency or RAM is binding
Sequence splitting (1,000-call segments): >94% accuracy
P3.1
P3.1
Segment API traces at 1,000 calls; validated against raw truncation
relevant to edge deployment. Random Forest matched deep learning accuracy (97.06%) at near-zero RAM cost, reinforcing the FDM recommendation of classical ensemble models for low-Q2 deployments. Multi-class complexity and model ranking reversal (P2.1). A notable reversal emerged in multi-class classification: XGBoost achieved 79.03% test accuracy while BiLSTM fell to 72.27%, a gap of nearly 7 pp. This directly contradicts the assumption that sequence-aware deep models are always superior for API-based tasks. The reversal arises because the current training corpus, split across nine simultaneous classes, provides insufficient per-class signal for recurrent models to generalise. This finding indicates that the FDM recommendation path for multi-class settings (many families, limited perclass data) should favour classical ensemble methods unless training data is substantially larger; hierarchical or ensemble deep architectures are identified as a direction for future work. Update frequency (Q4). Transfer learning on Malimg image data reduced average training time by 2.14× (mean 401.3 s pretrained vs. 857.6 s scratch) with no statistically significant accuracy difference (93.84% vs. 93.99%). The benefit is strongest for ResNet50, which converged in 4.21× fewer seconds (317 s vs. 1,336 s). However, on API-call feature maps and Android manifest vectors, pre-trained ImageNet weights provided limited advantage, confirming that the FDM should condition the MA1 (transfer learning) recommendation on the feature type (P3), not on the update interval alone. Class-incremental learning experiments support the MA2 (continuous-update) path: EfficientNetB0 and MobileNetV2
exhibited catastrophic forgetting of only 0.65 and 0.54 percentage points respectively across the incremental steps, while maintaining final accuracy above 98.9%. Detection sensitivity and pre-processing efficiency (Q2, Q3, Q5). The autoencoder pipeline delivered a 14.2× training speedup (117.7 min → 8.3 min) and a 4.3× inference speedup (8.51 s → 1.97 s), at the cost of 0.86 pp in test accuracy and 0.11 pp in validation accuracy. This trade-off is well-characterised: when SR is low (FP-sensitive or latencycritical deployments), the speedup substantially outweighs the marginal accuracy loss; when SR is high (critical-infrastructure or low-tolerance environments), the full CNN pipeline remains warranted. This operating-point decision is directly encodable in FDM via Q3 and Q5. B. Limitations While the FDM provides a structured, mathematically grounded configuration methodology, six categories of deployment scenario fall outside its current scope. Federated and privacy-preserving deployments. The framework assumes training data can be centralised. Organisations subject to data-sovereignty regulations requiring federated learning (where only model updates, not raw samples, are exchanged) are not supported. A future input dimension (e.g., Data Locality Constraint) would address this. Adversarial and evasion-aware deployments. The framework does not account for malware crafted to evade ML detectors via feature-space perturbations or model inversion.
17
Practitioners facing advanced persistent threats should treat P2.1–P2.3 as a baseline and layer adversarial hardening (e.g., adversarial training, certified defences) on top. Explainability requirements. FDM does not include explainability as an input dimension; P2.1 does not distinguish black-box models (e.g., CNNs) from interpretable ones (e.g., decision trees). Practitioners in regulated sectors (HIPAA, NIS2) must apply post-hoc methods such as SHAP or LIME, or manually override P2.1 recommendations. VIII. C ONCLUSION This paper presented the Framework for Decision-making (FDM) for building ML-based malware detection systems. The FDM formalises the configuration selection process through the Weighted Configuration Compatibility Score (WCCS), a multi-criteria scoring function that maps five quantifiable operational parameters, namely platform constraint (Q1), resource budget (Q2), response latency (Q3), update frequency (Q4), and detection sensitivity (Q5), to ranked recommendations across nine configuration dimensions covering data acquisition, model selection, hyperparameter optimisation, feature engineering, and model updating. Empirical evaluation across five experimental tasks on three heterogeneous datasets (Windows API call sequences, Malimg grayscale images, and Android manifest permissions) validated the framework’s recommendations. In binary malware classification, XGBoost achieved the highest accuracy-toresource ratio (97.46%, <70 MB RAM), consuming approximately 40× less memory than recurrent deep models while delivering competitive accuracy. Critically, in multi-class classification, this ranking reversed: XGBoost (79.03%) outperformed BiLSTM (72.27%) by nearly 7 pp, confirming that the optimal model choice is task-complexity-dependent, a distinction the FDM explicitly encodes. For model update strategies, transfer learning reduced average training time by 2.14× on imagebased malware data with no accuracy cost (validating the MA1 periodic-update path), while class-incremental learning with EfficientNetB0 maintained 99.13% final accuracy with only 0.65 percentage-point degradation across the incremental update steps (validating the MA2 continuous-update path). The autoencoder pre-processing trade-off (a 14× training speedup against a 0.86 pp accuracy loss) was quantified as a concrete operating point for latency- or resource-constrained deployments. Taken together, these results confirm the FDM’s central premise: no single ML configuration dominates across all operational contexts, and structured, constraint-aware selection is both necessary and feasible. Future work will focus on two primary directions: (1) extending the WCCS input space to include dimensions for data-locality, explainability, and adversarial robustness; and (2) developing a federated and cross-platform variant of the recommendation algorithm. R EFERENCES [1] J. Geng, J. Wang, Z. Fang, Y. Zhou, D. Wu, and W. Ge, “A survey of strategy-driven evasion methods for pe malware: transformation, concealment, and attack,” Computers & Security, vol. 137, p. 103595, 2024.
[2] C. Catalano, A. Chezzi, M. Angelelli, and F. Tommasi, “Deceiving aibased malware detection through polymorphic attacks,” Computers in Industry, vol. 143, p. 103751, 2022. [3] K. Brezinski and K. Ferens, “Metamorphic malware and obfuscation: a survey of techniques, variants, and generation kits,” Security and Communication Networks, vol. 2023, no. 1, p. 8227751, 2023. [4] L. Zhang, P. Liu, Y.-H. Choi, and P. Chen, “Semantics-preserving reinforcement learning attack against graph neural networks for malware detection,” IEEE Transactions on Dependable and Secure Computing, vol. 20, no. 2, pp. 1390–1402, 2022. [5] Y. Assor, “Anti-vm and anti-sandbox explained,” May 2023. [6] A. Kakareka, “Chapter 1 - detecting system intrusions,” in Network and System Security (Second Edition) (J. R. Vacca, ed.), pp. 1–27, Boston: Syngress, 2 ed., 2014. [7] L. Nataraj, S. Karthikeyan, G. Jacob, and B. S. Manjunath, “Malware images: visualization and automatic classification,” in Proceedings of the 8th International Symposium on Visualization for Cyber Security, VizSec ’11, (New York, NY, USA), Association for Computing Machinery, 2011. [8] E. C. Bayazit, O. K. Sahingoz, and B. Dogan, “A deep learning based android malware detection system with static analysis,” in 2022 International Congress on Human-Computer Interaction, Optimization and Robotic Applications (HORA), (Ankara, Turkey), pp. 1–6, 2022. [9] M. N. AlJarrah, Q. M. Yaseen, and A. M. Mustafa, “A contextaware android malware detection approach using machine learning,” Information, vol. 13, no. 12, pp. 1–25, 2022. [10] R. H. Mahdi and H. Trabelsi, “Detection of malware by using yara rules,” in 2024 21st International Multi-Conference on Systems, Signals & Devices (SSD), pp. 1–8, IEEE, 2024. [11] Y. Wu, M. Li, Q. Zeng, T. Yang, J. Wang, Z. Fang, and L. Cheng, “Droidrl: Feature selection for android malware detection with reinforcement learning,” Computers & Security, vol. 128, p. 103126, 2023. [12] K. Grosse, N. Papernot, P. Manoharan, M. Backes, and P. McDaniel, “Adversarial perturbations against deep neural networks for malware classification,” 2016. [13] A. F. Agarap, “Towards building an intelligent anti-malware system: A deep learning approach using support vector machine (svm) for malware classification,” 2019. [14] J. Suaboot, Z. Tari, A. Mahmood, A. Zomaya, and W. Li, “Sub-curve hmm: A malware detection approach based on partial analysis of api call sequences,” Computers & Security, vol. 92, pp. 1–15, Feb 2020. [15] S. Tobiyama, Y. Yamaguchi, H. Shimada, T. Ikuse, and T. Yagi, “Malware detection with deep neural network using process behavior,” in 2016 IEEE 40th Annual Computer Software and Applications Conference (COMPSAC), vol. 2, pp. 577–582, 2016. [16] M. Ali, S. Shiaeles, G. Bendiab, and B. Ghita, “Malgra: Machine learning and n-gram malware feature extraction and detection system,” Electronics, vol. 9, no. 11, pp. 1–21, 2020. [17] S. Y. Yerima and S. Sezer, “Droidfusion: A novel multilevel classifier fusion approach for android malware detection,” IEEE Transactions on Cybernetics, vol. 49, no. 2, pp. 453–466, 2019. [18] J. Jeon, B. Jeong, S. Baek, and Y.-S. Jeong, “Hybrid malware detection based on bi-lstm and spp-net for smart iot,” IEEE Transactions on Industrial Informatics, vol. 18, no. 7, pp. 4830–4837, 2021. [19] J. Bai and Q. Shi, “Malware detection method based on dynamic variable length api sequence,” in 2019 12th International Symposium on Computational Intelligence and Design (ISCID), vol. 2, (Hangzhou, China), pp. 285–288, 2019. [20] M. Ahmadi, D. Ulyanov, S. Semenov, M. Trofimov, and G. Giacinto, “Novel feature extraction, selection and fusion for effective malware family classification,” in Proceedings of the Sixth ACM Conference on Data and Application Security and Privacy, CODASPY ’16, (New York, NY, USA), p. 183–194, Association for Computing Machinery, 2016. [21] S. Gülmez and I. Sogukpinar, “Graph-based malware detection using opcode sequences,” in 2021 9th International Symposium on Digital Forensics and Security (ISDFS), (Elazig, Turkey), pp. 1–5, 2021. [22] Ö. Aslan and R. Samet, “A comprehensive review on malware detection approaches,” IEEE Access, vol. 8, pp. 6249–6271, 2020. [23] VirusTotal, “YARA - Getting Started.” https://yara.readthedocs.io/en/ stable/gettingstarted.html, 2024. Accessed: 2025-01-15. [24] R. A. Yunmar, S. S. Kusumawardani, F. Mohsen, et al., “Hybrid android malware detection: A review of heuristic-based approach,” IEEE Access, vol. 12, pp. 41255–41286, 2024. [25] A. Mehtab, W. B. Shahid, T. Yaqoob, M. F. Amjad, H. Abbas, H. Afzal, and M. N. Saqib, “Addroid: rule-based machine learning framework for android malware analysis,” Mobile Networks and Applications, vol. 25, pp. 180–192, 2020.
18
[26] A. H. Lashkari, A. F. A. Kadir, L. Taheri, and A. A. Ghorbani, “Toward developing a systematic approach to generate benchmark android malware datasets and classification,” in International Carnahan Conference on Security Technology (ICCST), (Quebec, Canada), pp. 1–7, 2018. [27] M. Kalash, M. Rochan, N. Mohammed, N. D. B. Bruce, Y. Wang, and F. Iqbal, “Malware classification with deep convolutional neural networks,” in 2018 9th IFIP International Conference on New Technologies, Mobility and Security (NTMS), pp. 1–5, 2018. [28] M. G. Zhangjie Fu, Yongjie Ding, “An lstm-based malware detection using transfer learning,” Journal of Cyber Security, vol. 3, no. 1, pp. 11– 28, 2021. [29] W. K. Wong, F. H. Juwono, and C. Apriono, “Vision-based malware detection: A transfer learning approach using optimal ecoc-svm configuration,” IEEE Access, vol. 9, pp. 159262–159270, 2021. [30] Sudhakar and S. Kumar, “Mcft-cnn: Malware classification with finetune convolution neural networks using traditional and transfer learning in internet of things,” Future Generation Computer Systems, vol. 125, pp. 334–351, 2021. [31] N. Owoh, J. Adejoh, S. Hosseinzadeh, M. Ashawa, J. Osamor, and A. Qureshi, “Malware detection based on api call sequence analysis: A gated recurrent unit–generative adversarial network model approach,” Future Internet, vol. 16, no. 10, 2024. [32] A. Kumar and K. Kaur, “A novel MCDM-based framework to recommend machine learning techniques for diabetes prediction,” International Journal of Engineering and Technology Innovation, vol. 14, no. 1, pp. 29–43, 2024. [33] X. Dong, D. J. Kedziora, K. Musial, and B. Gabrys, “Automated deep learning: Neural architecture search is not the end,” Foundations and Trends in Machine Learning, vol. 17, pp. 767–920, 02 2024. [34] H. Benmeziane, K. El Maghraoui, H. Ouarnoughi, S. Niar, M. Wistuba, and N. Wang, “Hardware-aware neural architecture search: Survey and taxonomy.,” in IJCAI, vol. 2021, pp. 4322–4329, 2021. [35] D. Marculescu, D. Stamoulis, and E. Cai, “Hardware-aware machine learning: Modeling and optimization,” in Proceedings of the International Conference on Computer-Aided Design, pp. 1–8, 2018. [36] S. Santos, A. L. Ottoni, R. Borgo, D. Ferreira, and E. Nepomuceno, “A systematic review of green machine learning: practices and challenges for sustainability,” Artificial Intelligence Review, 2026. [37] G. Menghani, “Efficient deep learning: A survey on making deep learning models smaller, faster, and better,” ACM Computing Surveys, vol. 55, no. 12, pp. 1–37, 2023. [38] S. Liu, B. Guo, C. Fang, Z. Wang, S. Luo, Z. Zhou, and Z. Yu, “Enabling resource-efficient aiot system with cross-level optimization: A survey,” IEEE Communications Surveys & Tutorials, vol. 26, no. 1, pp. 389–427, 2023. [39] Z. You, Y.-W. Si, D. Zhang, X. Zeng, S. C. Leung, and T. Li, “A decision-making framework for precision marketing,” Expert Systems with Applications, vol. 42, no. 8, pp. 3357–3367, 2015. [40] A. Narayanan, M. Chandramohan, L. Chen, and Y. Liu, “Contextaware, adaptive, and scalable Android malware detection through online learning,” IEEE Transactions on Emerging Topics in Computational Intelligence, vol. 1, no. 3, pp. 157–175, 2017. [41] R. Feng, S. Chen, X. Xie, G. Meng, S.-W. Lin, and Y. Liu, “A performance-sensitive malware detection system using deep learning on mobile devices,” IEEE Transactions on Information Forensics and Security, vol. 16, pp. 1563–1578, 2021. [42] Z. Alwaisi, T. Kumar, E. Harjula, and S. Soderi, “Securing constrained iot systems: A lightweight machine learning approach for anomaly detection and prevention,” Internet of Things, vol. 28, p. 101398, 2024. [43] M. Merluzzi, P. Di Lorenzo, and S. Barbarossa, “Wireless edge machine learning: Resource allocation and trade-offs,” IEEE Access, vol. 9, pp. 45377–45398, 2021. [44] W. Xu, Z. Yang, D. W. K. Ng, M. Levorato, Y. C. Eldar, and M. Debbah, “Edge learning for b5g networks with distributed signal processing: Semantic communication, edge computing, and wireless sensing,” IEEE journal of selected topics in signal processing, vol. 17, no. 1, pp. 9–39, 2023. [45] M. Tedla, S. Kulkarni, and K. Vaidhyanathan, “Ecomls: A selfadaptation approach for architecting green ml-enabled systems,” in 2024 IEEE 21st International Conference on Software Architecture Companion (ICSA-C), pp. 230–237, IEEE, 2024. [46] T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, “Optuna: A nextgeneration hyperparameter optimization framework,” in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 2623–2631, ACM, 2019. [47] S. Dadkhah, O. D. Okey, S. A. Maret, Y.-W. Lo, A. Firouzi, R. Kuki, T. Sasaki, K. Yoshioka, T. Ban, S. Ozawa, et al., “Cic-ynu-iotmal: A
comprehensive multilayer dataset for static and dynamic analysis of iot malware behavior,” Information Systems, p. 102722, 2026. [48] VirusShare, “Virusshare.com: A repository of malware samples.” https: //virusshare.com/, 2026. Accessed: 2025-05-01. [49] L. Yang, A. Ciptadi, I. Laziuk, A. Ahmadzadeh, and G. Wang, “Bodmas: An open dataset for learning based temporal analysis of pe malware,” in 2021 IEEE Security and Privacy Workshops (SPW), pp. 78–84, IEEE, 2021. [50] H. S. Anderson and P. Roth, “EMBER: An open dataset for training static PE malware machine learning models,” arXiv preprint arXiv:1804.04637, 2018. [51] C. Zhou and R. C. Paffenroth, “Anomaly detection with robust deep autoencoders,” in Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 665–674, 2017. [52] T. Vhito, J. Suaboot, W. Werapun, and T. Kliangsuwan, “Effects of malware detection parameters on classical vs deep ml techniques,” in 2023 7th International Conference on Information Technology (InCIT), pp. 440–445, IEEE, 2023.