HilEnT: Hilbert, Entropy Transformed Image Based Malware Detection Rahul Kale, Thesath Wijayasiri, Kar Wai Fok, Vrizlynn L. L. Thing
arXiv:2607.04772v1 [cs.CR] 6 Jul 2026
Cybersecurity Strategic Technology Centre, ST Engineering, Singapore.
*Corresponding author(s). E-mail(s): [email protected]; Contributing authors: [email protected]; [email protected]; [email protected]; Abstract With the increasing threat of malware across various software related domains, malware detection and classification is critical to determine the response actions. Different strategies have been adopted to address the challenge of malware detection. With the advent of deep learning techniques, malware detection using image processing has garnered research attention. In this work, we proposed a novel malware binary to image transformation technique HilEnT based on a combination of Hilbert curve based transformation of malware binary and the entropy feature comparison of malware file with benign and malware classes. Three grayscale images produced during this process are combined to form a three-channel colored image which is then used for malware detection using machine learning techniques. We performed supervised binary and multiclass classification to evaluate the effectiveness of our proposed HilEnT. We also evaluated a few-shot learning technique to assess the robustness of our proposed HilEnT in a practical setting where the number of available class samples is limited. Furthermore, we investigated the benefits of combination of Histogram of Oriented Gradients and Principal Component Analysis for time performance improvements through feature reduction techniques. We evaluated our proposed methodology on four datasets: Dike, Michael Lester Dataset, MicrosoftBIG 2015 and a self-collected dataset, and achieved the state-of-the-art results. Keywords: Malware Detection, Machine Learning, Malware Classification, Neural Network
1 Introduction
the number of new vulnerabilities is still increasing every month in the year 2024. As the malware
Malicious Software (Malware) has been an everdetection systems are developed based on the growing problem in software-dependent industries existing available malware, new malware designed as they can potentially infect various computing by attackers may evade detection on such detecand networking devices. According to a survey tion systems. With such a large number of large (Bensaoud et al. 2024) published recently in 2024, 1
monthly additions, distinguishing and classifying
Converting malware binary features to images,
different types of malware is crucial to deter-
also known as malware visualization (Nataraj
mine the response actions for safeguarding the
et al. 2011), is one of the popular machine-
systems under attack. This process also assists in
learning (ML) approaches adopted recently for
identifying their potential impacts on the system
malware detection and classification(Nataraj et al.
and selecting a defense mechanism for protection
2011; Cui et al. 2018; Hemalatha et al. 2021; Lo
against them.
et al. 2019; Luo and Lo 2017; Roseline et al.
Different
malware
detection
approaches
2020; Vinayakumar et al. 2019; Vu et al. 2020).
(Nataraj et al. 2010; Bensaoud and Kalita 2024;
The key idea here is to tackle the malware
Natani and Vidyarthi 2013; Chuang and Wang
detection problem as an image processing prob-
2015; Li et al. 2024; Wang and Li 2023) have
lem with Convolutional Neural Networks (CNNs)
been explored in the literature. In practical appli-
being commonly used for such applications. In
cations, most of the network traffic is generally
this paper, we adopted this approach of convert-
benign. With the constantly evolving nature of
ing the file’s binary sequence into an image which
the malware, it is generally difficult to obtain
is then analyzed using machine learning tech-
large number of malware samples of each class
niques for malware detection and classification.
for thorough training of supervised malware clas-
We proposed a novel malware binary to image
sification algorithms. However, obtaining a few
transformation technique HilEnT based on a com-
labeled anomaly samples is comparatively less
bination of Hilbert curve transformation, benign
expensive and more practical. With few-shot
class entropy cutoff comparison and malware class
detection approach, the limited labeled data is
entropy cutoff comparison. Hilbert curve transfor-
leveraged for effective malware classification.
mation converts the malware binary into grayscale
This can be achieved either by fine-tuning the
Hilbert curve pattern based image, while benign
pre-existing models with handful of labeled data
class entropy and malware class entropy based cut-
or by performing data augmentation of existing
off comparison provide one image each containing
data. Additionally, these models can poten-
corresponding regions of interests for both classes.
tially be used for detecting unknown classes not
These three grayscale images are combined to
used/seen during training. Some recent works
form a three-channel colored image which is then
(Hsiao et al. 2019; Bai et al. 2020; Wang et al.
used for malware detection using machine learn-
2021; Conti et al. 2022) explore few-shot learning
ing techniques. We performed supervised binary
based malware detection. 2
and multiclass classification to evaluate the effec-
Section 4, the details about the dataset and exper-
tiveness of our proposed HilEnT. Furthermore, we
imental setup are discussed. The evaluation exper-
investigated the benefits of combination of His-
iments and their results are presented in Section
togram of Oriented Gradients (HOG) and Princi-
5. Finally, the paper is concluded in Section 6.
pal Component Analysis (PCA) for time performance improvements through feature extraction
2 Related Works
and dimensionality reduction techniques. We also evaluated few-shot learning technique to assess the
2.1 General Machine
robustness of our proposed HilEnT in a practical
Learning-based approaches for setting where number of available class samples
Malware Detection
are limited. To summarize, in this work we first proposed
Nataraj et al. (2010) utilized bigram-based fea-
a novel binary file to image transformation tech-
tures and support vector machines to distinguish
nique for malware visualization, called HilEnT
packed and unpacked executables. The classifica-
based on Hilbert curve and entropy approaches.
tion was performed on the raw binary data with
We then evaluated its effectiveness using a super-
a focus on the fast processing time. For mal-
vised learning based CNN and robustness using
ware classification, Bensaoud and Kalita (2024)
a few-shot learning based technique. The per-
first extracted opcode sequences and Application
formance evaluation was conducted using four
Programming Interface (API) calls from Windows
datasets: Dike, Michael Lester Portable Exe-
malware files and then transformed them into N-
cutable(PE) Dataset, MicrosoftBIG 2015 and a
gram sequences. CNN and Long Short-Term Mem-
self-collected dataset.
ory was then utilized for classification task. Natani
The rest of the paper is organized as follows: In
and Vidyarthi (2013) identified that various mal-
Section 2, we review recent related works in liter-
ware behaviours are associated with the func-
ature about malware detection approaches includ-
tions utilized by compromised files through system
ing malware visualization and few-shot learn-
library calls. Therefore, they used API function
ing based methods along with some general
frequency as feature vector with ensemble clas-
approaches for malware detection. The proposed
sifiers to perform malware classification. Chuang
HilEnT framework is described in Section 3. In
and Wang (2015) utilized API calls for Android malware analysis. A Support Vector Machine (SVM)-based hybrid model was proposed which 3
was trained using separate malware-preferred fea-
similar to this existing work, our approach uses
ture set and benign-preferred feature set. For
PCA as a second step for dimensionality reduc-
Android malware detection and classification,
tion. This existing work uses autoencoder for
SynDroid model was proposed by Li et al. (2024).
dimensionality reduction. PCA may provide per-
This model was designed to address class imbal-
formance benefit for dimensionality reduction.
ance by first generating high-dimensional samples
Our proposed work converts the malware binary
using CTGAN-SVM and then utilize Random For-
features into 3-channel images instead of a single
est for malware classification. Wang and Li (2023)
channel grayscale images. We also utilize few-shot
first preprocessed the malware, and subsequently
classification method using CNN implementation
HOG of the grayscale malware image was obtained
for malware classification whereas Wang and Li
followed by the autoencoder for dimensionality
(2023) utilizes the ensemble learning based detec-
reduction. Finally an ensemble of Extra Trees,
tion approach.
XGBoost and Random Forest was used with voting for final classification decision. The dataset
2.2 Malware Detection with
used for this work was Microsoft’s 2015 Malware
Malware Visualization
Classification Challenge. During malware preprocessing, the malware binary file is converted to
Nataraj et al. (2011) were the first to propose the
grayscale using B2M algorithm, and image scaling
idea of malware visualization using bytes of a mal-
is performed using nearest-neighbour interpola-
ware file to represent pixels within an image. CNN
tion algorithm. The main purpose of the deep
was used for classification of the grayscale images
autoencoder is for dimensionality reduction as
obtained by transforming the malware binary files.
HOG may generate redundant features due to
Cui et al. (2018) utilized deep learning for mali-
overlapping blocks. A related hybrid architecture
cious code detection using malware visualization.
was proposed by Kumar and Kumar (2024), who
The malware binary bit strings were converted to
combined convolutional feature extraction with an
grayscale images and Bat algorithm was used for
autoencoder to improve classification performance
data augmentation to address the class imbalance.
for image-transformed malware in Industrial IoT
Finally, CNN was utilized for malware classifica-
settings. This further highlights the growing trend
tion. Hemalatha et al. (2021) used deep learning
of deep models integrating dimensionality reduc-
models for addressing data imbalance and achieve
tion modules for malware analysis. Though our
high accuracy for malware classification on four
proposed approach evaluates HOG as a first step
datasets. Malware binaries were transformed into 4
two-dimensional images and DenseNet was uti-
grayscale image array was passed through 1-
lized for the classification. Lo et al. (2019) pro-
dimensional convolutional and pooling layers fol-
posed a deep learning method for classification of
lowed by LSTM and fully connected layers for
grayscale malware images obtained by transform-
malware classification. The method proposed by
ing malware binaries. The deep learning model
Vu et al. (2020) focused on pixel encoding and
utilized was based on the CNN with Xception
byte arrangement for malware visualization of the
model which reduces the overfitting issues. Their
malware binaries. Space-filling curves were used
proposed model achieved high accuracy for image
to enhance these images with statistical features
based malware classification and outperformed
which assisted in improved detection of malware.
other ML methods. Luo and Lo (2017) proposed
More recently, Andriani et al. (2025) proposed
a 3-step method for malware classification. In
CNN-AutoMIC, which also uses Nataraj style
the first step, malware binary was transformed
grayscale malware images as input but focuses on
into a grayscale malware image and then reorga-
a robust two-stage architecture combining a CNN
nized in a 3 × 3 grid. In the second step, local
feature extractor, a nonlinear autoencoder that
binary pattern (LBP) features were extracted
compresses the features into a 2D latent space,
through these reorganized grids. In the final step,
and a KNN classification engine. Their emphasis
a CNN was utilized for malware classification.
is on cross dataset robustness and explainability
Roseline et al. (2020) proposed a layered ensemble
rather than proposing a new visualization method.
approach for malware detection and classification
IMCFN, introduced by Vasan et al. (2020), fine-
based on malware visualization. This approach
tunes a deep CNN backbone for malware images
was designed to reduce model complexity by
and remains one of the strongest recent CNN-
removing the need of hyperparameter tuning or
only visualization-based baselines, showing that
backpropagation. After transforming the malware
convolutional models trained on bytecode images
binaries to grayscale images, sliding window scan-
can achieve competitive performance. Similarly,
ning and cascade layering was used for malware
Ashawa et al. (2024) demonstrated that enhanced
classification. Vinayakumar et al. (2019) proposed
CNN architectures continue to be highly effec-
a novel image processing technique specifically
tive for image-based malware detection, further
tuned for deep learning methods to obtain a
reinforcing the relevance of visual feature extrac-
robust zero-day malware detection model. In their
tion pipelines. This validates that deep CNN
proposed approach, malware binary was converted
models tend to show remarkable results when
to a grayscale image first. Then the flattened
combined with malware visualization. Transfer 5
learning approaches have also gained attention,
an approach which combined malware visualiza-
with Panda et al. (2023) showing that IoT mal-
tion technique and few-shot learning approach.
ware images can be classified effectively using
The malware feature visualization technique gen-
pretrained CNN backbones.
erated a 3-channel image by fusing Entropy Image, Markov Image and Gray-level Images obtained from malware binary. The 3-channel fused image was then used with few-shot classifiers. Authors
2.3 Malware Detection with proposed two few-shot classification methods,
Few-shot Learning
Convolutional Siamese Neural Network(CSNN)
Fei-Fei et al. (2006) were the first to propose
and baseline Feature Extractor(baselineFE). They
the idea of few-shot learning. Hsiao et al. (2019)
performed the evaluation experiment for one-shot
adopted Siamese neural networks for malware
and ten-shot support sets which indicates the
image classification. During the preprocessing
number of training samples available for each
step, grayscale images were obtained from mal-
class. CSNN approach for one-shot classification
ware binaries and subsequently classified using
was utilized for the case with limited amount
average hash. One-shot learning using Siamese
of training data. For 10-shot classification, the
networks was performed for malware classifica-
baselineFE approach was utilized which outper-
tion. For Android malware classification, Bai et al.
formed the CSNN approach but required more
(2020) used few-shot learning by adopting Siamese
training data. Additionally, both methods were
neural network. Multi-layer Perceptron network
tested for novel/unseen class cases. Authors eval-
was trained for transforming malware into a latent
uated their proposed method on three datasets:
representation in a continuous vector space using
Malimg, Microsoft BIG 2015 and Malbaz. The
Siamese neural networks which improved clas-
first two datasets are publicly available whereas
sification performance. Wang et al. (2021) pro-
the third dataset was compiled using a dataset
posed ‘SIMPLE’, which was a few-shot malware
consisting of recently published malware executa-
classification model and utilized multi-prototype
bles on a public malware repository Malware-
modeling. It was based on their observation of
Bazaar. More recent work such as Alfarsi (2024)
multimodal data distribution for the behaviors
has reaffirmed the usefulness of few-shot learning
of malware within same malware family. Their
for malware classification, particularly in scenar-
proposed model achieved high accuracy for 5-
ios where labelled samples per family are scarce.
way 5-shot tasks. Conti et al. (2022) proposed
Meta learning approaches have also emerged, 6
including Mi-MAML by Ma et al. (2024), which
CSNN similar to Conti et al. (2022) however that
adapts few-shot optimisation strategies to handle
is a popular approach in literature for few-shot
rapidly evolving malware families. Our proposed
classification. In addition, we also perform super-
HilEnT malware visualization technique based on
vised classification and HOG-PCA combination
grayscale Hilbert curve transformation, as well as
evaluation on overall detection system.
entropy cutoff comparison approach differs from approach in Conti et al. (2022). In their work, Markov image is generated by using the bigrams within the binary file. Each pixel intensity value in
3 Proposed Architecture for
the resultant image represents frequency of value
Malware Detection
of the bigram represented by the row and column within that image. Gray-level matrix image is
We will now formally define the problem stategenerated by first creating the standard grayscale ment for the malware classification. image from the binary, and then combining cooccurrence matrix at different rotations to form the resultant image. Instead of standard grayscale, space-filling curves can potentially provide better basis for classification (Vu et al. 2020). There-
3.1 Problem Formulation
fore, we have chosen Hilbert curves as one of First, we will formulate the problem for the malthe three image channels. The entropy graph
ware detection. Let X = {xi }, i = 1, . . . , K , x ∈
plot image used in their work captures only the
Rn be the dataset containing the input samples.
block or file level entropy information whereas our This dataset contains benign samples and samples entropy-based images are designed to capture file from different malware families. The objective of level as well as class level entropy information the malware detection framework is to generate to identify the regions of interests with respect label yi for classification of data sample xi such to both the benign and malware classes in terms that: of entropy. This utilization of class level entropy information to identify such regions along with
yi =
grayscale Hilbert curve is the novelty of our work.
Benign/M alware, Binary M alwareF amily,
Multiclass
For our few-shot implementation, we also utilize where yi denotes the predicted label. 7
(1)
Fig. 1: HilEnT Malware Classification Framework
3.2 HilEnT Malware Classification
In the second stage of the framework, machine learning based methods were trained to produce
Framework
output as per the classification tasks. For binary As shown in Fig. 1, the HilEnT framework consists classification task, the output label will be benign of two main stages: binary file to image transforor malware. For multiclass classification task, the mation and malware classification using suitable output label will be class label for the malware approach. In the binary to image transformation family. stage, the input received was a raw binary file For the machine learning based classificawhich contains a sequence of bytes. Hilbert curve tion, we proposed a simple CNN model and transformation was utilized to draw a distinct benchmarked its performance against Support pattern based on input sequence of bytes. The Vector Machines (SVM) and Multilayer Percepmain motivation behind selection Hilbert curve tron(MLP). The simple supervised CNN classifitransformation was to identify and capture byte cation option was designed to have minimal comsequences that characterize a particular malware plexity. Combination of HOG-PCA was utilized family. By creating these locality based patterns for feature selection and dimensionality reduceffectively, the classifiers in the further stages tion to provide a time-efficient alternative with a of the framework pipeline can potentially better potential accuracy trade-off in supervised classifiunderstand the features associated with malware cation. Additionally, we conducted few-shot learnfamilies during training. We proposed a novel ing based classification experiments to evaluate way to include entropy information within the the versatility of the proposed HilEnT transforcorresponding transformed image as described in mation method even in situations with limited Section 3.3.2. Finally, the combination of three malware samples. grayscale or single channel images serves as a sinFrom a practical application standpoint, once gle three channel transformation or visualization the machine learning based classification networks of the input binary file. in the second stages are trained, only detection
8
will be performed during the deployment. Hence,
Pixel color and pixel mapping are the two main
we have presented a detection time per sam-
components that constitute the conversion of fea-
ple metric during our performance evaluations to
tures to images. For a grayscale image, a value
enable informed selection of detection algorithm
ranging from 0 to 255, is used to represent each
based on application requirements. We will now
pixel within the image; for instance, a black pixel
describe each stage within the framework.
would be represented by a value of 0, and a white pixel would be represented by a value of 255. A grid with a preset size is then utilized to arrange these pixels. Pixel mapping describes the arrangement of these pixels which determines the final
3.3 Binary to Image Transformation
appearance of the image.
3.3.1 Hilbert curve-based
Malware visualization method proposed in
Transformation
Nataraj et al. (2011) is the most popular in liter-
We opted for the Hilbert curve in our image
ature. In this Nataraj method, PE files were first
transformation approach based on the space-filling
represented as sequences of sets of 8 binary values.
curve technique proposed by Vu et al. (2020).
Each set of 8 binary values was then converted to
Fundamentally, this method was designed to cap-
its equivalent decimal value between the range 0
ture subtle distinctions through improved locality
to 255. This decimal value represents the grayscale
retention of the malware binary data to be trans-
pixel intensity for that set. Similarly, grayscale
formed. Evaluating the global features of the
pixel intensity values were calculated for all the
malware binary as a whole, while reducing the pre-
sets. For the pixel mapping step, first the width of
processing steps, was also a key motivation behind
the target image was fixed. Then, the pixels were
opting for this method. Through global feature
mapped from left to right till the row width is
evaluation, contextual information can be assessed
reached, after which the pixel mapping proceeded
holistically, which may improve model robustness
to the next row in the image from left to right.
and reliability for practical scenarios of malware
This technique generated images with fixed width
data evolving over time. By minimizing the pre-
but different lengths as the number of pixels gener-
processing steps, computational time and resource
ated was determined by the input malware binary
requirements are reduced which may enable the
size. According to Vu et al. (2020), this approach
deployment on resource-constrained clients such
may not identify the localized patterns effectively,
as edge or mobile computing devices. 9
and recommended using space-filling curves as an
byte value. Size of the generated Hilbert image
alternative. Space-filling curves are used for map-
is proportional to the input file size. Final image
ping one-dimensional arrays onto two-dimensional
size is fixed at 256x256 and is obtained through
space and find common utilization in image trans-
bilinear interpolation of generated Hilbert image.
formation applications. Therefore, we opted for
This two step process is crucial to maintain the
Hilbert curve as the space-filling curve with the
shape of family/class specific pattern irrespective
objective of improved identification of localized
of the file size. For example, if the file size is small,
patterns during transformation of PE binary files
after interpolation, the pattern will be blown up
to images (Wijayasiri et al. 2025).
along with the rest of the neighboring information.
An example of Hilbert curve transformed
So the detection algorithm will be able to cap-
image is shown in Fig. 2.
ture the overall pattern in a more meaningful way and avoid the potential issues due to varying file sizes. The distinct patterns that potentially characterize a particular malware family may serve as key features for the classification algorithms such as CNN. Compared to color-based Hilbert image generation in Wijayasiri et al. (2025), we adopt
Fig. 2: Hilbert curve transformed grayscale image generated from a Microsoft BIG 2015 malware sample.
a grayscale Hilbert curve based image generation approach.
3.3.2 Entropy Based Transformation
Hilbert curves are created for a block of byte sequence data within the input binary file. As
Entropy is a commonly used tool in the malware
seen in the figure, by creating these locality based
detection domain. In malware binary file con-
patterns effectively, the classifiers in the further
text, entropy can be considered as a measure of
stages of the framework pipeline can potentially
the unpredictability of the file’s data. Techniques
better understand the features associated with
such as payload encryption or file obfuscation are
malware families during training. Length of the
used to hide the malware data. Actions such as
sequence determines the Hilbert curve pattern
data compression will lower the unpredictability
which is mapped to the nearest exponent of 4.
of data, thereby raising the entropy. In general,
Each byte will represent a pixel and the intensity
the greater the entropy, the more likely the data
of that pixel will be given by the corresponding
is obfuscated or encrypted, and the more probable 10
the file is malicious. Therefore, patterns of low and
Our aim was to capture and encode entropy com-
high entropy can be considered as regions of inter-
parison information into image transformation.
est for the input sample. Our proposed entropy based novel image transformation technique is
Sensitivity to Dataset Composition
based on the observation that benign files gener-
The entropy cut off values EN Tben and EN Tmal
ally have lower overall average entropy compared
are computed as average entropy values over the
to malware files whereas malware regions within
benign and malware samples in the training set,
the malware files tend to have higher entropy than
respectively. These values are therefore not fixed
benign regions (Bang et al. 2024). Hence, to cap-
thresholds, but adapt naturally to the entropy
ture comparative information with respect to both
characteristics of the dataset under considera-
benign and malware images within a dataset, we
tion. In datasets where benign software exhibits
generated two images based on entropy values for
relatively high entropy, for example due to com-
each sample file: Benign Comparison Image and
pression or encryption, the benign cut off increases
Malware Comparison Image.
accordingly, reducing the risk of benign regions
For our proposed entropy transformation, we
being incorrectly highlighted as malicious. Simi-
will now describe the associated key terms. Block
larly, in datasets dominated by lightly obfuscated
entropy refers to the calculated entropy for a given
malware with lower overall entropy, the malware
block of byte sequence data from input sample
cut off shifts downward, preserving meaningful
file. Average file entropy refers to the average
contrast in the malware entropy channel. In prac-
of entropies of all the blocks within the given
tice, we observed that relative entropy contrast,
binary input file. These metrics are associated
rather than the absolute cut off values them-
with the sample entropy information. Average
selves, plays the dominant role in discrimination.
class entropy refers to the average of all the
This makes HilEnT robust to moderate variations
file entropies within the given broad class, that
in dataset composition without requiring manual
being the benign and malware classes. Benign
threshold tuning.
Entropy cutoff EN Tben refers to the average class
To generate these images, we first divided the
entropy of the benign class. Malware Entropy cut-
file binary byte sequence into the blocks of size
off EN Tmal refers to the average class entropy of
256. In benign comparison image, only the pix-
the malware class. These two metrics are associ-
els corresponding to blocks with entropy values
ated with the general class entropy information.
below the average file entropy were populated. The intensity or pixel value of the pixel to be 11
populated can be calculated using ((EN Tben −
value)/(EN Tben )) ∗ 255 which captures the benign class information within the dataset via the
EN Tben parameter. The pixel value represents how much lower the
(a) (b) Fig. 3: Entropy cut off comparison images for a Microsoft BIG 2015 malware sample. (a) Benign entropy contrast map highlighting low entropy regions relative to the benign class average. (b) Malware entropy contrast map highlighting high entropy regions relative to the malware class average.
block entropy is compared to EN Tben , normalized over the range [0, EN Tben ]. By extension, lower the block entropy value, larger will be the populated pixel intensity or pixel value. This process is carried out for all the blocks within the image to obtain the Benign Comparison Image. An example of Benign Comparison Image is shown in Fig.3a. On the flip side for the Malware Comparison
3.3.3 Combined Image
Image, for each block, a pixel corresponding to the block is populated only if its entropy is greater
The combination of Fig.2 as red channel, Fig.3a
than average file entropy. The value of the popu-
as green channel and Fig.3b as blue channel rep-
lated pixel is calculated as ((value−EN Tmal )/(8−
resents the three-channel Red-Green-Blue colored
EN Tmal )) ∗ 255. The intensity/pixel value is how
(RGB) HilEnT transformed image as shown in
much higher the block entropy value is compared
Fig.4.
to the EN Tmal , normalized between the range
EN Tmal to 8. Higher the block entropy, larger will be the pixel intensity/pixel value. An example of Malware Comparison Image is shown in Fig.3b with some low intensity pixels populated in the upper half of the image. In both pixel value calculations, factor of 255 is used for scaling the
Fig. 4: Final three channel HilEnT image for a Microsoft BIG 2015 malware sample obtained by combining Hilbert, benign entropy, and malware entropy channels.
grayscale image.
12
3.4 Machine Learning-based
The dimension of the rightmost fully connected layer denoted by (*) depends on the binary(=2)
Classification
or multiclass classification(=number of classes)
3.4.1 Supervised Classification:
requirement.
Convolutional Neural Network 3.4.2 Histogram of Oriented Gradients
For supervised classification, we proposed a CNN
(HOG)
as shown in Fig.5 The input layer is followed
The Histogram of Oriented Gradients (HOG) is a technique utilized commonly for feature extraction in image processing. HOG feature extraction process can be described as follows: First, image is divided into cells of size S × S pixels. Then ori-
Fig. 5: Lightweight CNN architecture used for supervised malware classification across all datasets.
entation of each pixel is calculated. Further, these orientations are aggregated in a histogram of orientations. Finally, these histograms are combined
by three convolutional and pooling layers. The
to obtain the required feature vector. An example
convolutional layer performs the process of fea-
of HOG can be found in Del Coco et al. (2015).
ture extraction through convolution between data
3.4.3 Principal Component Analysis
and the kernels. Pooling layers are used to obtain
(PCA)
higher level and more abstract features from the learned representations. To reduce overall com-
Dimensionality reduction maps the data onto a
putations, we have utilized max-pooling in this
space with fewer dimensions while preserving the
paper to obtain the overall maximum value from
key relationships within the data. It helps in
the group of selected feature map values. For acti-
alleviating the high dimensional data processing
vation function, rectified linear unit (ReLU) will
challenges such as more computational resources
be used. The convolutional-pooling layer combi-
and longer processing times (Savakis et al. 2014).
nations are followed by fully-connected layers. To
Additionally dimensionality reduction may help
calculate the activation of a neuron in a fully
in reducing overfitting by reducing the noisy and
connected layer, the weighted sum of the inputs
redundant features. Principal component analysis
is compared with a threshold. All activations
(PCA) reduces the dimensionality by preserving
from previous layer are connected to this neuron.
certain number of initial principal components 13
such that majority of the variation in the original
a dataset but with balanced distribution across
dataset is captured. PCA is generally an effective
different classes.
technique for dimensionality reduction. However,
The CSNN is trained with image pairs and
while using PCA, it is important to note that sig-
the pairs could either be formed by images from
nificantly reducing the dimension using PCA may
same family or images from different families. For
potentially affect the classification performance
a pair of images from same family, the CSNN
of the machine learning based methods. Hence,
should ideally generate the output as 1 whereas
there may be a potential trade-off between pro-
for a pair of images from different malware fami-
cessing time and classification accuracy due to
lies, the CSNN should ideally generate the output
dimensionality reduction.
as 0. This is based on the assumption that samples from a particular malware family generally tend to exhibit similar features or properties.
3.4.4 Few-shot Learning Based
During training, the CSNN is designed to iden-
Classification
tify the similarities and dissimilarities between
Limited availability of samples from different mal-
the pairs of input data rather than each individ-
ware families poses a significant challenge for
ual sample from the dataset. Structurally, CSNN
malware classification (Wang et al. 2021). Con-
can be considered as two parallel CNNs operat-
ventional supervised machine learning based algo-
ing side by side. The weights are shared among
rithms require sufficiently large amount of training
these CNNs and the output of these two CNNs is
data and can perform classification only within
integrated using a similarity metric. The weights
the classes or families observed during training.
are concurrently updated for both CNNs during
To enable the conventionally trained supervised
training and Euclidean similarity is used as the
model for detection of unseen or novel class, this
similarity metric (Conti et al. 2022). Each CNN
model would need to be trained again with large
generates a latent representation or embedding
number of new class samples. To address these
of the corresponding image from the input image
challenges, Convolutional Siamese Neural Net-
pair. Through training of CSNN, these generated
work (CSNN) model can be utilized. Previous
latent representations become similar for image
research works (Conti et al. 2022; Bai et al. 2020;
pairs from same malware family and distinct
Hsiao et al. 2019) have utilized the Siamese Neu-
for image pairs from different malware families.
ral Network to achieve good results for malware
As the Euclidean similarity metric compares the
classification with limited number of samples in
latent representations of these two images within 14
4 Experimental Setup
a pair, higher similarity score would be obtained if the images belong to the same malware family
4.1 Datasets
and lower similarity score would be obtained if the images within a pair belonged to different mal-
The four datasets used for performance evaluation
ware families. For few-shot learning, in general,
will now be described in brief.
the support set is a combined set of data which consists of reference samples obtained from each
4.1.1 Dike Dataset
class. These reference samples serve as the repreDike Dataset is a labelled dataset which contains sentation of the corresponding classes or families. benign and malicious Portable Executable (PE) The number of instances from each class present files and Object Linking and Embedding (OLE) in the support set is termed as K-shot. Query set files(DikeDataset Project 2021). There are 1082 is the set of target unlabelled samples for which benign files (982 PE, 100 OLE) and 10841 malithe classification is to be performed. For determincious files (8970 PE, 1871 OLE). The samples for ing the label or class of a sample from query set, this dataset are obtained from sources such as a pair of the target query image and each image MalwareBazaar, DuckDuckGo. In this work, we from the support set is provided to the CSNN. are only considering the PE files from both benign The pair which demonstrates maximum similarand malicious types. Additionally, we are using ity score, implies that the query image is similar this dataset only for binary classification experito support set image within that pair. Therefore, ment and hence only benign and malicious labels the family class of that support image is identified are considered. as the family class of the query image. As CSNN does not require significant amount of training
4.1.2 Michael Lester PE Dataset
data(Conti et al. 2022), it addresses the limited data availability issue. Additionally, if at least one
This dataset was created by sampling PE files
sample of a novel class is available in the support
from PE Malware Dataset by Michael Lester
set, CSNN can identify similar samples belonging
which was published by Practical Security Analyt-
to that class within the query set based on sim-
ics(Lester 2021). The dataset is primarily sourced
ilarity score without needing to train using that
from VirusShare, MalShare and TheZoo. The
class.
dataset contains around 200000 samples with 86812 benign and 114737 malicious samples. We randomly sampled 10000 malicious and 10000 15
benign files from the total dataset to create a bal-
Table 1: Sample Distribution for MicrosoftBIG Dataset
anced dataset with comparable size to rest of our
Ramnit Lollipop Kelihos v3 Vundo Simda Tracur Kelihos v1 Obfuscator Gatak
Sample Count 1541 2478 2942 475 42 751 398 1228 1013
Total
10868
Family
datasets. This will also be used only for binary classification experiment and as such only benign and malicious labels are considered.
4.1.3 Microsoft BIG 2015 Microsoft introduced this dataset in 2015 in a
Distribution(%) 14.2 22.8 27.1 4.4 0.4 6.9 3.6 11.3 9.3 100
Kaggle competition(Ronen et al. 2018). There are total 21,741 malware samples in the dataset which are further categorized as a labelled training set 2023). The dataset contains 4549 malware samples
of 10,868 samples and an unlabelled set of 10,873
sourced from the well-known malware database
samples. In this work, we have only utilized the
MalwareBazaar, spanning the years 2020 to 2023.
labelled training set, which consists of 9 malware
VirusTotal, a widely-used commercial malware
families. Additionally, we have only utilized the
detection service that utilizes 72 malware detec-
.bytes files provided in the dataset. Furthermore,
tors, is further employed to scan all the malware
some files within the dataset contain an unrecog-
samples, retaining those detected by more than 40
nizable symbol represented by the “??” in the byte
engines. The malware samples encompass total 8
sequence. Some files containing only the unrecog-
malware families. The sample distribution for the
nizable symbol were removed from the dataset.
self collected dataset is shown in Table 2.
The different classwise split for the dataset is shown in the Table 1.
4.1.4 Self-collected Dataset
Table 2: Sample Distribution for Self Collected Dataset
PE files are the main focus of our experiment. Dur-
GuLoader RedLineStealer Heodo TrickBot Loki GrandCrab SmokeLoader IcedID
Sample Count 2053 493 464 351 341 340 265 242
Total
4549
Family
ing the PE file collection for self-collected dataset, the size limitation is set to 5 MB for both malware and benign samples. In line with the static analysis procedure described in Krčál et al. (2018), packed malware samples are excluded from the dataset by using Detect It Easy(Detect It Easy Project 16
Distribution(%) 45.2 10.8 10.2 7.7 7.5 7.5 5.8 5.3 100
From the datasets under evaluation, Microsoft
RAM, Nvidia RTX3070 GPU(8GB) and Ubuntu
Dataset and Self-collected Dataset did not have
20.04 OS.
their own benign samples for binary classification.
Implementation Details for Reproducibility Therefore, we utilized the same set of benign samFor all supervised experiments, we used the same
ples from the Lester PE dataset as those samples
lightweight CNN architecture shown in Fig. 5.
are publicly available. Also, the count of selected
The network consists of three convolutional layers
benign samples from the Lester PE dataset is
with 3 × 3 kernels and ReLU activations, each fol-
roughly similar to Microsoft Dataset which pro-
lowed by max pooling, and three fully connected
vides a balanced dataset for binary classification.
layers for classification. Training was performed using the Adam optimizer with a learning rate
4.2 Evaluation Metrics
of 1 × 10−3 and a batch size of 32. For feature
For the evaluation of malware detection, we have
based experiments, Histogram of Oriented Gradi-
adopted accuracy metric for binary and multi-
ents was computed using 8 × 8 pixel cells with
class classification performance comparison as we
block normalization, and Principal Component
have a balanced dataset with similar number of
Analysis was applied to retain 95% of the cumu-
normal and malicious samples. For a fair compar-
lative variance. All parameters were kept fixed
ison, the training set and test set will be kept
across datasets and experiments to ensure fair
identical for each method under the evaluation.
comparison and reproducibility.
Experiment-specific dataset splits are explained in
5 Experiments and Results
the corresponding relevant experiments in Section 5. In certain experiments, we also provided detec-
The performance of our proposed HilEnT frametion time per sample metric to evaluate the impact work was evaluated across three main experiof HOG-PCA step on overall time performance. ments: Binary Classification, Multiclass Classification and Unseen Class Classification experiment.
4.3 System Setup
It is important to note that binary classifica-
Sklearn(Pedregosa et al. 2011), Keras(Chollet
tion here refers to classification with only two
et al. 2015) and PyTorch(Paszke 2019) were uti-
classes (benign and malware). Supervised classi-
lized for the implementation of the proposed
fication setting was used to produce either some
framework in Python. The evaluation system spec-
or all the results within all three experiments.
ifications include: Intel i7-10700K CPU, 64GB
On the other hand, few-shot setting was used 17
to perform only the multiclass classification and
this experiment, 5-fold cross validation was used
unseen class classification experiment. Within the
with final results obtained using average of 5-fold
supervised classification setting, we generated two
results. We also discuss the detection time per
result points for each possible case: with and with-
sample for all methods under consideration in this
out HOG-PCA. Within the few-shot setting, we
binary classification task.
have evaluated two support set sizes: 1 and 10.
The results are summarized in Table 3. The
To provide a baseline comparison for our proposed
’Image Txm Method’ refers to the Image Transfor-
HilEnT method, we generated Nataraj malware
mation method used to convert the byte sequence
visualization image(Nataraj et al. 2011) for cer-
into image files. ‘NO’ HOG + PCA rows in the
tain experiments. In addition to proposed CNN,
table refer to the simple supervised learning case.
we also implemented two popular machine learn-
The SVM (Gray), MLP (Gray) and CNN (Gray)
ing algorithms: Multilayer Perceptron (MLP) and
are used for establishing baseline for the Hilbert
Support Vector Machines (SVM) for baselines.
curve or Nataraj transformed grayscale images
MLP is constructed using multiple layers of basic
and therefore, these methods are not utilized for
processing units called perceptrons. We will now
the HilEnT transformed images. The results for
describe our individual experiments and discuss
the HilEnT method are shown in the separate
their results.
column for easier comparison with the remaining models.
5.1 Binary Classification
As seen from the Table 3, the HilEnT dis-
For this experiment, samples from different mal-
plays best performance across different datasets
ware types were considered under single mali-
for binary classification. Generally, combination
cious category. As Dike and Lester PE (Lester)
of HOG and PCA tends to show similar accu-
are binary datasets, no further processing was
racy performance irrespective of the transfor-
required. For Microsoft BIG (MS) and Self Col-
mation method. However, we observed that the
lected (Self) Dataset, samples from different mal-
HOG-PCA combination is around two orders of
ware families were considered as a single malicious
magnitude faster in detection time per sample
category. Only simple supervised and HOG-PCA
metric across all datasets compared to the meth-
enhanced supervised learning based experiment is
ods not using PCA as seen in Table 4. Therefore,
carried out for binary classification experiment.
as intended, a trade-off can be achieved between
The few-shot learning based method was not
accuracy and detection time performance with
used for the binary classification experiment. For
pre-processing step of HOG-PCA combination. 18
Table 3: Experiment 1: Binary classification accuracy for Hilbert and Nataraj baselines with and without HOG and PCA, compared against the proposed HilEnT method. Results are averaged over five fold cross validation using balanced benign and malware splits. Image HOG Accuracy Dataset Txm + SVM MLP CNN HilEnT Method PCA (Gray) (Gray) (Gray) YES 0.97 0.98 0.97 Hilbert NO 0.98 0.98 0.98 Dike 0.99 YES 0.97 0.98 0.97 Nataraj NO 0.98 0.98 0.98 YES 0.85 0.85 0.84 Hilbert NO 0.86 0.84 0.88 Lester 0.94 YES 0.85 0.84 0.83 Nataraj NO 0.86 0.85 0.88 YES 0.93 0.93 0.92 Hilbert NO 0.95 0.96 0.97 MS 0.99 YES 0.93 0.93 0.91 Nataraj NO 0.93 0.94 0.96 YES 0.95 0.96 0.94 Hilbert NO 0.94 0.96 0.97 Self 0.99 YES 0.95 0.96 0.92 Nataraj NO 0.94 0.96 0.97
Table 4: Experiment 1: Per sample detection time for binary classification using Hilbert and Nataraj baselines with and without HOG and PCA, compared against the proposed HilEnT method. Dataset
Image Txm Method Hilbert
Dike Nataraj Hilbert Lester Nataraj Hilbert MS Nataraj Hilbert Self Nataraj
HOG + PCA YES NO YES NO YES NO YES NO YES NO YES NO YES NO YES NO
Detection Time per sample(sec) SVM MLP CNN HilEnT (Gray) (Gray) (Gray) 4.1e-5 6e-7 2.8e-5 0.34 5.1e-5 9.1e-3 6.1e-4 3.7e-5 5.8e-7 2.5e-5 0.26 4.3e-5 9.2e-3 6.6e-4 2.7e-7 3.3e-4 0.59 3.3e-5 9.5e-3 4.7e-4 4.1e-4 2.7e-7 3.2e-4 0.39 3.2e-5 9.6e-3 2.5e-4 2.3e-7 3e-5 0.42 5.6e-5 8.9e-3 4.2e-4 2.6e-4 2.7e-7 3.2e-5 0.29 7.2e-5 9e-3 1.2e-4 2.4e-7 3.2e-4 0.47 6.1e-5 2.1e-3 5.4e-4 1.4e-4 2.3e-7 3.5e-4 0.35 5.1e-5 2.2e-3
SVM performance was mostly among the slowest
other in most cases. Nataraj and Hilbert curve
across different datasets and methods. MLP and
transformation methods mostly perform on par
CNN performance is generally on par with each
as well for most cases both in terms of accuracy
19
and detection time performance. Among the four
the table refer to the simple supervised learn-
datasets, lowest binary classification accuracy per-
ing case. The results for the HilEnT method and
formance was observed for Lester dataset across
HilEnT with Few-shot learning method are shown
all machine learning and image transformation
in the separate columns for easier comparison
methods.
with the remaining models. The MLP (Gray) and CNN (Gray) indicate that the image transforma-
5.2 Multiclass Classification
tion method for those algorithms consisted only
For this experiment, samples from different mal-
of Hilbert curve or Nataraj transformed image
ware families were considered under their own
exclusively without the entropy based transformed
malicious family category. As Dike and Lester PE
images.
(Lester) are binary datasets, those datasets are
As seen from the Table 5, the HilEnT displays
not considered for the multiclass experiment. All
best performance across both datasets for mul-
three algorithms: simple supervised and HOG-
ticlass classification. Unlike binary classification,
PCA enhanced supervised learning and few-shot
combination HOG and PCA generally demon-
learning based experiment is carried out for mul-
strates higher accuracy performance irrespective
ticlass classification experiment. For this exper-
of the transformation method for grayscale image
iment, the 80% of main dataset was used for
based multiclass classification. This could be
training and the remaining 20% of main dataset
attributed to the relevant feature extraction via
was used for test data. We did not report the
HOG for multiclass classification. Nataraj and
detection time per sample for this experiment as it
Hilbert curve transformation methods mostly per-
is similar to the results obtained in Experiment 1.
form on par for most cases both in terms of
Additionally, since SVM does not have an inherent
accuracy performance. Few-shot learning based
multiclass classification option and SVM time per-
classification shows only marginally lower accu-
formance in previous experiment was significantly
racy performance despite being trained with only
slower compared to the other methods, we did
fraction of data (around 10%) compared to other
not report the results for the SVM in multiclass
supervised methods. It indicates that few-shot
classification experiment.
based learning algorithm was able to under-
The results are summarized in Table 5. The
stand the relevant class comparison patterns from
’Image Txm Method’ refers to the Image Transfor-
limited samples available. However, the perfor-
mation method used to convert the byte sequence
mance difference is slightly larger for self collected
into image files. ‘NO’ HOG + PCA rows in
dataset. 20
Table 5: Experiment 2: Multiclass classification accuracy for Hilbert and Nataraj baselines with and without HOG and PCA, compared against the proposed HilEnT and few shot learning methods. Image HOG Accuracy Dataset Txm + MLP CNN FewHilEnT Method PCA (Gray) (Gray) Shot YES 0.93 0.93 Hilbert NO 0.90 0.91 MS 0.97 0.95 YES 0.94 0.94 Nataraj NO 0.91 0.93 YES 0.91 0.92 Hilbert NO 0.90 0.91 Self 0.95 0.92 YES 0.92 0.93 Nataraj NO 0.90 0.92
5.3 Unseen Class Classification
as the test set. The remaining dataset, containing benign samples and malicious samples from
Along with existing known families or types of
the remaining classes, formed the training set. The
malware, many new malware emerge regularly,
training set was categorized in binary classes i.e.
and it is not practical to obtain training data for
benign and malicious samples. On the other hand,
all such malware. Hence unseen class detection is
the entire test set contained only malicious sam-
an important set of evaluations to identify how the
ples, as all the samples belong to the single unseen
malware detection models perform when unseen
malware family class.
or novel class samples e.g. zero-day attacks are
To perform this set of experiments for the few-
presented for detection. This experiment is per-
shot learning based setting, during each round of
formed only with simple supervised learning and
experiment, one of the malware classes was treated
few-shot learning (FSL) based methods. Though
as unseen class. Only 150 samples from each seen
we are displaying the results for both methods in
class are available for training. No samples from
the same table, there is a small difference between
unseen class are used during training. During test-
the method in which this experiment is performed
ing, support set consisted of samples from the all
for simple supervised learning and FSL. As Dike
the classes. Only the unseen class samples were
and Lester PE are binary datasets, those datasets
present in the query set during testing. Hence,
are not considered for the unseen class experiment.
unlike simple supervised setting, benign samples
To perform this set of experiments for simple
are not used. Two support set sizes were consid-
supervised learning, during each round of the
ered: 1 (Sup=1) and 10 (Sup=10). The results
experiment, one of the malware family classes was
reported here are accuracy/recall value for the
removed from main dataset to serve exclusively
unseen class. 21
Table 6: Experiment 3: Unseen malware class detection accuracy for the Microsoft BIG 2015 and self collected datasets using supervised CNN, few shot learning with one and ten support samples, and the proposed HilEnT method. a. Accuracy for Microsoft BIG Dataset CNN FSL FSL Unseen Class HilEnT (Hilbert) (Sup=1) (Sup=10) Ramnit 0.96 0.72 0.62 0.57 Lollipop 0.76 0.82 0.68 0.39 Kelihos v3 0.85 0.99 1.00 1.00 Vundo 0.99 0.99 0.66 0.63 Simda 0.98 0.97 0.66 0.69 Tracur 0.99 0.97 0.39 0.30 Kelihos v1 0.99 0.93 0.88 0.93 Obfuscator 0.99 0.94 0.77 0.71 Gatak 0.99 0.98 0.73 0.64 Weighted Acc 0.89 0.90 0.77 0.67 b. Accuracy for self-collected Dataset CNN FSL FSL Unseen Class HilEnT (Hilbert) (Sup=1) (Sup=10) GuLoader 0.94 0.86 0.62 0.84 RedLineStealer 0.82 0.86 0.86 0.83 Heodo 0.99 0.93 0.81 0.81 TrickBot 0.28 0.96 0.87 0.88 Loki 0.93 0.96 0.89 0.89 GrandCrab 0.75 0.99 0.89 0.90 SmokeLoader 0.97 0.98 0.86 0.90 IcedID 0.53 0.25 0.90 0.8 Weighted Acc 0.88 0.87 0.72 0.85 The results for this experiment are shown
family samples seems to be most difficult to clas-
in Table 6. For simple supervised setting, per-
sify when they are unseen. For self collected,
formance for both grayscale Hilbert curve and
overall consistent results across different support
HilEnT image is consistent across different classes.
set sizes are observed except for class 1 with
Similar observations can be made for self-collected
support set size 1.
dataset performance. However, for IcedID family,
Overall drop in performance for unseen class
HilEnT performs significantly lower compared to
experiment is anticipated. Simple supervised
other methods.
method has only two possible classes as the output
For few-shot experiment, the results for the
for unseen data: benign or malware. Therefore,
MicrosoftBIG dataset show inconsistent results
potential for missclassifcation is small. On the
across different support set sizes. Kelihos v3 and
other hand, in few-shot learning setting, the num-
Kelihos v1 are overall easier to identify. Tracur
ber of possible output classes depends on the
22
Table 7: Accuracy comparison between the proposed HilEnT method and prior state of the art approaches on the Microsoft BIG 2015 malware dataset. Accuracy SoTA Classification SoTA HilEnT Binary NA NA (Lo et al. 2019) Multiclass 0.992 0.985 Binary 0.977 0.991 (Hemalatha et al. 2021) Multiclass 0.985 0.973 Binary NA NA (Conti et al. 2022) Multiclass 0.986 0.979 number of classes present in the support set which
provide any benign samples, each SoTA work uses
can reach up to total number of known classes
their own set of benign files.
within the dataset. Thus the potential to misclas-
Our proposed HilEnT achieves comparable
sify is higher in few-shot case owing to its inherent
results to the SoTA. It performs better in binary
setup.
classification but performs slightly lower in multi-
This unseen class experiment enabled us to test
class classification tasks.
our proposed methodologies for robustness against
Hemalatha et al. (2021) uses DenseNet net-
unseen class data.
work which consists of at least 98 convolution layers based on the network information provided in their paper. Xception model used by Lo et al. (2019) uses at least 36 convolutional layers. Our
5.4 State-of-the-art (SoTA)
proposed CNN network uses 3 convolutional lay-
Comparison
ers followed by 3 fully connected layers. Thus, Table 7 demonstrates the performance comparison
the number of total layers in our network can be
of our proposed model with some SoTA works for
considered 6.
Microsoft BIG 2015 dataset. For each comparison,
Focus for Hemalatha et al. (2021) and Lo et al.
we have replicated the train-test split or number of
(2019) was using deeper convolutional networks
benign samples utilized. As these SoTA models are
for malware classification. Those works have used
not publicly available, we have directly compared
standard approach to convert malware binaries to
our results with the results from their correspond-
images. Our focus was to propose a novel method
ing paper. Cells with ‘NA’ entries in Table 7
to transform malware binary to images such that
indicate that binary classification results were not
even shallow CNNs are able to perform the classifi-
reported for the corresponding SoTA work. Also
cation. Our proposed method utilizes combination
note that, as Microsoft BIG 2015 dataset does not 23
of Hilbert curve transformation and entropy based
Nataraj grayscale visualization across all evalu-
transformations to extract and represent the key
ated CNN backbones and datasets. Bold values
features directly during transformation. This step
indicate the higher accuracy within each pairwise
enables our approach to utilize shallower networks
comparison, and in most cases the improvements
for malware classification. Despite using shallower
are incremental and consistent with normal archi-
convolutional neural networks, we can achieve
tectural variability. However, entries marked with
comparable performance for multiclass classifica-
a superscript * denote cases where the abso-
tion and even better performance for binary classi-
lute improvement exceeds 1%, indicating a more
fication than the SoTA. Using shallower networks
notable increase in discriminative capability for
generally lowers the computational requirements
certain model–dataset combinations. These mod-
and improves the detection time performance.
erate but meaningful gains demonstrate that while
Compared to algorithm proposed by Conti
both visualization methods can support effective
et al. (2022), we proposed a different file binary to
CNN-based malware detection, HilEnT introduces
image transformation technique but still achieved
clearer structural cues that particularly benefit
comparable results. Our proposed method encodes
some backbones without disrupting overall perfor-
benign and malware class entropy information
mance stability.
within the transformation process which is novel
Although HilEnT achieves slightly lower accu-
to the best of our knowledge. This novel approach
racy on the Lester dataset compared to some of
which includes class entropy comparison has a
the deeper CNN backbones evaluated in Table 8,
potential to provide fairer entropy based compar-
this reduction is modest and occurs primarily in
ison in cases where the malware families within
cases where the competing models rely on sub-
certain datasets exhibit atypical behaviour. For
stantially larger architectures. In contrast, Table 9
instance, unlike general assumption where benign
demonstrates that HilEnT offers a clear advan-
files exhibit lower entropy values, if a dataset con-
tage in inference efficiency, processing samples
tains benign files with high entropy values, this
in 5.4e-4 seconds, significantly faster than any
class behaviour will be captured under benign
of the CNN-based models. The runtimes across
comparison images through class threshold for our
the compared backbones show that deeper net-
proposed HilEnT method.
works such as VGG19, ResNet152, and ViT-B/16
As shown in Table 8, the performance of
incur noticeably higher computational cost, while
HilEnT is generally comparable to the standard
even the lightweight models remain slower than HilEnT. This highlights a favorable trade off: 24
Table 8: Backbone Comparison of Malware Visualization Methods: Nataraj (baseline) vs. HilEnT (ours). Higher accuracy is bolded; improvements greater than 1% absolute accuracy are marked with *. Dataset
Dike
Lester
MS (Microsoft)
Self-Collected
Backbone Model AlexNet VGG16 VGG19 ResNet50 ResNet152 DenseNet121 EfficientNet-B0 MobileNetV3-Large ConvNeXt-Tiny ViT-B/16 AlexNet VGG16 VGG19 ResNet50 ResNet152 DenseNet121 EfficientNet-B0 MobileNetV3-Large ConvNeXt-Tiny ViT-B/16 AlexNet VGG16 VGG19 ResNet50 ResNet152 DenseNet121 EfficientNet-B0 MobileNetV3-Large ConvNeXt-Tiny ViT-B/16 AlexNet VGG16 VGG19 ResNet50 ResNet152 DenseNet121 EfficientNet-B0 MobileNetV3-Large ConvNeXt-Tiny ViT-B/16
Nataraj Acc. 0.99297 0.98995 0.98995 0.99146 0.99196 0.99498 0.99046 0.99347 0.99347 0.98744 0.94318 0.95533 0.93722 0.94541 0.94591 0.95484 0.95211 0.93995 0.95583 0.92854 0.98802 0.99521 0.99784 0.99664 0.99688 0.99616 0.99688 0.99425 0.99784 0.98873 0.98591 0.98591 0.98832 0.99278 0.99141 0.99072 0.99210 0.99141 0.99381 0.97629
HilEnT Acc. 0.99448 0.99648* 0.99498 0.99397 0.99397 0.99448 0.99548* 0.99598 0.99598 0.99046 0.94988 0.95906 0.95757* 0.96005* 0.95906 0.96154 0.96129 0.95931* 0.96501 0.93871 0.99449 0.99808 0.99784 0.99880 0.99856 0.99880 0.99880 0.99832* 0.99832 0.99616* 0.99107 0.99416* 0.99381 0.99485 0.99549 0.99513 0.99401 0.99462 0.99395 0.99086*
HilEnT delivers competitive accuracy across all
Furthermore, we test our image methods effi-
datasets while achieving the lowest inference time,
cacy using a state of the art classification pipeline
making it particularly suitable for real-time or
(Andriani et al. 2025). Table 10 reports the per-
resource-constrained malware detection scenarios.
formance of the CNN-AutoMIC architecture when trained on images generated using the Nataraj
25
favourable trade-off: comparable or improved clas-
Table 9: Per-sample inference time (seconds) on the self-collected dataset across several CNN backbones using the Nataraj visualization, compared with our HilEnT method. Dataset
Model AlexNet VGG16 VGG19 ResNet50 ResNet152 Self DenseNet121 EfficientNet-B0 MobileNetV3-Large ConvNeXt-Tiny ViT-B/16 HilEnT (Ours)
sification performance for most datasets while substantially reducing computational cost, making it particularly suitable for large-scale or real-time
Time (s) 8.81e-4 2.01e-3 2.73e-3 1.52e-3 3.15e-3 1.82e-3 1.06e-3 9.86e-4 2.03e-3 4.83e-3 5.4e-4
malware detection pipelines.
5.5 Limitations and Future Work While HilEnT achieves competitive performance across multiple datasets, several limitations remain that motivate future work. First, our experiments focus on Windows PE
grayscale visualization and our HilEnT repre-
binaries (and OLE files in the Dike dataset), with
sentation. As expected, the two visualization
a 5MB size limit and the exclusion of packed
strategies remain largely comparable across most
samples in the self-collected dataset. In addition,
metrics and datasets, reflecting the robustness
benign files for the Microsoft BIG 2015 and self-
of CNN-AutoMIC to different input encodings.
collected experiments are drawn from a single
Notably, HilEnT achieves consistently stronger
external benign corpus. Future work will extend
performance on the Lester, Microsoft, and self-
HilEnT to other executable and document formats
collected datasets, showing improvements across
(e.g. ELF, Android, office and PDF malware),
all five metrics. For the Dike dataset, Nataraj
relax strict size and packing constraints, and eval-
remains marginally higher in accuracy, preci-
uate on more diverse benign corpora that better
sion, F1, and AUC, while HilEnT produces a
reflect realistic deployment environments.
slightly higher recall, suggesting that the Hilbert-
Second, HilEnT currently relies on a sin-
entropy representation emphasizes different struc-
gle static modality: raw bytes transformed into
tural cues within the malware samples. Impor-
images. This is an intentional design choice that
tantly, these accuracy differences are relatively
avoids executing malware and enables low risk
small, while HilEnT offers significant advantages
deployment on endpoints without sandboxing.
in inference speed (Table 9), achieving the lowest
However, it also means that other informative
per-sample latency among all evaluated config-
static artefacts (such as PE header fields, section
urations. This indicates that HilEnT provides a
level metadata, imported API profiles, or string 26
Table 10: Performance comparison of CNN-AutoMIC using Nataraj visualization versus our HilEnT visualization across four datasets. Bold values indicate the higher score for each metric. Dataset rk dike rk lester rk ms rk self
Method Nataraj HilEnT Nataraj HilEnT Nataraj HilEnT Nataraj HilEnT
Acc 0.9874 0.9809 0.9238 0.9355 0.9851 0.9954 0.9811 0.9863
Prec 0.9697 0.9363 0.9238 0.9357 0.9850 0.9955 0.9787 0.9846
Rec 0.9589 0.9599 0.9238 0.9354 0.9852 0.9954 0.9773 0.9834
F1 0.9642 0.9477 0.9238 0.9355 0.9851 0.9954 0.9780 0.9840
AUC 0.9694 0.9641 0.9338 0.9397 0.9880 0.9971 0.9820 0.9850
6 Conclusion
features) are not yet exploited. A natural extension is to build multimodal static architectures.
Malware detection and classification are critical, This would combine HilEnT images with complewith the high number of new malware released mentary static representations, which would allow which target varying industries. For malware visuus to study how safely collected dynamic telemealization, we proposed a novel malware binary to try could be fused with HilEnT without changing image transformation technique HilEnT based on the static only nature of the endpoint detector. a combination of Hilbert curve transformation, Finally, we have not systematically analysed benign and malware entropy cutoff comparisons the robustness of HilEnT to subtle code maniputo obtain a three channel image. We further lation. In particular, the response of our methodutilized this image to perform malware detecology to very slight malware injection into benign tion using three different application driven algofiles. This may produce overlapping HilEnT rithms: simple CNN based supervised learning, images for infected goodware, clean benign, and HOG-PCA enhanced supervised learning for timefully malicious samples. Future work will invesperformance consideration and few-shot learning tigate the sensitivity of HilEnT to low footprint based approach for practical cases with limited injections, padding and other minor modificasamples. We evaluated the performance on four tions that preserve functionality. We also hope datasets to achieve comparable malware classificato study mechanisms such as multi-scale entropy, tion performance to the state-of-the-art methods local anomaly scoring, and uncertainty estimawith a focus on reduced inference time. tion to better separate these overlapping regimes and improve robustness in adversarial or grey area scenarios.
Abbreviations CNN Convolutional Neural Network 27
SVM Support Vector Machine
using deep convolutional neural networks. Elec-
MLP Multilayer Perceptron
tronics 13(20):4081
CSNN Convolutional Siamese Neural Network Bai Y, Xing Z, Li X, et al (2020) Unsuccessful
HOG Histogram of Oriented Gradients story about few shot malware family classifi-
PCA Principal Component Analysis cation and siamese network to the rescue. In:
PE Portable Executable Proceedings of the ACM/IEEE 42nd Interna-
RGB Red-Green-Blue tional Conference on Software Engineering, pp
FSL Few-shot Learning 1560–1571
Declarations
Bang J, Kim JN, Lee S (2024) Entropy sharing in ransomware: Bypassing entropy-based
• Availability of Data and Materials
detection of cryptographic operations. Sensors Publicly available datasets are referenced in the 24(5):1446 paper.
• Funding
Bensaoud A, Kalita J (2024) Cnn-lstm and trans-
Not applicable
fer learning models for malware classification
• Acknowledgments
based on opcodes and api calls. KnowledgeBased Systems 290:111543
We would like to express our sincere gratitude to the reviewers.
Bensaoud A, Kalita J, Bensaoud M (2024) A survey of malware detection using deep learning.
References
Machine Learning With Applications 16:100546
Alfarsi K (2024) Malware classification using fewChollet F, et al (2015) Keras. https://keras.io shot learning approach. Information 15(11):722 Chuang HY, Wang SD (2015) Machine learnAndriani S, Muniyandi RC, Balasundaram S ing based hybrid behavior models for android (2025) Cnn-automic: Combining convolutional malware analysis. In: 2015 IEEE International neural network and autoencoder to learn nonConference on Software Quality, Reliability and linear features for knn-based malware image Security, IEEE, pp 201–206 classification. Computers & Security 156:104507 Conti M, Khandhar S, Vinod P (2022) A few-shot Ashawa A, Lio P, Aro T, et al (2024) malware classification approach for unknown Enhanced image-based malware classification 28
family recognition using malware feature visu-
Hsiao SC, Kao DY, Liu ZY, et al (2019) Malware
alization. Computers & Security 122:102887
image classification using one-shot learning with siamese networks. Procedia Computer Science
Cui Z, Xue F, Cai X, et al (2018) Detection of 159:1863–1871 malicious code variants based on deep learning. IEEE Transactions on Industrial Informatics
Krčál M, Švec O, Bálek M, et al (2018) Deep
14(7):3187–3196
convolutional malware classifiers can learn from raw executables and labels only
Del Coco M, Carcagnı̀ P, Palestra G, et al (2015) Analysis of hog suitability for facial traits
Kumar S, Kumar A (2024) Image-based mal-
description in fer problems. In: Image Analy-
ware detection using convolutional neural net-
sis and Processing—ICIAP 2015: 18th Inter-
work with autoencoder in industrial internet
national Conference, Genoa, Italy, September
of things. Engineering Applications of Artificial
7-11, 2015, Proceedings, Part II 18, Springer, pp
Intelligence 133:108374
460–471 Lester M (2021) Pe malware machine learn-
practical
security
analytics
Detect It Easy Project (2023) Detect it easy.
ing
https://github.com/horsicq/Detect-It-Easy,
llc.
accessed: 2025-09-21
pe-malware-machine-learning-dataset/,
dataset.
https://practicalsecurityanalytics.com/
accessed: 2025-09-21 DikeDataset Project (2021) Dike dataset. https: //github.com/iosifache/DikeDataset, accessed:
Li J, He J, Li W, et al (2024) Syndroid: An
2025-09-21
adaptive enhanced android malware classification method based on ctgan-svm. Computers &
Fei-Fei L, Fergus R, Perona P (2006) One-shot Security 137:103604 learning of object categories. IEEE transactions on pattern analysis and machine intelligence
Lo WW, Yang X, Wang Y (2019) An xception
28(4):594–611
convolutional neural network for malware classification with transfer learning. In: 2019 10th
Hemalatha J, Roseline SA, Geetha S, et al (2021) IFIP international conference on new technoloAn efficient densenet-based deep learning model gies, mobility and security (NTMS), IEEE, pp for malware detection. Entropy 23(3):344 1–5
29
Luo JS, Lo DCT (2017) Binary malware image
Pedregosa F, Varoquaux G, Gramfort A, et al
classification using machine learning with local
(2011) Scikit-learn: Machine learning in python.
binary pattern. In: 2017 IEEE International
the Journal of machine Learning research
Conference on Big Data (Big Data), IEEE, pp
12:2825–2830
4664–4667 Ronen Ma Y, Li M, Zhang K (2024) Mi-maml: Classifying
R,
Radu
M,
Feuerstein
C,
et
al
(2018) Microsoft malware classification chal-
few-shot advanced malware using multi-head
lenge. arXiv preprint arXiv:180210135
meta-learning. Cybersecurity 7(1):45 Roseline SA, Geetha S, Kadry S, et al (2020) IntelNatani P, Vidyarthi D (2013) Malware detec-
ligent vision-based malware detection and clas-
tion using api function frequency with ensem-
sification using deep random forest paradigm.
ble based classifier. In: Security in Comput-
IEEE Access 8:206303–206324
ing and Communications: International SympoSavakis A, Sharma R, Kumar M (2014) Effisium, SSCC 2013, Mysore, India, August 22-24, cient eye detection using hog-pca descriptor. In: 2013. Proceedings 1, Springer, pp 378–388 Imaging and Multimedia Analytics in a Web Nataraj L, Jacob G, Manjunath BS (2010) Detect-
and Mobile World 2014, SPIE, pp 115–122
ing packed executables based on raw binary Vasan D, Alazab M, Wassan S, et al (2020) data. VRL, ECE 6 Imcfn: Image-based malware classification using Nataraj L, Karthikeyan S, Jacob G, et al (2011)
fine-tuned convolutional neural network archi-
Malware images: visualization and automatic
tecture. Computer Networks 171:107138
classification. In: Proceedings of the 8th interVinayakumar R, Alazab M, Soman K, et al (2019) national symposium on visualization for cyber Robust intelligent malware detection using deep security, pp 1–7 learning. IEEE access 7:46717–46738 Panda S, Tripathy S, Pattnaik R (2023) Transfer Vu DL, Nguyen TK, Nguyen TV, et al (2020) learning for image-based malware detection for Hit4mal: Hybrid image transformation for maliot. Sensors 23(6):3253 ware classification. Transactions on Emerging Paszke A (2019) Pytorch: An imperative style,
Telecommunications Technologies 31(11):e3789
high-performance deep learning library. arXiv preprint arXiv:191201703 30
Wang P, Tang Z, Wang J (2021) A novel few-shot malware classification approach for unknown family recognition with multi-prototype modeling. Computers & Security 106:102273 Wang Z, Li H (2023) Dae-ihog: An improved method for classification malware. In: Proceedings of the 15th International Conference on Digital Image Processing, pp 1–7 Wijayasiri T, Fok KW, Thing VL (2025) Enhanced consistency bi-directional gan (cbigan) for malware anomaly detection. arXiv preprint arXiv:250607372
31