Image-Based Malware Type Classification on MalNet-Image Tiny: Effects of Multi-Scale Fusion, Transfer Learning, Data Augmentation, and Schedule-Free Optimization
arXiv:2604.21153v1 [cs.CR] 22 Apr 2026
Ahmed A. Abouelkhaire, Waleed A. Yousef, and Issa Traoré Department of Electrical and Computer Engineering University of Victoria Victoria, BC, Canada
Abstract—This paper studies 43-class malware type classification on MalNet-Image Tiny, a public benchmark derived from Android APK files. The goal is to assess whether a compact image classifier benefits from four components evaluated in a controlled ablation: a feature pyramid network (FPN) for scale variation induced by resizing binaries of different lengths, ImageNet pretraining, lightweight augmentation through Mixup and TrivialAugment, and schedule-free AdamW optimization. All experiments use a ResNet18 backbone and the provided train/validation/test split. Reproducing the benchmarkstyle configuration yields macro-F1, denoted F 1macro , of 0.6510, consistent with the reported baseline of approximately 0.65. Replacing the optimizer with schedule-free AdamW and using unweighted cross-entropy increases F 1macro to 0.6535 in 10 epochs, compared with 96 epochs for the reproduced baseline. The best configuration combines pretraining, Mixup, TrivialAugment, and FPN, reaching F 1macro = 0.6927, Pmacro = 0.7707, AUCmacro = 0.9556, and Ltest = 0.8536. The ablation indicates that the largest gains in F 1macro arise from pretraining and augmentation, whereas FPN mainly improves Pmacro , AUCmacro , and Ltest in the strongest configuration. Index Terms—malware classification, static analysis, malware images, MalNet-Image Tiny, transfer learning, feature pyramid network, AdamW schedule-free
I. I NTRODUCTION Malware classification remains a difficult supervised learning problem because available corpora are heterogeneous, class distributions are imbalanced, and malicious software evolves faster than manually engineered feature sets. Imagebased static analysis offers a simple alternative representation pipeline: binaries are converted into images and processed by vision models without code execution. This line of work traces back to malware visualization, which showed that samples from the same family often exhibit similar textures and structural patterns in image form [1]. More recently, MalNetImage introduced a large public benchmark with more than 1.2 million malware images organized into 696 families and 47 types, together with the smaller MalNet-Image Tiny subset for faster experimentation [2]. This paper evaluates four components that have not been jointly studied on MalNet-Image Tiny in the cited literature:
i) a feature pyramid network (FPN) to mitigate scale variation introduced when binaries of different lengths are resized to a common image size, ii) schedule-free AdamW to reduce reliance on an explicit learning-rate schedule, iii) ImageNet pretraining, and iv) lightweight augmentation through Mixup and TrivialAugment. The full 47-type benchmark remains difficult: the MalNet-Image study reported type-level macroF1 values of only 0.47–0.48 for standard CNN backbones despite substantial differences in model size [2]. The objective here is therefore narrower and more controlled: to quantify the contribution of training strategy and multi-scale fusion on the fixed MalNet-Image Tiny split while keeping the backbone architecture compact. II. R ELATED W ORK AND DATASET C ONTEXT Prior work falls into three main groups. A. Sequence- and language-based malware modeling A first line of work treats opcode or API-call sequences as text-like data. Demirkiran et al. used ensembles of pretrained transformer models, including BERT and CANINE, and reported strong multiclass performance on Catak, Oliveira, VirusSample, and VirusShare datasets [3]. Yesir and colleagues compared fastText and BERT on API-call sequences extracted by dynamic analysis and found that fastText was often more efficient while remaining competitive on CSDMC, APIMDS, and a custom dataset [4]. Zhang et al. proposed CoDroid, which combines static opcode sequences and dynamic system-call sequences through a CNN–BiLSTM–attention architecture for Android malware detection [5]. These methods can capture sequential semantics, but they depend on sequence extraction pipelines and are not directly comparable to rawimage classification. B. Graph-based malware modeling A second line of work represents programs or behaviors as graphs. DLGraph combines function-call graphs and Windows API calls by embedding graphs with node2vec and then
TABLE I I MAGE - BASED MALWARE DATASETS RELEVANT TO THIS WORK . Dataset
Access
MalNet-Image [2] MalNet-Image Tiny [2] Virus-MNIST [12] Malimg [1] AndroDex [13] STAMINA [10] McAfee [14]
Public 1,262,024 Public 87,430 Public 51,880 Public 9,458 Public 24,746 Private 782,224 Private 367,183
Images Classes 696 43 10 25 179 2 2
learning joint representations with stacked denoising autoencoders [6]. AMalNet integrates graph convolutional networks with Independently Recurrent Neural Networks to model APIcall graphs and remote dependencies at scale [7]. GDroid formulates Android malware detection and family attribution as node classification in a heterogeneous graph connecting applications and Android APIs [8]. Graph methods capture structural dependencies that image models do not represent explicitly, but they also require graph construction and domainspecific parsers. C. Image-based malware classification Nataraj et al. introduced malware visualization and grayscale image classification using GIST descriptors on 9,458 samples from 25 families [1]. Alguliyev et al. combined grayscale malware images, the Radon transform, and transfer learning, reporting high accuracy on Microsoft Malware, IoTMalware, and MalNet-Image subsets [9]. STAMINA addressed file-size heterogeneity through file-size gating and separate image-width rules, then trained deep models on the resulting malware-as-image representation [10]. Kumar et al. evaluated fine-tuned ImageNet backbones such as VGG16, VGG19, ResNet50, and InceptionV3 for malware image classification [11]. Three gaps motivate this paper: limited use of the standard MalNet-Image Tiny split for controlled ablation, limited study of lightweight data-independent augmentation in malware imagery, and limited investigation of end-to-end multi-scale handling without manually deploying multiple models. D. Dataset coverage Table I summarizes image-based malware datasets relevant to this work. MalNet-Image Tiny is used because it preserves the public benchmark pipeline, including the fixed split protocol, while excluding the four largest types from the full dataset. This reduces computational cost and moderates, but does not remove, the class imbalance inherited from MalNet-Image. III. M ATERIALS AND M ETHODS A. Dataset and conversion pipeline MalNet-Image was constructed from Android APK files obtained from AndroZoo and labeled with Euphony, which aggregates antivirus labels from VirusTotal vendors [2]. The experiments use MalNet-Image Tiny, which contains 87,430 images from 43 classes with fixed splits of 61,201 for training,
Fig. 1. APK-to-image conversion pipeline. The DEX byte stream is reshaped into a two-dimensional grid and resized to 256×256; the three-channel variant assigns colors by DEX section.
8,743 for validation, and 17,486 for testing; these counts follow the benchmark’s 70/10/20 split design. The Tiny subset excludes the four largest types from the full benchmark and is therefore less imbalanced, although imbalance remains evident. The thesis source also notes apparent label inconsistencies, including near-duplicate type names such as “adsware” and “adware,” which may introduce additional label noise. Figure 1 summarizes the image-conversion pipeline. The DEX file is extracted from the APK, converted into a onedimensional array of 8-bit unsigned integers, reshaped into a two-dimensional grid according to the file-to-pixel-width mapping used in the dataset construction process, and then resized to 256 × 256 with a Lanczos filter. Two input variants are evaluated. The first uses a single grayscale channel (K = 1). The second uses three channels (K = 3) and color-codes byte positions by DEX section, including the header, identifier, class-definition, and data regions. B. Backbone, multi-scale extension, and transfer learning ResNet18 is used as the backbone because the MalNet benchmark reported a favorable trade-off between type-level performance and computational cost for this architecture [2], [15]. On the full benchmark, ResNet18 reached type-level macro-F1 of 0.47 with 12M parameters and 1.8 GFLOPs, whereas ResNet50 and ResNet101 reached 0.48 with 26M and 45M parameters and 3.9 and 7.6 GFLOPs, respectively [2]. These results motivate the use of ResNet18 as a compact backbone for isolating the effects of optimization, augmentation, and multi-scale fusion. To address scale changes induced by resizing binaries of different lengths, some experiments attach an FPN to intermediate ResNet stages. Let {C2 , C3 , C4 , C5 } denote the selected backbone feature maps and {P2 , P3 , P4 , P5 } the corresponding pyramid outputs, with P5 = Conv1×1 (C5 ). For i ∈ {2, 3, 4}, the top-down fusion rule is Pi = Conv1×1 (Ci ) + Upsample(Pi+1 ),
(1)
where Conv1×1 (·) denotes a lateral 1 × 1 projection and Upsample(·) doubles the spatial resolution. ImageNet pretraining is evaluated because prior work reported that pre-
trained visual representations remain useful even when the target images differ substantially from natural photographs [11], [16]. C. Optimization, losses, and augmentation The reproduced benchmark baseline uses AdamW, weighted cross-entropy, grayscale input, and ResNet18 [2]. The main optimizer studied here is schedule-free AdamW [17]. Let gt denote the stochastic gradient at iteration t, and let xt , yt , and zt denote the parameter sequences used by the optimizer. The second-moment estimate is vt , (2) vt = β2 vt−1 + (1 − β2 )gt2 , v̂t = 1 − β2t and the schedule-free update is yt = (1 − β1 )zt + β1 xt , ηt zt+1 = zt − √ gt − ηt λwd yt , v̂t + ϵ xt+1 = (1 − ct+1 )xt + ct+1 zt+1 ,
(3) (4) (5)
with η2 ct+1 = Pt t
ηt = η min 1,
, 2
i=1 ηi
t Twarmup
.
(6)
This optimizer is studied because it removes an explicit learning-rate schedule and the original method reports sensitivity to β1 , which was therefore included in the hyperparameter search. For class imbalance, both standard cross-entropy and weighted cross-entropy were evaluated. For a batch of size B and C classes, let pbc denote the predicted probability of class c for example b, and let ybc denote the corresponding one-hot target. The weighted loss is B
C
1 XX LWCE = − wc ybc log pbc , B c=1 b=1
Fig. 2. MalNet-Image Tiny class distribution. The Tiny subset is less imbalanced than the full MalNet-Image benchmark, but substantial imbalance remains across type–family pairs. TABLE II H YPERPARAMETER VALUES EXPLORED FOR SCHEDULE - FREE A DAM W. Hyperparameter
Values
Learning rate Weight decay Warmup steps β1 β2 Batch size
0.01, 0.001, 0.005 0.01 1000 0.9, 0.95 0.999 128
imbalance. For any class-wise metric Mc over C = 43 classes, the macro average is C
Mmacro = N wc = , Cnc
(7)
where N is the total number of training examples and nc is the number of training examples in class c. Standard cross-entropy is recovered by setting wc = 1 for all classes. The augmentation methods are Mixup [18], which forms convex combinations of two training pairs (x(a) , y(a) ) and (x(b) , y(b) ), x̃ = λmix x(a) + (1 − λmix )x(b) ,
(8)
ỹ = λmix y(a) + (1 − λmix )y(b) .
(9)
where λmix ∈ [0, 1], and TrivialAugment [19], which applies a single randomly selected augmentation operation with a sampled magnitude and does not require a search procedure. D. Evaluation protocol The experiments follow the benchmark-style fixed split. Performance is reported using macro-precision Pmacro , macrorecall Rmacro , macro-F1 F 1macro , and macro-AUC AUCmacro because macro-averaging assigns equal weight to each class and is therefore more appropriate than accuracy under class
1 X Mc . C c=1
(10)
This convention is used consistently for precision, recall, F1, and AUC. The selected checkpoint for each configuration is determined by validation F 1macro , with validation loss used only to break ties. Both optimization loss and F 1macro are reported because lower loss does not necessarily imply better class-balanced performance. IV. R ESULTS A. Experimental design and hyperparameter search The 15 experiments form a staged ablation. The sequence is: reproduce the published baseline; replace AdamW with schedule-free AdamW; compare weighted and unweighted cross-entropy; assess ImageNet pretraining; assess Mixup and TrivialAugment on grayscale and three-channel inputs; assess FPN; and finally combine the strongest settings. The schedule-free hyperparameter search covers learning rates {0.01, 0.001, 0.005}, weight decay 0.01, warmup steps 1000, β1 ∈ {0.9, 0.95}, β2 = 0.999, and batch size 128. The stronger schedule-free runs used β1 = 0.95.
Fig. 3. F 1macro versus Ltest across the ablation runs. Lower loss is associated with, but does not determine, higher F 1macro .
B. Full ablation results Table III reports the complete experiment set. Experiment 5 reproduces the benchmark-style baseline with F 1macro = 0.6510, which is consistent with the value reported for ResNet18 on MalNet-Image Tiny [2]. Experiment 6 shows that schedule-free AdamW with unweighted cross-entropy slightly exceeds this result at F 1macro = 0.6535 while requiring 10 epochs rather than 96. Across Experiments 9–15, the largest incremental improvements are associated with pretraining and augmentation. The strongest non-FPN model is Experiment 14 with F 1macro = 0.6927, and the strongest overall configuration is Experiment 15, which preserves the same F 1macro while improving Pmacro , AUCmacro , and Ltest . C. Observed effects of each component The reproduced baseline indicates that the implementation is consistent with the benchmark protocol. Comparing Experiments 1, 3, 4, and 6 suggests that weighted crossentropy does not provide a clear advantage under the schedulefree optimizer on the Tiny subset, which is less imbalanced than the full benchmark. Comparing Experiments 6, 9, and 13–15 indicates that pretrained weights are beneficial and that augmentation produces the largest gains in F 1macro . Relative to Experiment 9, Mixup alone increases F 1macro to 0.6698–0.6728, depending on the input representation; TrivialAugment alone increases F 1macro to 0.6764; and the combination reaches 0.6927. Adding FPN to the strongest pretrained and augmented configuration preserves F 1macro but improves Pmacro from 0.7670 to 0.7707, AUCmacro from 0.9552 to 0.9556, and Ltest from 0.8731 to 0.8536. V. D ISCUSSION The evidence in Table III suggests that the observed improvement is cumulative rather than attributable to a single change. Schedule-free AdamW mainly reduces the number of epochs required to reach baseline-level F 1macro . The larger gains are associated with ImageNet initialization and lightweight augmentation, which is consistent with prior reports that transfer learning can be effective for malware-image
classification [9], [11]. The comparison between grayscale and three-channel inputs is also informative: both representations remain competitive, and the results do not indicate a uniform advantage for one encoding across all settings. Instead, the strongest results arise when the input representation is combined with pretraining and augmentation rather than evaluated in isolation. The FPN hypothesis is only partially supported. Fixed resizing can change the apparent scale of similar bytecode regions across binaries of different lengths. Empirically, FPN improves Pmacro , AUCmacro , and Ltest under the strongest configuration, but it does not increase F 1macro beyond the best non-FPN model. This pattern suggests that multi-scale fusion may improve score calibration and class separation without materially changing the class-balanced error profile summarized by F 1macro . It is also consistent with the decision to study training strategy on top of a compact backbone rather than scaling depth alone, because the benchmark comparison among standard ResNet variants shows only small type-level differences at substantially higher computational cost [2]. Several limitations remain important. First, the experiments are systematic but not exhaustive. Second, evaluation is restricted to the provided MalNet-Image Tiny split rather than repeated resampling or cross-validation. Third, the dataset contains apparent label inconsistencies, such as near-duplicate type names, that may limit achievable performance. Fourth, optimization remains indirect because F 1macro is not used as the training objective. Figure 3 illustrates that lower loss does not always imply higher F 1macro . Finally, the reproduced performance on Tiny should not be interpreted as directly comparable to the full 47-type benchmark, because Tiny removes the four largest types and is less severely imbalanced. VI. C ONCLUSION On MalNet-Image Tiny, the reproduced ResNet18 baseline reaches F 1macro = 0.6510. Replacing AdamW with schedulefree AdamW and using unweighted cross-entropy increases F 1macro to 0.6535 while substantially reducing the number of training epochs. The best configuration combines ImageNet pretraining, Mixup, TrivialAugment, and FPN, yielding F 1macro = 0.6927, Pmacro = 0.7707, AUCmacro = 0.9556, and Ltest = 0.8536. Within the reported ablations, the principal gains arise from pretraining and augmentation, whereas FPN improves secondary metrics rather than F 1macro . Future work includes evaluation on the full MalNet-Image benchmark, improved handling of imbalance, refinement of the binary-to-image transformation, and dataset cleaning for label consistency. R EFERENCES [1] L. Nataraj, S. Karthikeyan, G. Jacob, and B. S. Manjunath, “Malware images: visualization and automatic classification,” in Proceedings of the 8th International Symposium on Visualization for Cyber Security, ser. VizSec ’11. New York, NY, USA: Association for Computing Machinery, Jul. 2011, pp. 1–7. [Online]. Available: https://doi.org/10.1145/2016904.2016908
TABLE III C OMPLETE ABLATION TABLE . PT: PRETRAINED ; I N : INPUT CHANNELS (1 = GRAYSCALE , 3 = RGB- LIKE DEX COLOR CODING ); TA: T RIVIAL AUGMENT; MU: M IXUP ; AF: SCHEDULE - FREE A DAM W; AW: A DAM W; CE: CROSS - ENTROPY; WCE: WEIGHTED CROSS - ENTROPY. M ETRICS ARE Pmacro , Rmacro , F 1macro , AND AUCmacro . ID PT FPN In TA MU Opt 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
N Y N N N N Y Y Y Y Y N Y Y Y
N N N N N N N Y N N N Y N N Y
1 3 1 1 1 1 3 3 1 3 1 3 3 3 3
N N N N N N N N N N N Y Y Y Y
N N N N N N N N N Y Y Y N Y Y
Loss
AF WCE AW CE AF CE AF CE AW WCE AF CE AW CE AF CE AF CE AF CE AF CE AF CE AF CE AF CE AF CE
Pmacro Rmacro F 1macro AUCmacro
Ltest
0.6455 0.6978 0.6715 0.6743 0.6651 0.6928 0.6907 0.6936 0.6951 0.7276 0.7315 0.7694 0.7098 0.7670 0.7707
1.7064 1.2115 1.7247 1.1375 1.8410 1.5553 1.3389 1.3520 1.1690 0.8586 0.8618 0.9023 1.0704 0.8731 0.8536
[2] S. Freitas, R. Duggal, and D. H. Chau, “MalNet: A Large-Scale Image Database of Malicious Software,” Sep. 2022, arXiv:2102.01072 [cs]. [Online]. Available: http://arxiv.org/abs/2102.01072 [3] F. Demirknran, A. Cayhr, U. Ünal, and H. Dağ, “An ensemble of pre-trained transformer models for imbalanced multiclass malware classification,” Computers & Security, vol. 121, p. 102846, Oct. 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ S0167404822002401 [4] S. Yesir and I. Sogukpinar, “Malware Detection and Classification Using fastText and BERT,” in 2021 9th International Symposium on Digital Forensics and Security (ISDFS), Jun. 2021, pp. 1–6. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/9486377 [5] N. Zhang, J. Xue, Y. Ma, R. Zhang, T. Liang, and Y.-a. Tan, “Hybrid sequence-based Android malware detection using natural language processing,” International Journal of Intelligent Systems, vol. 36, no. 10, pp. 5770–5784, 2021. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/int.22529 [6] H. Jiang, T. Turki, and J. T. L. Wang, “DLGraph: Malware Detection Using Deep Learning and Graph Embedding,” in 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), Dec. 2018, pp. 1029–1033. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/8614193 [7] X. Pei, L. Yu, and S. Tian, “AMalNet: A deep learning framework based on graph convolutional networks for malware detection,” Computers & Security, vol. 93, p. 101792, Jun. 2020. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0167404820300778 [8] H. Gao, S. Cheng, and W. Zhang, “GDroid: Android malware detection and classification with graph convolutional network,” Computers & Security, vol. 106, p. 102264, Jul. 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0167404821000882 [9] R. Alguliyev, R. Aliguliyev, and L. Sukhostat, “Radon transform based malware classification in cyber-physical system using deep learning,” Results in Control and Optimization, vol. 14, p. 100382, Mar. 2024. [Online]. Available: https://www.sciencedirect.com/science/ article/pii/S2666720724000122 [10] L. Chen, R. Sahita, J. Parikh, and M. Marino, “Stamina: Scalable deep learning approach for malware classification,” Intel and Microsoft, Tech. Rep., 2020. [Online]. Available: https://www.microsoft.com/ en-us/research/uploads/prod/2020/05/stamina.pdf [11] S. Kumar and K. Panda, “SDIF-CNN: Stacking deep image features using fine-tuned convolution neural network models for real-world malware detection and classification,” Applied Soft Computing, vol. 146, p. 110676, Oct. 2023. [Online]. Available: https://www.sciencedirect. com/science/article/pii/S1568494623006944 [12] D. Noever and S. E. M. Noever, “Virus-mnist: A benchmark malware dataset,” arXiv preprint arXiv:2103.00602, 2021. [Online]. Available: https://arxiv.org/abs/2103.00602 [13] A. Khan, M. Usama, B. B. Kamal, A. Ahmad, H. Malik, and S. Lee, “Androdex: Android dex images of obfuscated malware,” Scientific Data, 2024. [Online]. Available: https://www.nature.com/ articles/s41597-024-03027-3
0.6266 0.5975 0.6083 0.6320 0.6460 0.6301 0.6390 0.6499 0.6507 0.6411 0.6424 0.6269 0.6582 0.6537 0.6523
0.6276 0.6315 0.6317 0.6449 0.6510 0.6535 0.6536 0.6654 0.6656 0.6698 0.6728 0.6755 0.6764 0.6927 0.6927
0.9487 0.9434 0.9409 0.9475 0.9488 0.9451 0.9512 0.9492 0.9503 0.9519 0.9514 0.9521 0.9515 0.9552 0.9556
[14] M. Labs, “Mcafee dataset for malware detection,” 2020. [Online]. Available: https://www.mcafee.com/enterprise/en-us/assets/ white-papers/wp-machine-learning-malware-detection.pdf [15] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778. [16] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, “How transferable are features in deep neural networks?” Advances in neural information processing systems, vol. 27, pp. 3320–3328, 2014. [17] A. Defazio, Xingyu, Yang, H. Mehta, K. Mishchenko, A. Khaled, and A. Cutkosky, “The Road Less Scheduled,” May 2024, arXiv:2405.15682 [cs, math, stat]. [Online]. Available: http://arxiv.org/abs/2405.15682 [18] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond Empirical Risk Minimization,” Apr. 2018, arXiv:1710.09412 [cs, stat]. [Online]. Available: http://arxiv.org/abs/1710.09412 [19] R. Mueller and F. Hutter, “Trivialaugment: Tuning-free yet state-of-theart data augmentation,” arXiv preprint arXiv:2103.10158, 2021.