Self-Supervised Representations for Binary Program Clustering: From Empirical Study to Retrieval-Augmented Learning Martin Mocko∗† and Daniela Chudᇆ ∗ Faculty of Information Technology, Brno University of Technology, Brno, Czech Republic † Kempelen Institute of Intelligent Technologies (KInIT), Bratislava, Slovakia ‡ Faculty of Electrical Engineering and Information Technology, Slovak University of Technology, Bratislava, Slovakia
Abstract—Malware clustering is a critical task in cybersecurity that helps discover threats and analyze evolving malware families. While self-supervised learning (SSL) and tabular representation learning (TRL) have achieved breakthroughs in other domains, their application to binary program clustering (the task of clustering all incoming samples regardless of label) remains largely unexplored. This study presents the first systematic investigation of SSL and TRL methods for binary program clustering, conducted in two phases on the public Ember and Bodmas datasets. In Phase 1, we establish a performance ceiling by adapting prominent vision-based SSL models (BYOL, SimSiam, Barlow Twins, VICReg) for tabular data with supervised pair generation, finding that BYOL and SimSiam achieve performance comparable to fully supervised models, while Barlow Twins and VICReg significantly underperform. In Phase 2, we evaluate purely unsupervised TRL methods against strong baselines (PCA, Autoencoder, UMAP), demonstrating that VIME establishes a new state of the art for binary program clustering. Informed by these findings, we propose VIME-R, a retrieval-augmented extension of VIME that replaces random marginal-distribution corruption with retrieval-based augmentation to generate more informative training pairs. VIME-R further improves upon VIME, achieving 2.7%-5.8% higher Homogeneity on both datasets. Our results highlight retrieval-augmented tabular representation learning as a promising direction for enhancing automated malware analysis. Code will be made available. Index Terms—malware clustering, binary program clustering, self-supervised learning, tabular representation learning, retrieval-augmented learning
1. Introduction More than 80 million new malware samples were created in 2024 alone1 . Moreover, the average cost of recovery from a malware attack has risen from $1.82 million in 2023 to $2.73 million in 20242 . The trend of increasingly costly malware attacks motivates cybersecurity defenders to enhance 1. https://portal.av-atlas.org/malware/statistics 2. https://www.sophos.com/en-us/press/pressreleases/2024/04/ransomware-payments-increase-500-last-year-findssophos-state
100 95
Homogeneity
arXiv:2608.02348v1 [cs.CR] 3 Aug 2026
Email: [email protected], [email protected]
Class
Binary Program Clustering Quality Family 95.6%
90 85 80 75 70
77.9% 78.5% 71.8% 73.0%
Unsupervised Baseline
+6.0% vs BSL
78.7%
+5.5% vs BSL
Unsupervised TRL
Supervised Ceiling
Figure 1. Clustering quality of TRL methods for binary program clustering (averaged across datasets), measured by Homogeneity on class and family labels. Gains are shown as absolute percentages over strong unsupervised baselines (BSL). The ’Supervised Ceiling’ illustrates the performance gap between the group of fully supervised models/supervised SSLs and unsupervised solutions. Results computed as mean of the Homogeneity results for top-3 best methods for each method category.
their malware identification and prevention capabilities year after year. One task that can aid in the malware discovery and analysis process is malware clustering. It has the potential to identify new (potentially even zero-day) malware families [1], aid domain experts in malware analysis tasks3 , and can aid in creating compressed representative datasets [2], i.e., by sampling from the created clusters instead of using all collected samples. Moreover, binary program clustering - the task of clustering all incoming files, regardless of their potential label (i.e., malware, benign) was recently recognized to be a potential new way forward for clustering samples while still retaining high clustering quality [3]. However, to perform malware clustering efficiently, the input features (representations) presented to a clustering technique must be as representative and informative as possible. Such representations are often a result of a certain 3. https://www.humansecurity.com/learn/blog/satori-threat-intelligencedisruption-
set of conditions, i.e.: a) fully unsupervised - no labels are available, b) semi-supervised - few labels are available, c) fully supervised - all labels are available. In the last few years, many self-supervised (SSL) and contrastive learning (CL) approaches have been shown to work very well in the computer vision and natural language processing (NLP) domains [4]–[6]. However, the achievements in these domains do not automatically translate to the tabular data domain, specifically the malware domain. The main problem with transferring these methods is that they rely heavily on data augmentation to create pairs that serve as model inputs. These data augmentations aim to preserve semantic content while perturbing low-level (for example, image) statistics. Such augmentations are often not present in the case of tabular data. For NLP tasks, they are often sequential correlations (there is a certain sequential order in natural language text). The tabular domain does not contain these correlation advantages, and therefore creating data augmentations for tabular data is a much harder task. Nevertheless, multiple tabular representation learning (TRL) approaches were proposed over the last few years that also seem to contribute to the state of the art in self-supervised tabular representation learning [7]–[10]. Our work explores the potential of self-supervised and tabular representation learning in an unsupervised context to improve the quality of malware clustering, or rather, binary program clustering. For our baseline, we adopt and replicate the unsupervised representation learning methods used in [3] as a strong baseline (as confirmed by our experiments) and aim to beat it. We divide our experiments into two phases. The first phase of our experiments investigates the ceiling performance of well-known SSL methods that we adapted for the tabular domain. The ceiling is tested using pair creation based on the sample label, i.e., the pairs are supervised. The second phase of the experiments explores whether the reported improvements in recent tabular representation learning methods will also manifest in the malware domain in an unsupervised context (for binary program clustering). For the conducted experiments, we utilize the Homogeneity metric as our primary indicator of clustering quality. Multiple other malware clustering works [1], [3], [11]– [13] used Homogeneity or Purity (a very similar metric) as their main indicator of clustering quality in the past as well. The primary motivation is that, for clustering to be useful in the malware domain, it should first be able to create homogeneous clusters (of malware families). In addition, we report Homogeneity for both the class label (malware/benign) and the malware family label, emphasizing the differences between the two evaluation measures. Our results demonstrate that self-supervised tabular learning can significantly outperform existing state-of-theart approaches in binary program clustering. Specifically, we find that the performance ceiling of certain (originally vision-based) SSL approaches (when trained using supervised pairs) is comparable to fully supervised models, such as Multi-Layer Perceptrons (MLPs) trained on individual
family labels (see Figure 1). While we observe that classical SSL methods struggle without highly specific, familyaligned sample pairs, our best-performing fully unsupervised TRL solution (VIME-R) bridges this gap, offering a meaningful improvement over traditional unsupervised baselines and achieving new state-of-the-art results in binary program clustering on both Ember and Bodmas datasets, beating even vanilla VIME in the process. The contribution of our work is multi-fold and can be summarized as follows: • Our study is the first to investigate the applicability of self-supervised learning (SSL) and tabular representation learning (TRL) specifically for the task of binary program clustering (or, malware clustering, in general); • We demonstrate that unsupervised TRL methods, particularly VIME, can outperform strong traditional unsupervised baselines like PCA and Autoencoders, setting new state-of-the-art results for binary program clustering on the public Ember and Bodmas datasets; • By adapting foundational SSL methods via supervised pair generation, we establish a performance ’ceiling’ for binary program clustering. This benchmark enables a direct comparative analysis for novel data augmentation strategies, providing a baseline for their effectiveness in the malware domain; • We propose a novel retrieval-augmented corruption strategy for VIME, called VIME-R, that significantly improves the VIME method on the task of binary program clustering and achieves state-of-the-art results, beating all other tested unsupervised models. The rest of the paper is organized as follows. Section 2 explores the work related to this research. Section 3 introduces our approach undertaken to explore the possibilities of applying self-supervised and tabular representation learning approaches for the malware clustering task. In Section 4, results from the performed experiments are presented. It also introduces our proposed modification to VIME, called VIME-R. Section 5 draws conclusions from the experiments and discusses the broader implications of the work. Finally, section 6 concludes the work with final statements and takeaways.
2. Related Work 2.1. Self-Supervised and Contrastive Learning in the Malware Domain Despite numerous recent works in malware clustering [1], [3], [12]–[16], self-supervised, contrastive, and tabular representation learning remain under-explored. Mocko et al. [3] recently established baselines by comparing standard clustering and dimensionality reduction (PCA, Autoencoder, UMAP) on public benchmarks. Due to the scarcity of malware-specific clustering literature using these techniques, we include relevant detection and classification studies. Some utilize non-tabular graph structures (e.g., CFGs) [17], [18], while others transform binaries into grayscale or RGB
images to apply vision-based augmentations [19], [20] or ViT-style masking [21]. Other approaches employ specialized pre-training: Carter et al. [22] use Linux syscalls in a BERT-like contrastive framework, while Wang et al. [23] and Trizna et al. [24] apply masked language modeling to static Ember data and dynamic reports, respectively. To address concept drift, CADE [25] combines autoencoders with supervised contrastive loss, and EVOLIoT [26] utilizes BERT with dropout-based positive pairing.
2.2. Self-Supervised Learning Methods This subsection introduces impactful non-contrastive self-supervised learning (SSL) methods originally designed for computer vision. All of the methods utilize some form of a Siamese neural network architecture. Additionally, all of the methods also learn based solely on positive sample pairs. A primary challenge in non-contrastive SSL is avoiding ”collapsing” solutions where the network produces identical outputs. BYOL [4] addresses this using a momentumbased “teacher” network to bootstrap latent representations, while SimSiam [5] simplifies this by using a stop-gradient operation instead of a momentum encoder. Alternatively, Barlow Twins [27] prevents collapse by minimizing the cross-correlation matrix of distorted samples, and VICReg [28] introduces explicit Variance, Invariance, and Covariance terms into the loss function to ensure architectural independence. While these methods achieve state-of-the-art results using image augmentations, their efficacy on tabular malware data remains an open question.
2.3. Tabular Representation Learning Methods Tabular representation learning was solidified as a research area only a few years ago, with the 2022 workshop at NeurIPS titled the “First Table Representation Learning (TRL) Workshop”4 . Nevertheless, several TRL methods have emerged that gained popularity. Modern tabular representation learning often relies on feature-level corruption. VIME [7] introduces this by replacing feature subsets with values from the empirical marginal distribution, using a “mask estimator” and reconstruction loss to learn representations. SCARF [8] adopts VIME’s corruption technique but replaces the reconstruction objective with a contrastive InfoNCE loss, reportedly yielding superior results. Most recently, SwitchTab [9] utilizes an asymmetric autoencoder to decouple mutual and salient features between sample pairs, reportedly outperforming VIME by better isolating unique sample characteristics. To conclude the section, our focus is on improving malware clustering. Given the successes SSL and TRL models have achieved (admittedly in other domains), we consider them a potential way to improve the quality of malware clustering. Because the introduced SSL and TRL approaches may appear very similar, one could speculate 4. https://neurips.cc/virtual/2022/workshop/49995
that it is sufficient to utilize only TRL methods, as they are specifically aimed towards tabular data. However, it is important to emphasize that the situation is not as clear-cut. While TRL methods are designed for tabular data, they are not plug-and-play, as one cannot design their own samplepair creation strategies for them. The only methods that are independent of the pair-creation strategies (beyond requiring positive pair inputs) are the SSL methods: BYOL, SimSiam, BarlowTwins, and VICReg. These methods support custom data augmentation (pair creation) strategies. Therefore, future improvements via custom augmentation techniques designed specifically for the malware domain could favor SSL methods over TRL methods. For TRL methods such as VIME, SCARF, and SwitchTab, the pair creation (or, rather, pair corruption) strategies are strictly defined. Therefore, tampering with them means altering the core of what makes the methods work so well. Moreover, the research conducted using SSL and TRL methods to improve malware clustering is demonstrably lacking. We are not aware of any malware clustering works that focus on improving clustering via SSL or TRL methods. Even the use of these methods on tabular data, even in the field of malware detection, has so far been scarce. Finally, we were unable to find any works that used most of the proposed SSL methods, even for tabular data in general (across other domains). All of the aforementioned facts serve as motivation for our work.
3. Clustering Malware and Benign Programs using TRL and SSL Representations First, we reiterate that the ultimate goal of this work is to investigate the potential of increasing the quality of malware clustering (specifically, binary program clustering). To achieve this, we aim to improve the underlying representation that the clustering model operates on. Recall that self-supervised learning and tabular representation learning have both made contributions regarding the state-of-theart in representation learning, as mentioned in Section 2. Therefore, we investigate whether the utilization of selfsupervised and tabular representation learning methods has the potential to improve the quality of malware clustering. More precisely, we consider the following two research questions: • RQ1: Do all of the proposed popular SSL methods achieve a similar level of Homogeneity when utilizing supervised pair generation? • RQ2: Is it possible to beat a strong unsupervised baseline and improve binary program clustering quality by utilizing self-supervised or tabular representation learning methods? Based on the two research questions we divide the research into two phases - Phase 1 and Phase 2. The methodologies for Phase 1 and Phase 2 are the same in terms of the utilized datasets, preprocessing aproach, and baselines. Afterwards, the methodologies of Phase 1 and Phase 2 diverge. In Phase 1, we aim to experimentally measure the potential ceiling of the devised popular self-supervised learning
methods. More broadly, this performance level could constitute the ceiling for all methods that learn representations based on supervised positive pairs. These SSL methods heavily rely on image augmentations and operate solely on positive pairs. In the malware domain, augmentation techniques to synthesize new malware/benign sample pairs are scarce. Moreover, they are often relatively simple (see [29]) and, more importantly, defined only for PE samples. This makes it impossible for researchers without access to the original binaries to perform the augmentations and observe the effect on the resulting vectorized features (which, arguably, can sometimes even be negligible). Therefore, to fully measure the ceiling of SSL methods, we use labeled data to generate positive sample pairs. Phase 2, on the other hand, operates solely in the unsupervised regime. Therefore, no class (i.e., malware/benign) or malware family labels are used at any point in the model training process. The two phases of our experiments use the same clustering approach and the same evaluation method. We reiterate that our specific malware clustering scenario is the binary program clustering scenario, where we cluster all the (incoming) samples, regardless of their label. Thus, we do not rely on a pre-filtering step that removes benign programs from the dataset. A high-level view of our methodological framework is shown in Figure 2.
Datasets Clustering
Preprocessing
Representation learning
Pair creation
Sampling from uniform distribution
Evaluation
Binary program clustering downstream task
Sampling from empirical marginal distribution
Label-based pair creation
Corruption by zero
Figure 2. The general process of our methodology. Different representation learning methods are used for Phase 1 and Phase 2. Four distinct methods of pair creation are utilized overall. This is mainly due to the corruptions defined for VIME, SCARF, and SwitchTab.
TABLE 1. BASIC STATISTICS ABOUT THE DATASETS UTILIZED FOR THIS STUDY. M/B RATIO REPRESENTS THE RATIO BETWEEN MALWARE AND BENIGN SAMPLES IN THE RESPECTIVE DATASET. Dataset
# features
Dataset size
M/B ratio
# families
Bodmas
2381
134 435 total
43:57
581
2381
600 000 train 200 000 test
50:50
∼3000
Ember v2
Bodmas rely on static features, which may offer a constrained view of PE behavior, they remain the standard for ensuring experimental repeatability and facilitating crossstudy comparisons. Summary information about the datasets is presented in Table 1.
3.2. Preprocessing & Baseline Preprocessing With regards to the preprocessing steps, we use the approach utilized by [3], who established the state-of-the-art results in malware clustering on Bodmas and Ember datasets. This preprocessing approach is undertaken in order to ensure comparability of results between these two works. Thus, we first utilize scalers from the scikit-learn5 preprocessing module in the following order: RobustScaler, StandardScaler, and MinMaxScaler. Furthermore, we remove the same set of Ember features (for each dataset) identified as harming representation learning in [3], which we gathered from their GitHub repository6 . This leaves us with 2235 Ember features for each dataset. Baseline For both of the utilized datasets, we adopt the unsupervised baselines (PCA using scikit-learn5 , Autoencoder using PyTorch7 , and UMAP using cuML8 ) from [3]. Similarly to [3], we also set the number of components (dimension size after dimensionality reduction) to 10. In the case of UMAP, we set the n neighbors parameter to 20, min dist to 0.1 and init to spectral. The baselines that we utilize are strong unsupervised baselines, as will be shown later in the experimental results. Each baseline method is run 5 times to obtain robust metric information during evaluation.
3.3. Phase 1: Supervised Pair Learning 3.1. Datasets Ember is a de facto standard for Windows PE static analysis [30]. Ember consists of features extracted from raw binaries. This study utilizes the Ember v2 2018 edition, which comprises 600 000 train and 200 000 test samples across approximately 3000 AVclass [31] families. Each sample is represented by a 2381-dimensional feature vector. The dataset includes the original feature extraction source code to ensure reproducibility. Bodmas, released in 2021, improves upon existing benchmarks by providing malware samples, expert-curated family labels, and temporal metadata (first-seen timestamps) [32]. It utilizes the full Ember feature set, ensuring compatibility between the two datasets. While both Ember and
SSL Models The aim of Phase 1 is to explore the potential ceiling of popular self-supervised learning methods in the yet-unexplored context of malware clustering (and in the tabular domain). Thus, the SSL methods used in this experiment are BYOL, SimSiam, BarlowTwins, and VICReg. All methods learn their representations by training on positive samples (samples that should be close in the representation space). The motivation for using these methods is that they should learn good representations as long as they receive strong signals from the positive pairs. At the time 5. https://scikit-learn.org 6. https://github.com/kinit-sk/clustering-ares-2025 7. https://pytorch.org/ 8. https://docs.rapids.ai/api/cuml/stable/
of writing this paper, we are not aware of any implementations of these methods for tabular data, so we created our own implementations while drawing inspiration from several GitHub repository implementations91011 . In addition to the SSL methods, we also train an MLP classifier intending to have another “ceiling baseline” which can be compared with the ceilings of the SSL methods. It is trained on the task of predicting malware families (with benign samples included), which could be viewed as similar to the supervised pairlearning task using SSL methods. Afterwards, we extract its latent feature vectors and cluster them, the same as with other SSL methods. Most of the implementation work on SSL methods consisted of replacing the (often CNN-based) model backbones with simple fully connected (FC) neural network layers, followed by a non-linear activation function. All methods use a Siamese neural network architecture. BYOL and SimSiam share the same loss function, whereas BarlowTwins and VICReg each have their own. The loss functions of BarlowTwins and VICReg additionally depend on one and three hyperparameters, respectively. We determine these hyperparameters based on preliminary experimentation. For BarlowTwins, we set λ to 0.16 for both datasets. For VICReg, we set the hyperparameter triple (invariance, variance, covariance) to (19, 30, 1) for Bodmas and (5, 100, 5) for Ember. The final embeddings, which can be used for downstream tasks (e.g., clustering), can then be extracted from the Backbone or the Projector. For our case, we extract the embeddings from the Projector. Nevertheless, we set the embedding size to 10 for both the Backbone and the Projector. We train all of the SSL methods with a batch size of 1024 for 800 epochs and save the model weights every 50 epochs. Moreover, to ensure robust metrics, we also run each (representation, dataset) combination five times. Supervised Pair Sampling We adopt the same supervised pair sampling approach for training all of the selected SSL methods. Let D = {(xi , yi )}N i=1 represent a dataset consisting of N labeled samples, where each sample xi ∈ X is associated with a label yi ∈ Y . For the label space Y it holds that yi ∈ {malware family labels, benign label}. Furthermore, we define a binary relation R on the set D as follows: ( R((xi , yi ), (xj , yj )) =
1, 0,
if yi = yj otherwise
We define one epoch, E , of training the SSL methods (on positive n pairs) as: (xi , xj ) | ∃j such that R((xi , yi ), (xj , yj )) = o 1, ∀xi ∈ D, 1 ≤ i ≤ N . E =
9. https://github.com/PatrickHua/SimSiam 10. https://github.com/facebookresearch/barlowtwins 11. https://github.com/facebookresearch/vicreg
3.4. Phase 2: Unsupervised Pair Learning SSL & TRL Models Phase 2 aims to explore the potential of improving malware clustering quality in a purely unsupervised manner. This makes the SSL methods utilized in Phase 1 potentially unfit for the task, as-is, because they rely solely on learning from positive pairs. Data augmentations for tabular data, in general, do not guarantee the creation of positive (or negative) samples. In fact, in the case of tabular representation learning, the “data augmentations” that are employed are often referred to as data corruptions [7]–[9]. Consequently, one of the tasks the TRL methods often solve is to reconstruct the feature vector of sample xi from a corrupted input x̄i [7], [9]. For Phase 2, we primarily use tabular representation learning methods that do not rely on positive pairs. We utilize popular tabular representation learning methods VIME, SCARF, and SwitchTab. Additionally, to preserve some level of comparability with Phase 1, we utilize the two best SSL methods used in Phase 1 - BYOL and SimSiam. We train the methods for 800 epochs and save the model weights every 50 epochs, as in Phase 1. We also run each (representation, dataset) combination five times to ensure robust metrics. Unsupervised Pair Generation This is the most crucial step for SSL and TRL methods as it greatly influences the quality of the learned embeddings. The way unsupervised pairs are generated differs across the TRL methods we use. VIME corrupts the input features by sampling from each feature’s empirical marginal distribution. Formally, we define the overall generating process of corruptions for the TRL methods as [7]: x̃ = g(x, m) = m ⊙ x̄ + (1 − m) ⊙ x
(1)
For all of the TRL methods, m corresponds to a binary masked vector m = [m1 , . . . , md ] sampled from a Bernoulli distribution with a probability of pm . In the case of VIME, the j -th feature of x̄ sampled from the empirical disPis N tribution p̂Xj = N1 i=1 δ(xj = xi,j ) where xi,j is the j -th feature of the i-th sample in D (i.e., the empirical marginal distribution of each feature). We set VIME’s rate of corruption, pm , to 0.3, the same as in the original work [7]. The same pair creation/corruption approach is used to generate pairs for BYOL and SimSiam. The same general description of a corruption formula applies to SCARF as in Equation 1. Moreover, SCARF originally uses the same approach for data corruption as VIME. However, during preliminary experimentation, we discovered that it learns better with a different corruption approach. Therefore, for SCARF, the j -th feature of x̄ is sampled from a uniform distribution created from the minimum and maximum of the feature values, specifically x¯j ∼ U (min(xj ), max(xj )). We use a corruption rate, pm , of 0.3 for SCARF. SwitchTab, on the other hand, uses a corruption-by-zero approach, in principle the same as for Denoising Autoencoders. Therefore, we can also define its corruption process using Equation 1 where the whole corrupted feature vector is
After the representations from Phases 1 and 2 have been trained, we extract 10-dimensional embeddings and cluster them using a clustering algorithm. Inspired by the positive results using K-Means in [3], we employ the K-Means algorithm via the CUDA-accelerated cuML library. Unlike the study of [3], however, we do not utilize an amount of clusters that would be much higher than the reported number of malware families. Instead, we keep the number of clusters on the level of the number of malware families in the train portion of the respective datasets - 560 in the case of Bodmas and 2750 in the case of Ember. The remaining hyperparameters are kept at their default values, specifically 300 iterations (n iter) and the scalable k-means++ initialization scheme. The k-means++ initialization scheme improves convergence stability and reduces the variance of the final clustering error. We cluster all representations saved after every 50 epochs of SSL/TRL training for each run of the (representation, dataset) combinations. Afterwards, we perform clustering quality evaluation as described in Section 3.6.
runs for each (representation, dataset) combination. This effectively results in 80 total model checkpoints. To ensure objective clustering quality evaluation, we report robust metrics for both Phase 1 and Phase 2. For Phase 1, we report the maximum Homogeneity (other metrics as well) per run, averaged across five runs, together with standard deviation. This is natural since in Phase 1 we operate in a setting where label information is available. Therefore, it is easy to pick the best-trained model state (for each run of the representation methods). In Phase 2, the situation is more complicated since we don’t assume access to labeled information where a practitioner could pick the best-trained model state based on such information. Therefore, we report mean Homogeneity averaged across five model runs (each run consists of 16 checkpoints), together with its standard deviation. Moreover, we also report maximum Homogeneity in the fashion as for Phase 1, also together with the standard deviation. These metrics are computed on the train set of the datasets. Other metrics like Completeness and V-Measure are also reported in the same fashion. We run our experiments on a machine with 26 CPU cores, 225 GB of RAM, 1 Nvidia H100 GPU, 1 TB of disk space, and a Linux LTS 24.04 operating system. We divide the H100 GPU into 4 MiG instances to effectively isolate and run experiments in parallel across 4 separate GPU parts.
3.6. Evaluation & Setup
4. Experimental Results
We view the utility and quality of a malware clustering model primarily in its function to group samples that are strongly correlated. Therefore, we determine our primary metric to be Homogeneity. As stated in Section 1, in the literature, there are plenty of other malware clustering works that place similar significance on the metric of Homogeneity/Purity [1], [3], [11]–[13]. Given that H represents the Shannon Entropy, we define Homogeneity h as: ( 1, if H(C, K) = 0 h= (2) 1 − H(C|K) , otherwise H(C)
In this section, we present an evaluation of the conducted experiments. We discuss and give answers to each of our research questions: (RQ1) Supervised Pair Learning: SSL model comparison for malware clustering compared with unsupervised state-of-the-art baselines, and (RQ2) Unsupervised Pair Learning: comparison of state-of-the-art unsupervised malware clustering baselines together with new unsupervised TRL approaches.
where H(C) represents class entropy of the true class labels and H(C|K) represents Conditional Entropy of the true class labels, C , given the cluster assignments, K . To also ensure a more robust evaluation, as well as to provide readers with additional metric information, we also compute Completeness and V-Measure. All of the reported metrics are computed for both the class labels (malware/benign) and the family labels. We define “family labels” for all samples in the dataset. Malware samples are assigned fine-grained family labels (e.g., Emotet, TrickBot), while all legitimate software is assigned the single, coarse-grained label of “benign”. As mentioned in the previous subsection, clustering is computed every 50 epochs for the methods used (except for the baselines, for some of which this is not possible). Using 800 epochs for training yields 16 measurements (checkpoints) for each method. Moreover, there are five
Table 2 summarizes the results for Phase 1 of our experiments. We present only the best results, as this task assumes the availability of labels; thus, choosing the best-trained representation is straightforward (based on label information; as opposed to unsupervised representation learning and clustering, where we presume no label availability). The best results are represented in the form of a “mean of maxes”, i.e. maximum Homogeneity that is averaged across five runs for all of the models. We include all the unsupervised baselines from [3]. The Autoencoder baseline is now, however, trained on 800 epochs to align the results with SSL models, which all utilized 800 epochs for training. Moreover, we train an MLP classifier (for 800 epochs) to evaluate results from a fully supervised perspective, directly optimizing for the malware family classification task. After the MLP classifier is trained, its internal latent representation is extracted and used as input to the clustering algorithm.
a vector of zeros, i.e. x̄ = 0d , where d is the dimensionality of the feature space. The feature corruption ratio, pm , was set to 0.3 as in the original work [9].
3.5. Binary Program Clustering
4.1. Phase 1: Supervised Pair Learning
TABLE 2. R ESULTS FOR P HASE 1 - S UPERVISED PAIR L EARNING . P ERFORMANCE OF BASELINES AND SSL METHODS REPORTED VIA H OMOGENEITY ON CLASS AND FAMILY LABELS . VALUES ARE REPORTED IN PERCENTAGES . Homogeneity (Mean of Maxes) Dataset
Method
Class-level
Family-level
BODMAS
PCA Autoencoder UMAP Barlow Twins BYOL MLP SimSiam VICReg
80.25 ± 1.09 84.48 ± 0.32 82.34 ± 0.60 77.40 ± 15.17 99.42 ± 0.32 97.94 ± 0.53 98.46 ± 1.05 38.88 ± 53.23
71.96 ± 0.79 76.51 ± 0.30 80.43 ± 0.28 68.39 ± 13.43 84.92 ± 1.61 87.85 ± 1.18 80.86 ± 2.01 28.85 ± 39.55
PCA Autoencoder UMAP Barlow Twins BYOL MLP SimSiam VICReg
58.02 ± 0.19 58.80 ± 0.12 67.13 ± 0.32 60.97 ± 2.96 91.36 ± 2.64 96.09 ± 0.54 90.40 ± 2.28 42.54 ± 16.23
66.13 ± 0.24 67.44 ± 0.23 75.55 ± 0.12 61.41 ± 5.21 75.87 ± 0.60 66.01 ± 6.86 76.53 ± 2.20 47.63 ± 12.10
EMBER
MLP does not always win. Despite being optimized directly for malware classification, the MLP-derived latent representations do not consistently yield superior clustering results in all aspects. On Bodmas, MLP dominates family Homogeneity with 87.85%, an almost 3% lead over the second best performer, BYOL. However, MLP’s performance is only top-3 in class Homogeneity on said dataset. Even though MLP can dominate class Homogeneity on Ember, its performance diminishes on family Homogeneity, arguably the more important indicator of overall clustering quality. These findings suggest that MLP’s discriminative power does not always generalize across different malware distributions. BYOL outperforms other SSL methods. Among the four SSL methods utilized in Phase 1 experiments, BYOL is the most consistent in achieving top results. It achieves the best results in terms of Homogeneity on class (99.42%) and Homogeneity on family (84.92%) among the four SSL methods on Bodmas. On Ember, it even outperforms MLP on family Homogeneity by a very large margin, more than 9%. In addition, SimSiam deserves an honorable mention, ranking second among the four SSL methods. SimSiam is even able to beat other models in family Homogeneity on Ember. Overall BYOL and SimSiam seem to perform relatively similarly. Barlow Twins and VICReg Underperform. Unlike BYOL and SimSiam, these methods significantly underperformed across all metrics. Both of these methods did not manage to beat the baselines on Bodmas. It is only on Ember that BarlowTwins is able to beat PCA and Autoencoder in class Homogeneity. VICReg performed worst overall, with its performance not being able to keep up with even the weakest of baselines. The high standard deviations say it all - the methods are hit-and-miss. Despite extensive tuning to prevent model collapse or poor performance, we
were unable to make BarlowTwins and VICReg perform consistently well across multiple runs. If this is not possible to do while providing them with the best signal they can get (supervised pairs), it is not a worthwhile effort to try and make them work in an unsupervised context (Phase 2). Supervised Pair Learning ceiling for binary program clustering established. Looking at the results of the experiments from a high-level perspective, three major top models seem to emerge - MLP classifier, BYOL, and SimSiam, although not without some occasional hiccups (mainly MLP underperforming on family Homogeneity for Ember). Based on the results, we establish the current ceiling for supervised pair learning (or fully supervised learning) for both datasets. On Bodmas, it is around 99% class Homogeneity and 87.85% family Homogeneity. On Ember, it is just over 96% class Homogeneity and 76.53% family Homogeneity. This, of course, holds true for the respective amounts of clusters that we utilized (560 for Bodmas, 2750 for Ember). The established ceilings serve as useful guides for assessing the remaining headroom for the unsupervised approaches used in Phase 2, as well as for any future experiments where researchers might want to try out their own custom data augmentation techniques. RQ1 Finding: The four utilized SSL methods are not created equal. BYOL and SimSiam greatly outperform BarlowTwins and VICReg on the task of binary program clustering when trained based on supervised pairs.
4.2. Phase 2: Unsupervised Pair Learning This section covers experiments conducted for Phase 2 using SCARF, SwitchTab, and VIME as promising tabular representation learning methods. Table 3 shows both the mean and the maximum Homogeneity of the Phase 2 methods (together with unsupervised baselines) averaged over five method runs (i.e. “Mean of Means” and “Mean of Maxes”). This is also the case for V-Measure. We remind the reader that one run results in 16 model snapshots (i.e. checkpoints) collected during the training over 800 epochs. The main reason for showing the mean of the Homogeneities (or other metrics) is that it better captures the average performance of the models when no reasonable guess can be made about the point (i.e., the epoch) at which the model training should be stopped (and the model selected). Difficulty in identifying the optimal training stopping point is common during pure unsupervised training, especially when the model loss is poorly correlated with downstream task performance. The maximum Homogeneity showcases the representation’s full capabilities. Moreover, we also use the two best-performing SSL models from Phase 1 (Section 4.1), i.e., BYOL and SimSiam, for these experiments. Since this is an unsupervised pair learning experiment, both BYOL and SimSiam receive pairs via VIME’s pair creation mechanism, i.e., corruption via sampling from the empirical marginal distribution. This showcases the mechanism purely as a “data augmentation”
TABLE 3. P HASE 2 R ESULTS : H OMOGENEITY AND V-M EASURE ANALYSIS OF LEARNED UNSUPERVISED REPRESENTATIONS . W E COMPARE THE AVERAGE (M EAN ) AND BEST (M AX ) PERFORMANCE ACROSS SSL AND TRL METHODS ON THE B ODMAS AND E MBER DATASETS , AVERAGED OVER FIVE EXPERIMENT RUNS . VALUES ARE REPORTED IN PERCENTAGES . Homogeneity ↑ Class-level
V-Measure ↑ Family-level
Class-level
Family-level
Dataset
Method
Mean of Means
Mean of Maxes
Mean of Means
Mean of Maxes
Mean of Means
Mean of Maxes
Mean of Means
Mean of Maxes
BODMAS
PCA Autoencoder UMAP SCARF SwitchTab VIME BYOLVIME SimSiamVIME
80.25 ± 1.09 82.37 ± 0.31 82.34 ± 0.60 84.35 ± 0.41 82.93 ± 0.45 87.02 ± 0.62 67.52 ± 5.02 72.60 ± 2.00
80.25 ± 1.09 84.48 ± 0.32 82.34 ± 0.60 85.50 ± 0.28 84.17 ± 0.41 87.85 ± 0.66 73.11 ± 3.91 77.80 ± 1.61
71.96 ± 0.79 74.60 ± 0.32 80.43 ± 0.28 76.41 ± 0.15 79.10 ± 0.42 80.87 ± 0.22 68.22 ± 3.21 72.73 ± 1.44
71.96 ± 0.79 76.51 ± 0.30 80.43 ± 0.28 76.93 ± 0.15 79.78 ± 0.47 81.66 ± 0.37 71.71 ± 2.53 75.25 ± 1.11
18.14 ± 0.21 18.24 ± 0.08 17.73 ± 0.16 18.51 ± 0.08 17.73 ± 0.05 18.94 ± 0.23 15.35 ± 1.10 16.03 ± 0.40
18.14 ± 0.21 18.67 ± 0.07 17.73 ± 0.16 18.77 ± 0.06 18.04 ± 0.07 19.13 ± 0.26 16.46 ± 1.03 17.11 ± 0.36
44.28 ± 0.40 45.17 ± 0.22 47.63 ± 0.25 45.94 ± 0.08 46.59 ± 0.14 48.30 ± 0.33 42.14 ± 1.49 43.93 ± 0.74
44.28 ± 0.40 46.06 ± 0.18 47.63 ± 0.25 46.24 ± 0.06 47.17 ± 0.19 48.78 ± 0.43 43.90 ± 1.35 45.25 ± 0.61
EMBER
PCA Autoencoder UMAP SCARF SwitchTab VIME BYOLVIME SimSiamVIME
58.02 ± 0.19 58.32 ± 0.30 67.13 ± 0.32 60.41 ± 0.05 64.56 ± 0.59 66.04 ± 0.41 51.09 ± 1.61 53.55 ± 0.74
58.02 ± 0.19 58.80 ± 0.12 67.13 ± 0.32 60.82 ± 0.06 65.29 ± 0.67 66.80 ± 0.38 52.92 ± 1.63 56.41 ± 0.92
66.13 ± 0.24 67.08 ± 0.37 75.55 ± 0.12 69.24 ± 0.05 72.03 ± 0.19 73.61 ± 0.50 64.82 ± 1.10 67.34 ± 0.43
66.13 ± 0.24 67.44 ± 0.23 75.55 ± 0.12 69.45 ± 0.05 72.57 ± 0.24 74.36 ± 0.32 66.24 ± 1.28 68.86 ± 0.87
11.00 ± 0.02 10.97 ± 0.04 11.36 ± 0.05 11.33 ± 0.01 11.57 ± 0.07 12.12 ± 0.11 9.41 ± 0.20 9.79 ± 0.12
11.00 ± 0.02 11.05 ± 0.03 11.36 ± 0.05 11.40 ± 0.01 11.70 ± 0.08 12.23 ± 0.11 9.78 ± 0.27 10.17 ± 0.12
41.96 ± 0.10 42.31 ± 0.17 43.94 ± 0.06 43.57 ± 0.02 43.79 ± 0.14 45.59 ± 0.16 40.24 ± 0.40 41.57 ± 0.23
41.96 ± 0.10 42.49 ± 0.14 43.94 ± 0.06 43.72 ± 0.04 44.06 ± 0.14 45.81 ± 0.18 41.02 ± 0.45 42.16 ± 0.19
technique aimed at creating positive samples, since BYOL and SimSiam do not employ any reconstruction or maskprediction mechanism, as in VIME.
RQ2 Finding 1: Tabular representation learning methods can beat strong unsupervised baselines and improve the purely unsupervised quality of deep representations for binary program clustering, achieving new state-ofthe-art results.
VIME beats strong unsupervised baselines. VIME achieves state-of-the-art results in Phase 2 experiments. Moreover, VIME consistently performs best among the tested TRL methods. On Bodmas, it beat out all unsupervised baselines and all TRL method contenders in a strong fashion - with a 2.35% lead in class Homogeneity and 1.2% lead in family Homogeneity. On Ember, it manages to get very close to the surprisingly strong UMAP baseline, achieving a class Homogeneity of 66.80% and family Homogeneity of 74.36%. UMAP was surprisingly strong on Bodmas as well, showing that it is, in fact, a very strong baseline and that UMAP belongs to the repertoire of modern representation learning methods. These very strong top-1 and top-2 placements of VIME point to the data corruption technique used by VIME as very effective for unsupervised tabular representation learning, and to VIME itself as a stateof-the-art method for binary program clustering.
VIME versus Supervised Pair Results. A striking observation is that VIME, despite being fully unsupervised, comes remarkably close to the supervised ceiling on family Homogeneity. On Bodmas, the gap between VIME (81.66% max) and the best supervised method (MLP, 87.85%) is only 6.2%. On Ember, this gap narrows further: VIME achieves 74.36% (max) versus SimSiam’s ceiling of 76.53%, a difference of merely 2.2%. This is particularly noteworthy given that SimSiam received explicit family-level supervision for pair generation, while VIME operated without any label information. These results demonstrate that VIME’s corruption-based learning captures much of the discriminative structure that supervised methods exploit, and suggest that improvements to the corruption strategy could close the remaining gap entirely. SCARF: consistent but limited performance. SCARF ranks third among the tested TRL methods, underperforming compared to SwitchTab and VIME in family Homogeneity on both Bodmas and Ember. However, it consistently outperforms the baselines of PCA and Autoencoder on both datasets and exhibits remarkable stability; with its standard deviation in mean and maximum family Homogeneity on both datasets being very low. These results suggest that while contrastive loss on unsupervised pairs may be sub-optimal for the high similarity found in binary program clusters, SCARF remains a robust, consistent choice for broader tabular domains. BYOLVIME and SimSiamVIME reveal both good and bad news. The bad news about BYOL’s and SimSiam’s performance when utilizing VIME’s data augmentation for pair learning is that they are, on average, not able to reach the performance of even our unsupervised baselines. Their
SwitchTab claims second place among TRL methods. SwitchTab is a very strong contender to VIME primarily in family Homogeneity, where their results are often very similar (around 1% to 2% apart). Even though SwitchTab never beat VIME, it is very close in performance mainly on the Ember dataset. The performance difference between VIME and SwitchTab is more significant on class Homogeneity, where SwitchTab performs 1.5%-4% worse. Overall, SwitchTab deserves its place as the second-best and shows that the use of corruption by zeros, paired with decoupling mutual and salient features, helps learn highly informative unsupervised representations.
best performance (according to Table 3) sometimes matches that of PCA and Autoencoder baselines. However, this holds mainly for family Homogeneity and does not hold for class Homogeneity, where both SimSiamVIME and BYOLVIME often lose by more than 5 percentage points. The good news, on the other hand, is that the VIME “data augmentation” was strong enough for BYOLVIME and SimSiamVIME to reach a relatively good performance, considering that we presume to be learning on positive pairs; which is not necessarily the case with VIME augmentation (corruption) strategy. RQ2 Finding 2: Positive-pair-based SSL methods using VIME data augmentation fail to match the performance of unsupervised baselines and state-of-the-art tabular representation learning methods on the binary program clustering task.
TABLE 4. N EIGHBOR PURITY ANALYSIS ON THE E MBER DATASET. F OR EACH VALUE OF k , WE REPORT THE AVERAGE NUMBER OF SAME - CLASS AND OPPOSITE - CLASS SAMPLES AMONG THE k NEAREST NEIGHBORS , ALONG WITH THE PURITY ( PERCENTAGE OF NEIGHBORS SHARING THE SAME CLASS LABEL ). Malware samples
Benign samples
k
Same
Opp.
Purity
Same
Opp.
Purity
1 5 10 50 100 500 1000 2000
1.00 4.74 9.31 44.80 88.03 418.19 811.29 1574.83
0.00 0.26 0.69 5.20 11.97 81.81 188.71 425.17
100.0% 94.8% 93.1% 89.6% 88.0% 83.6% 81.1% 78.7%
1.00 4.69 9.20 44.03 86.40 408.89 788.81 1526.82
0.00 0.31 0.80 5.97 13.60 91.11 211.19 473.18
100.0% 93.8% 92.0% 88.1% 86.4% 81.8% 78.9% 76.3%
feature of x̄i from the local empirical marginal distribution conditioned on the neighborhood Nk (xi ):
4.3. VIME-R: Retrieval-Augmented Corruption of Binary Programs
k
1X p̂Xj |Nk (xi ) = δ(xj = xil ,j ) k
(4)
l=1
4.3.1. Motivation. The results from Phase 2 identify VIME as the strongest unsupervised method among those that were tested. However, VIME’s corruption mechanism samples from marginal distributions without regard to sample similarity (as well as class label or malware family information). We hypothesize that retrieval-based corruption, replacing features with values from semantically similar samples, can produce more informative augmentations, even more so in the domain of malware. In theory, the samples that are the closest to another sample have the best chances of belonging to the same, or a very similar, malware family. We speculate that this could, therefore, be an optimization of VIME for improving the model’s recognition of malware families. The hypothesis is further supported by an analysis of nearest neighbors on the Ember dataset presented in Table 4. It clearly shows that the closest neighbors which share the same label are usually found somewhere in the Top-1 to Top-100 range. After that point, label purity drops off. We thus propose VIME-R, Value Imputation and Mask Estimation with Retrieval. The method is a modification of VIME where the corruption process is augmented by sampling feature values only from the marginal distribution of the k nearest neighbors of a given sample, rather than from a global marginal distribution. To our best knowledge, ours is the first work to utilize this alternative approach for the corruption process of VIME. VIME-R modifies the corruption process by restricting the sampling pool to a sample’s local neighborhood. Let Nk (xi ) = {xi1 , xi2 , . . . , xik } denote the set of k nearest neighbors of sample xi in D. The corruption process follows the same general form as Equation 1: x̃i = g(xi , m) = m ⊙ x̄i + (1 − m) ⊙ xi
(3)
However, unlike VIME, which samples from the global empirical marginal distribution, VIME-R samples the j -th
where xil ,j is the j -th feature of the l-th nearest neighbor of xi . The mask vector m is sampled identically to VIME from a Bernoulli distribution with probability pm . The aim for VIME-R is to be a purely unsupervised method in our context, the same as VIME. Therefore, we conduct experiments on VIME-R according to the methodology of Phase 2. We set the corruption rate, pm , to 0.3 (the same as for vanilla VIME). Based on preliminary experiments with k ∈ {50, 100, 200, 500} we found k = 100 to provide the best balance between neighborhood purity and corruption diversity. Smaller values produced overly conservative corruptions, while larger values diluted the locality signal. Therefore, the number, k , of the k-nearest neighbors is set to 100 neighbors for both Bodmas and Ember datasets. We train VIME-R for 800 epochs, collect 16 evaluation checkpoints on each run, and perform five runs of the method on each dataset overall. Method evaluation also follows the methodology of Phase 2. 4.3.2. Experimental Results. The results of the VIMER experiments are presented in Table 5. In it, we report relevant Homogeneity metrics for all three baselines and for the best performing method of Phase 2, which is VIME. For brevity and purposes of the best comparison, we report the mean Homogeneity across the five experimental runs of the methods (on both class and family labels), together with the respective standard deviations. The results tell a clear story: VIME-R is consistently able to beat the best performing methods of Phase 2 and thus achieve new state-of-the-art results. VIME-R outperforms all baselines on both datasets. Moreover, it beats VIME in all Homogeneity metrics, improving over VIME by 2.7% (class) and 3.0% (family) on BODMAS, and 5.8% (class) and 3.9% (family) on EMBER. This constitutes a significant improvement over the state-of-the-art.
TABLE 5. H OMOGENEITY ANALYSIS OF VIME-R AGAINST OTHER REPRESENTATIONS USED IN P HASE 2. Homogeneity (Mean of Means) Dataset
Method
Class-level
Family-level
BODMAS
PCA Autoencoder UMAP VIME VIME-R
80.25 ± 1.09 82.37 ± 0.31 82.34 ± 0.60 87.02 ± 0.62 89.77 ± 0.31
71.96 ± 0.79 74.60 ± 0.32 80.43 ± 0.28 80.87 ± 0.22 83.83 ± 0.54
EMBER
PCA Autoencoder UMAP VIME VIME-R
58.02 ± 0.19 58.32 ± 0.30 67.13 ± 0.32 66.04 ± 0.41 71.81 ± 0.45
66.13 ± 0.24 67.07 ± 0.37 75.55 ± 0.12 73.61 ± 0.50 77.48 ± 0.48
VIME-R surpasses the supervised ceiling on Ember. Perhaps the most remarkable finding is that VIME-R achieves 77.48% family Homogeneity on Ember, surpassing the supervised ceiling established by SimSiam (76.53%) in Phase 1. This means that a purely unsupervised method, relying solely on geometric proximity in the feature space, can produce clustering representations that are more informative than those derived from explicit family-label supervision. We attribute this to the locality bias introduced by neighborhood-restricted corruption: by ensuring that corrupted features remain within the local data manifold, VIME-R learns representations that preserve fine-grained cluster structure more effectively than global approaches, whether supervised or unsupervised. VIME-R Finding: Retrieval-augmented corruption via local neighborhood sampling (VIME-R) consistently and significantly outperforms vanilla VIME and all other unsupervised methods on both datasets, achieving new state-of-the-art results for binary program clustering. On EMBER, VIME-R is the only TRL method to surpass the strong UMAP baseline and the even stronger supervised ceiling.
5. Discussion 5.1. Interpretation and Implications of Findings The collective results from the conducted experiments on RQ1 and RQ2 indicate that both self-supervised and tabular representation learning models can be used for malware clustering (or, rather, binary program clustering). The SSL approaches used in Phase 1, which aimed to assess the quality of the learned representations of the most prominent vision-based SSL methods, demonstrated strong performance in the supervised pair-learning context. Moreover, “ceiling” for pair learning methods was established. If reasonable data augmentation strategies (capable of creating positive pairs) for the malware domain are discovered (applicable not just to PE samples but also to vectorized representations), this would catalyze the mass adoption of
such SSL methods in the malware domain. This holds mainly for the purely unsupervised use of SSL methods, where strong data augmentation is necessary. This is the way forward to getting closer to the established supervised pairlearning ceiling. Moreover, even the SSL methods utilizing supervised pair learning could become more frequently utilized in the future in the malware domain, as the (MLP versus BYOL) result on Ember indicates that, in certain cases, the representations extracted from SSL methods can outperform representations from fully-supervised models. Our results validate the efficacy of self-supervised tabular models, which achieved state-of-the-art performance in the binary program clustering task, beating strong unsupervised baselines by around 0.5-8% on class Homogeneity and 0.4-9% on family Homogeneity, depending on the specific dataset and method combinations. Notably, while UMAP provided very competitive results on both of the tested datasets, VIME remains more viable for large-scale applications. VIME scales efficiently with increasing sample sizes, whereas UMAP’s computational overhead becomes prohibitive as the dataset expands, presenting a significant limitation for real-time or high-volume malware analysis. The data augmentation approach applied to BYOL and SimSiam in Phase 2, based on VIME, points to a lack of data augmentations for tabular datasets. Even more so, effective data augmentations for malware domain features are also scarce. Augmentation approaches that could specifically create positive pairs in the malware domain (for both malicious and benign samples) are highly needed for SSL methods like BYOL and SimSiam, since both methods were unable to achieve clustering quality even at baseline levels of Autoencoder or PCA. The introduction of VIME-R further reinforces the importance of the corruption strategy in tabular representation learning. By restricting the sampling pool to a sample’s k nearest neighbors, VIME-R bridges the gap between fully global (VIME) and fully supervised (Phase 1) corruption strategies. The improvements of 2.7–5.8% in class Homogeneity and 3.0–3.9% in family Homogeneity over vanilla VIME demonstrate that even a simple locality-aware modification to the corruption process can yield substantial gains. Crucially, VIME-R is the only unsupervised TRL method to surpass UMAP on the Ember dataset, a baseline that proved surprisingly resilient throughout our experiments. This result suggests that incorporating structural information about the data manifold into the corruption process is a highly effective inductive bias for binary program clustering. Moreover, VIME-R achieves this without any label information, relying solely on geometric proximity in the feature space. This positions retrieval-augmented corruption as a promising middle ground between domain-agnostic corruptions (which may introduce too much noise) and supervised pair generation (which requires costly labeling).
5.2. Limitations Our work’s first limitation is the reliance on static features (Ember featureset) for both datasets. While static
analysis is computationally efficient, it may fail to capture complex runtime behaviors that dynamic analysis would reveal. Furthermore, as the threat landscape evolves, specific static features can become obsolete as attackers develop new evasion techniques to mimic benign file structures. The Ember (2018) and Bodmas (2020) datasets, while standard benchmarks in recent literature [1], [3], represent specific temporal snapshots. Given the rapid evolution of malware, these samples may not fully capture the most recent polymorphic or fileless threats. However, they remain the most robust public benchmarks available for evaluating large-scale representation learning in this domain. To isolate the impact of feature representations on clustering quality, we restricted our downstream evaluation to the K-Means algorithm. While alternative clustering methods (e.g., HDBSCAN or GMM) exist, using a consistent baseline ensures that observed improvements are attributable to the SSL/TRL representations rather than the clustering logic. Finally, while we conducted extensive preliminary experiments, the hyperparameter space for loss-function coefficients (e.g., in Barlow Twins and VICReg) and corruption probabilities (in VIME, VIME-R, and SCARF) is vast; further optimization might yield marginal gains but requires significant additional time investment and computational resources.
5.3. Future work Several promising directions exist for extending this research. First, incorporating dynamic analysis datasets would provide a more holistic view of model behavior, though this requires generating larger benchmarks to overcome current public data scarcity. Second, the hybrid VIMER architecture can be refined by implementing adaptive neighborhood sizes, distance-weighted sampling, or domainaware retrieval corruption to further close the gap to the supervised ceiling. Third, extending the framework into a semi-supervised learning (SSL) scenario would leverage its ability to handle small amounts of labeled data (5–10%), closely mirroring real-world cybersecurity triage. Finally, future iterations could transition from sampling marginal distributions toward modeling the joint probability distribution of features (inspired by [33]) to capture complex interfeature dependencies.
6. Conclusion Our study builds on recent developments in selfsupervised and tabular representation learning. This is the first research work to investigate the use of SSL and TRL methods for binary program clustering (an extension of malware clustering). We advance malware feature representation by achieving state-of-the-art results in binary program clustering through specialized tabular representation learning on the Bodmas and Ember datasets. Moreover, we explicitly report Homogeneity for both class (malware/benign) and family labels (e.g., Emotet, Mirai, etc.), which was often not the case in
past research. This shows the differences in how well the representations can distinguish between malware and benign software, as well as how they perform at distinguishing individual malware families. Phase 1 investigates popular SSL methods and their efficacy in supervised pair learning. We establish state-ofthe-art results in supervised pair learning for binary program clustering across both datasets and multiple SSL methods. BYOL achieves the best results among the SSL methods, achieving 99.42% class Homogeneity and 84.92% family Homogeneity on Bodmas, even beating the MLP classifier. This allowed us to establish a performance “ceiling” for binary program clustering. This benchmark enables a direct comparative analysis for novel data augmentation strategies, providing a baseline for their effectiveness in the malware domain. Furthermore, we find that not all of the popular SSL methods are created equal. BarlowTwins and VICReg significantly underperform BYOL and SimSiam. Phase 2 evaluates unsupervised binary program clustering using tabular and self-supervised representation learning. VIME established a new state-of-the-art, surpassing the leading binary program clustering baseline and, remarkably, outperforming a supervised MLP on family Homogeneity on Ember. SwitchTab similarly beat the baseline, confirming tabular representation learning as a top-tier approach for malware. While applying VIME’s feature corruption to BYOL and SimSiam did not match unsupervised baselines, it maintained reasonably high Homogeneity across class and family labels. This validates VIME’s corruption/augmentation process while highlighting a critical need for domainspecific data augmentation tailored to vectorized malware features. Finally, we proposed and experimentally validated VIME-R, a modification of VIME by introducing a retrievalaugmented corruption approach. VIME-R learns by corrupting samples via empirical marginal distribution only based on samples that are in their close local neighborhood. VIME-R managed to achieve new state-of-the-art results and do it convincingly, with 2.7% to 5.8% differences in Homogeneity over the second-best unsupervised method, VIME. Introduction of retrieval-based data augmentation is the first step towards supervised-level clustering quality for the unsupervised task of binary program clustering (and malware clustering).
Acknowledgment Acknowledgments have been omitted for the doubleblind peer-review process.
LLM Usage Statement The authors used an AI assistant to improve readability and language during manuscript preparation. The authors reviewed and edited the output and take full responsibility for the content of the publication.
References [1]
O. Jurečková, M. Jureček, M. Stamp, F. Di Troia, and R. Lórencz, “Classification and online clustering of zero-day malware,” Journal of Computer Virology and Hacking Techniques, vol. 20, no. 4, pp. 579–592, 2024.
[18] Y. Su and X. Wang, “An graph neural network approach with selfsupervised learning for malware detection,” in 2024 IEEE 7th Information Technology, Networking, Electronic and Automation Control Conference (ITNEC), vol. 7. IEEE, 2024, pp. 775–778. [19] S. J. I. Ismail, B. Rahardjo, T. Juhana, Y. Musashi et al., “Malssl—self-supervised learning for accurate and label-efficient malware classification,” IEEE Access, vol. 12, pp. 58 823–58 835, 2024.
[2]
U. Bayer, P. M. Comparetti, C. Hlauschek, C. Kruegel, and E. Kirda, “Scalable, behavior-based malware clustering.” in NDSS, vol. 9, 2009, pp. 8–11.
[3]
M. Mocko, J. Ševcech, and D. Chudá, “Clustering malware at scale: A first full-benchmark study,” in International Conference on Availability, Reliability and Security. Springer, 2025, pp. 231–251.
[20] F. Wang, Y. Chen, H. Gao, Q. Li, and C. Wang, “Self-supervised contrastive representation learning for classifying internet of things malware,” Engineering Applications of Artificial Intelligence, vol. 150, p. 110299, 2025.
[4]
J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar et al., “Bootstrap your own latent-a new approach to self-supervised learning,” Advances in neural information processing systems, vol. 33, pp. 21 271–21 284, 2020.
[21] F. Wang, X. Shi, F. Yang, R. Song, Q. Li, Z. Tan, and C. Wang, “Malsort: Lightweight and efficient image-based malware classification using masked self-supervised framework with swin transformer,” Journal of Information Security and Applications, vol. 83, p. 103784, 2024.
[5]
X. Chen and K. He, “Exploring simple siamese representation learning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 15 750–15 758.
[6]
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pretraining of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), 2019, pp. 4171–4186.
[22] J. Carter, S. Mancoridis, P. Protopapas, and B. Mitchell, “contrastbert: Behavioral anomaly detection for malware using contrastive learning,” in International Conference on Game Theory and AI for Security. Springer, 2025, pp. 167–186.
[7]
J. Yoon, Y. Zhang, J. Jordon, and M. Van der Schaar, “Vime: Extending the success of self-and semi-supervised learning to tabular domain,” Advances in neural information processing systems, vol. 33, pp. 11 033–11 043, 2020.
[23] Y. Wang, M. Xu, K. Luo, H. Tong, C. Jin, and B. Xie, “Bibe: A selfsupervised contrastive learning architecture for malware detection,” in 2023 IEEE 11th International Conference on Computer Science and Network Technology (ICCSNT). IEEE, 2023, pp. 251–257. [24] D. Trizna, L. Demetrio, B. Biggio, and F. Roli, “Nebula: Self-attention for dynamic malware analysis,” IEEE Transactions on Information Forensics and Security, vol. 19, pp. 6155–6167, 2024. [25] L. Yang, W. Guo, Q. Hao, A. Ciptadi, A. Ahmadzadeh, X. Xing, and G. Wang, “{CADE}: Detecting and explaining concept drift samples for security applications,” in 30th USENIX Security Symposium (USENIX Security 21), 2021, pp. 2327–2344.
[8]
D. Bahri, H. Jiang, Y. Tay, and D. Metzler, “Scarf: Self-supervised contrastive learning using random feature corruption,” arXiv preprint arXiv:2106.15147, 2021.
[9]
J. Wu, S. Chen, Q. Zhao, R. Sergazinov, C. Li, S. Liu, C. Zhao, T. Xie, H. Guo, C. Ji et al., “Switchtab: Switched autoencoders are effective tabular learners,” in Proceedings of the AAAI conference on artificial intelligence, vol. 38, no. 14, 2024, pp. 15 924–15 933.
[26] M. Dib, S. Torabi, E. Bou-Harb, N. Bouguila, and C. Assi, “Evoliot: A self-supervised contrastive learning framework for detecting and characterizing evolving iot malware variants,” in Proceedings of the 2022 ACM on Asia conference on computer and communications security, 2022, pp. 452–466.
[10] T. Ucar, E. Hajiramezanali, and L. Edwards, “Subtab: Subsetting features of tabular data for self-supervised representation learning,” Advances in Neural Information Processing Systems, vol. 34, pp. 18 853–18 865, 2021.
[27] J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny, “Barlow twins: Self-supervised learning via redundancy reduction,” in International conference on machine learning. PMLR, 2021, pp. 12 310–12 320.
[11] Y. Fang, W. Zhang, B. Li, F. Jing, and L. Zhang, “Semi-supervised malware clustering based on the weight of bytecode and api,” IEEE Access, vol. 8, pp. 2313–2326, 2019. [12] Z. Wilkins and N. Zincir-Heywood, “Cougar: clustering of unknown malware using genetic algorithm routines,” in Proceedings of the 2020 genetic and evolutionary computation conference, 2020, pp. 1195– 1203. [13] O. Jurečková, M. Jureček, and M. Stamp, “Online clustering of known and emerging malware families,” in Machine Learning, Deep Learning and AI for Cybersecurity. Springer, 2025, pp. 37–59. [14] M. Ali, J. Hagen, and J. Oliver, “Scalable malware clustering using multi-stage tree parallelization,” in 2020 IEEE International Conference on Intelligence and Security Informatics (ISI). IEEE, 2020, pp. 1–6. [15] N. MacAskill, Z. Wilkins, and N. Zincir-Heywood, “Scaling multiobjective optimization for clustering malware,” in 2021 IEEE Symposium Series on Computational Intelligence (SSCI). IEEE, 2021, pp. 1–8. [16] A. Mishra and M. Stamp, “Cluster analysis and concept drift detection in malware: A. mishra, m. stamp,” Journal of Computer Virology and Hacking Techniques, vol. 21, no. 1, p. 27, 2025. [17] Y. Gao, H. Hasegawa, Y. Yamaguchi, and H. Shimada, “Malware self-supervised graph contrastive learning with data augmentation,” Int. J. Adv. Secur, vol. 16, pp. 116–125, 2023.
[28] A. Bardes, J. Ponce, and Y. LeCun, “Vicreg: Variance-invariancecovariance regularization for self-supervised learning,” arXiv preprint arXiv:2105.04906, 2021. [29] L. Demetrio, S. E. Coull, B. Biggio, G. Lagorio, A. Armando, and F. Roli, “Adversarial exemples: A survey and experimental evaluation of practical attacks on machine learning for windows malware detection,” ACM Transactions on Privacy and Security (TOPS), vol. 24, no. 4, pp. 1–31, 2021. [30] H. S. Anderson and P. Roth, “Ember: an open dataset for training static pe malware machine learning models,” arXiv preprint arXiv:1804.04637, 2018. [31] M. Sebastián, R. Rivera, P. Kotzias, and J. Caballero, “Avclass: A tool for massive malware labeling,” in International symposium on research in attacks, intrusions, and defenses. Springer, 2016, pp. 230–253. [32] L. Yang, A. Ciptadi, I. Laziuk, A. Ahmadzadeh, and G. Wang, “Bodmas: An open dataset for learning based temporal analysis of pe malware,” in 2021 IEEE Security and Privacy Workshops (SPW). IEEE, 2021, pp. 78–84. [33] J. Li, Y. Zhang, Y. Huang, and K. Leach, “Malmixer: Few-shot malware classification with retrieval-augmented semi-supervised learning,” in 2025 IEEE 10th European Symposium on Security and Privacy (EuroS&P). IEEE, 2025, pp. 268–288.
Dataset
bodmas bodmas bodmas bodmas bodmas bodmas bodmas bodmas bodmas bodmas bodmas bodmas bodmas bodmas
ember ember ember ember ember ember ember ember ember ember ember ember ember ember
Method
autoencoder simsiamvime vicreg vime vime-r byolvime umap simsiam pca switchtab scarf mlp barlowtwins byol
autoencoder switchtab umap scarf simsiamvime vicreg pca vime barlowtwins vime-r simsiam byol mlp byolvime
Family 76.51 ± 0.30 75.25 ± 1.11 28.85 ± 39.55 81.66 ± 0.37 84.77 ± 0.65 71.71 ± 2.53 80.43 ± 0.28 80.86 ± 2.01 71.96 ± 0.79 79.78 ± 0.47 76.93 ± 0.15 87.85 ± 1.18 68.39 ± 13.43 84.92 ± 1.61 67.44 ± 0.23 72.57 ± 0.24 75.55 ± 0.12 69.45 ± 0.05 68.86 ± 0.87 47.63 ± 12.10 66.13 ± 0.24 74.36 ± 0.32 61.41 ± 5.21 77.79 ± 0.53 76.53 ± 2.20 75.87 ± 0.60 66.01 ± 6.86 66.24 ± 1.28
Class 84.48 ± 0.32 77.80 ± 1.61 38.88 ± 53.23 87.85 ± 0.66 90.83 ± 0.37 73.11 ± 3.91 82.34 ± 0.60 98.46 ± 1.05 80.25 ± 1.09 84.17 ± 0.41 85.50 ± 0.28 97.94 ± 0.53 77.40 ± 15.17 99.42 ± 0.32 58.80 ± 0.12 65.29 ± 0.67 67.13 ± 0.32 60.82 ± 0.06 56.41 ± 0.92 42.54 ± 16.23 58.02 ± 0.19 66.80 ± 0.38 60.97 ± 2.96 72.43 ± 0.36 90.40 ± 2.28 91.36 ± 2.64 96.09 ± 0.54 52.92 ± 1.63
Homogeneity
6.10 ± 0.02 6.43 ± 0.04 6.20 ± 0.02 6.29 ± 0.01 5.59 ± 0.07 26.17 ± 41.27 6.08 ± 0.01 6.74 ± 0.07 6.03 ± 0.26 6.90 ± 0.02 9.01 ± 0.15 8.88 ± 0.26 9.93 ± 0.59 5.39 ± 0.14
10.50 ± 0.04 9.61 ± 0.21 67.10 ± 45.05 10.74 ± 0.15 10.59 ± 0.03 9.27 ± 0.59 9.93 ± 0.09 12.54 ± 0.24 10.23 ± 0.12 10.11 ± 0.04 10.54 ± 0.03 11.89 ± 0.15 9.33 ± 1.92 12.25 ± 0.20
Class
31.02 ± 0.11 31.67 ± 0.17 30.98 ± 0.05 31.91 ± 0.05 30.44 ± 0.06 52.09 ± 27.88 30.73 ± 0.05 33.31 ± 0.24 26.83 ± 1.96 32.86 ± 0.15 33.83 ± 0.73 32.53 ± 0.26 30.18 ± 2.48 29.82 ± 0.29
32.96 ± 0.15 32.42 ± 0.47 77.19 ± 31.35 34.78 ± 0.37 34.43 ± 0.13 31.71 ± 1.03 33.83 ± 0.21 35.24 ± 0.69 31.97 ± 0.26 33.50 ± 0.14 33.07 ± 0.04 37.18 ± 0.36 29.10 ± 6.21 35.87 ± 0.40
Family
Completeness Class
11.05 ± 0.03 11.70 ± 0.08 11.36 ± 0.05 11.40 ± 0.01 10.17 ± 0.12 12.85 ± 2.12 11.00 ± 0.02 12.23 ± 0.11 10.98 ± 0.47 12.60 ± 0.05 16.38 ± 0.29 16.18 ± 0.48 18.00 ± 0.96 9.78 ± 0.27
Family
42.49 ± 0.14 44.06 ± 0.14 43.94 ± 0.06 43.72 ± 0.04 42.16 ± 0.19 37.54 ± 4.73 41.96 ± 0.10 45.81 ± 0.18 37.33 ± 2.87 46.20 ± 0.23 46.92 ± 1.10 45.53 ± 0.34 41.40 ± 3.57 41.02 ± 0.45
46.06 ± 0.18 45.25 ± 0.61 21.28 ± 29.22 48.78 ± 0.43 48.97 ± 0.23 43.90 ± 1.35 47.63 ± 0.25 49.01 ± 1.07 44.28 ± 0.40 47.17 ± 0.19 46.24 ± 0.06 52.25 ± 0.49 40.69 ± 8.36 50.41 ± 0.59
V-Measure
18.67 ± 0.07 17.11 ± 0.36 11.99 ± 16.43 19.13 ± 0.26 18.96 ± 0.06 16.46 ± 1.03 17.73 ± 0.16 22.24 ± 0.39 18.14 ± 0.21 18.04 ± 0.07 18.77 ± 0.06 21.20 ± 0.25 16.65 ± 3.39 21.79 ± 0.32
TABLE 6. C LUSTERING EVALUATION RESULTS ( MEAN OF MAXES ) ACROSS METHODS AND DATASETS .