Measuring Duplication, Leakage, and Rendering Dependence in Malware-as-Image Benchmarks

Measuring Duplication, Leakage, and Rendering Dependence in Malware-as-Image Benchmarks Stefan Larson and Kevin Leach

arXiv:2610.39351v1 [cs.CR] 29 Oct 2026

Vanderbilt University, Nashville, TN, USA {stefan.larson, kevin.leach}@vanderbilt.edu

Keywords:

Malware Classification, Image Classification, Byteplot Images, Dataset Duplication, Test-Train Leakage, Benchmark Evaluation.

Abstract:

Classifying malware binaries rendered as images is a popular approach to malware family identification, and papers on the standard benchmarks for this task report accuracies of 97–99%. In this paper we audit ten of these benchmarks for near-duplicate images and test-train leakage, and retrain standard classifiers under naive and content-aware splits. Deduplicating by file hash, which is the standard practice, does not prevent leakage, because distinct binaries render to identical images. Deduplicating on image content reduces top-1 accuracy by up to 8 points and macro-F1 by up to 15 points on the released benchmarks we find leaky. The rendering step also adds duplication, since byteplots of the same binaries contain 19.7% near-duplicates where static feature vectors contain 2.2%, and changing the width at which bytes are wrapped into rows reduces accuracy by 20 to 33 points, although no byte changes. We recommend that content-level duplication and a non-image baseline be reported alongside accuracy.

1

INTRODUCTION

Representing malicious code (malware) as an image is a popular approach to malware family classification. A binary is the file of raw bytes that contains a compiled program, and it can be read as a stream of unsigned 8-bit values and wrapped into rows of fixed width, which produces a grayscale image known as a byteplot. These images are used for family classification: a malware family is a group of programs derived from one code base, released as many variants by recompiling it, reconfiguring it, or running it through an obfuscator (a tool that rewrites a program so that its code differs but its behavior does not). Nataraj et al. (2011) observed that programs in the same family produce visually similar textures under the byteplot encoding, and reported 98.08% accuracy across 25 families using GIST descriptors and a kNN classifier. A large body of work has followed, and several benchmarks built to support it are now standard, including MaleVis (Bozkir et al., 2019), Dumpware10 (Bozkir et al., 2021), Virus-MNIST (Noever and Miller Noever, 2021), MalNet-Image (Freitas et al., 2022), AndroDex (Aurangzeb et al., 2024), MalVis (Makkawy et al., 2025) and MaleX (Mohammed et al., 2021). Papers that train CNN and Vision Transformer models on these benchmarks rou-

tinely report accuracies in the 97–99% range, and a recent survey of more than 100 papers in the area (Brosolo et al., 2025a) and a systematization of the same literature (Brosolo et al., 2024) both report this range. Despite this volume of work, the quality of these benchmarks has received little scrutiny. Duplication is one concern, since the programs in a malware family are variants of a common program and their shared code produces identical or near-identical images. The rendering is a second concern, since the width at which bytes are wrapped into rows is an arbitrary choice by the benchmark creators, and whether a model’s accuracy depends on that choice has not been tested. Leakage through duplicates is documented across machine-learning-based science (Kapoor and Narayanan, 2023), and it is present even in canonical vision benchmarks, where 3.3% of CIFAR-10 and 10% of CIFAR-100 test images have a near-duplicate in training (Barz and Denzler, 2020). The same defect is measured in document classification (Larson et al., 2023, 2026), face recognition (Wu et al., 2024), license plate recognition (Laroca et al., 2023) and open-domain question answering (Lewis et al., 2021), and most of the studies that report accuracy find that models score lower on test data that does not

have a counterpart in training. Studies of Android malware datasets of extracted features report mixed results on both effects (Pendlebury et al., 2019; Zhao et al., 2021), and whether either holds for the malware-as-image benchmarks has not been measured. In this paper, we measure duplicate content and test-train overlap in ten malware-as-image benchmarks and quantify the effect of both on reported accuracy. We identify duplicates by the decoded pixels of each image, because benchmark creators deduplicate by the hash of the program file, and two different files can render to the same image. On MalNetTiny, whose authors deduplicated by file hash, 14.9% of the test images are pixel-identical to a training image. We then build deduplicated, family-grouped splits and retrain standard classifiers on them, and we examine the rendering step itself, both as a source of duplication that the underlying corpus does not contain and as a choice on which the trained model turns out to depend. We make our audit code, deduplicated splits and per-run results publicly available at https://github.com/gxlarson/malware-image-audit. This paper is a benchmark audit, and we do not propose a new model, training procedure or dataset. Our contribution is to measure data quality inside the image benchmarks themselves and to quantify its effect on reported accuracy. This type of paper is in the tradition of prior work that has revealed similar problems in computer vision (Barz and Denzler, 2020; Radenović et al., 2018; Northcutt et al., 2021), NLP (Lewis et al., 2021) and document understanding (Larson et al., 2023, 2026) benchmarks. To summarize, the main contributions of this paper are: 1. We audit ten malware-as-image benchmarks, finding duplication rates from 0 to 35.7% and test-set leakage as high as 50.6%, and show that deduplicating by file hash, which is the standard practice, does not remove the leakage, since distinct files can contain identical code and render to identical images. On MalNet-Tiny, file-hash deduplication does not remove any images, so it does not recover any of the 8.44-point drop in accuracy that we measure under content-aware deduplication. 2. We construct content-aware, family-grouped splits and retrain three architectures on the seven benchmarks small enough for us to train on repeatedly, and find that accuracy drops by 3.7 to 10.5 percentage points on five of them, while the drop on the other two is not significant. A training set enlarged with further copies of its own images is no more accurate, so the drop comes from test images that have a copy in training and not from

the smaller training set. 3. We measure the contribution of the image encoding itself on the two corpora where this measurement is possible. MOTIF provides gold labels, raw binaries and static features together. MOTIF’s feature vectors contain 2.2% near-duplicates and do not lose accuracy under deduplication, while our byteplots of the same samples contain 19.7% and lose 10.52 ± 2.23 points. Dumpware10 provides the same binaries at four byteplot widths, and training at one width and testing at another reduces accuracy by 20.53 ± 2.71 points, although the bytes do not change, and by 33.08 ± 1.74 points on our renderings of MOTIF at five widths. We do not claim that malware-as-image accuracy is uniformly an artifact of duplicated data, since Malimg loses less than one point under our deduplicated split. Our claim is that inflation must be measured per benchmark, and that the image encoding is not a neutral re-representation of the binary.

2

BACKGROUND AND RELATED WORK

Task setup. Malware family classification is posed as n-way image classification. Each sample is one program rendered as one image, and its label is the family to which the program is assigned. A model is trained on a training split and evaluated by top-1 accuracy on a held-out test split. The benchmarks differ in which part of the program is rendered. Most are rendered from the Windows PE (Portable Executable) file on disk, Dumpware10 from the memory that a program occupies while it runs, Virus-MNIST from only the first kilobyte of the file, and the Android benchmarks from the DEX (Dalvik Executable) bytecode inside the Android application package (APK), which is not native machine code. Table 1 lists the ten that we audit, with the part of the program from which each is rendered, its encoding and the source of its labels. The task column is the label that we train and evaluate on: the malware family, the malware type (for example adware or trojan), a broader category, a k-means cluster, or benign versus malicious (detection). “Single AV” in the label column means the verdict of one antivirus vendor, while MOTIF’s labels come from the threat reports of 14 vendors and are alias-resolved by the MOTIF authors, so several reported family names map to one label. AndroDex contains each program twice, before and after obfuscation, and except where we state otherwise our AndroDex measurements use the 10,281 originals and

hold out the obfuscated variants. File hashes and image content. Malware corpora are conventionally deduplicated by hashing the distributed file. Consider two Android applications that differ only in a signing certificate and a version string. Their APK files have different SHA-256 hashes, so deduplication by file hash keeps both, but their bytecode is identical and both render to the same image. If one copy is in the training split and the other in the test split, the model is evaluated on an image that it has already seen in training. We therefore distinguish an image’s file hash, over the file as distributed, from its content hash, the SHA-256 of the decoded pixel array on which the model is trained. Malware as images. Nataraj et al. (2011) introduced the byteplot and reported 98.08% accuracy with GIST features and k-NN. They also noted that when variants of one family are processed by the same packer (a tool that compresses a program and wraps it in a stub that restores it at run time), “the images of the newly packed malware (of same family) are also similar,” which is one source of the near-duplicates that we measure. Later authors replaced GIST with CNNs and transformers and extended the idea to Android DEX bytecode and memory dumps (Bozkir et al., 2019, 2021; Freitas et al., 2022; Makkawy et al., 2025; Aurangzeb et al., 2024; Mohammed et al., 2021), reporting accuracies of 97– 99% on their benchmarks. Two recent surveys organize this literature by model and rendering scheme (Brosolo et al., 2025a, 2024), but neither examines duplication or leakage. Duplication in code and security datasets. There is a specific reason to expect duplication in malware datasets, since their samples are code and their classes are defined by shared code. Allamanis (2019) shows that duplicated code inflates the measured performance of models of code, and Croft et al. (2023) find data quality problems in vulnerability datasets that affect reported detection performance. TESSERACT (Pendlebury et al., 2019) shows that ignoring temporal and spatial constraints inflates Android malware detection results. Closest to our own measurement, Zhao et al. (2021) deduplicate Android datasets of extracted features and find that the accuracy of supervised classification is largely unchanged. We find the same for the feature vectors of MOTIF, which do not show a drop under deduplication, whereas a byteplot rendering of the same samples loses ten points (Section 5), so the two results differ in the representation and not in the corpus. Zhao et al. (2021) deduplicate

extracted features, whereas the image benchmarks are deduplicated, if at all, by file hash, and we identify duplicates in the decoded images. Prior work does not measure duplication and leakage inside the image datasets themselves.

3

AUDITING THE BENCHMARKS

We first set out how we index and compare images, then report what we find in the audit of the ten benchmarks.

3.1

Method

We audit the ten benchmarks listed in Table 1. Nine are distributed as image files, which we decode into the pixel array on which a model is trained. MOTIF is distributed as binaries and feature vectors, so we render its binaries as byteplots ourselves, using the convention described below. MALITE (Anand et al., 2025) also renders MOTIF’s binaries as grayscale byteplots and trains vision models on them, alongside Malimg, Dumpware10 and MalNet, so MOTIF is already used for image-based classification. Of MOTIF’s 454 malware families, 131 contain a single sample, and we train on the remaining 323 families, 2,964 samples in all. All of the benchmarks are byteplots, with MalVis being the exception, which encodes entropy over a 32-byte window (red and blue channels) and n-gram frequencies (green) into a fixed 256 × 256 RGB image, and we include MalVis because it allows the duplication question to be asked of a different encoding. MaleX distributes each of its programs both as a byteplot and as a bigram-DCT image, and we audit its byteplots and compare the two encodings in Section 5. A byteplot’s width is a fixed constant chosen by the benchmark’s creator, and its height is then determined by the file size, which means a larger program produces a taller image. We measure the effect of this choice of width on accuracy in Section 5. Because image heights therefore vary within a benchmark while CNNs expect a fixed input, every image is resized to 224 × 224 before training, without preserving aspect ratio, as the creators of MaleVis and MalNet-Image also do (Bozkir et al., 2019; Freitas et al., 2022). The resizing discards the file-size information encoded in the image height and rescales the texture by a perimage factor, which is relevant to the cross-rendering result in Section 5. For each image we record its content hash and file hash (Section 2), two perceptual hashes (pHash and dHash, which give visually similar images simi-

Table 1: The ten malware datasets analyzed in this paper. Rendered from Encoding

Malimg (Nataraj et al., 2011) whole PE file MaleVis (Bozkir et al., 2019) whole PE file Virus-MNIST (Noever and Miller Noever, 2021) first 1 KB Dumpware10 (Bozkir et al., 2021) memory dump MalNet-Tiny (Freitas et al., 2022) Android DEX MalNet-Image (Freitas et al., 2022) Android DEX MalVis (Makkawy et al., 2025) Android DEX AndroDex (Aurangzeb et al., 2024) Android DEX MOTIF (Joyce et al., 2021) whole PE file MaleX (Mohammed et al., 2021) whole PE file

Images

byte → gray 3 bytes → RGB byte → gray, 322 3 bytes → RGB byte → gray byte → gray entropy/n-gram byte → gray byte → gray byte → gray, 2562

lar hashes) and its dimensions. Two images are exact content duplicates when their content hashes match. Hashing the decoded pixels also makes the measure independent of re-encoding, so mirrors of a benchmark that differ only in compression yield the same content hashes. Two images are near-duplicates when the Hamming distance between their 64-bit pHashes is at most a threshold τ, which we set to 4. Below we check that this threshold matches copies of one program and that our conclusions do not depend on τ. Figure 1 shows the leakage in the MaleVis published split for τ from 0 to 10, which is 38.3% at τ = 0, where only hash-identical images are matched, 50.6% at τ = 4, and at least 38% across the whole range. At τ = 0, which merges only images with identical pHashes, the drop in accuracy on MaleVis is 3.67 percentage points (Section 4), so the drop does not depend on admitting near matches, and τ sets how much of it we measure. We checked in two ways that τ = 4 groups copies of one program and does not group different programs. First, if two images within distance 4 have different labels, the pair is either a false match or a labeling disagreement, so the share of such pairs is an upper bound on the rate of false matches. On five of the seven benchmarks that we retrain on (Section 4) this share is at most 0.15%, against 50–99% for random pairs. On Malimg and MalNet-Tiny it is 18.7% and 10.5%, but it is already 20.1% and 7.7% at τ = 0, so these pairs come from the labels and do not increase with the threshold: the Malimg pairs are all in one group of 106 Autorun.K and 800 Yuner.A images, and the largest mixed groups in MalNet-Tiny contain images with overlapping type labels such as adware and spyware. Second, on MOTIF we can compare the binaries behind each pair of images, using the distance between their TLSH digests (Oliver et al., 2013), a locality-sensitive hash of a file’s bytes for which a lower distance means more similar files. The median TLSH distance is 1 for image pairs at pHash distance 0 and 4 for pairs at distance 1 to 4, against 280 for other pairs from the same family and 390 for

Task

Classes

Label source

9,347 family 25 families single AV 14,226 family 26 classes single AV (Comodo) 51,880 cluster 10 classes k-means pseudo 4,294 family 11 classes single AV 87,430 type 43 types Euphony/VirusTotal 1,262,025 type, family 47 types, 696 fam. Euphony/VirusTotal 1,300,822 category 11 classes AndroZoo aggregate 21,507 detection 2 (benign/mal.) dataset construction 3,095 family 454 families threat reports 1,044,394 detection, family 2; 100 families Avast, BitDefender

τ=4

60

test images with a train near-duplicate (%)

Dataset

50 40 30 20

pHash

10

exact-content floor (5.1%)

0

0

2

4

dHash

6

both agree

8

pHash / dHash Hamming threshold τ

10

Figure 1: Leakage in the MaleVis published split against the pHash threshold τ.

pairs from different families. We then merge near-duplicate pairs transitively (union-find over content-distinct representatives) into near-duplicate groups, each containing one distinct image together with every copy of that image. From here on we work with these groups, reducing each group to a single representative when building deduplicated splits and counting a test image as leaked when its group also contains a training image. Where a benchmark provides a train/test split we measure leakage on that split, and where it does not, we use the same stratified random split as the papers that report results on it, and measure leakage there. Figure 2 shows one pair of distinct entries per benchmark: (a) two MalNet-Tiny APKs with different SHA-256 hashes, identical pixels and opposite sides of the published split, (b) an exact duplicate in our rendering of MOTIF, which is deduplicated by construction, (c) one of 41 identical Virus-MNIST images, and (d) a Dumpware10 pair that the procedure reports as distinct, as a negative control.

3.2

Duplication Audit Results

Table 2 reports the audit over the ten benchmarks described in Table 1. The Exact column is the share of all images that have at least one pixel-identical copy, and the Near column is the share of content-distinct images (one per exact group) that have at least one

Table 2: Duplication audit results. Test leakage is exact / including near-duplicates, n/m is not measured and — is not applicable. ‡ Hamming distance 0, so a lower bound. ¶ Cross-validated, so there is no single split. Dataset

Split

Exact

Near Test leakage

Malimg random 15.1% 22.5% 14.6% / 33% MaleVis published 7.3% 59.5% 5.1% / 50.6% Virus-MNIST — 35.7% 11.0% — Dumpware10 published 0% 22% 0% / 14.6% MalNet-Tiny published 16.9% 26.4% 14.9% / 36.2% MalNet-Image published 16.6% 36.8%‡ 14.4% / 40.7% MalVis random 15.6% 50.8%‡ 14.2% / 48.3%‡ AndroDex (all) — 18.5% 16.3% — AndroDex (originals) random 35.7% 19.3% 32.3% / 44.3% MOTIF 5-fold CV 2.2% 19.7% n/m¶ MaleX random 13.3% 32.9% 13.1% / 40.5% (a) MalNet-Tiny

(b) MOTIF

(c) Virus-MNIST

(d) Dumpware10

2 distinct APK hashes

pixel-identical

41 identical

distinct

Figure 2: Duplicate pairs found by the audit, one benchmark per column. Orange frames mark duplicates and blue the negative control.

other distinct image within pHash distance 4, so Near does not count exact copies and can be smaller than Exact. Rows marked published show the leakage in the benchmark’s own partition, which applies to any paper that uses that partition, and rows marked random show it for a stratified random 80/20 split of the kind the literature on that benchmark constructs, averaged over three seeds. For MalNet-Image we report its type split, and its family and binary splits have leakage within 0.1 points of it. Virus-MNIST provides per-binary hashes only for its test split, so we cannot measure its leakage. For comparison, leakage in the leaky benchmarks here is three to fifteen times the 3.3% and 10% near-duplicate rates measured in CIFAR-10 and CIFAR-100 (Barz and Denzler, 2020), while the cleanest, Dumpware10 at 14.6%, is within five points of CIFAR-100. We report three findings: (1) Deduplication by file hash. The authors of MalNet-Image report removing any sample whose APK file hash had already been collected, and we confirm this practice in our audit, finding 0% file-hash reuse and no image whose file hash appears on both sides of the split. Duplication in the pixel content re-

mains, however, and in MalNet-Tiny, the subset small enough to retrain on repeatedly, 16.9% of images are exact content duplicates and 14.9% of the test set is pixel-identical to training data, and the full 1.26Mimage set has a similar rate of 16.6%. MalVis shows the same gap between file hashes and image content independently, at 1.3M images and under a different encoding. Each file hash occurs once and under a single family label, yet 15.6% of its images are content duplicates and 14.2% of a naive test split is pixel-identical to training data. Because MalVis is not a byteplot, it is the stronger corroboration, since the duplication remains under a rendering that discards byte order and encodes entropy and n-gram statistics. MaleX shows the same gap for Windows PE files, over 1.04M programs: each SHA256 occurs once, yet 13.3% of its byteplots have a pixel-identical copy, and 17,317 of them are in groups of pixel-identical byteplots that contain both malware and benign programs. This gap arises because distinct APKs can contain identical DEX bytecode, as with repackaged applications, shared libraries and rebuilds that change only signing or manifest metadata, and identical bytecode renders to an identical image, so a hash of the distributed file does not identify the model’s input. Because MalNet-Tiny does not contain any filehash reuse, deduplicating it by file hash does not remove any images and leaves the naive split intact, so a file-hash pipeline does not recover any of the drop in accuracy that we measure under content-aware deduplication (Section 4). AndroDex is the converse case, over its 10,281 originals with the obfuscated variants set aside: file and content hashing remove the same 26.0% of the corpus, while near-duplicate grouping removes 34.4% (these are removal rates, the share of images removed when each group is reduced to one image, and they are lower than the rates in Table 2, which count every image in a group), and of the 44.3% of test images that have a duplicate in training in its naive split, file-hash deduplication removes only the 32.3 points that are pixel-identical. Between them, the two cases show that neither identity hashing nor exact-content hashing is sufficient. (2) Duplication by rendering scheme. The choice of which bytes to render accounts for the entire range of duplication rates in Table 2, from 0 to 35.7%. Memory dumps (Dumpware10) do not contain any exact duplicates, since runtime state differs even between samples of one family; whole-file byteplots have rates of 7–15% as shared headers and packer stubs recur; Android DEX renderings have rates of roughly 17%; and truncated-byte images are worst,

since Virus-MNIST renders only the first 1 KB and shared prologues produce identical 32 × 32 images. MalVis, which is not a byteplot, is in the middle of that range, so the effect is not specific to byte-to-pixel mapping. (3) Leakage in published splits. Under the published train/test partitions, 50.6% of the MaleVis test images, 36.2% of the MalNet-Tiny test images and 40.7% of the MalNet-Image test images have an exact or near-duplicate in training, meaning that published work using these partitions is evaluated with this leakage. Malimg does not have an official split, and in the random partitions used by papers on Malimg roughly 33% of the test images have a near-duplicate in training. Label quality. We also find problems with labels in our audit, although duplication is the issue that we quantify end-to-end. Virus-MNIST family labels are unsupervised k-means pseudo-labels, and re-running k-means on raw bytes recovers 77% of the published labels (ARI 0.69), which means that reported accuracy is substantially circular. Malimg contains degenerate classes such as Obfuscator.AD, which is 142 images forming a single near-duplicate cluster, and Autorun.K, which has the same structure. MaleVis has an Other class that is not a family. The 100 families of MaleX are Avast detection names: five families each appear as two classes that differ only in a suffix (Win32:Allaple and Win32:Allaple [Wrm]), 17 classes are generic detections such as Win32:Trojan-gen, and 19 programs are listed both as malware and as benign.

4

EFFECT OF DEDUPLICATION ON ACCURACY

The audit in Section 3 measures how much duplicated content each benchmark contains. In this section we retrain standard classifiers on splits in which no nearduplicate group spans training and test, and measure how much of the reported accuracy the duplication accounts for.

4.1

Protocol

We compare three splits per dataset, holding the training regime fixed: naive stratified random 80/20, applied uniformly so that the three conditions are comparable across benchmarks. The published ratios differ (MaleVis

provides a fixed 9,100/5,126 partition, MalNetTiny a 70/10/20 one and Dumpware10 an 80/20 one), so this condition represents the standard random split and is not the same as each benchmark’s own split. We also train and evaluate on the published splits of MaleVis, MalNet-Tiny and Dumpware10 (Section 4.2). dedup near-duplicate groups (exact content ∪ pHash ≤ 4) collapsed to one representative, then partitioned per family so that no near-duplicate spans the split. ctrl a control for training-set size, evaluated on the dedup test set. Its training set has the same size as that of dedup but is drawn at random from all images outside the dedup test set, so unlike dedup it can contain duplicates of other training images and of test images. Deduplication reduces the size of the training set, and MaleVis—the worst case—loses 58% of its images and is left with 4,757 for training, and Dumpware10’s deduplicated training set is the smallest at 2,813. Fine-tuning an ImageNet-pretrained network is the standard pipeline in this literature, as in the 97.48% that MaleVis’s authors report with DenseNet121 (Bozkir et al., 2019) and in the surveys of the area (Brosolo et al., 2025a, 2024). We therefore fine-tune ImageNet-pretrained ResNet-50 (He et al., 2016), DenseNet-121 (Huang et al., 2017) and ViTB/16 (Dosovitskiy et al., 2021), chosen to span a residual CNN, a densely connected CNN and a vision transformer, for 15 epochs with AdamW (Loshchilov and Hutter, 2019) at 10−4 , and report top-1 accuracy over three seeds with 95% Student-t intervals. Each dataset is evaluated on the task listed in Table 1, and all differences in accuracy are given in percentage points of top-1 accuracy. Because deduplication changes the class distribution of the test set, we also report macro-F1 and balanced accuracy (the mean of the per-class recalls). These metrics need per-image predictions, which the runs behind Table 3 did not save, so we repeated the ResNet-50 runs on the same naive, dedup and ctrl splits and saved them. The mean top-1 accuracy of the repeated runs under naive and dedup is within 0.32 points of Table 3 on every dataset. We also run three shallow baselines that do not learn a representation: LightGBM and logistic regression over each image downsampled to a 32 × 32 grayscale thumbnail and flattened to a 1,024dimensional vector, and 1-NN over its pHash. We compute the 95% interval of a drop between two conditions fromq the half-widths h1 and h2 of their two intervals, as h21 + h22 .

Table 3: Top-1 accuracy under the naive, dedup and ctrl splits, mean ± 95% CI over three seeds (∗ six, ‡ ten). Bold drops exclude zero. Dataset ResNet-50 Malimg Dumpware10 MaleVis MalNet-Tiny Virus-MNIST AndroDex‡ MOTIF∗

naive

dedup

drop

ctrl

99.61 ±0.08 97.98 ±0.88 96.40 ±1.95 78.73 ±0.42 92.30 ±0.97 96.31 ±0.45 63.43 ±1.51

99.11 ±0.56 +0.49 ±0.57 96.69 ±1.13 +1.30 ±1.44 90.28 ±1.48 +6.12 ±2.45 70.28 ±1.89 +8.44 ±1.93 87.05 ±1.78 +5.25 ±2.03 92.66 ±0.89 +3.65 ±1.00 52.91 ±1.65 +10.52 ±2.23

98.91 ±0.34 96.12 ±3.28 85.24 ±1.88 68.31 ±0.57 86.17 ±0.89 93.85 ±1.21 50.99 ±2.13

DenseNet-121 Malimg 99.18 ±1.69 Dumpware10 98.21 ±0.83 MaleVis 96.97 ±0.90 MalNet-Tiny 78.87 ±0.61 Virus-MNIST 92.78 ±0.42 AndroDex 96.03 ±1.37 MOTIF∗ 64.81 ±3.61

99.30 ±0.51 −0.12 ±1.77 96.97 ±2.04 +1.25 ±2.20 91.46 ±2.35 +5.51 ±2.51 71.00 ±1.41 +7.88 ±1.53 87.69 ±0.91 +5.09 ±1.00 91.08 ±4.68 +4.95 ±4.88 53.38 ±2.96 +11.43 ±4.67

— — — — — — —

ViT-B/16 Malimg Dumpware10 MaleVis MalNet-Tiny Virus-MNIST AndroDex MOTIF∗

98.23 ±2.46 +0.61 ±2.63 94.55 ±0.41 +1.76 ±3.70 89.80 ±0.67 +5.91 ±1.08 67.00 ±1.63 +8.28 ±2.20 87.10 ±1.03 +5.31 ±1.38 88.76 ±5.05 +5.31 ±5.29 47.67 +10.85 ±11.60

— — — — — — —

4.2

98.84 ±0.94 96.31 ±3.67 95.71 ±0.84 75.28 ±1.48 92.41 ±0.92 94.07 ±1.57 58.52

Results

Table 3 reports all three architectures over three seeds, except ResNet-50 on MOTIF and AndroDex, to which we added seeds (six and ten) because the interval on the drop was wide at three. We run ctrl for ResNet-50 only, over three seeds (six for MOTIF), and take its values from the repeated runs described above. MalNet-Image, MalVis and MaleX exceed a million images each and are audited but not retrained, and the ViT-B/16 intervals on MOTIF are too wide to be informative (naive alone is 58.52 ± 10.92). The datasets that lose accuracy are the ones in which the audit found leakage: the two in which it found little or no leakage do not lose a measurable amount (Malimg +0.49 ± 0.57, Dumpware10 +1.30 ± 1.44), while the four in which it found substantial leakage lose between 5.3 and 10.5 points. AndroDex is an intermediate case, since its ResNet50 interval excludes zero only at ten seeds (+2.85 ± 4.75 → +3.65 ± 1.00), while its DenseNet-121 and ViT-B/16 intervals remain at three seeds and still include zero, so the drop is established for one architecture of the three. On the five datasets that lose accuracy, the shallow baselines lose more than the CNN (6.6–17.5 points for LightGBM, 16.8–33.8 for 1-NN over pHash), so the deep model generalizes beyond texture matching.

Macro-averaged metrics. Families differ in how much of their content is duplicated, so deduplication changes the class distribution of the test set. In seed 0, the share of the largest class in the test set increases from 12.9% to 29.4% on MaleVis (the Other class) and from 31.7% to 45.9% on Malimg, and families with very few distinct images do not have a test image after deduplication (Autorun.K, Obfuscator.AD and Yuner.A in Malimg, VBA in MaleVis). The drop in macro-F1 excludes zero on the same five datasets as the drop in top-1 accuracy: it is 6.53 ± 2.86 points on MalNet-Tiny, 15.07 ± 1.27 on Virus-MNIST, 4.31 ± 1.33 on AndroDex, 11.11 ± 3.20 on MOTIF and 13.36±11.18 on MaleVis, while the drops on Malimg (2.19 ± 5.44) and Dumpware10 (1.02 ± 3.79) include zero. Balanced accuracy divides the datasets in the same way, and the per-split values of all three metrics are in our released results. Leaked and clean test images. Within one naive run the model and its training set are fixed, so we measure the effect of leakage without any change in training-set size by comparing the accuracy of the model on the test images that have a near-duplicate in training with its accuracy on the remaining test images. The leaked test images are classified correctly 94.1–100% of the time on every dataset. Because a leaked image may belong to an easier family than a clean one, we reweight the clean accuracy to the family mix of the leaked images, over the families that have both kinds of test image. After reweighting, accuracy on clean images is lower than on leaked images by 20.75 ± 1.16 points on MalNetTiny, 32.94 ± 4.48 on MOTIF, 14.40 ± 5.53 on VirusMNIST, 20.41±12.94 on MaleVis and 4.78±1.75 on AndroDex, while the differences on Malimg (5.57 ± 6.17) and Dumpware10 (2.78 ± 5.57) include zero. Training on the published training sets of MaleVis, MalNet-Tiny and Dumpware10 and evaluating on their published test sets yields differences of 15.81 ± 3.02, 21.96 ± 0.66 and 1.77 ± 1.99 points, so papers that report results on these splits are affected in the same way. Training-set size. Deduplication reduces the size of the training set, so part of the drop could come from training on fewer images. The ctrl condition does not separate the two effects, since its training set has fewer distinct images than that of dedup and can contain copies of test images. We therefore fix the dedup test set of each seed, remove from the pool every nearduplicate group with a member in that test set, and draw two kinds of training set from the rest: one image per group, as in dedup, or random images to-

gether with their duplicates. We train ResNet-50 on each kind at 25, 50, 75 and 100% of the size of the dedup training set, and also on the whole remaining pool, over three seeds. The training sets with duplicates are not significantly more accurate at any size, and at 100% they are less accurate than the training sets of distinct images by 4.73±2.13 points on MaleVis, 2.20±0.69 on MalNet-Tiny and 2.21±1.33 on MOTIF, while VirusMNIST (1.12 ± 2.05) and AndroDex (0.00 ± 2.58) do not show a difference. The whole remaining pool contains the same groups as the dedup training set and 1.2 to 2.3 times as many images, and training on it yields accuracy within 1.3 points of dedup on all five datasets (MaleVis 90.64 against 90.25, MalNetTiny 69.99 against 70.58, Virus-MNIST 86.04 against 86.76, MOTIF 52.16 against 52.39 and AndroDex 93.85 against 92.61). Accuracy therefore depends on the number of distinct groups in the training set, and adding duplicates of the training images does not raise it toward the naive accuracy, so the drop from naive to dedup comes from test images that have a copy in training and not from the smaller training set. Our result agrees with that of Dias et al. (2026) in face recognition, where removing 15% of the MS1MV3 training images as pHash near-duplicates does not reduce verification accuracy. Dumpware10 has the smallest deduplicated training set, 2,813 images, and its drop (+1.30 ± 1.44) is not significant. Temporal split. We also split MOTIF by the month of first appearance that its metadata records, training on the samples that appeared up to January 2020 and testing on the 299 samples in 53 families that appeared from February 2020 to May 2021, and compare against random splits with the same number of training and test samples per family. In the temporal split 2.3% of the test images have a near-duplicate in training, against 17.4–18.4% in the random splits, and ResNet-50 is 20.62 ± 5.84% accurate against 45.37 ± 3.84% over three seeds. The temporal split both removes most of the leakage and tests on later malware, so the 25-point difference combines the two effects, and it agrees with the lower accuracy that TESSERACT (Pendlebury et al., 2019) reports for Android detection under temporal splits. Relation to published results. A drop measured from a weaker training setup would not apply to published results, so we check that our naive accuracy matches each benchmark’s published result. Our naive accuracy exceeds the published result on three of the four benchmarks that report one, Malimg 99.61 against 98.08% (Nataraj et al., 2011), Dump-

ware10 97.98 against 96.39% (Bozkir et al., 2021), and Virus-MNIST 92.30 against roughly 80% (Noever and Miller Noever, 2021), and on the fourth, MaleVis, it is within half a point of the published result when measured in the same way, as we describe below. Three rows do not have a comparable published result: MalNet-Image reports macro-F1 rather than top-1 (Freitas et al., 2022), AndroDex is a data descriptor that does not report a benchmark result of its own (Aurangzeb et al., 2024), and MOTIF-asbyteplot is our rendering. The 97.48% that the MaleVis authors report with DenseNet-121 is a closed-set result on their published validation split, which leaves out the legitimatesoftware class Other (Bozkir et al., 2019). On the 3,644 malware images of that split our ResNet-50 yields an accuracy of 97.10%, or 97.26% when its prediction is restricted to the 25 malware families. Including the 1,482 Other images lowers top-1 accuracy to 87.66%, while balanced accuracy is 95.80%, because the Other images make up 29% of the published validation images but 4% of the training images, and 64.5% of them are classified correctly. The drops in Table 3 are therefore measured from the reported operating point. Sensitivity to the near-duplicate threshold. The deduplication criterion has one free parameter, the pHash Hamming threshold τ, fixed at 4 throughout. Figure 1 shows that the measured leakage is stable in τ, and so is the drop: rebuilding the MaleVis splits at τ ∈ {0, 2, 8} and retraining yields drops of 3.67, 5.87, 6.12 and 7.34 points at τ = 0, 2, 4, 8, monotonic in τ and never zero. At τ = 0, which collapses only hashidentical images, the drop is still 3.67 points, so τ sets how much of the drop is measured but not whether there is one. We quote point estimates because the per-family test composition is fixed across seeds, so other seeds would not measure the variation of this sweep. Generality across architectures. All three architectures lose accuracy on the same datasets, on identical splits. MaleVis loses 6.12 / 5.51 / 5.91 points and MalNet-Tiny 8.44 / 7.88 / 8.28 under ResNet-50 / DenseNet-121 / ViT-B/16, while Malimg and Dumpware10 remain within noise of zero on all three, so the drop depends on the dataset and not on the architecture. Shallow baselines. On naive splits, LightGBM over the thumbnails is within 5.1 points of the CNN on six of the seven datasets (98.4% against 99.6% on Malimg) and 15.6 points below it on MOTIF, while lo-

gistic regression is within 5 points only on Malimg and Virus-MNIST. Deep visual representation learning therefore does not appear necessary for most of the reported accuracies. Removal rates. Reducing each near-duplicate group to one image removes 31.6% of Malimg, 58% of MaleVis, 39.4% of Virus-MNIST and 34.4% of AndroDex. Obfuscation. AndroDex includes obfuscated variants of its originals, which an obfuscator has rewritten so that their code differs while their behavior is unchanged. Training on the originals and testing on the obfuscated variants, with zero exact-duplicate leakage between the two, reduces accuracy by 13.6 points (92.76 → 79.17), but accuracy remains far above chance, so the byteplot contains some signal that is invariant to obfuscation. The accuracy that remains is inconsistent with the model relying only on recognizing a fixed, repeated byte pattern such as a packer stub or a shared header.

5

THE EFFECT OF THE IMAGE REPRESENTATION

Section 4 does not establish whether the image encoding is itself responsible for the inflation, because the benchmarks differ in corpus, labeling and rendering at once, so we separate the effect of the rendering from the other two in two ways.

5.1

Comparing Representations on MOTIF

The experiment is a paired comparison of two representations of one corpus: we take a fixed set of samples, represent each sample both as a static feature vector and as a byteplot image, train a model on each representation over identical folds, and compare top-1 accuracy. Because the samples, the folds and the seeds are shared, any difference between the two is attributable to the representation and not to the corpus, the labels or the split. MOTIF (Joyce et al., 2021) is the one corpus on which this comparison is possible, since it provides raw binaries and EMBER-v2 (Anderson and Roth, 2018) feature vectors for the same samples, the standard static-feature baseline for PE malware. MOTIF’s family labels are also derived from published threat reports rather than from antivirus votes, and its samples are deduplicated by construction. We render exactly those

Table 4: Representations of the same MOTIF samples, 5fold CV over seeds 0–2 with identical folds in every row; n/m is not measured. Model

Representation

LightGBM EMBER-v2 (2,381-d) Logistic reg. EMBER-v2 features 1-NN EMBER-v2 features ResNet-50 LightGBM MalConv2

top-1 top-5 73.93 ±0.51 82.01 73.25 ±0.37 n/m 69.01 ±1.32 n/m

byteplot image 63.63 ±0.71 72.99 byteplot 32 × 32 pixels 48.44 ±0.68 58.90 raw byte sequence 48.20 ±0.95 59.62

samples as byteplots and emit fold lists from an identical StratifiedKFold call, so that the image and non-image numbers are paired on samples, folds and seeds. The MOTIF authors distribute its binaries disarmed so that they cannot execute, which has been given as a reason that MOTIF is “unsuitable as a malware image dataset” (Bruzzese, 2024). Disarming sets two fields of the PE header to zero, the machine type and the subsystem, and restoring them to the values in MOTIF’s metadata produces a file with the MD5 hash of the original for 3,091 of the 3,095 binaries and for all 2,964 that we train on. The rest of the header, the region to which a byteplot CNN attends most (Brosolo et al., 2025b), is unchanged, so a byteplot of a disarmed file differs from one of the original file in four pixels. Retraining ResNet-50 on byteplots of the restored files over the same folds and seeds yields 63.55 ±1.21, against 63.63 ±0.71 for the disarmed files (Table 4), so disarming does not affect the comparison with MOTIF’s EMBER features, which were computed from the original files. On those folds (Table 4) the byteplot CNN scores 63.63 ± 0.71 against 73.93 ± 0.51 for LightGBM on static features computed from the same binaries, roughly ten points less. It exceeds the weaker bytelevel models, LightGBM on 32 × 32 pixels (48.44) and MalConv (Raff et al., 2018) (48.20), by roughly 15 points, so the byteplot contains usable signal. MalConv is usually trained on 105 –106 samples, so its accuracy on 2,964 is not a ceiling. MOTIF’s feature representation has 0.1% exact and 2.2% near-duplicate rows and does not show a drop under deduplication. Rendering those same samples as byteplots produces 19.7% near-duplicates (Table 2) and a drop of 10.52 ± 2.23 points (Table 3). Since the corpus is already deduplicated, the duplication must be introduced by the rendering step, as shared headers, packer stubs and padding all render to near-identical textures. The rendering is therefore itself a source of duplication, which curation of the corpus cannot prevent. MaleX (Mohammed et al., 2021) allows the same

5.2

Cross-Rendering Transfer

The width at which a byteplot is wrapped is an arbitrary choice made by the benchmark’s creator, and changing it changes the image without changing any byte of the program. If a model depends on that choice, then part of what it has learned is a property of the rendering and not of the malware. Dumpware10 allows a test of this dependence, since it provides the same binaries √ rendered at four byteplot widths (224, 300, 4096, filesize). Re-wrapping a byte stream at a different width is content-preserving, in that no byte changes and only the positions of the row breaks change. The four renderings of a given binary are near-indistinguishable by eye, sharing global texture statistics to within 1.8 gray levels, yet they differ in 87–90% of pixels and correlate at only r = 0.44– 0.51. We train one model per width and evaluate each against every width’s test split. Every rendering is resized to 224 × 224, so the width changes where each byte is placed in the image and not the size of the model’s input. Test-set membership is identical across all four renderings (861 binaries; per-binary filename Jaccard 1.000 in every pair), so each cell varies only in how the same binaries were drawn. In the transfer matrix of Figure 3, the mean inrendering accuracy is 96.84 ± 0.48 and the mean cross-rendering accuracy is 76.31 ± 2.84, a drop of 20.53 ± 2.71 points with a worst cell at 57.07. A substantial part of what the model uses is therefore the 2D geometric alignment produced by the reshape. The structure of the transfer matrix is consistent with a dependence on geometry, since the two narrow widths transfer to each other reasonably well, and 4096 and

100

224

96.7

90.7

73.1

70.9

300

81.3

97.3

77.0

77.0

4096

59.0

66.7

97.3

92.3

√ size

64.8

73.1

89.7

96.1

pooled

98.5

98.2

98.5

98.5

224

300

4096

√ size

80

70

top-1 (%)

90

trained on width

comparison on an independent corpus of 1.04M Windows PE programs, since it provides each program in two encodings: a 256 × 256 byteplot and a bigramDCT image, the two-dimensional discrete cosine transform of a 256 × 256 histogram of consecutive byte pairs. Over the same programs, 13.3% of the byteplots and 6.4% of the bigram-DCT images are exact content duplicates, and of the 66,710 programs whose bigram-DCT image has an exact copy, 93% also have an exact copy as a byteplot, so the byteplot encoding of this corpus has twice the duplication of the other encoding. We compare only exact duplicates here, because pHash does not separate families on the bigram-DCT images, where 41.8% of the pairs at pHash distance 0 have different family labels, against 2.3% for the byteplots. MOTIF-as-byteplot is only 63% accurate and still drops 10 points under deduplication, so a low absolute accuracy does not indicate a clean benchmark.

60

50

tested on rendering width

Figure 3: Cross-rendering transfer on Dumpware10 (ResNet-50, mean of three seeds). The outlined diagonal is in-rendering accuracy, and the pooled row is one model trained on all four widths.

√ filesize transfer well to each other (89.70, 92.30) but poorly to the narrow widths (59.04–73.05), which suggests that the learned features are specific to a range of widths. MOTIF is not distributed as images, so we render its binaries ourselves at five widths, the four above and the file-size-dependent width of Nataraj et al. (2011) that our MOTIF audit uses, and train ResNet-50 at each width on the first fold of our cross-validation over three seeds. The mean inrendering accuracy is 63.78 ± 4.69 and the mean cross-rendering accuracy 30.70 ± 3.04, a drop of 33.08 ± 1.74 points, which is larger than on Dumpware10. Pooling renderings during training. Training a single model on all four renderings pooled yields an accuracy of 98.22–98.49% on every width, exceeding each width-specialized model on its own width by 1– 2 points. On MOTIF the pooled model is 64.9–65.9% accurate on every width, 0.1–3.3 points above each width-specialized model. The dependence on width can therefore be removed in training, and drawing the same binary at several widths is an augmentation that does not change any bytes or require additional data. We read this observation as one about how benchmarks are built: each benchmark is distributed in a single rendering, so models trained and evaluated on it can depend on that rendering without the dependence being reported. Brosolo et al. (2025b) augment over a transformation that changes content (packing) and recover accuracy, whereas we augment over one that preserves content and both recover transfer and improve accuracy on the diagonal.

6

DISCUSSION

The inflation of a benchmark depends on both its corpus and its rendering, so the inflation has to be measured for each benchmark, and we release the tooling for doing so. Malimg loses only 0.49 ± 0.57 points to content-aware deduplication, an interval that includes zero, and Dumpware10’s memory-dump images do not contain any exact duplicates, which makes it the cleanest representation we measured. A benchmark that passes a deduplication audit can still pose an easy task, as on Malimg, where gradient boosting over 32 × 32 thumbnails is 98.4% accurate and each of two families consists of a single near-duplicate group. We make three recommendations based on our measurements. (1) Deduplicate on the model’s input, reporting content-hash and perceptual-hash duplication rates alongside accuracy, since MalNet, MalVis and MaleX show that a file-hash deduplication pipeline still leaves 13–17% of images as content duplicates. (2) Render at multiple widths and pool, which removes the drop under a change of width on both Dumpware10 and MOTIF and improves inrendering accuracy (Figure 3), and we do not know of a benchmark that does this. (3) Report a nonimage baseline on the same samples, since static features with gradient boosting beat a fine-tuned CNN by 10 points on the one corpus where the comparison is properly paired, so the benefit of an image encoding needs to be demonstrated.

7

CONCLUSION

In this paper, we audited ten malware-as-image benchmarks for duplication and test-train overlap, rebuilt their splits without the duplicates, and retrained standard classifiers to measure the effect on accuracy. We find that deduplicating by file hash, which is the standard practice, does not prevent pixel-level leakage, since distinct files render to identical images. Content-aware splits reduce accuracy by 3.7 to 10.5 points on five benchmarks, while the drop on the other two is not significant. The rendering introduces duplication that the corpus does not contain, as well as a dependence on wrap width that accounts for a drop of 20 to 33 points. On MOTIF, where the comparison can be paired, the EMBER-v2 static features outperform the image encoding by 10 points on the same samples. Duplication of the model’s inputs and dependence on the encoding are not specific to malware, since wherever a corpus is converted to a different representation before training, deduplicating the distributed

files does not ensure that the model’s inputs are distinct, and the encoding can introduce structure that the corpus does not contain. Duplication of the inputs and dependence on the encoding can both be detected at low cost, by reporting duplication over the model’s input and by running one baseline that does not use the encoding, and these two checks would have detected each of the effects we report.

REFERENCES Allamanis, M. (2019). The adverse effects of code duplication in machine learning models of code. In Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!). Anand, S., Mitra, B., Dey, S., Rao, A., Dhar, R., and Vaidya, J. (2025). MALITE: Lightweight malware detection and classification for constrained devices. IEEE Transactions on Emerging Topics in Computing. Anderson, H. S. and Roth, P. (2018). EMBER: An open dataset for training static PE malware machine learning models. arXiv preprint arXiv:1804.04637. Aurangzeb, S., Aleem, M., Khan, M. T., Loukas, G., and Sakellari, G. (2024). AndroDex: Android Dex images of obfuscated malware. Scientific Data, 11(1):212. Barz, B. and Denzler, J. (2020). Do we train on test data? Purging CIFAR of near-duplicates. Journal of Imaging, 6(6):41. Bozkir, A. S., Cankaya, A. O., and Aydos, M. (2019). MaleVis: A dataset for vision based malware recognition. In IEEE Signal Processing and Communications Applications Conference (SIU). Bozkir, A. S., Tahillioglu, E., Aydos, M., and Kara, I. (2021). Catch them alive: A malware detection approach through memory forensics, manifold learning and computer vision. Computers & Security. Brosolo, M., Asmitha K. A., Conti, M., Rafidha Rehiman K. A., Muhammed Shafi K. P., Nicolazzo, S., Nocera, A., and Vinod P. (2025a). Security through the eyes of AI: How visualization is shaping malware detection. arXiv preprint arXiv:2505.07574. Brosolo, M., Puthuvath, V., Asmitha K. A., Rafidha Rehiman K. A., and Conti, M. (2024). SoK: Visualizationbased malware detection techniques. In Proc. 19th International Conference on Availability, Reliability and Security (ARES). Brosolo, M., Vinod P., and Conti, M. (2025b). The road less traveled: Investigating robustness and explainability in CNN malware detection. arXiv preprint arXiv:2503.01391.

Bruzzese, R. R. (2024). Building visual malware dataset using VirusShare data and comparing machine learning baseline model to CoAtNet for malware classification. In Proc. 16th International Conference on Machine Learning and Computing (ICMLC).

Lewis, P., Stenetorp, P., and Riedel, S. (2021). Question and answer test-train overlap in open-domain question answering datasets. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL).

Croft, R., Babar, M. A., and Kholoosi, M. M. (2023). Data quality for software vulnerability datasets. In Proceedings of the 45th International Conference on Software Engineering (ICSE).

Loshchilov, I. and Hutter, F. (2019). Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR).

Dias, A., Pinto, J. R., Proença, H., and Neves, J. C. (2026). Different defects, different effects: A controlled audit of face recognition training data. arXiv preprint arXiv:2610.28766. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR). Freitas, S., Duggal, R., and Chau, D. H. (2022). MalNet: A large-scale image database of malicious software. In Proceedings of the 31st ACM International Conference on Information and Knowledge Management (CIKM). arXiv:2102.01072. He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Huang, G., Liu, Z., van der Maaten, L., and Weinberger, K. Q. (2017). Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Joyce, R. J., Amlani, D., Nicholas, C., and Raff, E. (2021). MOTIF: A malware reference dataset with ground truth family labels. arXiv preprint arXiv:2111.15031. Kapoor, S. and Narayanan, A. (2023). Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 4(9):100804. Laroca, R., Estevam, V., Britto, A. S., Minetto, R., and Menotti, D. (2023). Do we train on test data? the impact of near-duplicates on license plate recognition. In Proceedings of the International Joint Conference on Neural Networks (IJCNN). Larson, S., Lim, G., and Leach, K. (2023). On evaluation of document classification with RVL-CDIP. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics (EACL). Larson, S., Nagy, A., Desai, S., Desai, C., Lima, N. C., Yuan, Y., Betala, S., Prajapati, K. K., Suleiman, J. T., Duwal, S., and Leach, K. (2026). Revising RVL-CDIP: Quantifying errors and test-train overlap. In Proceedings of the 26th ACM Symposium on Document Engineering (DocEng).

Makkawy, S. J., De Lucia, M. J., and Barner, K. E. (2025). MalVis: A large-scale image-based framework and dataset for advancing Android malware classification. arXiv preprint arXiv:2505.12106. Mohammed, T. M., Nataraj, L., Chikkagoudar, S., Chandrasekaran, S., and Manjunath, B. (2021). Malware detection using frequency domain-based image visualization and deep learning. In Proceedings of the 54th Hawaii International Conference on System Sciences, page 7132. Nataraj, L., Karthikeyan, S., Jacob, G., and Manjunath, B. S. (2011). Malware images: Visualization and automatic classification. In Proc. 8th International Symposium on Visualization for Cyber Security (VizSec). Noever, D. and Miller Noever, S. E. (2021). VirusMNIST: A benchmark malware dataset. arXiv preprint arXiv:2103.00602. Northcutt, C. G., Athalye, A., and Mueller, J. (2021). Pervasive label errors in test sets destabilize machine learning benchmarks. In Proceedings of the 35th Conference on Neural Information Processing Systems, Datasets and Benchmarks Track. Oliver, J., Cheng, C., and Chen, Y. (2013). TLSH – a locality sensitive hash. In 2013 Fourth Cybercrime and Trustworthy Computing Workshop, pages 7–13. Pendlebury, F., Pierazzi, F., Jordaney, R., Kinder, J., and Cavallaro, L. (2019). TESSERACT: Eliminating experimental bias in malware classification across space and time. In USENIX Security Symposium. Radenović, F., Iscen, A., Tolias, G., Avrithis, Y., and Chum, O. (2018). Revisiting Oxford and Paris: Large-scale image retrieval benchmarking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Raff, E., Barker, J., Sylvester, J., Brandon, R., Catanzaro, B., and Nicholas, C. (2018). Malware detection by eating a whole EXE. In AAAI Workshop on Artificial Intelligence for Cyber Security. Wu, H., Tian, S., Gutierrez, J., Bhatta, A., Öztürk, K., and Bowyer, K. W. (2024). Identity overlap between face recognition train/test data: Causing optimistic bias in accuracy measurement. arXiv preprint arXiv:2405.09403. Zhao, Y., Li, L., Wang, H., Cai, H., Bissyandé, T. F., Klein, J., and Grundy, J. (2021). On the impact of sample duplication in machine-learning-based Android malware detection. ACM Transactions on Software Engineering and Methodology, 30(3):40.

Related documents

• Measuring Duplication, Leakage, and Rendering Dependence in Malware-as-Image Benchmarks#1490746
• Federated multimodal malware classification under non-IID data#413282
• SLIFER: Investigating performance and robustness of malware detection pipelines#644196
• EXHIB: A Benchmark for Realistic and Diverse Evaluation of Function Similarity in the Wild#1200896
• A multistage verification method for malicious URL detection using feature selection heuristic rules and machine learning#306099