Q-MALDetect: Quantum Image-Based Malware Detection and Adversarial Robustness Analysis

Q-MALDetect: Quantum Image-Based Malware Detection and Adversarial Robustness Analysis Walid El Maouaki† , Mohammad Izen† , Nouhaila Innan†§ , Alberto Marchisio†§‡∗ , Muhammad Shafique†§ † eBrain Lab, Division of Engineering, New York University Abu Dhabi, PO Box 129188, Abu Dhabi, UAE § Center for Quantum and Topological Systems (CQTS), NYUAD Research Institute, NYUAD, Abu Dhabi, UAE ‡ SDU Microelectronics, Institute of Mechanical and Electrical Engineering, University of Southern Denmark, Odense, Denmark

arXiv:2610.26240v1 [cs.CR] 19 Oct 2026

Emails: {walid.el.maouaki, mi2527, nouhaila.innan, muhammad.shafique}@nyu.edu, chisio@sdu.dk

Abstract—Quantum machine-learning (QML) models are increasingly considered for security applications, yet their adversarial robustness in malware analysis remains insufficiently understood. This paper presents Q-MALDetect, a hybrid convolutional and variational-quantum detector for byte-image representations of x86-64 Executable and Linkable Format (ELF) binaries, and evaluates it under feature-space and a range of Grad-CAM-guided, executable-aware evasion attacks. The model achieves 97.2% training accuracy, but its robustness deteriorates as attack strength increases, with the strongest evaluated featurespace attack reaching a 78.7% attack success rate (ASR). Among executable-aware attacks, Grad-CAM-guided no-operation (NOP) insertion produces the highest conditional ASR of 32.9%. Dynamic adversarial training is the most effective evaluated defense, reducing feature-space ASR to as low as 5.0%, executableaware ASR to 7.7%, and the sequential-attack ASR to 34.0%. In contrast, convolutional-autoencoder preprocessing increases executable-aware ASR to 39.7%, coinciding with a rise in GradCAM attribution mass within the mutable .text region from 35.2% to 45.7%. Ablation results reveal no monotonic relationship between qubit count and robustness, while the encoding interface modifies the learned attack surface without neutralizing the strongest attack. These findings demonstrate that the effectiveness of attacks and defenses depends strongly on the perturbation constraints and evaluation conditions. Reliable deployment of QML malware detectors therefore requires assessment against both feature-space and executable-aware attacks rather than reliance on clean accuracy or generic denoising alone. Index Terms—Quantum machine learning, malware detection, ELF binaries, adversarial robustness, hybrid quantum-classical neural networks, adversarial training.

I. I NTRODUCTION Linux malware has become a critical security concern because servers, embedded platforms, and IoT devices commonly rely on Executable and Linkable Format (ELF) instances. Static malware detectors based on deep learning can process binaries quickly and can avoid the cost of dynamic sandbox execution. A widely adopted approach converts executable bytes into images and applies convolutional neural networks (CNNs), allowing models to learn discriminative bytelayout patterns without hand-crafted features [1]. However, CNNs are vulnerable to adversarial examples, i.e., small, carefully chosen perturbations can cause high-confidence misclassification [2], [3]. In malware detection, this threat is ∗ This work has been carried out while at NYUAD.

especially subtle because arbitrary feature-space perturbations may not correspond to a valid executable. Recent ELF-focused work, such as ADVeRL-ELF, has shifted toward problem-space attacks that maintain both executability and malicious behavior while evading detection [4]. Quantum machine learning (QML) offers a new design space for security analytics because parameterized quantum circuits can act as trainable feature maps with representations that differ from classical neural networks. Prior studies have investigated the robustness of quanvolutional and hybrid quantum-classical models [5]–[7]. Yet most QML robustness studies use canonical image data, while adversarial malware research focuses predominantly on classical models. Consequently, it remains unclear how a hybrid QML malware detector behaves when exposed to both unconstrained feature-space perturbations and executable-aware binary mutations. This paper addresses that question through a vulnerability study of Q-MALDetect, a hybrid convolutional and variationalquantum ELF classifier built on established hybrid QML detector designs [8]–[10]; the contribution is a joint featurespace and executable-aware robustness evaluation of the complete pipeline. This study asks four questions: (i) whether the hybrid model fits the clean data to a high-recall operating point; (ii) which feature-space and executable-aware attacks are effective, and under what conditions; (iii) whether adversarial training and CAE preprocessing provide consistent protection; and (iv) how perturbation budget, qubit count, encoding interface, and simulated hardware noise affect the observed attack surface. The main contributions are: We formulate a critical-domain robustness evaluation for a hybrid quantum-classical ELF malware detector and separate clean classification, feature-space sensitivity, and executable-aware evasion. • We benchmark a spectrum of attacks that combines FGSM, PGD, and an FGSM-initialized PGD variant with two Grad-CAM-guided NOP-insertion attacks. The latter attacks use attribution to align a constrained mutation budget with model-sensitive executable regions; one selects NOP actions randomly and one tests whether tabular Q-learning improves on it.

•

We compare two standard defenses, dynamic adversarial training and CAE preprocessing, and identify an asymmetric defense outcome: the CAE attenuates additive feature-space perturbations but can increase Grad-CAMguided problem-space ASR by shifting attribution toward the mutable .text region. • We evaluate adversarial training and ablations over perturbation budget, qubit count, and encoding strategy, showing that quantum-interface choices materially affect attack success.

model-confidence feedback for evaluating candidate mutations, restricted to semantics-neutral NOP insertion with ELF repair. The dataset contains 1,300 x86-64 ELF binaries from the Labeled-ELFs repository [29], including 600 benign and 700 malware samples. The training set contains all 600 benign samples and 600 malware samples. The remaining 100 malware samples are held out from training and reserved exclusively for adversarial attack evaluation. Clean metrics are computed on the balanced training set and fix each model’s operating point; every ASR is measured on these unseen malware, matching the malware-to-benign evasion threat model. II. BACKGROUND AND R ELATED W ORK Each file is represented by its first 224 × 224 = 50,176 Byte-image malware classification maps executable bytes bytes. Short files are zero-padded and long files are truncated; into grayscale or color images and applies image classifiers [1]. byte values are divided by 255 to obtain x ∈ [0, 1]224×224 . Classical malware evasion studies have moved from Truncation means that attacks and explanations concern only unrestricted image perturbations toward functionality-aware file regions visible within the first 50,176 bytes. Figure 2 binary transformations. COPYCAT injects adversarial content illustrates both the resulting byte image in Figure 2(a) and a into malware representations [11]; reinforcement-learning Grad-CAM map in Figure 2(b) used to guide domain-space approaches search over executable transformations [12]; attacks. EvilELF studies structural modifications to Linux binaries [13]; Clean performance is reported using accuracy, precision, and ADVeRL-ELF inserts semantic NOP sequences into ELF malware recall, and false-positive rate. Attack success rate is code regions to evade an image-based ResNet detector [4]. defined over initially correct malware: These studies demonstrate that executable constraints usually #{i : f (xi ) = 1, f (xadv i ) = 0} reduce attack freedom but produce a more operationally relevant ASR = × 100%, (1) #{i : f (x ) = 1} i threat model. Quantum methods have been studied for malware, intrusion, where label 1 denotes malware. Restricting the denominator network-security, and side-channel analysis tasks [8]–[10], [14]– to initially correct samples prevents clean-model errors from [18]. In parallel, adversarial QML research has established being counted as attack successes. Results are summarized as attacks, training-based defenses, robustness verification, mean ± standard deviation across seeds. universal perturbations, and experimental demonstrations on quantum hardware [19]–[25]. More recent studies investigate B. Hybrid Quantum-Classical Model Detector architecture, expressibility, transferability, circuit metrics, and The CNN frontend contains three stride-2 convolutional federated robustness [5]–[7], [26]–[28]. These results motivate layers with ReLU activations, followed by adaptive average attack-aware QML design, but they do not establish robustness pooling to 4 × 4 and flattening to h ∈ R1024 . For the default for executable malware. four-qubit amplitude-encoding model, the fully connected Research gap. Existing work does not provide a unified bridge produces evaluation of a hybrid QML malware detector under both z = π tanh(W1 h), z ∈ [−π, π]16 . (2) differentiable feature-space attacks and attribution-guided executable mutations, nor does it examine whether defenses Amplitude embedding requires a unit-norm vector, so the circuit can protect against one threat class while exposing another. input is z This paper focuses on that gap. e z= , (3) max(∥z∥2 , δ) III. S TUDY D ESIGN AND M ETHODOLOGY with a small numerical constant δ > 0. A single PennyLane Figure 1 summarizes the complete protocol, from StronglyEntanglingLayers block [30] processes the ELF preprocessing and hybrid inference through attacks, four-qubit state. Four Pauli-Z expectation values are passed to defenses, ablations, and evaluation. The protocol maintains a linear layer and sigmoid to produce malware probability ŷ; separate model-development and held-out attack pools and the decision threshold is 0.5. The CNN and bridge use learning distinguishes feature-space perturbations from reconstructed rate 0.001, while the quantum parameters and output head use binary mutations. 0.01. Training uses Adam, binary cross-entropy, batch size 32, and 10 epochs. A. Threat Model and Dataset •

We study non-targeted evasion: an adversary starts from malware that the model classifies correctly and seeks a benign prediction. All attacks assume access to the trained model. FGSM and PGD require end-to-end gradients. The GradCAM-guided attacks require gradients for attribution and

C. Feature-Space Attacks FGSM applies one loss-maximizing step and clips the result to the valid input range: xFGSM = Π[0,1] (x + ε sign(∇x L(x, y))) ,

(4)

① Input Preprocessing

② Hybrid QNN Detector

③ Adversarial Attacks

ELF Binary Dataset 1,300 x86-64 binaries 600 benign | 700 malware Labeled-ELFs repo

CNN Frontend 3 × Conv2D (stride 2) + ReLU Adaptive AvgPool → 4 × 4 Flatten → h ∈ R^1024

Train / Test Split Train: 1,200 samples Test: 100 malware samples (Held out)

FC Bridge Linear layer maps h to z z = π · tanh(W1h) z ∈ [-π,π]16

Feature-Space Attacks (white-box gradients) FGSM PGD Single step Iterative projected xadv = x + ε·sign(∇xL) gradient descent Compound (FGSM→PGD): FGSM init + PGD refine

④ Defense Strategies

Adversarial Training Augment training with adversarial samples Substantially reduces ASR

⑤ Evaluation Output

Detection Rate Accuracy

Attack Effectiveness ASR across attacks

Byte → Image Truncated/padded to 224 x 224 grayscale Normalized to [0,1]

Variational Quantum Layers 4-qubit amplitude encoding StronglyEntanglingLayers Pauli-Z readout: q = [<Z0>,...,<Z3>]

Sigmoid Output ŷ ≥ 0.5 → Malware ŷ < 0.5 → Benign BCE loss + Adam

Problem-Space Attacks (executable-preserving) GradCAM + Q-Learning Agent Agent selects NOP actions Reward: evasion success Executable-valid mutations

Convolutional Autoencoder (CAE) Pre-filter reconstructs input Reduces adversarial noise

Robustness Analysis Impact of circuit design and defenses

GradCAM + NOP Injection Locate influential byte regions Insert semantic NOP sequences Preserves executability

Ablation Studies Qubits (2, 4, 8) | perturbation ε | encoding

Research Contribution Executable-preserving attacks on QML malware detection

Fig. 1: Q-MALDetect evaluation workflow. ELF binaries are converted to 224 × 224 normalized byte images and divided into a 1,200-sample model-development pool and a 100-malware attack holdout. A CNN bridge supplies normalized amplitudes to a variational quantum circuit. The evaluation covers three feature-space attacks and two Grad-CAM-guided, executable-aware NOP-insertion attacks; dynamic adversarial training and CAE preprocessing are then tested under the same attack families. Ablations vary perturbation budget, qubit count, encoding interface, and NOP length, while the final stage reports clean metrics, ASR, attribution behavior, and noise-model sensitivity. a) Original Malware Image

b) Base Model GradCAM

c) CAE Model GradCAM

modify normalized byte-image values and are not asserted to correspond to valid executable files. D. Executable-Aware Grad-CAM Attacks

1

2

Fig. 2: Representative attribution shift under CAE preprocessing. (a) Original malware byte image. (b) The base-model Grad-CAM map is spatially diffuse. (c) The CAE-model map concentrates high activation in a narrow horizontal band aligned with the analyzed .text region. The two numbered callouts mark segments of this same high-activation band, not separate processing stages. Across 495 paired maps, .text attribution mass increases from 35.2% to 45.7%.

with ε = 0.01 in the main experiment. PGD applies 50 steps with α = 0.001 and projects after every update:   x(t+1) = ΠB∞ (x,ε)∩[0,1] x(t) + α sign(∇x(t) L) . (5) The compound attack applies FGSM first and then runs PGD from the FGSM output. In the reported implementation, the second projection is centered on that intermediate point. Each stage uses the same nominal ε, so the final displacement from the original input can reach 2ε before clipping. The compound result is therefore a stronger sequential threat model, not a same-budget comparison with FGSM or PGD. These attacks

1) Single-Point Insertion and Reconstruction: The two domain attacks use Grad-CAM [31] at the third convolutional layer to rank influential image locations. Candidate locations are mapped back to byte offsets, restricted to the ELF .text section, and snapped to a nearby instruction boundary identified through Capstone disassembly. Boundaries crossed by short relative branches are rejected. All NOPs for one sample accumulate at a single accepted boundary, reducing the repair problem to one cut position and one total shift. After inserting a block of N bytes, the procedure expands .text, shifts subsequent section and program-header offsets and virtual addresses, updates relevant ELF header fields, and adjusts relative branches, calls, RIP-relative references, relocations, symbols, and dynamic-table addresses that cross the cut. LIEF is then used to reparse the rebuilt binary. Since multibyte NOPs leave the architectural state unchanged and all listed cross-cut references are repaired, the mutation is semanticsneutral by design; every rebuild is verified for structural validity and re-read consistency, and smoke tests on rebuilt benign utilities confirm runnability. An executable-aware success is thus a structurally valid, semantics-neutral rebuild classified as benign. Let I denote initially correct attack malware and R ⊆ I the samples for which a safe boundary is found and a structurally

TABLE I: Experimental configuration summary.

valid rebuild is produced. The plotted domain metric is ASRexec =

{i ∈ R : f (xrebuilt ) = 0} i × 100%. |R|

(6)

The total success rate is ASRexec × |R|/|I|. The executable rate (ASRexec ) acts as the maximum possible overall rate, achieved at 100% reconstruction coverage. Feature-space and executable-aware ASRs measure distinct vulnerabilities and are not directly comparable. 2) Grad-CAM-Guided Random NOP Insertion: The random baseline selects among six architecturally valid multi-byte x8664 NOP encodings of length 6-11 bytes. At each turn, it inserts one randomly selected NOP at the accumulating GradCAM-guided boundary, re-reads the first 50,176 bytes, and queries the detector. The attack stops after a label flip or 200 turns. A successful image-level evasion is counted only after reconstruction, reparsing, and rescoring of the rebuilt ELF. 3) Grad-CAM-Guided Q-Learning: The learned variant tests whether selecting a particular NOP encoding improves on random choice. The state is one of 10 confidence buckets, s = min (9, ⌊10c⌋) ,

(7)

where c ∈ [0, 1] is the current malware confidence. The six NOP encodings form the action space. With   corig − cnew , −1, 1 , (8) p = clip corig − 0.5 + δ

Category

Configuration

Dataset Split Training Learning rate Architecture Default circuit Gradient attacks FGSM PGD Domain attacks Insertion NOP set Turns per sample Reconstruction Reported ASR RL agent RL state / action Adv. training CAE Qubit ablation Encoding ablation Epsilon ablation Seeds

1300 ELF (224 × 224) 600 benign, 700 malware 10 epochs, batch 32, Adam, BCE CNN 0.001, quantum 0.01 CNN + FC + QNN + FC 4 qubits, 1 layer, Amplitude FGSM, PGD, Compound ε = 0.01 ε = 0.01, α = 0.001, 50 iter GradCAM+NOP, GradCAM+RL single safe .text boundary 6 canonical NOPs, 6–11 bytes up to 200 Capstone + LIEF, structure-valid image (upper bound) + executable tabular Q-learning, 8 passes 10 conf. buckets / 6 NOPs dynamic PGD/epoch + NOP/RL aug., 5 ep. 4-layer, latent 128, 20 epochs n ∈ {2, 3, 4, 5, 6, 7, 8, 9} Amplitude, Angle ε ∈ [10−3 , 101 ], 10 pts 3 seeds for all experiments unless specified

convolutional encoder that progressively downsamples the 224 × 224 image to a 14 × 14 × 128 representation, which is compressed into a 128-dimensional latent vector. A symmetric four-layer transposed-convolution decoder reconstructs the the reward is  input with a sigmoid output. The CAE is trained for 20 epochs  10, cnew < 0.5, on clean model-development data using mean-squared error (9) r = 2p, p > 0,  and Adam with a learning rate of 0.001. It is then frozen,  p, p ≤ 0. and the classifier is trained on reconstructed inputs. All whitebox feature-space attacks differentiate through the complete The Q-table update is h i CAE-classifier pipeline. ′ ′ Q(s, a) ← Q(s, a) + αQ r + γ max Q(s , a ) − Q(s, a) , We also study the impact of three design variables a′ (10) on adversarial vulnerability. The qubit sweep uses n ∈ with αQ = 0.1 and γ = 0.9. Exploration uses ϵRL -greedy {2, 3, 4, 5, 6, 7, 8, 9}. Because amplitude encoding requires 2n selection, decaying from 0.9 to 0.05 by a factor of 0.995 inputs and the readout returns n expectation values, this sweep per episode. Policy fitting uses eight passes over model- grows the bridge from 4,096 to 524,288 weights (1024 · 2n ) development malware, with at most 200 turns per episode. while the circuit grows from 6 to 27 rotation angles (3n). The frozen policy acts greedily on the attack holdout, and only The encoding study compares amplitude embedding with fourreconstructed evasions contribute to Eq. (6). feature angle embedding at four qubits; it therefore evaluates complete encoding interfaces, in which the bridge dimension E. Defenses, Ablations, and Noise-Model Evaluation (16 vs. 4), gradient flow, and encoding gates change jointly. To improve Q-MALDetect robustness, we apply adversarial The perturbation study evaluates 10 logarithmically spaced ε training (AT) following the min-max formulation of Madry et values from 10−3 to 101 . Since inputs are clipped to [0, 1], al. [3]. At each of five fine-tuning epochs, fresh PGD examples budgets at or above 1 are nominal values that primarily expose are generated against the current model and replace 50% of the saturation behavior rather than progressively larger feasible ℓ∞ corresponding development samples. Static reconstructed NOP regions. and RL bypass examples generated from model-development A final transfer experiment evaluates adversarial inputs malware are also included so that hardening covers both threat generated against the noiseless model under 1,024-shot classes. The held-out attack malware remain excluded from FakeManilaV2 simulation, a local backend whose noise model fine-tuning. is derived from a device snapshot; it thus measures sensitivity As a second line of defense, we prepend a Convolutional to stored device noise and finite-shot sampling. Table I Autoencoder (CAE) to the Q-MALDetect pipeline as a consolidates the default settings across the main comparison preprocessing stage. The CAE consists of a four-layer and ablations.

Fig. 3: Training convergence for one illustrative run. Loss decreases from 0.6815 to 0.0605 while training accuracy increases from 56.41% to 98.44% over 10 epochs.

All classical components are implemented in PyTorch. PennyLane provides the differentiable quantum simulation, while LIEF and Capstone support ELF parsing and disassembly. IV. R ESULTS AND D ISCUSSION A. Clean Performance and Undefended Attack Surface

Attack Success Rate (%)

100% 80%

Undefended CAE-Defended Adversarial Training

78.7 52.3

42.3

50.7

60% 40% 20% 0%

34.0

32.939.7

29.837.2

27.3

25.3 5.0

FGSM

5.7

PGD

9.0

7.7

Compound GradCAM Attack +NOP Attack Type

GradCAM +RL

Fig. 4: ASR under the undefended, CAE-defended, and adversarially trained conditions. FGSM, PGD, and Compound are featurespace ASRs; Grad-CAM+NOP and Grad-CAM+RL are conditional executable ASRs. Bar labels round the corresponding means to whole percentages. AT reduces all five attacks, whereas the CAE reduces feature-space ASR but increases both executable-aware rates.

Executable ASR (%)

0.7 0.6 Train Accuracy 0.5 0.4 Train Loss 0.3 0.2 0.1 0.0 1 2 3 4 5 6 7 8 9 10 Epoch

Loss

Accuracy

1.0 0.8 0.6 0.4 0.2 0.0

80% 70% 60% 50% 40% 30% 20% 10% 0%

Mean exe ASR

The illustrative run in Fig. 3 shows stable optimization: the training loss decreases from 0.6815 at epoch 1 to 0.0605 at epoch 10, while the training accuracy increases from 56.41% to 98.44%. Across the evaluated runs, the model achieves 6 7 8 9 10 mean training-set accuracy of 97.2% ± 1.0%, precision of NOP length (bytes), constant 2520-byte budget 95.1% ± 1.4%, malware recall of 99.6% ± 0.8%, and a falsepositive rate of 5.2% ± 1.5%. These results confirm that the model fits the clean training data and provides a high-recall Fig. 5: Conditional executable ASR versus NOP length with total inserted volume fixed at 2,520 bytes, averaged over five seeds. Faint starting point for the robustness analysis. lines show individual seeds and the bold line shows the mean. The In Figure 4, under the undefended model, FGSM reaches absence of a monotonic length trend, together with large seed variation, 42.3% ASR and PGD reaches 52.3%. The sequential compound indicates that total displacement and detector fragility dominate the attack is strongest at 78.7%, consistent with its two-stage identity of the selected NOP. optimization and potentially doubled displacement budget. The executable-aware attacks attain conditional ASRs of non-monotonic and the between-seed spread is larger than any 32.9% for Grad-CAM+NOP and 29.8% for Grad-CAM+RL. consistent length trend. Inserting bytes at one early .text These rates are lower than the compound feature-space result boundary shifts all subsequent visible bytes, so total layout but operationally more concrete: each counted success is a displacement and the chosen boundary provide a stronger reparsed, semantics-neutral rebuild that the detector scores as mechanism than the semantic identity of a particular NOP. benign. Comparisons with prior ELF attacks [4], [13], [32], An effective learned attacker should therefore control location, [33] must account for different models, architectures, mutation volume, or transformation family. sets, budgets, denominators, and functionality checks. The random and learned NOP attacks perform similarly: the B. Defense Effectiveness 1) Dynamic Adversarial Training: Dynamic AT is the 3.1-point mean difference in favor of random selection lies within the seed variation, although the Q-tables learn state- strongest defense in Figure 4. FGSM drops from 42.3% to dependent preferences (mean within-state action spread of 2.2 5.0% (a 37.3-point reduction), PGD from 52.3% to 5.7% (46.6 value units, 1.6–3.0 across seeds). These preferences cannot points), and the compound attack from 78.7% to 34.0% (44.7 raise ASR because the actions change NOP form and length points). Grad-CAM+NOP falls from 32.9% to 9.0%, and Gradbut not the insertion location or the dominant image-layout CAM+RL from 29.8% to 7.7%. Figure 6 shows that these reductions occur across the paired seeds rather than being effect. The fixed-volume diagnostic in Figure 5, an ablation over produced by one outlying run. constant-action policies at matched volume, supports this The result follows the composition of the fine-tuning data: interpretation. With total inserted volume held at 2,520 bytes, fresh PGD examples harden the differentiable attack surface, mean conditional executable ASR is 35.1%, 44.8%, 26.5%, while reconstructed NOP/RL examples expose the model 40.5%, and 34.6% for NOP lengths 6-10. The sequence is to domain mutations. The residual 34.0% compound ASR

remains important. PGD-based hardening does not fully cover The sweep therefore characterizes architecture width a sequential attack whose feasible displacement can exceed the rather than quantum resources in isolation: changing n single-stage training budget. These gains quantify resistance simultaneously changes the amplitude-input dimension from to malware evasion, the objective of the threat model; the 2n , the number of measured expectations, parameter count, and clean-accuracy and false-positive cost of hardening is the optimization geometry. Wide uncertainty bands further show complementary deployment metric. that initialization and data partitioning are consequential. This 2) Convolutional-Autoencoder Preprocessing: The CAE architecture-coupled behavior is consistent with prior QML produces a threat-dependent reversal. Relative to the robustness studies in which circuit design and learned geometry, undefended model, FGSM falls from 42.3% to 25.3%, PGD rather than qubit count alone, govern attackability [5], [7], [35]. 3) Encoding Interface: At four qubits, amplitude and angle from 52.3% to 27.3%, and the compound attack from 78.7% to 50.7%. The bottleneck removes part of the additive image- encoding both attain 99.0% training accuracy, but they induce space perturbation, but it does not eliminate the sequential different attack surfaces (Figure 9). The angle-encoded interface lowers FGSM from 42.3% to 30.7% and PGD from 52.3% to attack. 33.3%. It does not protect against the compound attack, which For executable-aware attacks, the same preprocessing is changes from 78.7% to 81.0%. Conditional executable ASR harmful: Grad-CAM+NOP rises from 32.9% to 39.7%, and also falls, from 32.9% to 21.8% for NOP insertion and from Grad-CAM+RL rises from 29.8% to 37.2%. The attribution 29.8% to 15.4% for RL. analysis provides a coherent mechanism. Across 495 paired The domain changes are model-dependent rather than maps, the fraction of Grad-CAM activation mass inside .text independent of encoding: changing the bridge and circuit increases from 35.2% to 45.7%, a 10.5-point mean shift, and changes the classifier gradients and thus the Grad-CAM 60% of samples move in that direction. As Figure 2 illustrates, insertion site. Because amplitude encoding receives 16 bridge the CAE model exhibits pronounced activation confined to features and angle encoding four, feature dimensionality, a narrow band that is spatially aligned with the mutable code region (in panel c), as indicated by circular markers gradient flow, and the optimization landscape change together 1 and 2 in the figure. This does not by itself prove causation, with the gates; the result therefore characterizes the complete but it is consistent with the measured increase in attacks angle-encoded interface, which is less vulnerable to four of the that deliberately target .text. The observation complements five attacks. The compound result demonstrates that encoding interpretation-based ELF defenses [34]: attribution can identify choice alone is insufficient. 4) Shot-Based Noise-Model Sensitivity: Figure 10 compares adversarial structure, but a defense that changes attribution can noiseless inference with FakeManilaV2 shot-based simulation. also expose a more concentrated mutation target. FGSM changes from 42.0% to 48.0%, PGD from 52.0% to 56.0%, and the compound attack from 79.0% to 81.0%. C. Sensitivity and Ablation Studies The largest mean shift is 6 points. The ranking is preserved: 1) Perturbation Budget: Figure 7 evaluates a ten-seed run compound remains strongest, followed by PGD and FGSM. over 10 logarithmically spaced budgets. In the transition The transferred adversarial examples therefore remain between approximately ε = 8 × 10−3 and 2 × 10−2 , ASR effective under device-derived noise and finite sampling. Prior rises sharply and uncertainty is widest; the compound attack work shows that noise can alter quantum-classifier robustness increases first, followed by PGD and FGSM. PGD and the under specific assumptions [20]; here, no uniform protective or compound attack approach saturation only over the subsequent harmful effect appears, and the mean attack ranking is stable. points, roughly 6 × 10−2 to 2 × 10−1 , rather than at ε = 0.02 exactly. FGSM plateaus in the mid-80% range and declines D. Interpretation slightly at the largest nominal budgets. Three conclusions emerge from the combined results. First, The high-budget regime must be read with input clipping. threat-model validity changes the meaning of ASR. FeatureFor ε ≥ 1, the [0, 1] cube, rather than the nominal ℓ∞ radius, space attacks reveal differentiable sensitivity, but they do determines the feasible input. The curves therefore describe not produce executable malware. The Grad-CAM attacks sign saturation and iterative clipping, not meaningful growth operate under substantially tighter binary constraints, yet their to a radius of 10 in normalized byte space. The slight FGSM conditional executable rates remain nontrivial. Recent ELF decline is not evidence that very large perturbations restore work similarly shows that effective evasion depends on where robustness. and how a binary can be transformed [4], [32], [33]. In Q2) Qubit Count: Figure 8 shows no monotonic relation MALDetect, the fixed-volume experiment indicates that byte between circuit width and adversarial robustness. FGSM and displacement at an influential boundary matters more than PGD fluctuate substantially across 2-9 qubits, with local lows which NOP encoding is selected, explaining why the six-action around the four- and eight-qubit configurations and renewed RL policy does not outperform random choice. increases at nine qubits. The compound attack remains the Second, defense quality is threat-dependent. Dynamic AT largest curve at every width, ranging approximately from 60% performs best because its training distribution explicitly to 87%. The domain attacks remain lower in this cohort, mostly includes both gradient and reconstructed domain examples. The within 5-20%, but also vary non-monotonically. CAE is mismatched: it filters part of the additive perturbation

Fast, single-step perturbation

Executable-aware (random NOP insertion)

Iterative, stronger optimization

Executable-aware (RL agent)

Most effective feature-space attack

100% 80% 60% 40% 20% 0%

FGSM PGD Compound 10 3

10 2

10 1

100

101

Epsilon (log scale) Fig. 7: Feature-space ASR versus nominal perturbation budget for FGSM, PGD, and the sequential compound attack. Shaded bands denote ±1 standard deviation. The 10 values are logarithmically spaced from 10−3 to 101 ; values at or above 1 primarily reveal clipping-induced saturation. FGSM PGD Compound

100%

GradCAM+NOP GradCAM+RL

Standard attacks GradCAM attacks

ASR (%)

100% 80%

78.7 52.3

42.3

60% 40%

32.9 30.7

FGSM

29.8

33.3 21.8

20% 0%

Amplitude Encoding Angle Encoding

81.0

PGD

15.4

Compound GradCAM GradCAM Attack +NOP +RL Attack Type

Fig. 9: ASR for amplitude- and angle-encoded four-qubit models. The angle-encoded interface lowers FGSM, PGD, and both conditional executable rates, while the sequential compound attack remains approximately 80%. The bridge dimension changes with the encoding, so the comparison covers the complete interface rather than encoding gates alone.

was designed to remove. Third, the quantum design variables do not yield a simple robustness law. Neither more qubits nor a different encoding guarantees protection, and the sequential attack remains strong across configurations. This agrees with the broader adversarialQML literature: robustness is a property of the full dataencoding-circuit-optimizer system [5], [7], [19], [35].

80% 60% 40% 20% 0%

Attack Success Rate (%)

ASR (%)

Fig. 6: Paired ASR before and after dynamic adversarial training. Faint lines show individual seeds, bold lines show means, and shaded regions show ±1 standard deviation. AT sharply reduces both feature-space and executable-aware attacks, although the sequential compound attack retains a 34.0% mean ASR.

2

3

4

5 6 Number of Qubits

7

8

9

Fig. 8: ASR across 2-9 qubits. Solid curves denote feature-space attacks and dashed curves denote conditional executable ASR. No attack exhibits a monotonic qubit trend; the compound attack remains highest at every width. Because the bridge and readout dimensions change with n, the plot is an architecture-width ablation rather than an isolated test of quantum resources.

but concentrates the classifier’s evidence in the same mutable section targeted by the executable-aware attacks. Robust preprocessing must therefore be evaluated against adaptive, problem-space mutations, not only against the perturbations it

V. C ONCLUSION Q-MALDetect demonstrates that a hybrid quantum-classical malware detector with high training accuracy can retain a broad adversarial attack surface. The undefended model reaches 78.7% ASR under the stronger sequential feature-space attack and conditional executable ASRs of 32.9% and 29.8% under Grad-CAM-guided NOP variants. Dynamic adversarial training reduces all five attacks, although the compound attack remains at 34.0%. CAE preprocessing lowers feature-space ASR but raises executable-aware ASR while shifting attribution toward .text, establishing a concrete defense mismatch. Qubit count has no monotonic robustness effect, angle encoding

Attack Success Rate (%)

Simulator FakeManilaV2

100% 80%

42

48

52

79

81

56

60% 40% 20% 0%

FGSM

PGD

Attack Type

Compound Attack

Fig. 10: Feature-space ASR under noiseless simulation and 1,024shot FakeManilaV2 noise-model simulation. Mean shifts are at most 6.0 percentage points, while the attack ranking is unchanged. FakeManilaV2 is a local simulator backend, not physical-hardware execution.

does not neutralize the compound attack, and FakeManilaV2 simulation preserves the attack ranking without constituting hardware validation. Robust QML malware detection therefore requires an explicit, multi-level threat model, reconstruction and functionality accounting, defense-specific adaptive testing, and matched classical comparisons; clean accuracy or generic denoising alone cannot support a security claim. ACKNOWLEDGMENT This work was supported in part by the NYUAD Center for Quantum and Topological Systems (CQTS), funded by Tamkeen under the NYUAD Research Institute grant CG008, and the Center for Cyber Security (CCS), funded by Tamkeen under the NYUAD Research Institute Award G1104. R EFERENCES [1] L. Nataraj et al., “Malware images: Visualization and automatic classification,” in Proceedings of the 8th International Symposium on Visualization for Cyber Security (VizSec), 2011, pp. 1–7. [2] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” International Conference on Learning Representations (ICLR), 2015. [3] A. Madry et al., “Towards deep learning models resistant to adversarial attacks,” in International Conference on Learning Representations (ICLR), 2018. [4] A. Ravi, V. Chaturvedi, and M. Shafique, “ADVeRL-ELF: Adversarial elf malware generation using reinforcement learning,” in 2025 62nd ACM/IEEE Design Automation Conference (DAC). IEEE, 2025, pp. 1–7. [5] W. El Maouaki, A. Marchisio, T. Said, M. Bennai, and M. Shafique, “AdvQuNN: A methodology for analyzing the adversarial robustness of quanvolutional neural networks,” in 2024 IEEE International Conference on Quantum Software (QSW). IEEE, 2024, pp. 175–181. [6] W. El Maouaki, A. Marchisio, T. Said, M. Shafique, and M. Bennai, “RobQuNNs: A methodology for robust quanvolutional neural networks against adversarial attacks,” in 2024 IEEE International Conference on Image Processing Challenges and Workshops (ICIPCW). IEEE, 2024, pp. 4090–4095. [7] W. El Maouaki, A. Marchisio, T. Said, M. Shafique, and M. Bennai, “Designing robust quantum neural networks via optimized circuit metrics,” Advanced Quantum Technologies, p. 2400601, 2025. [8] G. Barrué and T. Quertier, “Quantum machine learning for malware classification,” in Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 2023, pp. 245–260. [9] R. Liu, M. Eren, and C. Nicholas, “Can feature engineering help quantum machine learning for malware detection?” arXiv preprint arXiv:2305.02396, 2023.

[10] Z. Wang, K. W. Fok, and V. L. Thing, “Network attack traffic detection with hybrid quantum-enhanced convolution neural network,” Quantum Machine Intelligence, vol. 7, no. 1, p. 50, 2025. [11] A. Khormali et al., “COPYCAT: Practical adversarial attacks on visualization-based malware detection,” in arXiv preprint arXiv:1909.09735, 2019. [12] H. S. Anderson et al., “Learning to evade static pe machine learning malware models via reinforcement learning,” arXiv preprint arXiv:1801.08917, 2018. [13] A. Kosikowski, D. Cho, M. Ninan, A. Ralescu, and B. Wang, “EvilELF: Evasion attacks on deep-learning malware detection over elf files,” in 2023 International Conference on Machine Learning and Applications (ICMLA). IEEE, 2023, pp. 1702–1709. [14] D. Abreu, C. E. Rothenberg, and A. Abelém, “Qml-ids: Quantum machine learning intrusion detection system,” in 2024 IEEE Symposium on Computers and communications (ISCC). IEEE, 2024, pp. 1–6. [15] W. El Maouaki, N. Innan, A. Marchisio, T. Said, M. Bennai, and M. Shafique, “Quantum clustering for cybersecurity,” in 2024 IEEE International Conference on Quantum Computing and Engineering (QCE). IEEE, 2024, pp. 5–10. [16] R. Bhatnagar et al., “Meta-quantum ensemble framework for robust network intrusion detection,” arXiv preprint arXiv:2605.28879, 2026. [17] W. El Maouaki, A. Marchisio, and M. Shafique, “Q-LEAK: Quantumbased LEAKage verification for side-channel countermeasures,” in 2026 IEEE International Conference on Quantum Computing and Engineering (QCE). IEEE, 2026. [18] W. El Maouaki, A. Marchisio, and M. Shafique, “QGCL: Quantum-guided clause learning for cryptanalytic SAT,” arXiv preprint arXiv:2605.25756, 2026. [19] H. Liao, I. Convy, W. J. Huggins, and K. B. Whaley, “Robust in practice: Adversarial attacks on quantum machine learning,” Physical Review A, vol. 103, no. 4, p. 042427, 2021. [20] Y. Du et al., “Quantum noise protects quantum classifiers against adversaries,” Physical Review Research, vol. 3, no. 2, p. 023153, 2021. [21] J. Guan, W. Fang, and M. Ying, “Robustness verification of quantum classifiers,” in International Conference on Computer Aided Verification. Springer, 2021, pp. 151–174. [22] M. Weber et al., “Optimal provable robustness of quantum classification via quantum hypothesis testing,” npj Quantum Information, vol. 7, no. 1, p. 76, 2021. [23] W. Gong and D.-L. Deng, “Universal adversarial examples and perturbations for quantum classifiers,” National Science Review, vol. 9, no. 6, p. nwab130, 2022. [24] W. Ren et al., “Experimental quantum adversarial learning with programmable superconducting qubits,” Nature Computational Science, vol. 2, no. 11, pp. 711–717, 2022. [25] N. Innan et al., “Quantum neural networks under threat: Modeling security risks and attack vectors,” in International Conference on Quantum Engineering Sciences and Technologies for Industry and Services. Springer, 2025, pp. 129–139. [26] W. El Maouaki, N. Innan, A. Marchisio, T. Said, M. Bennai, and M. Shafique, “QFAL: Quantum federated adversarial learning,” arXiv preprint arXiv:2502.21171, 2025. [27] W. El Maouaki, N. Innan, A. Marchisio, T. Said, M. Shafique, and M. Bennai, “RobQFL: Robust quantum federated learning in adversarial environment,” arXiv preprint arXiv:2509.04914, 2025. [28] W. El Maouaki and M. Shafique, “Q-RAIL: A reliability-aware framework for quantum federated learning on heterogeneous noisy hardware,” in 2026 IEEE International Conference on Quantum Computing and Engineering (QCE). IEEE, 2026. [29] nimrodpar, “Labeled-Elfs: A collection of well labeled elf binaries,” GitHub repository, 2019, available: https://github.com/nimrodpar/LabeledElfs. [30] V. Bergholm et al., “PennyLane: Automatic differentiation of hybrid quantum-classical computations,” arXiv preprint arXiv:1811.04968, 2018. [31] R. R. Selvaraju et al., “Grad-CAM: Visual explanations from deep networks via gradient-based localization,” in Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017, pp. 618–626. [32] M. Xue et al., “A reinforcement learning-based ELF adversarial malicious sample generation method,” IEEE Journal on Emerging and Selected Topics in Circuits and Systems, vol. 14, no. 4, pp. 743–757, 2024. [33] L. Hrdonka and M. Jureček, “Adversarial malware generation in linux elf binaries via semantic-preserving transformations,” arXiv preprint arXiv:2604.22639, 2026. [34] Y. Qiao et al., “Adversarial ELF malware detection method using model interpretation,” IEEE Transactions on Industrial Informatics, vol. 19, no. 1, pp. 605–615, 2023. [35] M. T. West et al., “Benchmarking adversarially robust quantum machine learning at scale,” Physical Review Research, vol. 5, no. 2, p. 023186, 2023.

Related documents

• Quantum computing and Artificial Intelligence: Toward a new computational paradigm#112930
• Training Variational Quantum Algorithms Is NP-Hard, Even Locally#1201141
• The Convergence of Quantum Computing and Machine Learning: A Path to Accelerating AI Solutions In#72262
• Hybrid Quantum-Classical GANs for the Generation of Adversarial Network Flows#168310
• Hybrid Variational Quantum-Classical Framework with Adaptive Weighting and Efficiency Assessment#1076148