1
Breaking TinyML: Why Quantized Neural Networks Need Domain-Specific Security Analysis
arXiv:2606.14427v1 [cs.CR] 12 Jun 2026
Jacob Huckelberry∥∗† , Andrea Mattia Garavagno∥∗‡§ , Yuke Zhang¶ , Peter A. Beerel¶ , James Mickens∗ , Vijay Janapa Reddi∗ ∗ Harvard University, † Draper Laboratory, ‡ University of Genoa, § Scuola Superiore Sant’Anna, ¶ University of Southern California ∥ These authors contributed equally to this work and share first authorship
Abstract—Most TinyML hardware focus on supporting Quantized Neural Networks (QNNs) to meet stringent constraints on power consumption, size, and cost. Despite this, the security aspects of quantization within TinyML hardware remain largely unexplored. Although previous studies indicate that QNNs demonstrate similar or enhanced robustness when compared to full-precision Deep Neural Networks (DNNs) against typical evasion attacks, no attack strategies tailored specifically for TinyML hardware have been proposed yet. This paper addresses the aforementioned shortfall by demonstrating how a two-step attack pipeline can surpass the current state-of-the-art in the QNN context and shows the need for more hardware-aware security research. Index Terms—Tiny Machine Learning (TinyML), TinyML Security, Model Extraction, Evasion, Quantization
I. I NTRODUCTION The rapid proliferation of Tiny Machine Learning (TinyML) has introduced unique security challenges that fundamentally differ from traditional machine learning deployments. TinyML mandates quantization to meet the stringent resource constraints of commodity electronics. Furthermore, most TinyML hardware accelerators, including Google’s Coral Edge TPU, Qualcomm’s Hexagon NPU, and NE16, exclusively support Quantized Neural Networks (QNNs) [1], [2], creating a distinct computational paradigm that necessitates specialized security analysis for hardware-accelerated AI systems. Despite the widespread adoption of quantization in TinyML hardware, the security implications of this hardware-software co-design choice remain underexplored. Current understanding of QNN security relies heavily on applying state-of-the-art Deep Neural Network (DNN) attack methodologies to quantized models. Preliminary studies [3] have crafted adversarial examples for QNNs using model evasion attacks originally designed for DNNs. These studies revealed that QNNs show no loss in model robustness and, in some cases, demonstrate better robustness than their full-precision counterparts [4]. However, a critical gap exists in the current security landscape: no attack specifically designed to exploit the hardware characteristics of QNN accelerators has been proposed [4]. The unique properties of quantized hardware implementations, including discrete value spaces, gradient discontinuities, and specialized computational patterns, create potential attack vectors that conventional DNN attacks cannot exploit, yet this hardware-specific attack surface remains unexplored.
The practical implications are significant. TinyML devices are deployed at massive scale (smart doorbells, parking monitors, industrial sensors) and firmware updates are difficult or impossible for many embedded systems. An attacker who extracts a model once can then craft reusable physical adversarial perturbations: a sticker on a license plate that makes a car invisible to parking fee detectors, or a pattern printed on clothing that defeats person detection in security cameras. Unlike digital attacks requiring per-image computation, these physical perturbations work persistently whenever the deployed model encounters them, amplifying the impact across millions of devices. To address this critical gap, we apply for the first time a well-studied two-step attack pipeline for DNNs [5], [6] to circumvent the typical gradient discontinuities preventing the application of white-box attacks in the QNN domain. Our approach achieves up to 47.30% accuracy reduction on CIFAR-10 (reducing model accuracy from 85.30% to 38.00%, compared to only 19.70% for the best gray-box attack) while requiring only 50,000 total queries. In contrast, black-box attacks like Boundary Attack and GeoDA require thousands of queries per adversarial example; generating adversarial examples for 1,000 test images would require millions of total queries, making our approach orders of magnitude more efficient. Moreover, this attack effectiveness is hardware-specific: when applied to full-precision models, our methodology underperforms compared to existing attacks. This performance differential challenges the assumption that quantization inherently improves model robustness. Instead, our findings suggest that conventional attacks may simply be ill-suited for the quantized hardware domain, creating an artificial impression of enhanced security. The contributions of this work are fourfold: 1) We demonstrate the first successful circumvention of hardware-imposed gradient discontinuities in QNNs through surrogate-based white-box attacks achieving 8284% fidelity. 2) We show, on CIFAR-10, superior performance against quantized hardware models compared to state-of-the-art gray-box attacks (over 20% better accuracy reduction) successfully overcoming gradient-masking, while maintaining practical query efficiency.
This work has been accepted for publication in IEEE Micro. © 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. Final publication is available at https://doi.org/10.1109/MM.2026.3666128.
2
3) We benchmark our approach on commercial TinyML hardware and demonstrate the fastest interaction time compared to the most query-efficient state-of-the-art alternative. 4) We prove that the attack’s effectiveness is hardwarespecific, thereby establishing the critical need for domainspecific security research in TinyML systems. These findings have significant implications for the rapidly expanding TinyML hardware ecosystem, where resource constraints often preclude the implementation of comprehensive security measures, making hardware-aware attack methodologies and corresponding defenses increasingly important. II. R ELATED W ORKS The intersection of DNN quantization and adversarial robustness has emerged as an important research area, though current understanding remains limited by the application of conventional attack methodologies to quantized models deployed on TinyML hardware. A. Quantization and Adversarial Robustness Early investigations into the security implications of quantization revealed counterintuitive findings regarding model robustness. Previous studies demonstrated that quantization of neural networks does not inherently lead to loss in model robustness and can introduce resilience against the transferability of adversarial attacks [3]. Building on these preliminary findings, recent comprehensive work by Costa et al. [4] found that 8-bit QNNs deployed on edge hardware are more robust to some state-of-the-art model evasion attacks than their fullprecision counterparts. Their study systematically evaluated both gray-box attacks, which can query the model an unlimited number of times while obtaining complete output information in the form of confidence scores, and black-box attacks, which can only access the predicted class labels. However, their evaluation was limited to directly applying existing DNN attacks without considering hardware-specific vulnerabilities. B. State-of-the-Art Attack Methodologies The evaluation of QNN security has relied on adapting existing DNN attack frameworks to the quantized domain. Among the state-of-the-art gray-box attacks applied to QNNs on TinyML hardware, the Zeroth Order Optimization (ZOO) attack and the Square Attack (SA) have shown varying degrees of effectiveness. The ZOO attack leverages zeroth order stochastic coordinate descent along with dimension reduction, hierarchical attack, and importance sampling techniques to craft adversarial perturbations for given input images. In contrast, the SA attack employs random search to craft squared perturbations capable of fooling the classifier for a given input. While effective against DNNs, our results show these methods underperform when targeting quantized hardware due to gradient masking effects. For black-box attack evaluation, researchers have applied the Boundary Attack (BA) and the Geometric Decision-based Attack (GeoDA) to QNNs. The BA starts from a large adversarial perturbation and then seeks to reduce the perturbation while maintaining its adversarial properties, whereas GeoDA
exploits the low mean curvature of the decision boundary in the vicinity of data samples to effectively estimate the normal vector to the decision boundary and produce minimal perturbations to cross it. While these achieve high success rates, they require prohibitive query counts unsuitable for practical hardware or TinyML deployment scenarios. C. Limitations of Current Approaches An important characteristic distinguishing QNN security evaluation from traditional DNN analysis is the absence of white-box attack assessments. Black-box attacks can require large amounts of queries to craft adversarial examples, while gray-box attacks can be more efficient in this aspect by leveraging confidence scores of the attacked DNN. The barrier to white-box analysis in QNNs is the gradient discontinuity problem. Quantization operations map continuous inputs to discrete output levels, creating a staircase activation function. In this environment, exact gradient calculation fails because: Zero Gradients: On the flat sections of the ’steps,’ the derivative is exactly zero, providing no directional information for the attack algorithm to update its noise; • Undefined Gradients: At the vertical jumps between steps, the derivative is undefined. •
This limitation represents a significant gap in current QNN security analysis, as white-box attacks typically achieve higher success rates than their gray-box and black-box counterparts in the full-precision domain. The inability to compute exact gradients has effectively eliminated an entire class of potent attack methodologies from QNN security evaluation on TinyML hardware. Our work circumvents this limitation through surrogate model construction. D. Research Gap and Motivation The current state of QNN security research reveals a fundamental limitation: all existing approaches apply attack methodologies originally designed for full-precision DNNs to quantized models running on specialized hardware. This approach assumes that the security characteristics of hardwareaccelerated QNNs can be adequately understood through the lens of conventional attacks, potentially overlooking vulnerabilities specific to the quantized hardware domain. Our work addresses this gap by proposing for the first time the usage of a well-studied two-step attack pipeline for DNNs [5], [6], in the QNN hardware domain, enabling the usage of white-box attacks through surrogate model construction. While surrogate-based model extraction has been explored extensively in the DNN domain [7], [8], no prior work has applied this pipeline specifically to circumvent gradient discontinuities in hardware-deployed QNNs. In the first phase, a surrogate model is built using extraction attacks, then whitebox attacks are used to craft adversarial examples leveraging exact gradient calculations from the surrogate. Finally the crafted adversarial examples are transferred to the target QNN running on TinyML hardware. The latter methodology overcomes state-of-the-art gray-box attacks on MLPerf Tiny models [9], a TinyML hardware
3
benchmark, while providing a more practical and scalable alternative to direct black-box attacks thanks to a one-time query performed during the model extraction phase. Importantly, the performance of the adopted methodology falls behind in the full-precision context, highlighting the need for hardwareaware attacks and challenging assumptions about the inherent robustness advantages of quantized models on accelerators. III. T HREAT M ODEL We define a threat model outlining the adversary’s objectives, capabilities, and knowledge, providing a practical framework for evaluating attacks against TinyML hardware. A. Adversary Goals The adversary’s primary objective is to compromise the integrity of the targeted quantized neural network by slightly perturbing benign inputs causing the model to misclassify them. While there are multiple pathways to achieve model compromise [5], [6], this work focuses specifically on a two-step attack sequence that systematically combines model extraction and evasion attacks to exploit vulnerabilities unique to the quantized hardware domain. B. Adversary Capabilities The attacker has temporary physical access to the deployed TinyML device, which is sufficient to perform queries and inject crafted inputs during the model extraction phase, but not necessarily to extract firmware or permanently modify it. We assume a Man-in-the-Middle (MitM) or direct injection setup where the adversary can bypass the physical sensor and inject digital signals directly into the hardware interface (e.g., SPI, I2C, or MIPI CSI). Furthermore, regarding output observation, the attacker is restricted to observing external device behaviors. This includes monitoring GPIO state transitions (e.g., a ’detect’ pin going high), analyzing power consumption traces, or observing network side-channels (e.g., WiFi packet timing). C. Adversary Knowledge To make this analysis as realistic as possible while remaining practically relevant, we assume the attacker operates under a black-box threat model with several important constraints and assumptions that align with real-world TinyML hardware deployment scenarios. The attacker does not have access to the model architecture details, trained weights, or gradient information. They may only interact with the model by feeding it inputs and observing the corresponding outputs, simulating the typical interaction pattern available to external adversaries targeting deployed TinyML accelerators. We assume a set of unlabeled samples drawn from the Original Domain Data (ODD) distribution is known to the attacker for a one-time model extraction phase. This reflects a realistic scenario where an attacker targeting a specific application, such as a commercial person-detection doorbell, can collect imagery from well-known TinyML person detection datasets. The attacker does know that the target model employs 8bit integer quantization. This assumption is reasonable given that quantization is a well-known characteristic of TinyML hardware deployments, often documented in accelerator specifications or inferable from hardware constraints.
Fig. 1: A figure describing the two-step attack. First, the attacker performs a model extraction attack using Copycat CNN. For this attack, the adversary uses the ODD to query the target model, and uses the input output pairs to train a surrogate model. The attacker then uses the surrogate model to craft adversarial examples using PGD and FGSM. These adversarial examples are then transferred to the target model.
Additionally, we assume the model only outputs hard labels rather than the full softmax probability distribution. This constraint aligns well with TinyML hardware deployment practices that require minimization of computational overhead and information leakage. Many TinyML accelerators implement this restriction as both a performance optimization and a security measure, making our assumption highly realistic for practical hardware deployments. IV. T INY ML M ODEL E XTRACTION AND E VASION The attack pipeline operates by first approximating the target model through a model extraction attack, then leveraging the resulting surrogate model to craft white-box adversarial examples using established methods such as the Fast Gradient Signed Method (FGSM) [10] and Projected Gradient Descent (PGD) [11]. These adversarial examples are subsequently transferred to the target model running on TinyML hardware to evaluate their effectiveness. A comprehensive visual depiction of this attack sequence is presented in Figure 1. A. Model Extraction Step The foundation of our attack pipeline lies in conducting a strategic model extraction attack to create a surrogate model that strongly approximates the decision boundaries and behavior of the target quantized model deployed on hardware. While several approaches exist for model extraction, we selected Copycat CNN [8] based on its compatibility with our defined threat model and its minimal operational requirements. Copycat CNN is particularly well-suited to our threat model because it operates under minimal assumptions that align with realistic TinyML hardware deployment constraints. The method requires only that the target model outputs hard labels, the attacker has access to unlabeled data from an appropriate distribution, and that the attacker has selected a suitable surrogate model architecture for the extraction process. The Copycat CNN extraction process utilizes the ODD to systematically query the target model and collect the inputoutput pairs necessary for surrogate training. ODD refers to the dataset on which the model was originally trained and
4
validated, providing the most direct approximation of the target’s training distribution. The collected input-output pairs are then used to train a surrogate model. Importantly, the surrogate model need not perfectly replicate the target’s outputs; it only needs to approximate the target’s decision boundary geometry. A surrogate achieving 82-84% fidelity (agreement on roughly 5 out of 6 predictions) is sufficient for adversarial transfer because adversarial examples exploit the structure of decision boundaries rather than exact weight values. Perturbations that cross boundaries in the surrogate will likely cross similar boundaries in the target, enabling effective attacks despite imperfect extraction. B. Model Evasion Step Once the attacker has successfully crafted a sufficiently accurate surrogate model, or ensemble of surrogate models, they can proceed to the adversarial example generation phase. The key advantage of this approach is that the attacker now has white-box access to the surrogate model(s), enabling the use of gradient-based white-box evasion attacks to generate adversarial examples that can be transferred to the target model on the hardware accelerator. For the evasion component of our attack pipeline, we selected FGSM [10] and PGD [11] due to their demonstrated effectiveness and strong transferability properties across different model architectures and hardware domains. Both attacks represent well-established methods in the adversarial machine learning literature and provide complementary approaches to adversarial example generation. Both FGSM and PGD generate perturbations constrained by the L∞ norm, creating perturbations that are dense and uniform across the target image. This perturbation characteristic is particularly important for transferability, as the uniform nature of the modifications means they are applied broadly across the input space and do not rely on specific input features that may be unique to the surrogate model architecture. C. Benchmark Selection and Target Models For our experimental evaluation, we selected the benchmark from the TinyML industry-standard MLPerf Tiny suite [9], which provides models and datasets specifically designed for hardware accelerator deployments. We target two key models representative of real-world hardware constraints: a customized ResNetv1 for CIFAR-10, and a MobilenetV1-0.25 for Visual Wake Words (VWW). These models are optimized for different hardware accelerator architectures and application domains within the TinyML ecosystem. D. Model Extraction Implementation In the model extraction phase, we use Copycat CNN [8] as implemented in the Adversarial Robustness Toolbox (ART) to generate surrogate models that approximate the quantized behavior of models running on TinyML hardware. The selection of surrogate architectures balances approximation quality with the computational constraints typical of attacking deployed hardware systems. For the CIFAR-10 hardware reference model, we employ ResNet50 as the surrogate, while for the VWW hardware
model, we use MobileNetV1. Both utilize ImageNet pretrained weights from Keras Applications, providing a foundation for capturing hardware-specific behaviors. We systematically query the target hardware with 50,000 samples from the ODD training set. The resulting input-output pairs fine-tune the surrogate models for 10 epochs, balancing extraction quality with the query limitations imposed by real hardware deployments. E. Hardware Implementation A microcontroller, programmed to emulate the sensor’s interface, injects samples to the target TinyML hardware and then monitors the GPIO activity to detect the predicted label, according to the threat model proposed in section III. The microcontroller communicates via SPI with a host computer and is used both to query the target TinyML hardware in the model extraction phase and to test the adversarial examples crafted by the adopted attack. F. Hardware Selection We use three commercial TinyML hardware platforms to evaluate the cost of the model extraction phase, namely: the NUCLEO-F401RE, the NUCLEO-G474RE, and the NUCLEO-H743ZI2. An OV7670 image sensor is connected to each board. I2C interface and DMA are set on every board to automatically transfer the data acquired from the sensor to the memory buffer used as input of the programmed neural network without impacting the computational power available to the TinyML hardware. The OV7670 outputs a maximum of 30 frame per second. G. Adversarial Example Generation and Transfer Following surrogate model creation, we generate adversarial examples using both PGD and FGSM attacks. We craft adversarial examples using 1,000 test samples with ϵ = 8/255, the standard for evaluating hardware robustness [12]. The ϵ parameter defines the limit on how much the attacker is allowed to change the original image to fool the model, ensuring the attack remains ”imperceptible” to the human eye. These examples are then transferred to target models running on both int-8 quantized hardware (actual deployment) and full-precision implementations to assess hardware-specific vulnerabilities. V. E VALUATION We evaluate our hardware-aware attack pipeline on two MLPerf Tiny benchmarks, demonstrating that our two-step approach effectively exploits vulnerabilities specific to quantized models deployed on TinyML hardware. Our comprehensive analysis examines each attack phase across both benchmarks and compares performance against state-of-the-art methods. A. Evaluation Methodology We assess our attack pipeline along four key dimensions for each benchmark dataset (CIFAR-10 in Section V-B and VWW in Section V-C): Model Extraction Fidelity measures how accurately our surrogate models replicate the behavior of hardware-deployed
5
TABLE I: Copycat CNN results on the target ResNetV1 using the ODD dataset CIFAR-10 (upper section of the table) and on target MobileNetV1 with the alpha parameter set at 0.25 using the ODD dataset VWW (lower section of the table).
precision extraction fidelity, as Copycat CNN relies only on hard-label outputs which are preserved across quantization levels.
Target Model
Surr. Arch.
Figure 2a reveals a key finding: adversarial examples crafted on our surrogate models transfer equally well to both int-8 quantized hardware and full-precision models, with the solid (int-8) and dotted (full-precision) lines largely overlapping. This contradicts previous work suggesting poor transferability to quantized models [4], and indicates that our extraction process successfully captures the quantized model’s behavior, enabling effective white-box attacks on hardware that would otherwise be impossible due to gradient discontinuities.
Fidelity
Target Acc.
Surr. Acc.
ResNetV1 (Int-8) ResNetV1 (FP-32) ResNetV1 (Int-8) ResNetV1 (FP-32)
CIFAR-10 ResNet-50 82.70% ResNet-50 83.00% MbNetV1 83.80% MbNetV1 83.40%
84.30% 84.30% 84.30% 84.30%
83.00% 82.30% 84.40% 84.80%
MbNet025 (Int-8) MbNet025 (FP-32) MbNet025 (Int-8) MbNet025 (FP-32)
Visual Wake Words ResNet-50 79.40% ResNet-50 81.40% MbNetV1 82.10% MbNetV1 80.80%
85.60% 85.60% 85.60% 85.60%
69.60% 71.9% 72.00% 70.60%
2) Phase 2: Adversarial Transferability
3) Performance Against State-of-the-Art targets. High fidelity, where the surrogate mirrors the hardware model’s decision boundaries, is essential for generating effective adversarial examples against accelerators. Adversarial Transferability evaluates whether adversarial examples crafted on surrogate models successfully fool both int-8 quantized hardware and full-precision models. This reveals whether our attack exploits hardware-specific vulnerabilities or general model weaknesses. Comparison with State-of-the-Art benchmarks our approach against existing gray-box and black-box attacks, demonstrating superior effectiveness and query efficiency critical for practical hardware deployments. Feasibility on TinyML Hardware measures the interaction time with the target TinyML hardware required by the proposed attack. Throughout our evaluation, we compare attack performance between int-8 quantized models (actual hardware deployment) and full-precision counterparts. This comparison confirms that our approach specifically targets hardware-induced vulnerabilities, as detailed in Section V-D. B. CIFAR-10 Results Our attack pipeline achieves significant accuracy reduction on the CIFAR-10 hardware benchmark, with PGD reducing model accuracy from 85.30% to 38.00% (47.30% reduction) and FGSM achieving 35.30% reduction on int-8 quantized models. This attack effectiveness is domain-specific: when applied to full-precision models, our approach underperforms compared to state-of-the-art attacks, highlighting that we are exploiting hardware-specific vulnerabilities rather than general model weaknesses. Furthermore, our approach requires only 50,000 queries total, making it orders of magnitude more practical than black-box alternatives that can require millions of queries. Next, we now examine each phase of the attack to understand how this effectiveness is achieved. 1) Phase 1: Model Extraction Fidelity The basis of our attack’s success lies in achieving high fidelity during model extraction. As shown in Table I, our extracted surrogate models achieve 82.70% to 83.80% fidelity with the target hardware model, indicating strong approximation of the hardware-specific decision boundaries. Interestingly, there is minimal difference between int-8 and full-
Table II positions our approach within the landscape of existing attacks. Against int-8 hardware models, we outperform all gray-box attacks by over 20% in accuracy reduction, likely because our surrogate-based approach circumvents the gradient masking that affects direct gray-box methods like ZOO. While black-box attacks achieve higher success rates, they require thousands of queries per image versus our onetime 50,000 query cost. Most importantly, our attack’s effectiveness diminishes on full-precision models, confirming that we are exploiting vulnerabilities specific to quantized hardware implementations rather than general model weaknesses. 4) Feasibility on TinyML Hardware Table III, in the CIFAR-10 section, showcases the query efficiency of our approach, which translates to the fastest interaction time with the target TinyML hardware. Only 0.5 hours of querying is required with the fastest hardware, the NUCLEO-H743ZI2 (H7), while 3.4 hours is required with the slowest hardware, the NUCLEO-F401RE (F4). When compared to the most query-efficient alternative of the Square Attack, the range increases up to 67.8 hours for the slowest hardware and 18.5 hours for the fastest hardware. C. Visual Wake Words Results Our attack pipeline demonstrates moderate effectiveness on the Visual Wake Words hardware benchmark, achieving 12.70% accuracy reduction with FGSM and 10.40% with PGD on int-8 quantized models. While less dramatic than CIFAR-10, these results reveal important insights about hardware vulnerabilities in binary classification tasks. Notably, our approach outperforms some state-of-the-art gray-box attacks while maintaining the same practical advantage of requiring only 50,000 queries. The attack again shows hardware specificity, with reduced effectiveness against full-precision models. We examine each phase to understand the factors affecting attack performance on this binary classification task. 1) Phase 1: Model Extraction Fidelity Model extraction results for VWW (Table I, lower section) show fidelity ranging from 79.40% to 82.10%, slightly lower 1 The time has been computed multiplying the average query time of each TinyML hardware, measured over 1000 queries, by the number of queries required by each attack.
6
(a) Target ResNetV1 using CIFAR-10
(b) Target MobileNetV1 0.25x using VWW Fig. 2: Model evasion results on the target ResNetV1 using CIFAR-10 and on MobileNetV1 0.25x using VWW. The results for FGSM are on the left, and the results for PGD are on the right. Solid lines denote the adversarial accuracy of int-8 quantized models, while dotted lines denote the adversarial accuracies of full-precision models. There are separate sets of these lines for whether the same architecture of the target model was used for creating the surrogate model.
than CIFAR-10 but still indicating strong approximation of the hardware model. However, surrogate accuracy is noticeably lower than the target model accuracy (69.60%-72.00% vs 85.60%), likely due to our query budget being half the original training set size (50,000 queries vs 100,000 training samples). Despite lower surrogate accuracy, the high fidelity metrics suggest the surrogate captures the hardware-specific decision boundaries necessary for adversarial generation. 2) Phase 2: Adversarial Transferability Figure 2b shows more nuanced transferability patterns compared to CIFAR-10. In the FGSM plot, solid (int-8 quantized hardware) lines fall slightly below dotted (full-precision) lines, suggesting marginally better transferability to quantized hardware. Conversely, the PGD plot shows the opposite trend. These marginal differences likely stem from attack nondeterminism rather than fundamental differences between hardware and full-precision models.
The reduced attack effectiveness on VWW compared to CIFAR-10 likely stems from the fundamental geometry of binary classification. With only two classes (person vs. noperson), the model learns a single decision boundary rather than the complex multi-class boundaries in CIFAR-10’s tenclass problem. This simpler boundary structure means perturbations must push inputs further to cross into the opposite class, and there are no ”nearby” alternative classes that might capture misclassified inputs. Additionally, VWW’s binary output provides less information for the surrogate to learn from during extraction, potentially resulting in coarser approximation of the true decision boundary. 3) Performance Against State-of-the-Art Table II reveals a different competitive landscape for VWW. We acknowledge that Square-L2 achieves higher accuracy reduction than our approach (26.40% vs. 12.70%). However, this comparison requires important context: Square Attack
7
TABLE II: Comparison of two-step attack sequence results to direct gray-box and black-box attacks for VWW and CIFAR-10. The results for the gray-box and black-box attacks are pulled from [4], who used the same models and datasets as in our work Attack
Accuracy
CIFAR-10 Adv. Acc. Acc. Reduct.
L∞
Accuracy
Visual Wake Words Adv. Acc. Acc. Reduct.
Work
L∞
QNN (8-bit integer) Proposal: Extraction → White-Box PGD FGSM
85.30% 85.30%
38.00% 50.00%
47.30% 35.30%
0.03 86.10% 0.03 86.10% Gray-Box
75.70% 73.40%
10.40% 12.70%
0.03 0.03
This Work This Work
ZOO Square-L2 Square-L∞
87.50% 87.50% 87.50%
87.50% 67.80% 82.30%
0.00% 19.70% 5.20%
0.00 91.00% 0.22 91.00% 0.01 91.00% Black-Box
84.00% 64.60% 86.00%
7.00% 26.40% 5.00%
0.09 0.32 0.01
Costa et al. [4] Costa et al. [4] Costa et al. [4]
Boundary GeoDA
87.50% 87.50%
6.50% 13.00%
81.00% 74.50%
0.08 0.08
91.00% 91.00%
0.00% 0.00%
0.00 0.03
Costa et al. [4] Costa et al. [4]
91.00% 91.00%
DNN (32-bit floating-point) Proposal: Extraction → White-Box PGD FGSM
84.80% 84.80%
40.20% 51.10%
44.60% 33.70%
0.03 86.50% 0.03 86.50% Gray-Box
74.40% 75.50%
12.10% 11.00%
0.03 0.03
This Work This Work
ZOO Square-L2 Square-L∞
88.30% 88.30% 88.30%
24.70% 47.30% 6.10%
63.60% 41.00% 82.20%
0.05 91.30% 0.22 91.30% 0.01 91.30% Black-Box
54.00% 45.30% 37.60%
37.30% 46.00% 53.70%
0.20 0.28 0.01
Costa et al. [4] Costa et al. [4] Costa et al. [4]
Boundary GeoDA
88.30% 88.30%
7.70% 11.00%
80.60% 77.30%
0.01 0.03
91.30% 91.30%
0.00% 0.00%
0.00 0.03
Costa et al. [4] Costa et al. [4]
TABLE III: Time spent1 querying the target TinyML hardware by the two-step attack adopted in this paper and the Square Attack, the most query efficient alternative requiring only 1 query per iteration to generate an adversarial example. The hardware targets of this comparison are NUCLEO-F401RE (F4), NUCLEO-G474RE (G4), and NUCLEO-H743ZI2 (H7). The same setting adopted by [4] have been adopted. Attack
F4
CIFAR-10 G4
H7
Visual Wake Words F4 G4 H7
QNN (8-bit integer) Proposal Square-L2 Square-L∞
3.4h 135.6h 67.8h
1.8h 70.6h 35.3h
0.5h 18.5h 9.3h
2.2h 87.8h 43.9h
1.2h 47.2h 23.6h
0.5h 18.5h 9.3h
91.30% 91.30%
4) Feasibility on TinyML Hardware Furthermore our approach demonstrates to be better suited to interact with TinyML hardware by showcasing the fastest query time. On the slowest hardware the query time of our approach is only 2.2 hours, while the fastest query time of Square Attack on the same hardware is 67.8 hours, 30.8 times slower, according to table III, Visual Wake Words section. Notably the NUCLEO-H743ZI2 (H7) achieves the same query time as the CIFAR-10 setup, while the other targets show a faster query time. This can be explained by the 30 frames per second limit imposed by the OV7670 sensor, which acts in this case as a bottleneck for the overall system. D. The Cause Behind Hardware Specificity
requires access to confidence scores, which real TinyML hardware rarely exposes. More critically, black-box attacks (Boundary Attack, GeoDA) achieve 0% accuracy reduction on VWW because the binary classification boundaries defeat these methods entirely. This makes our approach the only effective option for attacking VWW on hardware that only exposes hard labels. The full-precision results again confirm hardware specificity: our attack performs comparably poorly on both quantized and full-precision VWW models, but state-of-the-art attacks show dramatically better performance on full-precision versions. This reinforces that existing attacks fail to exploit hardware-specific vulnerabilities, while our approach, though less effective on binary classification, still targets weaknesses unique to quantized implementations.
Previous work [4] argued that the underperformance of black- and gray-box attacks in the QNN setting is caused by quantization, which may erase small perturbations, making the gradient of the attack zero, or amplify bigger perturbations, making the gradient of the attack explode. Our approach circumnavigates this effect by attacking the surrogate model, which is not quantized. However, in the DNN domain, quantization is gone, but our approach still works on the surrogate model, acting as if quantization were still there, and therefore approaching the problem in a suboptimal way, falling behind the state-of-the-art. VI. C ONCLUSION This work demonstrates the critical need for domain-specific security analysis of TinyML systems by showing how a twostep attack pipeline, designed specifically for quantized neural networks, outperforms state-of-the-art methods on TinyML
8
hardware. By achieving high-fidelity model extraction followed by white-box adversarial generation, we circumvent the gradient discontinuities that have previously protected quantized models from gradient-based attacks. Our approach achieves significant accuracy reduction on MLPerf Tiny benchmarks while showcasing the smallest interaction time with the TinyML target hardware. Furthermore, it is the only attack able to affect the simpler decision boundaries of the model trained on the Visual Wake Words dataset without requiring access to the class probabilities, which are rarely exposed by real-world TinyML hardware. Most importantly, the attack’s effectiveness is specific to quantized implementations, confirming that we exploit hardware-induced vulnerabilities rather than general model weaknesses. As AI accelerators proliferate across edge computing applications, ranging from autonomous systems to IoT devices, our findings emphasize that hardware-aware security must become a fundamental design consideration rather than an afterthought. Future work should focus on developing comprehensive security frameworks that account for the unique computational characteristics of quantized neural networks and the constraints of hardware deployment. Only through such domain-specific approaches can we ensure the security and reliability of the rapidly expanding TinyML ecosystem.
Jacob Huckelberry received his B.Sc. in Computer Science from the United States Military Academy in 2023 and his M.Sc. in Data Science from Harvard University. Concurrently, Jacob is a Cyber Officer in the U.S. Army and a Draper Laboratory Scholar in their Cyber AI Tools group. His research interests lie at the intersection of machine learning, embedded systems, and cybersecurity. Contact him at [email protected].
Andrea Mattia Garavagno received his B.Sc. degree in Electronic Engineering from the University of Genova, Italy, in 2018 and his M.Sc. degree with honours in Embedded Computing Systems, from Scuola Superiore Sant’Anna and the University of Pisa, Italy, in 2022. He’s currently a Ph.D. student at the University of Genoa co-funded by Scuola Superiore Sant’Anna, Italy, and a Fellow of the Harvard John A. Paulson School of Engineering and Applied Sciences researching TinyML systems. He co-authored the book “Introduction to Microprocessor-Based Systems Design” published by Springer-Nature. Contact him at [email protected].
Yuke Zhang received the B.Eng. degree from the Beijing University of Posts and Telecommunications, Beijing, China, and the M.A.Sc. from Dalhousie University, Halifax, NS, Canada. She is currently pursuing the Ph.D. degree in electrical and computer engineering with the University of Southern California, Los Angeles, CA, USA. Her research interests include privacy-preserving machine learning and hardware security. Contact her at [email protected].
R EFERENCES [1] K. Seshadri, B. Akin, J. Laudon, R. Narayanaswami, and A. Yazdanbakhsh, “An evaluation of edge tpu accelerators for convolutional neural networks,” in 2022 IEEE International Symposium on Workload Characterization (IISWC). IEEE, 2022, pp. 79–91. [2] F. Conti, P. D. Schiavone, and L. Benini, “Xnor neural engine: A hardware accelerator ip for 21.6-fj/op binary neural network inference,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 37, no. 11, pp. 2940–2951, 2018. [3] H. Thorsteinsson, V. J. Henriksen, T. Chen, and R. Selvan, “Adversarial fine-tuning of compressed neural networks for joint improvement of robustness and efficiency,” arXiv preprint arXiv:2403.09441, 2024. [4] M. Costa and S. Pinto, “David and goliath: An empirical evaluation of attacks and defenses for qnns at the deep edge,” in 2024 IEEE 9th European Symposium on Security and Privacy (EuroS&P). IEEE, 2024, pp. 524–541. [5] N. Papernot, P. McDaniel, and I. Goodfellow, “Transferability in machine learning: from phenomena to black-box attacks using adversarial samples,” arXiv preprint arXiv:1605.07277, 2016. [6] M. Juuti, S. Szyller, S. Marchal, and N. Asokan, “Prada: protecting against dnn model stealing attacks,” in 2019 IEEE European Symposium on Security and Privacy (EuroS&P). IEEE, 2019, pp. 512–527. [7] T. Orekondy, B. Schiele, and M. Fritz, “Knockoff nets: Stealing functionality of black-box models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 4954–4963. [8] J. R. Correia-Silva, R. F. Berriel, C. Badue, A. F. De Souza, and T. Oliveira-Santos, “Copycat cnn: Stealing knowledge by persuading confession with random non-labeled data,” in 2018 International joint conference on neural networks (IJCNN). IEEE, 2018, pp. 1–8. [9] C. Banbury, V. J. Reddi, P. Torelli, J. Holleman, N. Jeffries, C. Kiraly, P. Montino, D. Kanter, S. Ahmed, D. Pau et al., “Mlperf tiny benchmark,” arXiv preprint arXiv:2106.07597, 2021. [10] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” arXiv preprint arXiv:1412.6572, 2014. [11] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” arXiv preprint arXiv:1706.06083, 2017. [12] F. Croce, M. Andriushchenko, V. Sehwag, E. Debenedetti, N. Flammarion, M. Chiang, P. Mittal, and M. Hein, “Robustbench: a standardized adversarial robustness benchmark,” arXiv preprint arXiv:2010.09670, 2020.
Peter A. Beerel is the Associate Chair of Graduate Programs in the ECE Department at USC and a Principal Distinguished Scientist and Research Director at the Information Sciences Institute. He co-founded TimeLess Design Automation in 2008, which was sold to Fulcrum Microsystems in 2010 then acquired by Intel in 2011. He has (co)authored 16 US patents and more than 190 conference and journal papers covering a wide range of topics in VLSI, CAD, and Machine Learning, several of which have won best paper awards. He has been on the program committee of various VLSI conferences and has been associate editor for IEEE TCAD and IEEE Transactions on Circuits and Systems I. Contact him at [email protected].
James Mickens is a Gordon McKay Professor of Computer Science at Harvard University. His research focuses on the performance and security of large-scale online services. He leverages techniques from isolated software architectures, applied cryptography, and secure hardware design. Prior to becoming a professor at Harvard, Dr. Mickens spent seven years at Microsoft Research, working in the Distributed Systems group and collaborating with various product teams from Azure and Windows; he was also a visiting professor at MIT’s Parallel and Distributed Operating Systems group. At Harvard, he serves on the Board of Directors for the Berkman Klein Center for Internet & Society. Contact him at [email protected].
Vijay Janapa Reddi expertise lies in computer architecture, machine learning systems, and autonomous agents. His leadership roles in MLCommons and the tinyML Foundation, along with his work in developing the MLPerf benchmarks, provide valuable insights into the practical challenges and opportunities of deploying TinyML in the real-world. Prof. Janapa Reddi’s experience in academia and industry collaborations ensures a comprehensive and practical approach to addressing the resource-constrained challenges of securing TinyML systems. Contact him at [email protected].