AMRM-Pure: Semantic-Preserving Adversarial Purification
Zhihao Dou1,2,∗ Zhiqiang Gao1,∗,† Dongfei Cui3 Weida Wang4 Qinjian Zhao1 1 5,† 6 7,† Dinggen Zhang Jun Yan Zeke Xie Shufei Zhang 1 Wenzhou-Kean University; 2 Case Western Reserve University; 3 Northeast Electric Power University; 4 Fudan University; 5 Shanghai Ocean University; 6 HKUST (Guangzhou); 7 Shanghai AI Lab
arXiv:2607.04474v1 [cs.CR] 5 Jul 2026
∗
Equal contribution.
Abstract Adversarial purification is a defense technique that employs generative models to remove adversarial perturbations. Current methods often rely on powerful generators, typically diffusion models, and focus on reducing the gap between adversarial and clean samples in the feature space, while overlooking semantic correlation within a single sample. To address this issue, we explore adversarial purification from the perspective of preserving semantic relationships among image patches. We employ an Attentive Mask Reconstruction Model (AMRM), which shows superior performance. Our theoretical and experimental analysis reveals that AMRM is highly sensitive to adversarial noise, as such noise significantly distorts patch relationships. Based on this observation, we propose AMRM-Pure, a purification framework that denoises adversarial inputs by preserving patch-level semantics, and formulate this process as a tractable optimization problem with respect to the input. To further enhance robustness, we finetune AMRM-Pure with classification loss to strengthen semantic consistency. We apply our insight to two AMRM architectures, including Mask Autoencoder (MAE) and MaskDiT. Extensive experiments confirm the effectiveness of our method, establishing new state-of-the-art performance across multiple benchmarks.
1
INTRODUCTION
Deep Neural Networks (DNNs) are vulnerable to adversarial examples (Carlini and Wagner, 2017; Song et al., Proceedings of the 29th International Conference on Artificial Intelligence and Statistics (AISTATS) 2026, Tangier, Morocco. PMLR: Volume 300. Copyright 2026 by the author(s).
†
Corresponding authors
2018; Fischer et al., 2017; Lyu et al., 2015), which are imperceptible to humans. However, these inputs with the malicious perturbations can cause DNNs to make erroneous predictions. Adversarial training (Madry et al., 2018; Zhang et al., 2019) is the state-of-the-art method for defending against adversarial attacks. However, the trade-off between generalization and robustness remains a concern (Zhang et al., 2019), especially against unseen adversarial examples. Furthermore, adversarial training incurs significantly higher computational costs compared to standard training. Alternatively, another notable defense strategy is adversarial purification, which attracts widespread attention. Adversarial purification can be broadly classified into two categories, including purification with generative models (Yoon et al., 2021; Nie et al., 2022; Lin et al., 2024; Bai et al., 2024; Zhang et al., 2024) and adaptation-based purification (Shi et al., 2021). Generative model-based approaches are the most widely used methods in adversarial purification, typically harnessing the powerful capabilities of generative models (e.g., diffusion) to transform the distribution of adversarial examples to that of clean samples (Nie et al., 2022). Future efforts will aim to further enhance the denoising capabilities of the purification model through various approaches. These include leveraging contrastive guidance to steer diffusion models (Bai et al., 2024), integrating classifier confidence guidance into the denoising process (Zhang et al., 2024), and fine-tuning the purification model with adversarial loss for robust optimization (Lin et al., 2024). The aforementioned methods primarily focus on aligning adversarial examples with the semantic distribution of clean samples, but neglect the semantic relationships among different patches within a sample. To fill this gap, we use an attentive mask reconstruction model (AMRM) to investigate how adversarial perturbations distort the semantic relationships among image patches. AMRM divides an image into multiple patches, masks a subset of them, and reconstructs the masked patches by using a self-attention (Vaswani et al., 2017) mecha-
AMRM-Pure: Semantic-Preserving Adversarial Purification
(a)
(b)
(c)
(d)
(e)
(f)
Figure 1: (a) Original image, (b) Masked image, (c) Clean image reconstruction from MAE, (d) Adversarial example under AutoAttack, (e) Reconstruction of adversarial example under AutoAttack from MAE, (f) Reconstruction of the denoised image under AutoAttack from MAE (denoised by our AMRM-PureMAE ). nism to explicitly capture dependencies among visible patches, such as the Mask Autoencoder (MAE) (He et al., 2022) or MaskDiT (Zheng et al., 2024). To design a robust purification method, we first identify an intriguing phenomenon of the simple MAE so that an easier analysis. Specifically, in the case of adversarial examples subjected to tiny, visually imperceptible perturbations, the reconstruction performance of MAE is severely compromised, dealing a devastating blow. As a typical example shown in Figure 1a and1d, although the clean example and adversarial example appear very similar, MAE’s reconstruction outputs in Figures 1c and 1e exhibit significant differences. The reconstruction of perturbed data, as illustrated in Figure 1e, still displays poor quality. These findings suggest that the reconstruction capability of MAE is highly sensitive to adversarial perturbations, although these perturbations are visually imperceptible. Motivated by this observation, we consider preserving the semantic relationships among image patches as a novel mechanism for adversarial purification, a direction that has not been fully explored in existing works. Based on such a research motivation, our proposed study aims to fill the gap. In this paper, through a series of analyses, we conjecture that this phenomenon results from adversarial perturbations that could easily distort semantic relations within patches, i.e., causing variation in the attention matrix (AMV), which leads to degraded image reconstruction quality of MAE. Through rigorous theoretical derivations and empirical experiments, we provide compelling evidence of the sensitivity of MAE to this AMV. Concretely, as the patch attention matrix essentially reflects how different semantic patches may be related to the masked patch, altering the attention matrix means a semantic change when the masked patch is reconstructed. Figures 3b and 3c show that the reconstruction of the target patch in the red square depends on the similar patches in the clean image, while for adversarial images, high importance is assigned to distant and dissimilar patches. This suggests that the adversarial perturbations alter the semantic relations among patches. Moreover, our findings reveal that the reconstruction loss of adversarial examples is lower-
bounded by the sum of the loss for clean examples and the AMV. Meanwhile, we reveal that the sensitivity of AMV is transferable to a diffusion-based AMRM, e.g., MaskDiT, as shown in empirical analysis in Figure 6. Drawing inspiration from this finding, we propose a novel AMRM-Pure method which purifies adversarial perturbations by minimizing AMV, ultimately resulting in a inter-patch semantic preserving framework. AMRM-Pure leverages the inherent sensitivity of AMV to adversarial noise, thereby achieving enhanced robustness. To realize AMRM-Pure, we introduce two variants: AMRM-PureMAE based on MAE (He et al., 2022) and AMRM-PureMaskDiT based on MaskDiT (Zheng et al., 2024). Furthermore, based on the insight of the previous work (Lin et al., 2024; Zhang et al., 2024), we propose a Robust AMRM-Pure based on MAE (RAMRM-PureMaskDiT ) and Robust MRM-Pure based on MaskDiT (RAMRM-PureMaskDiT ) that leverages classification loss to fine-tune the purification model, significantly improving its inter-patch semantic preservation capability. We have extensively evaluated our method by comparing the important adversarial training and adversarial purification methods on various challenging adaptive attack benchmarks. Our method achieves state-of-the-art (SOTA) performance on four datasets, e.g., CIFAR-10 (Krizhevsky et al., 2009), CIFAR-100 (Krizhevsky et al., 2009), SVHN (Netzer et al., 2011), and ImageNet (Deng et al., 2009). In summary, our main contributions are as follows: 1) We investigate the susceptibility of attention-based MRM to noise interference from both theoretical and empirical perspectives and disclose that the noise induces the deviation of semantic relations among patches, resulting in a degradation of the quality of the reconstruction. On the basis of our findings, we devise a novel and efficient purification technique, called AMRM-Pure, which is theoretically proven by rigorous analysis. 2) By successfully applying our approach to MAE and MaskDiT, we introduce AMRM-PureMAE and AMRMPureMaskDiT . Meanwhile, we further propose RAMRM-
Dou et al.
PureMAE and RAMRM-PureMaskDiT , which incorporates classification loss to fine-tune the purification model, significantly improving standard and robust accuracy.
�0adv
2.1
PRELIMINARIES AND RELATED WORK Adversarial Training
Adversarial training (AT) is a technique that enhances the robustness of a neural network by augmenting training samples with additional adversarial examples (Goodfellow et al., 2015; Kurakin et al., 2017; Tramèr et al., 2018; Zhang et al., 2019; Dou et al., 2024). Since AT typically involves a high computational cost, some studies (Wong et al., 2020; Liu et al., 2021; Vivek and Babu, 2020) have focused on exploring ways to accelerate the training process by a one-step training strategy. In addition, the diffusion model has been employed for extensive data augmentation for adversarial training in many proposals (Wang et al., 2023; Gowal et al., 2021; Sehwag et al., 2021), which enlarged the original dataset and enhanced the robust generalization. 2.2
Adversarial Purification
Generative models have shown great promise in purifying adversarial examples, drawing significant attention in robustness research. The early milestone study (Samangouei et al., 2018) introduced Defense-GAN, using GANs for purification. Song et al. (Song et al., 2017) proposed the PixelDefense method, which employs the autoregressive models to mitigate the perturbations. Score-based generative models have also been applied for defense (Yoon et al., 2021). Leveraging diffusion models, DiffPure (Nie et al., 2022) uses Stochastic Differential Equation (SDE) diffusion (Song et al., 2021) for the denoising procedure, achieving robustness. Recent works (Li et al., 2025; Liu et al., 2025) further improve robustness by fine-tuning diffusion models. Lin et al. (Lin et al., 2024) proposed a hybrid approach combining adversarial training with purification. It is significant to reconstruct the data without semantic information changes. IDC (Mei et al., 2025) redesigns diffusion models from generating high-quality images to producing distinguishable label images, proposing an efficient image-to-image diffusion classifier that significantly reduces computational cost while improving adversarial robustness. Thus, Bai et al. (Bai et al.,
AMRM
Rconstruction ��� loss ℒ���
reconstruction example
−� ���� ��
adversarial example �s−1 adv
3) Extensive experiments have been conducted to validate the effectiveness of our MRM-Pure on various benchmarks, showing that our approach consistently achieves favorable outcomes after denoising processes.
2
…
�sadv
�s−1 adv
+
…
Clip(�, �)
�Sadv
adversarial example �sadv
Purification Process
Figure 2: Overview of the proposed AMRM-Pure. 2024) introduced contrastive guidance in diffusion models to enhance purification while preserving semantics. The adversarial purification method can be combined with other machine learning paradigms. For example, the framework of Self-supervised Online Adversarial Purification (SOAP) (Shi et al., 2021) achieves notable results by integrating self-supervised tasks during training, further boosting robustness. There are several purification based on MAE (Zhou et al., 2023; Wu et al., 2022; You et al., 2023). DIR (Zhou et al., 2023) introduces a joint training framework of classifier and MAE under adversarial training, where the MAE restores robust features from unmasked patches to mitigate adversarial noise. Following the denoising autoencoder paradigm, DMAE (Wu et al., 2022) and NIM-MAE (You et al., 2023) incorporate Gaussian noise into masked image modeling, which is eliminated through the encoding–decoding process. Specifically, DMAE focuses on achieving robust pre-training against Gaussian noise to enhance generalization and robustness, though it shows limited effectiveness against adversarial perturbations. NIM-MAE leverages pretrained models to remove adversarial noise, yet its robustness performance still leaves room for improvement. Unlike previous methods, ARM-Pure leverages the sensitivity of semantic patch relationships to adversarial perturbations and employs optimization-based denoising to reduce AMV, effectively minimizing semantic variations in adversarial examples. This novel perspective has not been studied in previous research. 2.3
Preliminary of Masked Autoencoder (MAE)
MAE (He et al., 2022) is briefly introduced within the context of adversarial robustness in this subsection. A clean input sample x, drawn from the dataset X, is partitioned into n patch vectors of dimension d, forming x̄ ∈ Rn×d . The matrix x̄ can be randomly divided into m = (1 − ρ)n masked patch vectors and (n − m) visible patch vectors, where ρ is the mask ratio. MAE uses an encoder-decoder architecture. The encoder, f (·),
AMRM-Pure: Semantic-Preserving Adversarial Purification
(a)
(b)
(c)
(d)
Figure 3: The first column, Figure (a) represents the Mask Matrix. The second column, Figure (b) illustrates the Attention Weights for clean samples. The third column, Figure (c) depicts the Attention Weights for adversarial examples. The fourth column, Figure (d) showcases the Attention Weights for denoised samples (by our AMRM-PureMAE ). Patches with a deeper red color mean the elements with more attention. The data is sampled from the ImageNet dataset (Deng et al., 2009). produces Venc ∈ Rm×de , where Venc = f (x1 ), and x1 is the visible portion of input x. Here, de is the dimension of each patch feature in Venc . The decoder, g(·), maps Venc back to pixel space, producing Vdec ∈ R(n−m)×d , i.e., g(Venc ) = Vdec , which reconstructs masked patches x2 . Reconstruction quality is measured using Mean Squared Error (MSE) loss as follows: Lrec (x1 ) =
N X 1 ||g(f (x1,i )) − x2,i ||2 , N (n − m) i=1
(1)
architecture due to its simplicity and typical design, and investigate the link between AMV variation and decoder reconstruction loss, both theoretically and empirically. Given the structural similarity, we further extend this analysis to MaskDiT (Sec. 3.4), and our results confirm that the theory also holds, demonstrating strong transferability.
3.1
where N represents the sample number in X. The MAE structure consists of multiple self-attention layers, where attention captures semantic relationships between input patches. At the t-th layer, the input features are Zt ∈ Rnt ×dt , with nt patches and dt t t dimensional patch features. Weight matrices WQ , WK , t t t and WV generate the query Q , key K , and value Vt matrices, all in Rnt ×dt . The self-attention matrix At is computed as: t t T Q (K ) t √ A = softmax , dt quantifying similarities between Qt and Kt . The j-th output patch feature ej is a weighted sum of value vectors: ej =
n X i=1
atji vit ,
t t qji kji atji = Pn t , o=1 qjo
where atji indicates how much vit contributes to ej .
3
THEORETICAL ANALYSIS
In this section, we theoretically analyze how adversarial perturbations affect semantic relationships among patches. We take MAE as a representative AMRM
Adversarial Perturbation Induces Attention Matrix Variation in MAE
Given a clean sample x and its adversarial counterpart xadv , the attention matrices and input features at the t-th layer in MAE are At and Zt for x, and Atadv and Ztadv for xadv , respectively. To save the space, more defination can be founded in Appendix 2.3. The attention matrix variation (AMV) at layer t induced by adversarial perturbation is formally defined as Atadv − t t At , where WK , WQ ∈ Rnt ×dt are the weight matrices at the t-th layer, and N represents the number of training samples. AMV indicates a shift in MAE’s focal points on the image, reflecting a change in the inter-patch semantic information being captured, as Atadv misaligns attention toward irrelevant regions and distorts the overall interpretative context (see Figure 3). To quantify how such noise affects the attention matrix, we derive Theorem 3.1 to formally express the impact of perturbation δt on AMV, revealing the inherent AMV sensitivity of the MAE. Theorem 3.1. Let δt = Ztadv − Zt denotes the latent feature shift caused by the adversarial perturbation at layer t in MAE. With a set {ωi }ki=0 and kernel coefficient ωi ∈ N (0, Id ), it holds that:
t
t
||Aadv − A ||2 ≥ γ
t ⊤ t t ⊤ t Y − BQ WQ + Y − BK WK δt
, 2
Dou et al.
B=
k X
⊤ t t exp ωi (Q + K ) ,
i=0
exp
k X
⊤ t t exp ωi (Q + K ) ωi ,
i=0
γ=
Y=
∥Qt ∥2 +∥Kt ∥2 − 2
m
.
Proof. The proof can be seen in Appendix I.2. Theorem 3.1 suggests that even minor shifts in the latent features (δt ) may have the ability to cause disproportionately large changes in AMV, especially due to the high dimensionality dt of internal projection matrices. Notably, in MAE/AMRM, where dt ≫ input dimension, the sensitivity is further amplified. This analysis reveals the intrinsic vulnerability of MAE’s attention mechanism under adversarial conditions, offering a theoretical foundation for the empirical trends shown in Figure 4 (a-c) of Section 3.3. 3.2
Impact of Decoder Attention Shifts on Adversarial Reconstruction in MAE
To deepen the understanding of how attention pattern distortions affect output quality in MAE, we present a theoretical lower bound on the reconstruction loss under adversarial conditions. This analysis extends the discussion of AMV sensitivity in Section 3.1 and reveals how attention shifts in the decoder layer affect reconstruction loss. First, let Ladv rec denote the average reconstruction loss for adversarial examples, corresponding to the reconstruction loss Lrec for clean samples as Eq. (1) in Appendix 2.3. Theorem 3.2. Let Adec i,t denote the attention matrix at the t-th layer of the MAE decoder for the i-th sample in the dataset, and let Adec adv,i,t denote the corresponding attention matrix for the adversarial examples. With ratio constants CA and H, it holds that:
lead to increases in reconstruction loss Lrec , and Figure 4 provides further evidence of this phenomenon. Notably, the reconstruction loss is shown to be consistent with the degree of AMV, confirming a strong correlation between inter-patch semantic relationships and output degradation. This theorem builds a theoretical foundation of robust MAE-based purification. 3.3
Empirical Validation
This section empirically validates our theoretical analysis. We first assess the impact of MAE reconstruction on adversarial perturbations, then analyze the correlation between attention matrix variation and reconstruction loss. Finally, we show that adversarial perturbations alter semantic relationships within MAE patches and confirm the sensitivity of AMV to such perturbations. Perturbation Leads to Degraded Reconstruction Quality. To empirically investigate how perturbation influences reconstruction, an image is randomly selected from the SVHN dataset. We then compare the reconstruction results of the clean data and the adversarial example as shown in Figure 1. The adversarial example generated by the AutoAttack procedure (Croce and Hein, 2020b) in Figure 1d looks almost identical to its clean counterpart in Figure 1a visually. However, its reconstruction (Figure 1e) is significantly different from that of the clean sample (Figure 1c). Likewise, its reconstruction result also shows significant differences with a reconstruction of its clean sample (Figure 1c). These phenomena emphasize the substantial impact of adversarial perturbations on the overall outcome of the reconstruction.
Visualization of Attention Matrix Variation. To check how the semantic relationship between different patches changes under perturbation, we show the degree of importance of visible patches for reconstructing T X N X 2 1 1 dec Ladv Lrec + HCA Adec − crec the . masked patch in Figure 3. We select an image from rec ≥ adv,i,t − Ai,t 2 2N T t=1 i=1 ImageNet (Deng et al., 2009) and randomly generate a mask matrix. A specific masked patch is considered crec is the reconstruction bias, which symbolizes the as the target patch (marked red in a box) in Figure 3a. disparity between the output of MAE and the original, Then, the visible patches are fed into the MAE to perunmasked image. The definition of crec can be found form the reconstruction. The degree of importance in Appendix I.1. of visible patches for reconstructing the target patch, which is determined by the corresponding values of the Proof. The proof can be seen in the Appendix I.3. attention matrix, is illustrated in Figure 3b (e.g., the last layer attention of the MAE decoder). Meanwhile, It shows that the lower bound of the reconstruction we also display the corresponding visualizations of the loss for adversarial data can be decomposed into three adversarial example and the denoised example for the components: the average reconstruction loss for clean same mask matrix and target patch position. data Lrec , the average attention matrix variation for the PT PN MAE decoder at each layer 2N1 T t=1 i=1 ||(Adec − In generating attention weights for the target patch (red advi,t 2 Adec )|| , and constant terms. Theorem 3.2 demonbox), the approach involves using the patch itself as the i,t strates that adversarial distortions in AMV of decoder query vector and the remaining patches as key vectors.
AMRM-Pure: Semantic-Preserving Adversarial Purification
0.100
0.0005
0 100 Iteration step (d)
CIFAR-100 AMV 0.125 Loss 0.0010 0.100 0.0005 0 100 Iteration step (e)
0.200 0.175
ImageNet AMV 0.8 Loss
Loss
CIFAR-10 AMV Loss 0.0010
Loss AMV
0.125
Loss AMV
AMV
1.2 AMV Loss 1.0 0.0 0.01 0.05 Perturbation Range (c) 0.1
Loss
AMV 0.02 Loss 0.0 0.01 0.01 0.05 Perturbation Range (b)
0.2 AMV
0.1
ImageNet 0.03 Loss
AMV 0.002 Loss 0.0 0.001 0.01 0.05 Perturbation Range (a)
AMV
0.1
CIFAR-100 0.003 Loss
AMV
CIFAR-10
0.6 0.150 0 100 Iteration step (f)
Figure 4: Trends of MAE reconstruction loss and attention matrix variation under AutoAttack and during purification across multiple datasets. (a–c): Under AutoAttack on CIFAR-10, CIFAR-100, and ImageNet with different attack budget. (d–f): During the purification process with AMRM-PureMAE on CIFAR-10, CIFAR-100, and ImageNet. Through self-attention, the attention weights are determined. Higher weights indicate a stronger semantic similarity between the target patch and the patch itself. Figures 3b, 3c, and 3d show the importance degree of visible patches in the clean sample, adversarial example, and denoised example, respectively. Patches that are closer to red are more significant. As demonstrated in Figure 3b, when a hole in the tape is used as the target patch, the visible patch of another similar hole and the surrounding patches serve as the most important basis for reconstructing the target patch. However, as illustrated in Figure 3c, some distant and irrelevant visible patches with large color and shape differences are taken or focused to reconstruct the adversarial perturbed target patch. This indicates that the adversarial perturbation leads MAE to erroneous attention. More examples can be seen in Appendix G. Analysis of AMV Sensitivity and Consistency with Reconstruction Loss. To validate Theorem 3.2 and the effect of AMV on the reconstruction quality, we visualize the changes of reconstruction loss and AMV values with respect to the intensity of adversarial noise in Figure 4.PWe evaluate reconstruction loss and N dec average AMV N1 i=1 ||Adec adv,i − Ai ||2 on 150 images from CIFAR-10, CIFAR-100, and ImageNet with a 0.5 mask ratio, using AutoAttack to generate adversarial examples. As shown in Figure 4 (a–c), AMV is highly sensitive, rising sharply even under small perturbations (e.g., δ = 0.01), and then steadily increasing until reaching its softmax-bounded upper limit. Reconstruction loss and AMV follow the same trend, aligning with our theoretical analysis in Theorem 3.2 of their relationship and AMV’s sensitivity to perturbations. 3.4
A General AMRM Framework
The aforementioned analysis based on simple MAE reveals a key vulnerability in attentive masked image modeling: inter-patch semantic information (captured by AMV) is sensitive to noise. To show the universality of this observation, we further illustrate this sensitivity of AMV on a more powerful generative model, such as MaskDiT (Zheng et al., 2024), which is a powerful
diffusion-based AMRM that preserves its masked autoencoder architecture while integrating forward and reverse diffusion steps for high-quality image generation. Given their architectural alignment, the MaskDiT indeed agrees with the analysis presented in Theorem 3.1 in Sec 3.1. As shown in Figure 7 of Appendix E, empirical comparisons show that MaskDiT exhibits consistent patterns with MAE under adversarial settings. Specifically, the AMV of MaskDiT remains highly sensitive to noise, so that its reconstruction loss and AMV consistently maintain alignment. As such, MaskDiT continues to satisfy Theorem 3.1 and 3.2. Motivated by this phenomenon, we propose AMRM-PureMaskDiT , which leverages MaskDiT’s generative capacity while retaining MAE’s inter-patch semantic sensitivity, further improving purification performance by optimizing its reconstruction loss (Eq. (3)).
4
METHOD
4.1
Adversarial Purification with AMRM
As discussed in Section 3, the attention mechanism of AMRM is highly sensitive to adversarial perturbations, which distort inter-patch semantic relationships and degrade reconstruction quality. Based on this sensitivity, we propose a purification scheme, AMRM-Pure, which formulates denoising as an optimization problem that minimizes semantic variations. We denote the clean data as x, and the adversarial example as xadv . In the context of denoising, the objective is to induce a modification ∆ on xadv such that the attention matrix of the denoised examples, atten(xadv + ∆), closely aligns with the attention matrix of clean data samples atten(x). Therefore, the learning objective of denoising can be formed as an Attention Matrix Variation Minimization problem, which is denoted as: min L(∆) = ||atten(xadv + ∆) − atten(x)||2 , ∆
s.t. ||∆||∞ ≤ Ce , where Ce is a small constant.
(2)
Dou et al.
In addressing the AMV Minimization problem, the denoising process strives to mitigate adversarial perturbations on xadv . Its goal is to achieve a consistent attention matrix between the denoised images and the clean images of MAE. This alignment ensures that the patch relationship of the denoised image closely approximates that of clean samples, thereby minimizing the impact of adversarial perturbations in the denoising outcome. Since clean attention atten(x) is unavailable during inference, directly minimizing AMV Eq. (2) is intractable. Instead, inspired by Theorem 3.2, we minimize the AMRM reconstruction loss as a tractable surrogate. Reconstruction loss not only enables efficient optimization without requiring clean attention (tractability), but also can reduce AMV caused by perturbations (owing to the consistent trend presented in Theorem 3.2), thereby aligning with the inter-patch semantic structure of the clean input. As such, our approach instead minimizes the AMRM reconstruction loss for adversarial purification.
atten(x). The denoising algorithm and pipeline of our AMRM-Pure can be seen in Algorithm 1 (Supplement F) and the whole process is in Figure 2. For AMRMPure, when the mechanism is applied to the MAE framework, we denote it as AMRM-PureMAE ; when applied to the MaskDiT framework, we denote it as AMRM-PureMaskDiT . To empirically validate the theory of AMRM-Pure, we also plot the trends of the MAE reconstruction loss and AMV with increasing denoising iterations in Figure 4 (d-f) and Fig 7 (b) for MaskDiT. We randomly selected 100 adversarial examples from each dataset, perturbed using AutoAttack. For CIFAR10 and CIFAR100, the perturbation 8 magnitude is set to ℓ∞ = 255 , while for ImageNet, it 4 is set to ℓ∞ = 255 . As observed, both the AMV and loss exhibit a similar downward trend as the number of purification iterations increases. This supports the validity of our theory. Furthermore, we provide strict convergence analysis within the Appendix H.
min∆ Lrec (xadv + ∆) ⇐⇒ min∆ L(∆), s.t.
(3)
||∆||∞ ≤ Ce .
To address this problem, we employ the standard Projected Gradient Descent (PGD) method (Madry et al., 2018). In this approach, the modifications are iteratively added to adversarial examples, and the total number of iterations is denoted as S. At the s-th iteration, the denoising process is denoted as: xsadv = Clip(xs−1 adv − λ · ∆s , η),
(4)
∆s = sign(∇x Lrec (xs−1 adv )).
Table 1: Clean and robust accuracy (%) on CIFAR-10 obtained by different purification methods. WideResNet is commonly abbreviated as WRN. Method
Classifier
Std Acc
Robust Acc ℓ∞ ℓ2
Shi et al. (Shi et al., 2021) Yoon et al. (Yoon et al., 2021) Zhang et al. (Zhang et al., 2023) Diffpure (Nie et al., 2022) COUP (Zhang et al., 2024) ADBM (Li et al., 2025) ADDTw/Diffpure (Liu et al., 2025)
WRN-28-10 WRN-70-16 WRN-70-16 WRN-70-16 WRN-28-10 WRN-70-16 WRN-28-10
91.89 87.93 93.16 92.50 90.33 91.90 89.94
4.56 37.65 22.07 42.20 41.72 47.70 55.76
7.25 57.81 35.74 60.80 57.25 63.30 -
AMRM-PureMAE AMRM-PureMaskDiT RAMRM-PureMAE RAMRM-PureMaskDiT
WRN-28-10 WRN-28-10 WRN-28-10 WRN-28-10
88.57 92.03 90.09 93.11
40.53 50.57 45.15 62.13
53.50 64.53 60.72 73.57
Here Lrec signifies the AMRM reconstruction loss defined in Eq. (1), with λ representing the step size and η as the clipping threshold. The overall modification ∆ is composed of individual iteration modification ∆s for s ∈ [1, S]. The purpose of ∆ is to guide the attention distribution of adversarial examples atten(xadv ) towards the clean sample distribution
4.2
Robust Purification Model
As the study of AToP (Lin et al., 2024) shows, further fine-tuning a purification model using classification loss can enhance its robustness against both seen and unseen attacks. Following this insight, we propose a twostage fine-tuning method to develop Robust AMRMPureMAE (RAMRM-PureMAE ) and Robust AMRMPureMaskDiT (RAMRM-PureMaskDiT ) variants to enhance the semantic relationship-preserving capabilities. For more details about our method, refer to Appendix D.1. Figure 6a (see Appendix D.1) shows a quantitative analysis of enhanced semantic relationships using AMV as an evaluation metric. We randomly select 100 images from the CIFAR-10 dataset and examined the AMV of AMRM-PureMAE and RAMRM-PureMAE under the 8 AutoAttack with an attack budget of 255 across different purification iterations. Specifically, the initial AMV (without purification) of RAMRM-PureMAE is higher than that of AMRM-PureMAE . This suggests that Robust MAE is more sensitive to interpatch semantic information changes caused by adversarial attacks. However, with the progression of purification iterations, the AMV of RAMRM-PureMAE decreases significantly, highlighting its superior capability in preserving semantic integrity compared to AMRM-PureMAE . Furthermore, as shown in Figure 6b (Appendix D.1), both AMRM-PureMaskDiT and RAMRM-PureMaskDiT exhibit similar trends.
AMRM-Pure: Semantic-Preserving Adversarial Purification
Table 2: Clean and robust accuracy (%) on CIFAR100 obtained by different purification methods. The experiment are implemented on WideResNet-28-10. Method
Std Acc
Diffpure (Nie et al., 2022) COUP (Zhang et al., 2024) ADDTw/DDPM (Liu et al., 2025) AMRM-PureMAE AMRM-PureMaskDiT RAMRM-PureMAE RAMRM-PureMaskDiT
45.23 65.71 66.02 65.34 70.03 66.28 69.87
Robust Acc ℓ∞ ℓ2 11.57 15.22 18.85 14.28 24.39 19.53 29.91
31.53 34.28 36.57 29.29 36.51 31.58 43.27
Table 3: Clean and robust accuracy (%) on SVHN obtained by different purification methods. The experiment are implemented on WideResNet-28-10. Method
Std Acc
Diffpure (Nie et al., 2022) COUP (Zhang et al., 2024) ADBM (Li et al., 2025) AMRM-PureMAE AMRM-PureMaskDiT RAMRM-PureMAE RAMRM-PureMaskDiT
93.90 92.07 93.50 94.54 94.91 94.47 95.39
Robust Acc ℓ∞ ℓ2 39.70 41.62 47.90 27.59 46.57 39.15 55.90
63.30 63.97 65.70 55.29 66.38 60.51 70.18
scribed in (Li et al., 2025; Liu et al., 2025) to ensure a more robust evaluation of defense performance in our experiments. Our threat model is purifier followed by a classifier. To ensure fair comparison with adversarial training methods, we use AutoAttack with full gradient settings (Chen et al., 2024) as the evaluation protocol, ensuring objectivity and comparability. Evaluation Metrics. To evaluate the model’s performance, we employ two metrics for classification: robust accuracy (Robust Acc) and standard accuracy (Std Acc), which are tested respectively on adversarial examples and clean samples. Due to the high computational cost of testing models with multiple attacks, we follow previous work (Nie et al., 2022; Lin et al., 2024; Li et al., 2025) and randomly select 512 test samples from each testing dataset. All results are from 5 different random seeds and we use its average values. The standard deviations of the experiments will be provided separately in the Supplementary Materials (not in the Appendix). 5.2
5
EXPERIMENT
5.1
Experimental Setting
Datasets and Classifier. In this section, we validate the robustness of our purification method, AMRMPure, on four benchmark datasets, including CIFAR-10 (Krizhevsky et al., 2009), CIFAR-100 (Krizhevsky et al., 2009), SVHN (Netzer et al., 2011), and ImageNet (Deng et al., 2009). We use WideResNet-28-10 (Zagoruyko and Komodakis, 2016) as the main classifier for CIFAR10, CIFAR-100, and SVHN, and ResNet-101 (He et al., 2016) as the main classifier for ImageNet. Adversarial Attacks. Several studies (Chen et al., 2024; Li et al., 2025; Liu et al., 2025) show that the AutoAttack method (Croce and Hein, 2020b) tends to overestimate the robustness of diffusion models, primarily due to the presence of gradient obfuscation, which prevents the attack from effectively exploiting the true vulnerabilities of the model. To address this issue, recent studies (Chen et al., 2024; Li et al., 2025; Liu et al., 2025) have adopted the gradient checkpointing technique to efficiently extract complete gradients throughout the diffusion process. Furthermore, Li et al. (Li et al., 2025) have further demonstrated that, compared to AutoAttack, the combination of PGD + EOT is more effective in evaluating the adaptive defense mechanisms of diffusion models. In line with these studies (Li et al., 2025), we employ the PGD200 + 8 EOT20 configuration with ℓ∞ (ϵ = 255 ) and ℓ2 (ϵ = 1), utilizing the exact gradient computation method de-
Compare with the State-of-the-art
We compare our results with state-of-the-art methods across four datasets: CIFAR-10, CIFAR-100, SVHN, and ImageNet. Due to the space limitations, we provide detailed comparisons under the PGD200 + EoT20 attack for CIFAR-10, CIFAR100, and SVHN in main paper. More experimental results, including performance on ImageNet, transferability of the fine-tuned purification model, defense against extra attacks, ablation studies, sensitivity analysis, and evaluations between different classifiers, are also presented in Appendix C. CIFAR-10. Table 1 highlights the performance of various purification methods on the CIFAR-10 dataset in terms of Std Acc and Robust Acc. RAMRMPureMaskDiT excels with 62. 13% robust accuracy in ℓ∞ attacks, 73. 57% in ℓ2 attacks, and strong standard accuracy of 93. 11%. In contrast, traditional methods like the previous method (Shi et al., 2021) perform poorly, achieving only 4.56% under ℓ∞ attacks. In general, our method greatly improves adversarial robustness, with additional WideResNet-70-16 results provided in Supplementary C.5. CIFAR100. Table 2 summarizes the performance of various purification methods on CIFAR-100. Our proposed method achieves strong performance, with AMRM-PureMaskDiT reaching the highest standard accuracy of 70.03% and a robust accuracy of 24.39% under ℓ∞ attacks and 36.51% under ℓ2 attacks. RAMRMPureMaskDiT further improves robustness, achieving the best ℓ∞ robust accuracy of 29.91% and ℓ2 robust ac-
Dou et al. 8 Table 4: Comparison with adversarial training under Autoattack (ϵ = 255 ) SVHN Std Acc ℓ∞
Architecture
Rebuffi et al.(Rebuffi et al., 2021) Pang et al. (Pang et al., 2022) Wang et al.(Wang et al., 2023)
✓ ✓ ✓
WRN-28-10 WRN-28-10 WRN-28-10
87.33 88.10 91.12
60.73 61.51 63.35
62.41 62.08 68.06
32.06 31.40 35.65
94.34 – 95.19
60.90 – 61.85
AMRM-PureMAE AMRM-PureMaskDiT RAMRM-PureMAE RAMRM-PureMaskDiT
✗ ✗ ✗ ✗
WRN-28-10 WRN-28-10 WRN-28-10 WRN-28-10
88.57 92.03 90.09 93.11
40.65 64.97 47.15 75.83
65.34 70.03 66.28 69.87
16.77 33.51 22.15 40.13
94.54 94.91 94.47 95.39
47.03 64.15 50.03 66.12
SVHN. Table 3 shows that ADBM achieves strong robust accuracy (47.90% under ℓ∞ attacks and 65.70% under ℓ2 ) attacks but is outperformed by RAMRMPureMaskDiT , which achieves the best robust accuracy (55.90% and 70.18%) with comparable standard accuracy. While RAMRM-PureMAE leads in standard accuracy (95.39%), its robustness is lower. Overall, our MaskDiT-based methods better balance clean and robust performance than ADBM. Comparison with adversarial training
As shown in Table 4, our methods achieve competitive or superior robustness compared to adversarial training baselines, without using any extra data. In contrast, prior works (Rebuffi et al., 2021; Pang et al., 2022; Wang et al., 2023) rely on 1M additional training samples. Notably, RAMRM-PureMaskDiT achieves 75.83% robust accuracy on CIFAR10, outperforming all baselines and highlighting the effectiveness of our data-free approach. 5.4
CIFAR100 Std Acc ℓ∞
Extra data
curacy of 43.27%, outperforming other methods like Diffpure and COUP.
5.3
CIFAR10 Std Acc ℓ∞
Method
Inference Time Comparison
Table 5: Inference time (s) consumption comparison across different defense models on CIFAR-10 and ImageNet datasets. Defense Model
CIFAR10
Diffpure (Nie et al., 2022)
12.39
ImageNet 81.54
AMRM-PureMAE AMRM-PureMaskDiT RAMRM-PureMAE RAMRM-PureMaskDiT
18.25 32.85 11.77 29.73
31.51 79.27 27.38 62.52
Table 5 compares the inference time between different defense models in CIFAR-10 and ImageNet. We calculate the run-time for all methods with a batch size of 32, and our experiments are conducted on an A40 GPU. For CIFAR-10, RAMRM-PureMAE achieves the fastest time (11.77s), followed by Diffpure (12.39s) and AMRM-PureMAE (18.25s). On ImageNet, AMRM-PureMAE is the most efficient
(31.51s), significantly outperforming Diffpure (81.54s). The results highlight that the DiffPure model has the advantage of inference time for smaller datasets, while our proposed model performs better on the metrics of inference time on the ImageNet dataset.
6
CONCLUSION
This paper reveals the vulnerability of AMRM to subtle adversarial attacks, caused by adversarial noises that disrupt semantic relations in image patches. To address this, we propose the AMRM-Pure pipeline, a denoising method using Attention Matrix Variation Minimization, which iteratively refines adversarial examples by minimizing reconstruction loss to converge to clean images. We further enhance AMRM-Pure with classifier loss, introducing RAMRM-PureMAE . We apply two AMRM (MAE and MaskDiT) to our method. Extensive experiments demonstrate that our methods achieve state-of-the-art performance on multiple benchmarks.
7
ACKNOWLEDGEMENT
The work was partially supported by the following: the Zhejiang Provincial Natural Science Foundation – Exploration Project under No. LMS26F020007, the Wenzhou Applied Fundamental Research Program (Basic Research) under No. GG20250198, the WKU 2026 International Frontier Interdisciplinary Research Institute Talent Program under No. WKUTP2026002, the WKU 2025 International Collaborative Research Program under No. ICRPSP2025001.
AMRM-Pure: Semantic-Preserving Adversarial Purification
References M. Andriushchenko, F. Croce, N. Flammarion, and M. Hein. Square attack: a query-efficient blackbox adversarial attack via random search. In European conference on computer vision, pages 484–501. Springer, 2020. A. Athalye, L. Engstrom, A. Ilyas, and K. Kwok. Synthesizing robust adversarial examples. In International conference on machine learning, pages 284–293, 2018. M. Bai, W. Huang, T. Li, A. Wang, J. Gao, C. F. Caiafa, and Q. Zhao. Diffusion models demand contrastive guidance for adversarial purification to advance. Internal Conference on Machine Learning, 2024. S. Cao, P. Xu, and D. A. Clifton. How to understand masked autoencoders. arXiv preprint arXiv:2202.03670, 2022. N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. in 2017 ieee symposium on security and privacy (sp). in 2017 IEEE Symposium on Security and Privacy (SP), pp., page 39–57, 2017. H. Chen, Y. Dong, Z. Wang, X. Yang, C. Duan, H. Su, and J. Zhu. Robust classification via a single diffusion model. ICML, 2024. J. Chen and Q. Gu. Rays: A ray searching method for hard-label adversarial attack. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1739– 1747, 2020. K. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Davis, A. Mohiuddin, L. Kaiser, et al. Rethinking attention with performers. International Conference on Learning Representations, 2021. F. Croce and M. Hein. Minimally distorted adversarial examples with a fast adaptive boundary attack. In International Conference on Machine Learning, pages 2196–2205. PMLR, 2020a. F. Croce and M. Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameterfree attacks. In International conference on machine learning, pages 2206–2216. PMLR, 2020b. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. O. Dou, Z. Gao, H. Shen, Z. Yuan, S. Zhang, and K. Huang. Improving robust generalization with
diverging spanned latent space. Transactions on Machine Learning Research, 2024. V. Fischer, M. C. Kumar, J. H. Metzen, and T. Brox. Adversarial examples for semantic image segmentation. International Conference on Computer Vision, 2017. I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. International Conference on Learning Representations, 2015. S. Gowal, S.-A. Rebuffi, O. Wiles, F. Stimberg, D. A. Calian, and T. A. Mann. Improving robustness using generated data. Advances in Neural Information Processing Systems, 34:4218–4233, 2021. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000–16009, 2022. A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009. A. Kurakin, I. Goodfellow, and S. Bengio. Adversarial machine learning at scale. n International Conference on Learning Representations, 2017. Y. Lee, J. R. Willette, J. Kim, and S. J. Hwang. Visualizing the loss landscape of self-supervised vision transformer. arXiv preprint arXiv:2405.18042, 2024. X. Li, W. Sun, H. Chen, Q. Li, Y. Liu, Y. He, J. Shi, and X. Hu. Adbm: Adversarial diffusion bridge model for reliable adversarial purification. ICLR, 2025. G. Lin, C. Li, J. Zhang, T. Tanaka, and Q. Zhao. Adversarial training on purification (atop): Advancing both robustness and generalization. International Conference on Learning Representations, 2024. G. Liu, I. Khalil, and A. Khreishah. Using single-step adversarial training to defend iterative adversarial examples. In Proceedings of the Eleventh ACM Conference on Data and Application Security and Privacy, pages 17–27, 2021. Y. Liu, K. Liu, Y. Xiao, Z. Dong, X. Xu, P. Wei, and L. Lin. Towards understanding the robustness of diffusion-based purification: A stochastic perspective. In The Thirteenth International Conference on Learning Representations, 2025. C. Lyu, K. Huang, and H.-N. Liang. A unified gradient regularization family for adversarial examples. 2015 IEEE international conference on data mining, pages 301–309, 2015.
Dou et al.
A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018. H. Mei, M. Dong, and C. Xu. Efficient image-to-image diffusion classifier for adversarial robustness. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 6081–6089, 2025. S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard. Deepfool: a simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2574–2582, 2016. Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, A. Y. Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, page 7. Granada, Spain, 2011. T. Nguyen, M. Pham, T. Nguyen, K. Nguyen, S. Osher, and N. Ho. Fourierformer: Transformer meets generalized fourier integral theorem. Advances in Neural Information Processing Systems, 35:29319– 29335, 2022.
D. Song, K. Eykholt, I. Evtimov, E. Fernandes, B. Li, A. Rahmati, F. Tramèr, A. Prakash, and T. Kohno. Physical adversarial examples for object detectors. 12th USENIX Workshop on Offensive Technologies (WOOT 18), Aug. 2018. Y. Song, T. Kim, S. Nowozin, S. Ermon, and N. Kushman. Pixeldefend: Leveraging generative models to understand and defend against adversarial examples. International Conference on Learning Representations, 2017. Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations, 2021. F. Tramèr, A. Kurakin, N. Papernot, I. Goodfellow, D. Boneh, and P. McDaniel. Ensemble adversarial training: Attacks and defenses. International Conference on Learning Representations, 2018. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
W. Nie, B. Guo, Y. Huang, C. Xiao, A. Vahdat, and A. Anandkumar. Diffusion models for adversarial purification. International Conference on Machine Learning, 2022.
B. Vivek and R. V. Babu. Single-step adversarial training with dropout scheduling. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 947–956. IEEE, 2020.
T. Pang, M. Lin, X. Yang, J. Zhu, and S. Yan. Robustness and accuracy could be reconcilable by (proper) definition. In International Conference on Machine Learning, pages 17258–17277. PMLR, 2022.
Z. Wang, T. Pang, C. Du, M. Lin, W. Liu, and S. Yan. Better diffusion models further improve adversarial training. International Conference on Machine Learning, 2023.
D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536–2544, 2016.
E. Wong, L. Rice, and J. Z. Kolter. Fast is better than free: Revisiting adversarial training. International Conference on Learning Representations, 2020.
S.-A. Rebuffi, S. Gowal, D. A. Calian, F. Stimberg, O. Wiles, and T. Mann. Fixing data augmentation to improve adversarial robustness. arXiv preprint arXiv:2103.01946, 2021. P. Samangouei, M. Kabkab, and R. Chellappa. Defensegan: Protecting classifiers against adversarial attacks using generative models. International Conference on Learning Representations, 2018. V. Sehwag, S. Mahloujifar, T. Handina, S. Dai, C. Xiang, M. Chiang, and P. Mittal. Robust learning meets generative models: Can proxy distributions improve adversarial robustness? International Conference on Learning Representations, 2021. C. Shi, C. Holtz, and G. Mishne. Online adversarial purification based on self-supervision. International Conference on Learning Representations, 2021.
Q. Wu, H. Ye, Y. Gu, H. Zhang, L. Wang, and D. He. Denoising masked autoencoders help robust classification. ICLR, 2022. J. Yoon, S. J. Hwang, and J. Lee. Adversarial purification with score-based generative models. In International Conference on Machine Learning, pages 12062–12072. PMLR, 2021. Z. You, D. Liu, B. Han, and C. Xu. Beyond pretrained features: noisy image modeling provides adversarial defense. Advances in Neural Information Processing Systems, 36, 2023. S. Zagoruyko and N. Komodakis. Wide residual networks. In British Machine Vision Conference 2016, 2016. B. Zhang, W. Luo, and Z. Zhang. Enhancing adversarial robustness via score-based optimization. Advances in Neural Information Processing Systems, 36:51810– 51829, 2023.
AMRM-Pure: Semantic-Preserving Adversarial Purification
H. Zhang, Y. Yu, J. Jiao, E. Xing, L. El Ghaoui, and M. Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pages 7472–7482, 2019. M. Zhang, J. Li, W. Chen, J. Guo, and X. Cheng. Classifier guidance enhances diffusion-based adversarial purification by preserving predictive information. In ECAI 2024, pages 2234–2241. IOS Press, 2024. Q. Zhang, Y. Wang, and Y. Wang. How mask matters: Towards theoretical understandings of masked autoencoders. Advances in Neural Information Processing Systems, 35:27127–27139, 2022. H. Zheng, W. Nie, A. Vahdat, and A. Anandkumar. Fast training of diffusion models with masked transformers. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. D. Zhou, Y. Chen, N. Wang, D. Liu, X. Gao, and T. Liu. Eliminating adversarial noise via information discard and robust representation restoration. In International Conference on Machine Learning, pages 42517–42530. PMLR, 2023.
Checklist 1. For all models and algorithms presented, check if you include: (a) A clear description of the mathematical setting, assumptions, algorithm, and/or model. [Yes] (b) An analysis of the properties and complexity (time, space, sample size) of any algorithm. [Yes] (c) (Optional) Anonymized source code, with specification of all dependencies, including external libraries. [Yes] 2. For any theoretical claim, check if you include: (a) Statements of the full set of assumptions of all theoretical results. [Yes] (b) Complete proofs of all theoretical results. [Yes] (c) Clear explanations of any assumptions. [Yes] 3. For all figures and tables that present empirical results, check if you include: (a) The code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL). [Yes] (b) All the training details (e.g., data splits, hyperparameters, how they were chosen). [Yes]
(c) A clear definition of the specific measure or statistics and error bars (e.g., with respect to the random seed after running experiments multiple times). [Yes] (d) A description of the computing infrastructure used. (e.g., type of GPUs, internal cluster, or cloud provider). [Yes] 4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets, check if you include: (a) Citations of the creator If your work uses existing assets. [Yes] (b) The license information of the assets, if applicable. [Yes] (c) New assets either in the supplemental material or as a URL, if applicable. [Yes/No/Not Applicable] (d) Information about consent from data providers/curators. [Yes] (e) Discussion of sensible content if applicable, e.g., personally identifiable information or offensive content. [Yes] 5. If you used crowdsourcing or conducted research with human subjects, check if you include: (a) The full text of instructions given to participants and screenshots. [Not Applicable] (b) Descriptions of potential participant risks, with links to Institutional Review Board (IRB) approvals if applicable. [Not Applicable] (c) The estimated hourly wage paid to participants and the total amount spent on participant compensation. [Not Applicable]
AMRM-Pure: Semantic-Preserving Adversarial Purification: Supplementary Materials
A
Limitation
One limitation of AMRM-PureMAE lies in its linear memory consumption with respect to batch size, which may restrict scalability under limited GPU resources.
B
Broader impact
We are the first to systematically explore the relationship between adversarial noise and inter-patch semantic information. While existing defense methods primarily focus on suppressing pixel-level perturbations or enhancing model robustness structurally, our work takes a novel perspective by analyzing reconstruction consistency and semantic alignment. We reveal how adversarial perturbations disrupt inter-patch semantic relations and propose a reconstruction paradigm that restores this consistency. This new angle provides a valuable direction for future adversarial defense research and advances the theoretical and practical understanding of robustness from a structure-aware perspective.
C
Supplement Experiment
We have enhanced this section with additional experiments to provide a more comprehensive evaluation of our work. Specifically, we present ImageNet (Deng et al., 2009) results under PGD200 + EoT20 (Madry et al., 4 for ℓ∞ attack and ϵ = 0.5 for ℓ2 attack, using 2018; Athalye et al., 2018) with the perturbation budgets ϵ = 255 ResNet-101 (He et al., 2016) as the classifier. We also performed ablation studies on CIFAR-10 (Krizhevsky et al., 2009), CIFAR-100 (Krizhevsky et al., 2009), and SVHN (Netzer et al., 2011) using different backbones and evaluated diverse attack scenarios. C.1
Performance on ImageNet.
Table 6 presents the standard accuracy and robust accuracy of different purification methods on the ImageNet dataset under the ℓ∞ attack. As shown in the table, the ADDT method achieves the highest standard accuracy at 80.20%, slightly outperforming the other methods. However, in terms of robustness, RAMRM-PureMaskDiT stands out with a robust accuracy of 36.87%, surpassing all other methods, including ADDT. In contrast, AMRM-PureMAE and AMRM-PureMaskDiT demonstrate relatively lower robustness, achieving 24.75% and 32.29%, respectively. C.2
Transferability of finetuned purification on new classifiers
We fine-tune the RAMRM-PureMAE /RAMRM-PureMaskDiT model based on WideResNet-28-10 (Zagoruyko and Komodakis, 2016) and replace it with different classifiers for testing experiments. The WideResNet-7016 (Zagoruyko and Komodakis, 2016) and ResNet-50 (He et al., 2016) are selected for testing process to observe the transferability of our proposed method across different classifiers. The results in Table 7 highlight the superior performance of our method, particularly RAMRM-PureMaskDiT , which achieves the highest robust accuracy across both classifiers and attack norms. Notably, the fine-tuned models, initially trained on WideResNet-28-10, demonstrate strong transferability when applied to WideResNet-70-16 and ResNet-50 without the need for retraining. This finding underscores the practicality and scalability of our method, as its fine-tuned models can be seamlessly adapted to new classifiers, providing an efficient and robust defense against adversarial attacks.
AMRM-Pure: Semantic-Preserving Adversarial Purification
Table 6: Clean and robust accuracy (%) with ℓ∞ and ℓ2 attack on ImageNet obtained by different purification methods. Defense model
Std Acc
Robust Acc ℓ∞ ℓ2
Diffpure (Nie et al., 2022) ADDT (Liu et al., 2021)
77.51 80.20
30.15 35.83
44.15 -
AMRM-PureMAE AMRM-PureMaskDiT RAMRM-PureMAE RAMRM-PureMaskDiT
67.53 75.52 78.85 79.52
24.75 32.29 29.48 36.87
35.95 45.57 42.25 51.17
8 Table 7: Robust accuracy (%) of different purification methods against ℓ∞ (ϵ = 255 ) and ℓ2 (ϵ = 1) adversarial attacks across two classifiers: WideResNet-70-16 and ResNet-50. Here, our method are derived by fine-tuning on the WideResNet-28-10 classifier.
WideResNet-70-16 ℓ∞ ℓ2
ResNet-50 ℓ∞ ℓ2
Diffpure (Nie et al., 2022)
42.20
60.80
38.02
54.74
RAMRM-PureMAE RAMRM-PureMaskDiT
44.27 58.37
62.42 68.55
43.72 54.22
60.08 67.15
Classifier
C.3
Performance on unseen threats Table 8: Robust accuracy (%) against unseen threats with the setting of ℓ1 (ϵ = 12) and ℓ2 (ϵ = 1). Defense model
CIFAR-10 ℓ1 ℓ2
CIFAR-100 ℓ1 ℓ2
SVHN ℓ1
ℓ2
Diffpure (Nie et al., 2022) ADBM (Li et al., 2025)
44.30 49.60
60.80 63.30
13.51 -
27.53 -
46.10 51.20
63.30 65.70
RAMRM-PureMAE RAMRM-PureMaskDiT
44.41 65.11
60.72 73.57
12.97 41.15
29.58 43.27
47.09 55.53
60.51 70.18
For the three methods, ADBM, RAMRM-PureMAE , and RAMRM-PureMaskDiT , all of which are fine-tuned under the ℓ∞ norm, the ℓ1 and ℓ2 norms are considered as unseen threats. To verify the robustness of the proposed method, we will now conduct testing under these unseen threats. For a fair comparison on the CIFAR-10 dataset, we employ the WideResNet-70-16 architecture, and we use the WideResNet-28-10 architecture on the CIFAR-100 and SVHN datasets. Table 8 presents the robust accuracy (%) of different defense models against unseen threats (ℓ1 and ℓ2 attacks) on the CIFAR-10, CIFAR-100, and SVHN datasets. As a baseline method, DiffPure (Nie et al., 2022) performs moderately on CIFAR-10 and SVHN but poorly on CIFAR-100, especially under ℓ1 attacks (13.51%). ADBM outperforms DiffPure on CIFAR-10 and SVHN, but no data is provided for CIFAR-100, suggesting potential limitations or untested performance on this dataset. Our RAMRM-PureMAE method slightly outperforms DiffPure on CIFAR-10 and SVHN but underperforms on CIFAR-100 (12.97% vs. 13.51%), indicating some limitations on more complex datasets. In contrast, RAMRM-PureMaskDiT significantly outperforms all other methods across all datasets and attack types. On CIFAR-10, RAMRM-PureMaskDiT achieves accuracies of 65.11% and 73.57% under ℓ1 and ℓ2 attacks, respectively, far surpassing other methods. On CIFAR-100, although its performance under ℓ2 attacks is slightly lower than under ℓ1 , it still outperforms other methods. On the SVHN dataset, RAMRM-PureMaskDiT also demonstrates considerable robustness performance, particularly under
Dou et al.
ℓ2 attacks (70.18%). Overall, RAMRM-PureMaskDiT exhibits the strongest robustness against unseen threats, especially on CIFAR-10 and CIFAR-100, showcasing its superior generalization and defense capabilities, while RAMRM-PureMAE , though slightly less effective, still outperforms baseline methods in certain scenarios. C.4
Defense against adaptive attacks
Table 9 presents the robust accuracy (%) of various defense methods under different adversarial attacks in the adaptive ℓ2 (ϵ = 1)-norm setting on the CIFAR-10 dataset. The evaluated adaptive attacks include C&W (Carlini and Wagner, 2017)+EOT (Athalye et al., 2018), DeepFool (Moosavi-Dezfooli et al., 2016)+EOT, AutoAttack (Croce and Hein, 2020b)+EOT, and PGD (Madry et al., 2018)+EOT. Among the methods, Diffpure and ADBM represent baseline defense approaches, with ADBM generally outperforming Diffpure across all attacks. For instance, ADBM achieves 78.40% robust accuracy against C&W+EOT compared to Diffpure’s 74.80%. The pure methods (non-adversarial training approaches) show varying performance: AMRM-PureMAE exhibits the lowest robust accuracy across all attacks, while AMRM-PureMaskDiT demonstrates stronger performance, particularly against DeepFool+EOT (82.8%). RAMRM-PureMAE shows moderate results, and RAMRM-PureMaskDiT consistently outperforms all other methods, achieving the highest robust accuracy against every attack, with 80.58% for C&W+EOT, 86.11% for DeepFool+EOT, 73.59% for AutoAttack+EOT, and 69.57% for PGD+EOT. This indicates that RAMRM-PureMaskDiT is the most effective defense method in this setting, offering superior robustness across diverse adversarial attacks. Table 9: Robust Accuracy (%) of various defense methods under different attacks in the ℓ2 (ϵ = 1)-norm setting using the exact gradient with WideResNet-70-16 on CIFAR-10.
C.5
Method
C&W+EOT
DeepFool+EOT
AutoAttack+EOT
PGD+EOT
Diffpure(Nie et al., 2022) ADBM(Li et al., 2025)
74.80 78.40
78.40 84.30
63.90 66.80
60.80 66.30
AMRM-PureMAE AMRM-PureMaskDiT RAMRM-PureMAE RAMRM-PureMaskDiT
62.54 79.15 72.20 80.58
65.15 82.80 72.29 86.11
52.75 66.43 59.11 73.59
53.50 64.53 60.72 69.57
Extra experiments on different classifier
Table 10 shows our standard and robust accuracy using WideResNet-70-16 under CIFAR-10 and SVHN. Compared with WideResNet-28-10, it shows better results. It means the overparameterization contributes model’s robustness. Among all the methods, the effectiveness of the RAMRM-PureMaskDiT method is most notable. Table 10: Performance of standard accuracy and robust accuracy (%) using WideResNet-70-16.
C.6
Method
Architecture
AMRM-PureMAE AMRM-PureMaskDiT RAMRM-PureMAE RAMRM-PureMaskDiT
WideResNet-70-16 WideResNet-70-16 WideResNet-70-16 WideResNet-70-16
Std Acc
CIFAR10 ℓ∞
ℓ2
Std Acc
SVHN ℓ∞
ℓ2
89.66 94.91 91.07 93.85
42.21 52.07 47.92 63.94
56.77 66.55 60.95 75.50
94.97 94.93 94.78 95.91
17.11 46.03 40.79 56.02
32.75 63.77 60.51 69.18
Performance on Black-box Attack
To evaluate the effectiveness of against black-box attacks, we adopt three black-box attack methods: FAB (Croce and Hein, 2020a), Square (Andriushchenko et al., 2020), and Rays (Chen and Gu, 2020) on CIFAR-10 and SVHN. The black-box scenario implies that the attacker has no knowledge of the defense method. Table 11 shows
AMRM-Pure: Semantic-Preserving Adversarial Purification 8 the robustness of various methods against black-box ℓ∞ attacks with the perturbation budget ϵ = 255 using WideResNet-28-10. RMaskDiT still achieves the best results. 8 Table 11: Robust accuracy (% ) against different black-box attacks ℓ∞ (ϵ = 255 ) with WideResNet-28-10. The “Vanilla" setting represents the model trained on clean datasets without any defense.
CIFAR-10 SVHN Robust Acc Robust Acc Std Acc Std Acc Square FAB RayS Square FAB RayS Vanilla WideResNet-28-10 96.75 19.15 0.00 1.23 98.11 9.08 14.78 16.89 Diffpure (Nie et al., 2022) WideResNet-28-10 89.15 89.15 88.29 90.51 93.93 92.15 93.13 92.97 ADBM (Li et al., 2025) WideResNet-28-10 93.49 93.32 92.98 93.16 AMRM-PureMAE WideResNet-28-10 88.57 78.59 76.43 77.29 94.54 92.57 93.36 93.41 AMRM-PureMaskDiT WideResNet-28-10 92.03 90.96 92.25 93.39 94.91 92.80 92.59 92.73 RAMRM-PureMAE WideResNet-28-10 90.09 90.25 89.15 92.31 94.47 92.73 93.27 93.36 RAMRM-PureMaskDiT WideResNet-28-10 93.11 93.27 93.38 93.03 95.39 94.15 94.18 94.88 Method
C.7
Architecture
Robust under BPDA attack
We evaluate the robustness of our model under a strong white-box attack setting using BPDA combined with EoT set to 20. The results show as follow: 8 Table 12: Clean and robust accuracy (%) under BPDA attack (ϵ = 255 ) on CIFAR-10.
Method Diffpure (Nie et al., 2022) AMRM-PureMAE AMRM-PureMaskDiT RAMRM-PureMAE RAMRM-PureMaskDiT
Architecture WRN-28-10 WRN-28-10 WRN-28-10 WRN-28-10 WRN-28-10
Std Acc 89.20 88.57 92.03 90.09 93.11
Robust Acc (ℓ∞ ) 78.53 78.89 83.44 80.17 85.41
8 Table 12 presents the comparison of clean and robust accuracy under BPDA attack (ϵ = 255 ) on the CIFAR-10 dataset. While the conventional Diffpure method achieves decent robustness, our proposed methods demonstrate significant improvements in both clean and robust accuracy. In particular, RAMRM-PureMaskDiT achieves the highest clean accuracy (93.11%) and robust accuracy (85.41%), highlighting its superior purification capability and enhanced resistance to adversarial attacks. These results validate the effectiveness of our approach in improving semantic reconstruction and adversarial robustness.
C.8
A specific attacks
We consider a white-box adversarial attack that jointly optimizes reconstruction fidelity and classification error by minimizing a weighted combination of reconstruction loss and negative classification loss. Specifically, given an input x, the attacker generates an adversarial example xadv by solving the following optimization problem: xadv = arg min (1 − α) Lrec (x′ ) − α Lcls (x′ ) ′ x
(5)
where Lrec denotes the reconstruction loss and Lcls is the classification loss that encourages misclassification. The trade-off parameter α ∈ [0, 1] controls the balance between preserving reconstruction quality and inducing misclassification. The attack is implemented using a strong PGD-based procedure with sufficient iterations and step size tuning to avoid gradient obfuscation. This formulation generalizes standard adversarial attacks by incorporating a reconstruction constraint, aiming to produce adversarial examples that remain visually consistent while still fooling the classifier. Results and Analysis. We evaluate the effectiveness of the proposed joint-loss attack under different settings of the trade-off coefficient α and step size β. Effect of α. We vary α ∈ {0.1, 0.3, 0.5, 0.7, 0.9} to study the trade-off between reconstruction fidelity and adversarial strength. The results are shown in Table 13.
Dou et al.
Table 13: Effect of the trade-off parameter α on the joint-loss attack. α
0.1
0.3
0.5
0.7
0.9
AMRM-PureMaskDiT RAMRM-PureMaskDiT
89.77 90.17
88.29 89.25
87.54 89.57
87.10 89.11
87.24 88.27
As α increases, the adversarial objective becomes stronger, but the reconstruction quality degrades. Specifically, when α = 0.1, the reconstruction loss is low (Ladv rec = 0.04), but the negative classification loss remains small (−Lcls = −0.19), indicating a weak attack. In contrast, when α = 0.9, the attack becomes much stronger (−Lcls = 4.4), but the reconstruction loss increases significantly (Ladv rec = 0.19), leading to severely degraded reconstructions. These results reveal a clear trade-off between reconstruction fidelity and adversarial effectiveness. Effect of β. We further vary the step size β ∈ {0.01, 0.05, 0.1, 0.5, 1}, and report the results in Table 14. Table 14: Effect of the step size β on the joint-loss attack. β
0.01
0.05
0.1
0.5
1
AMRM-PureMaskDiT RAMRM-PureMaskDiT
84.49 85.59
83.22 87.92
87.54 89.57
88.59 88.62
89.27 89.54
In table 14, we observe that the attack becomes more effective when β is around 0.05–0.1. However, even under the best step size, the proposed attack still does not outperform the standard full-gradient white-box attack used in our paper. From the above results, we conclude that the joint-loss attack struggles to achieve a satisfactory balance between reconstruction fidelity and adversarial effectiveness. In particular, no choice of α can simultaneously ensure low reconstruction loss and strong adversarial impact. This suggests that the reconstruction objective and classification objective are inherently conflicting under this formulation, making the attack weaker than a standard full-gradient white-box attack. C.9 C.9.1
Ablation study Impact of time step number
To investigate the impact of time steps on the denoising process in MaskDiT, we conduct experiments by observing the robust accuracy at different time steps, aiming to understand how varying time steps influence the model’s ability to effectively remove noise and improve overall performance. Table 15: Impact of time step on CIFAR-10, and all configurations align with Table 1. Time steps
15
20
25
AMRM-PureMaskDiT RAMRM-PureMaskDiT
49.27 49.22
50.41 51.34
50.57 62.13
30 51.69 58.22
As shown in Table 15, our method exhibits a stable increase in robust accuracy as time steps increase, peaking at 51.69 when the metric of time steps is set as 30. In contrast, RAMRM-PureMaskDiT achieves its highest accuracy of 62.13 when the metric of time steps is set as 25, but experiences a slight drop at the 30th step, indicating its sensitivity to the optimal time step selection. C.9.2
Ablation on the Effectiveness of MaskDiT framework
To explore the origins of the enhanced performance observed with AMRM-PureMaskDiT and RAMRM-PureMaskDiT , we conducted a series of controlled experiments in Table 16. We begin with training two models, e.g., MaskDiT and RMaskDiT, used for purification. Firstly, following the DiffPure (Nie et al., 2022), we conduct the reconstructionbased purification processes for purification, which are MaskDiT/purification and RMaskDiT/purification . This
AMRM-Pure: Semantic-Preserving Adversarial Purification
reconstruction based purification involves a forward (noise addition) and backward (denoising) pass. The reconstructed images are treated as purified results to investigate whether the superior performance stems from generative ability of selected models. On the other hand, we then compared the above performances with the proposed AMV-based purification methods by using the same models, where the purification processes are named as AMRM-PureMaskDiT and RAMRM-PureMaskDiT . By keeping the model architecture consistent across all variants, we enabled a direct comparison of the effectiveness of different denoising strategies. It is note that MaskDiT and RMaskDiT indeed possess a certain capability to withstand stronger adversarial attacks; however, their effectiveness is still much lower than that of our proposed AMRM-PureMaskDiT and RAMRM-PureMaskDiT . Table 16: Ablation analysis on different denoising components across various datasets. Method
Architecture
CIFAR10 Std acc Robust acc
CIFAR100 Std acc Robust acc
Std acc
SVHN Robust acc
MaskDiT/purification RMaskDiT/purification AMRM-PureMaskDiT RAMRM-PureMaskDiT
WRN-28-10 WRN-28-10 WRN-28-10 WRN-28-10
91.13 90.57 92.03 93.11
63.29 64.15 70.03 69.87
92.28 93.03 94.91 95.39
40.07 45.19 46.57 55.90
C.10
42.99 47.57 50.57 62.13
13.57 18.55 24.39 29.91
Sensitivity Analysis
In this subsection, taking the CIFAR10 under same setting with Table 1, we analyze the impact of step size, mask ratio and step size. The result is described in Fig. 5. It confirms our theoretical analysis.