Statistical Adversaries: Natural Backdoor-like Features in Vision Datasets Paul K. Mandal1,2,3 [email protected]
Pavan Reddy4
Abstract
arXiv:2607.05516v1 [cs.CV] 6 Jul 2026
Tristan Malatyński5 [email protected]
systematic failure cases [8, 20, 35]. These findings suggest that ordinary, unpoisoned datasets may contain statistical structure that can be leveraged to cause model failures. Moosavi-Dezfooli et al. [15] demonstrated that a single adversarial direction can affect many images and transfer across different model architectures. Other work also shows that incorporating information about a target-class distribution can improve cross-model attacks [19]. These adversarial directions, however, are generally obtained through optimization against a single victim or surrogate model. Similarly, frequency-domain attacks use structured frequency components to efficiently search for adversarial perturbations [7]. These adversarial directions are identified through responses from the attacked model. Informationgeometric attacks similarly derive adversarial directions from Fisher geometry from a trained neural network [40]. Therefore, while prior work established that numerous different types of attack directions exist, these directions had to be identified through model-mediated signals. Our paper seeks to answer what these prior papers do not: can target-specific failure directions can instead be constructed from the statistics of the original source dataset, without optimizing a victim or surrogate attack objective? Establishing these directions would connect dataset-level statistics to model sensitivities that are shared across multiple different architectures. We refer to these data source derived directions that induce target-specific failures as statistical adversaries. Our study is focused on four major questions:
Model-specific adversarial attacks have been extensively studied. We study a different failure mode: naturally occurring statistical signals in vision data that can behave like backdoor-like triggers without being maliciously inserted. We call these signals statistical adversaries. We analyse Imagenet to find patterns that are strongly linked to certain labels. We then use statistical controls to remove random correlations from our candidate signals. Finally, we demonstrate that these signals directly and predictably alter model predictions. These statistical adversaries are more targeted than generic corruptions and transfer across different model architectures. This suggests that some vulnerabilities are driven by dataset structure and distribution rather than a single model’s idiosyncrasies. We conclude that ordinary datasets can contain exploitable adversarial surfaces even in the absence of poisoning, and suggest that dataset audits should treat spurious structure not only as a source of bias or interpretability failure, but also as a latent attack surface for vision models.
1. Introduction Although modern vision models achieve strong benchmark performance, these metrics do not guarantee that the features they use are semantically meaningful. Models often use spurious patterns for predictions; these models perform well on benchmarks, but fail to transfer over to real-world scenarios [4]. Ilyas et al. [10] further argues that adversarial examples are features learned from these poorly generalized, spurious patterns. Additionally, both natural and model audits have shown that ImageNet contains harmful spurious features, class-associated frequency shortcuts, and
RQ1. Do models trained on ordinary, unpoisoned data exhibit sensitivity to adversarial directions derived only from the dataset? RQ2. Can these directions be constructed without victimmodel gradients, queries to a model, or surrogate attack objectives? RQ3. Can the resulting directions induce target-specific, model-independent failures?
1 Neurint, LLC, Baton Rouge, LA, USA. 2 U.S. Army Cyber Corps, U.S. Army Reserve, USA. 3 Northwestern State University of Louisiana, Natchitoches, LA, USA. 4 Automata, Arlington, VA, USA.
To answer these questions, we construct bounded perturbation directions solely from class-conditional ImageNet statistics and evaluate them on held-out, targetnegative images across CNN and transformer classifiers.
5 AGH University of Krakow, Krakow, Poland.
Correspondence to Paul K. Mandal: [email protected]
1
We compare raw class-mean directions against more controlled band-pass diagonal-whitened and Hellingermotivated frequency-statistical directions, using random, low-pass, spectrum-matched, global-mean, and wrongtarget controls. We find that selected source-statistical directions increase evidence for absent target classes beyond repeated matched controls. These effects primarily appear as calibrated false-positive inflation, rank movement, and top-(k) entry rather than consistent top-1 target takeover. Our contributions are:
we explore whether this step can be completely eliminated and the direction determined solely based on the dataset.
2.2. Decision-Making Features of Neural Networks Deep learning classifiers tend to focus on features that differ from human perception and are not related to the target class, a behavior that was observed long before the current era of artificial intelligence [30]. The best-known example is the recognition of wolves in snowy scenes [4] using local texture statistics instead of a global shape representation [9, 25, 26]. Similar examples occur also on ImageNet, where the classifier learned e.g. to recognize birds based on the background [37]. Attributes characterized by imbalance between groups may dominate predictions for minority subgroups [22]. Large-scale audits have revealed the existence of thousands of spurious features—correlated with classes and recognizable by humans—directly within the ImageNet dataset [2, 8, 20, 23], such as by adapting to different frequency ranges that may misdefine a class [34, 35, 38]. SilverLining showed that both spatial and spectral shortcuts can simultaneously emerge in real datasets and that removing such cues requires explicitly controlling frequencydomain confounders [33]. These audits are framed as issues related to bias and interpretability: they explain why the model fails in the case of an atypical example, but they do not show that the same spurious structure can be transformed into a controlled, repeatable perturbation of the data able to fool the model. This literature points to the frequency domain as a natural place to explore classconditional structures and directly provides information on the statistics we use in our work to investigate potential gaps in the ImageNet dataset. Most work on shortcut learning focuses on analyzing failures under distribution shifts. We, however, investigate whether such statistical regularities themselves can serve as perturbation directions.
• We introduce and define statistical adversaries: directional perturbations derived only from the training data, which we evaluate through a held-out protocol with norm, frequency, and spectrum-matched controls. • We propose source-only direction families based on bandpass diagonal-whitened class contrasts and Hellingermotivated frequency statistics, requiring no victim-model gradients, queries, or surrogate attack optimization. • We show that selected source-statistical directions induce target-specific false-positive inflation, rank movement, and top-k entry across CNN and transformer classifiers, while raw class-mean directions provide a negative baseline.
2. Related Works 2.1. Adversarial Attacks The current literature on adversarial attacks comes from the observation that imperceptible perturbations can significantly alter the predictions of neural networks. Szegedy et al. were the first to demonstrate the existence of adversarial attacks [27]. Goodfellow et al. later found that vulnerability to adversarial attacks results from local linearity in multidimensional models, and introduced the Fast Gradient Sign Method (FGSM) [5]; however, Projected Gradient Descent (PGD) is considered the strongest type of attack and serves as the benchmark for evaluating model robustness [18]. More relevant to our evaluation settings are studies showing that a single perturbation can effectively attack a significant portion of the dataset for a given model [15], and in some cases even across different architectures by taking into account information about the target class distribution [19], by adjusting the statistics of intermediate features of a given class instead of directly attacking the logits [11] or by correlating decision boundaries with geometric properties [12]. Black-box attacks from a frequency-domain perspective [3, 7] and attacks based on information geometry from Fisher’s geometry of a trained network [40], further demonstrate that the shape of an effective perturbation is constrained by a structure that is not specific to any particular model. Nevertheless, these approaches determine perturbations using information from the model. In contrast,
2.3. Backdoor Attacks and Natural Triggers “Backdoor” attacks involve introducing triggering patterns during training to cause the model’s predictions to shift in the direction chosen by the attacker. BadNets was the first to demonstrate the effectiveness of “backdoor” poisoning [6]. Subsequent work introduced “clean-label” attacks [32], hidden triggers [24], and semantic backdoors [14]. However, poisoned examples leave a detectable spectral trace in feature space [31], although that study uses them to detect an injected trigger rather than to rule out naturally occurring triggers, which is the focus of our research. Ordinary, naturally photographed objects can act as triggers when associated with a target label [36], and naturally occurring reflection patterns on glass or water can serve the same function [14]. The most important findings come from [28, 29], showing that normally trained, unpoisoned models already contain input transformations resembling triggers,
2
Figure 1. Source-statistics overview. The figure summarizes the dataset-level statistics used to form class-conditioned directions.
3.1. Setup
which the authors refer to as natural backdoors. Our work shares the main thesis of this article, but differs in terms of mechanism and evaluation: instead of reverse-engineering a trigger for a specific, trained model, we derive a potential direction based solely on source-class statistics, without access to any victim model, and then verify whether this single, model-independent direction reproduces backdoor-like behavior across several held-out architectures at once. The literature on dataset bias shows that ImageNet exhibits a structure that can be exploited; the literature on frequency bias narrows down the range in which this structure typically occurs; the literature on transfer attacks shows that a single direction can generalize to different models; and the literature on natural backdoors shows that models not subjected to poisoning may already contain vulnerabilities resembling triggers. To the best of our knowledge, no previous work combines all four observations into a single process that (i) derives a potential direction exclusively from class-conditional source statistics, (ii) rules out trivial explanations for this effect using norm-matched control groups, frequency, and false targets - not just a pure baseline-and (iii) evaluates the remaining directions without fine-tuning or access to queries, on architectures where that direction was never built or tested during search. This is a gap that Statistical Adversaries are intended to fill.
Images and labels. Images are x ∈ X = [0, 1]C×H×W with C = 3 channels and H = W = 224, giving flattened dimension d = CHW . Class labels are drawn from Y = {1, . . . , K} with K = 1000. Source dataset. Perturbations are functions of a source c training set Dtr = {(xi , yi )}N i=1 alone; we write Dtr for its class-c subset and the number of training samples nc = c |. Evaluation uses a disjoint validation set Dval . |Dtr Victim models. We attack a fixed family of M pretrained (m) classifiers {f (m) }M : X → RK returns m=1 , where each f a vector of K class logits. For each target t, evaluation is restricted to the target-negative validation images Vt = {x ∈ Dval : y(x) ̸= t},
(1)
so movement toward t is an induced false positive rather than a correct prediction. Perturbation model. For each target t we build a single perturbation δ t ∈ Rd , apply it additively, and clamp the result to the image range under an ℓ∞ budget ε: x̃ = clip[0,1] (x + δ t ),
3. Problem Formulation
∥δ t ∥∞ ≤ ε,
We construct universal targeted perturbations from training-set statistics alone: a single perturbation is built per target class with no access to the victim models at construction time, and is then evaluated for its effect on a family of pretrained classifiers. Figure 1 summarizes the source statistics used to build these directions.
ε ∈ {8, 16, 32}/255.
(2) (3)
Construction is model-free: δ t = A(Dtr , t) with ∂A/∂f (m) ≡ 0, so it uses no gradients and issues no queries to any f (m) . The same δ t is reused across every image in Vt and across all M models, making the attack universal in both inputs and models.
3
Figure 2. Methodology overview. The figure illustrates the construction pipeline for source-statistical adversary directions.
3.2. Evaluation metrics
4.1. Source-statistical construction
(m) as a one-vs-rest deOur headline metric treats each ft
Class means. From the class subsets of Dtr we form the class, global, and complement means 1 X 1 X nc µc , (6) x, µ= µc = nc N c c
tector for class t. On the clean target-negative images we (m) fix a threshold τt at the (1 − α) empirical quantile of the clean target logits, with α = 0.05, and then hold that threshold fixed after perturbation.The resulting false-positive-rate shift is ∆FPRα = EVt 1
(m) (m) {ft (x̃) > τt } − α,
lift = 1 + ∆FPRα /α.
x∈Dtr
µt̄ =
X 1 nc µc . N − nt
(7)
c̸=t
(4)
The raw class-contrast direction is the difference between the target and non-target means,
(5)
We report ∆FPRα in percentage points and as extra false fires per 1,000 images. Secondary metrics record the target-logit shift, the target-rank improvement, and targeted ASR@1/5. Target-rank and ASR definitions are provided in Appendix B.1, and inference rules are provided in Appendix B.4.
g t = µt − µt̄ .
(8)
To prevent a few high-variance coordinates from dominating, we form the diagonally whitened contrast P 1/2 wt = g t ⊘(s+η), sj = N1 i (xi,j −µj )2 , (9)
4. Source-Statistical Adversary Construction
where ⊘ denotes coordinatewise division, s is the vector of per-coordinate standard deviations over Dtr , and η > 0 stabilizes low-variance coordinates.
Every perturbation is produced as follows: (i) form a targetspecific direction from first- and second-moment statistics of Dtr ; (ii) optionally reshape it with a linear operator O that retains only part of the signal; and (iii) project the result to the ℓ∞ budget. Figure 2 illustrates the full construction pipeline.
Operator family and projection. A perturbation family is defined by applying an operator O to either g t or wt . The final perturbation is obtained by scaling the resulting direction v t to the budget, vt δ t = Πε (v t ) = ε , ∥δ t ∥∞ = ε. (10) ∥v t ∥∞ 4
The confirmation experiments focus on FFT-Hellinger and bandpass diagonal-whitened directions. Appendix C gives the full operator definitions and the complete perturbationfamily table.
slice, and positions 15–24 form the confirmation slice. The candidate panel is fixed before the confirmation slice is evaluated, and only results from this slice are used for the headline confirmation analysis.
5. Experimental Protocol
Frozen confirmation panel. The confirmation panel contains 11 target–construction–budget combinations evaluated on all four victim models, giving 44 model-level evaluations. Twenty-four correspond to FFT-Hellinger and twenty correspond to bandpass diagonal-whitened mean. Since the panel was assembled during the development stages, the reported results summarize these selected candidates rather than every ImageNet target.
We test whether class-conditioned statistics estimated from ImageNet can be converted into universal perturbations that increase false positives for a chosen target class. The perturbations are constructed without using victim-model gradients, model-specific optimization, or image-specific updates. Our primary outcome is target-specific false-positiverate (FPR) inflation. We report targeted ASR@1 and ASR@5 as secondary outcomes. To determine whether an observed effect is specific to the proposed class-conditioned direction, we compare it with five controls: Gaussian-random noise, lowpassrandom noise, spectrum-random noise, the global ImageNet mean, and a direction constructed for the wrong target. Each control is evaluated on the same images and under the same perturbation budget as the corresponding proposed direction. The perturbation budget limits the largest change made to any individual RGB channel value.
Statistical tests and Benjamini–Hochberg correction. We test paired target-threshold crossings with a one-sided exact binomial test, equivalently a one-sided exact McNemar test, and apply Benjamini–Hochberg correction across the 44 confirmation cells. This testing protocol is detailed more thoroughly in Appendix D.
5.2. Victim Models and Controls Victim models. We evaluate four pretrained ImageNet classifiers: ResNet-50, ConvNeXt-Tiny, ViT-B/16, and Swin-T. All models use the default pretrained weights provided by Torchvision. Their parameters remain frozen, and inference is performed in evaluation mode. The same target-specific perturbation is evaluated on every model.
5.1. Data and Evaluation Source-statistic construction set. We compute the source statistics from all 1,281,167 images in the ImageNet1K training split. The split contains 1,000 classes, with between 732 and 1,300 training images per class. We use the training split only to estimate the statistics from which the perturbations are constructed. We do not train or finetune any of the victim models, and we do not compute FPR, ASR, or attack-success results on the training images. ImageNet-1K images do not have consistent dimensions. Thus, we resized each image using bicubic interpolation such that its shorter side was 256 pixels. We then centercropped the image to 224 × 224 and converted to an RGB tensor with values in [0, 1]. From these images, we calculate per-class and global mean images, first and second moments at each pixel location and color channel, and summaries of how image energy is distributed across spatial frequencies.
Perturbation application We add our perturbations in the RGB image space prior to ImageNet normalization. After adding the perturbation, pixel values are clipped to the valid RGB range of [0, 1]. We then apply standard ImageNet normalization. Our perturbation budgets, ℓ∞ are 8/255 and 16/255. No single RGB value for a given pixel may change more than the perturbation budget. Controls. Each proposed direction is compared with controls evaluated on the same target class, target negative images, victim model, and perturbation budget. The Gaussianrandom control uses 10 random pixel-space noise directions. The lowpass-random and spectrum-random controls use 30 seeds each. The global-mean and wrong-target controls are deterministic.
Candidate selection and validation sets. The ImageNet1K validation split contains 50,000 images, with exactly 50 images from each class. Candidate discovery requires searching over target classes, perturbation constructions, and perturbation budgets. Evaluating the final candidates on the same images used to select them would therefore overestimate their performance. Thus, we shuffle the 50 validation images within each class using seed 17 and divide the resulting ordering into non-overlapping slices. Positions 0–4 form the conceptcheck slice, positions 5–14 form the candidate-validation
6. Results We first report whether raw class averages are recognizable to the models and which frequency components of the mean-based directions appear useful. We then report the held-out confirmation results on the frozen candidate panel. We follow this with comparisons on the matched controls in addition to a breakdown by construction, architecture, 5
Figure 3. Representative confirmation-slice examples. Each triplet shows a clean target-negative image, a visualization of the corresponding source-statistical perturbation, and the perturbed image after applying the perturbation. Panel titles report the construction, perturbation budget, and victim model; text below the images reports the target class, clean and perturbed target logit, and clean and perturbed target rank. Table 1. Target-specific false-positive inflation on the frozen confirmation panel. Counts are condition-level prediction decisions rather than unique images. Corrected counts are after Benjamini–Hochberg correction. Construction All proposed FFT-Hellinger Bandpass-whitened
Cells
Evals.
Clean Perturbed FPR FPR
44 439,560 5.005% 24 239,760 5.005% 20 199,800 5.005%
FPR Net add. Extra FPs Corrected lift FPs /1,000 q ≤ .05
9.689% 1.94× 7.994% 1.60× 11.723% 2.34×
and secondary attack metrics. Figure 3 shows representative confirmation-slice examples used to illustrate these aggregate effects. The raw class-average images are rarely classified as their own class. Across the four victim models, own-class top-1 accuracy ranges from 1.1% to 2.5%, and own-class top-5 accuracy ranges from 4.2% to 5.7% . The median own-class rank is also far from the top of the prediction list, ranging from 212.5 to 282.0. This suggests that the final effect is not explained by class averages simply looking like recognizable examples of their corresponding classes. Instead, the useful signal appears to require the transformations described in Section 4, such as class contrasts, whitening, and frequency selection.
20,589 7,167 13,422
46.84 29.89 67.18
40/44 21/24 19/20
positives on our clean, unperturbed images. After applying the proposed directions, this increases to 42,589 target false-positive decisions. This is a net increase of 20,589 false positives, or 46.84 additional false positives per 1,000 target-negative images. In other words, the proposed directions increase target-specific FPR from 5.005% to 9.689%, a 1.94× lift over the clean baseline (Table 1). The increase is positive in 43 of 44 cells, and 40 of 44 clean-to-perturbed comparisons remain significant after Benjamini–Hochberg correction. The target-specific increase is not accompanied by a general increase in false positives for arbitrary wrong classes. The generic any-wrong-class FPR decreases from 5.005% to 4.452%. Figure 4 provides a qualitative view of one such target-directed change.
6.1. Proposed Directions Increase Target-Specific FPR
6.2. Controls Explain Some, but Not All, of the Effect
The frozen confirmation panel contains 11 targetconstruction-budget candidates evaluated on four models, for 44 model-candidate cells. Each cell is evaluated on 9,990 target-negative confirmation images, giving 439,560 condition-level image evaluations. These are conditionlevel evaluations rather than unique images, since the same validation image can appear under different targets, constructions, budgets, and models. Our confirmation panel contains 22,000 targeted false-
The matched controls demonstrate several findings with our controls. First, Gaussian-random noise does not reproduce the effect. Its average perturbed FPR is 4.792%, which is slightly below our clean baseline. Our proposed directions exceed the Gaussian-random mean in 43 of 44 cells. Because this control has only 10 seeds, the smallest possible empirical p-value is 1/11 = 0.0909. Therefore, our onesided empirical randomization test over control seeds can-
6
Figure 4. Representative interpretability views for clean and perturbed inputs. Each column pair shows the clean and perturbed image for the same target, construction, budget, and victim model. Rows show the image, LIME overlay, saliency overlay, and saliency heat map. In the LIME row, red marks regions that support the target class and blue marks regions against it; the saliency rows are unsigned, so brighter yellow means larger attribution magnitude, regardless of direction. These visualizations are qualitative; aggregate confirmation statistics are reported in Tables 1–4.
not produce p < .05 for this control. Lowpass-random perturbations produce some falsepositive inflation, raising FPR to 5.941%. The proposed directions remain larger by 3.748 percentage points on average and exceed the lowpass-random mean in 37 of 44 cells. Thirty-five of those comparisons remain significant after Benjamini–Hochberg correction. While this shows that smooth low-frequency noise can move target scores, its shift is relatively negligible and it significantly underperforms our proposed method. Spectrum-random is the strongest control. It raises FPR to 7.681%, which means that matching the proposed direction’s frequency magnitude accounts for part of the effect. Even so, the proposed directions remain larger on average by 2.008 percentage points and exceed the spectrumrandom mean in 37 of 44 cells. Although 21 comparisons have raw p ≤ .05, none survive Benjamini–Hochberg correction. Therefore, we treat spectrum matching as an important partial explanation as opposed to a ruled-out null. The deterministic controls are much weaker. The globalmean control barely outperforms the clean baseline, raising FPR only to 5.207%. The wrong-target control raises FPR to 6.209% over the 40 valid cells, but the proposed direction still exceeds it in 32 of those 40 cells. This indicates that target identity matters. Simply applying another
class-statistical direction is not equivalent to applying the direction for the intended target. Table 2 summarizes these matched-control comparisons.
6.3. FFT-Hellinger and Bandpass-Whitened Directions Both Contribute FFT-Hellinger and bandpass-whitened directions play different roles in the confirmation panel. FFT-Hellinger is the frequency-profile construction motivated by the classlevel frequency-energy summaries. It is positive in all 24 of its cells, increases FPR from 5.005% to 7.994%, and is Benjamini–Hochberg significant against clean in 21 of 24 cells. Bandpass-whitened is the stronger empirical construction in this panel. It increases FPR from 5.005% to 11.723%, is positive in 19 of 20 cells, and is Benjamini– Hochberg significant against clean in 19 of 20 cells. The confirmation panel is not balanced across methods, targets, and budgets. Thus, direct method comparisons should use matched subsets. For FFT-Hellinger, targets 267 and 789 appear at both 8/255 and 16/255. Within these matched cells, the 16/255 result is larger for 7 out of 8 of them. For bandpass-whitened, target 789 appears at both budgets. The 16/255 result is larger for all four models. At 16/255, FFT-Hellinger and bandpass-whitened share targets 140, 207, and 789. On those matched cells, bandpass-whitened is larger for 10 out of 12 comparisons. 7
Table 2. Comparison with clean inputs and matched controls. Stochastic control FPRs are averaged over seeds within each model–candidate cell. Corrected counts are after Benjamini–Hochberg correction. Comparator Clean Gaussian-random Lowpass-random Spectrum-random Global mean Wrong target†
FPR 5.005% 4.792% 5.941% 7.681% 5.207% 6.209%
∆FPR Prop.-comp. Prop. > comp. Raw p ≤ .05 Corrected q ≤ .05 0.000 pp −0.213 pp +0.936 pp +2.676 pp +0.202 pp +1.204 pp
+4.684 pp +4.897 pp +3.748 pp +2.008 pp +4.482 pp +3.588 pp
43/44 43/44 37/44 37/44 38/44 32/40
40/44 0/44 35/44 0/44 – –
40/44 0/44 35/44 21/44 – –
The Gaussian-random test has minimum attainable p = 1/11 = .0909. The lowpass-random and spectrum-random tests have minimum attainable p = 1/31 = .0323. † Four unavailable wrong-target comparisons are excluded.
Table 3. Matched budget and construction comparisons. Values are mean target-specific FPR changes in percentage points. Comparison
Matched subset
Setting A
Setting B
B>A
FFT-Hellinger budget Bandpass budget Construction at 16/255
Targets 267, 789; four models Target 789; four models Targets 140, 207, 789; four models
8/255: 1.071 8/255: 3.606 FFT-Hell.: 4.385
16/255: 5.231 16/255: 9.377 BP-whitened: 5.674
7/8 4/4 10/12
Table 4. Target-specific FPR inflation by victim architecture. Each model is evaluated on the same 11 frozen candidates. Corrected counts are after Benjamini–Hochberg correction. Model
Pert. FPR
tion decreases it. Highpass also produces negative average target-logit and target-rank movement. We therefore carry forward the frequency-restricted bandpass direction, along with the FFT-Hellinger construction, into the confirmation experiment. We use this ablation is to motivate method selection. It is exploratory and not used as evidence for the final claims.
∆FPR Positive Corrected cells q ≤ .05
ResNet-50 5.912% +0.907 pp ConvNeXt-T 6.117% +1.112 pp ViT-B/16 11.615% +6.610 pp Swin-T 15.111% +10.106 pp
10/11 11/11 11/11 11/11
9/11 9/11 11/11 11/11
7. Conclusion We show that source-statistical perturbations can induce target-specific false positives without model gradients, queries, or image-specific optimization. On the frozen confirmation panel, FFT-Hellinger and bandpass-whitened directions increase target-specific FPR from 5.005% to 9.689%, with positive movement in 43 of 44 model– candidate cells and 40 cells remaining significant after Benjamini–Hochberg correction. The effect is strongest for thresholded FPR, target logits, and target ranks. The controls clarify what this result does and does not show. Gaussian random noise and global mean remain near the clean baseline, while lowpass-random produces a smaller effect. Spectrum-random is the strongest control and explains part of the behavior, but the proposed directions remain larger on average and exceed the spectrum-random mean in most cells. The current evidence therefore supports the existence of selected, transferable target-specific false-positive vulnerabilities induced by class-conditioned image statistics.
These comparisons are descriptive because the candidate panel was selected during development. Table 3 summarizes these matched subsets.
6.4. The Effect Transfers Across Architectures The same frozen perturbations are evaluated across all four architectures. The effect is positive for nearly every architecture-candidate pair, but its magnitude differs sharply by model. ResNet-50 and ConvNeXt-Tiny show smaller average increases, while ViT-B/16 and Swin-T are much more sensitive. On Swin-T, the mean target-specific FPR rises to 15.111%, a 10.106 percentage-point increase over clean. On ViT-B/16, it rises to 11.615%. The two convolutional models also show positive movement, but at a smaller scale (Table 4).
6.5. Frequency ablation The exploratory frequency ablation shows that the effect is not driven by high-frequency residuals. Bandpass, raw mean, and zero-DC mean directions all increase targetspecific FPR on average, while the highpass mean direc8
References
[15] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 1, 2 [16] Konda Reddy Mopuri, Utsav Garg, and R. Venkatesh Babu. Fast feature fool: A data independent approach to universal adversarial perturbations. In Proceedings of the British Machine Vision Conference, pages 30.1–30.12. BMVA Press, 2017. 12 [17] Ningping Mou, Binqing Guo, Lingchen Zhao, Cong Wang, Yue Zhao, and Qian Wang. No-box universal adversarial perturbations against image classifiers via artificial textures. IEEE Transactions on Information Forensics and Security, 19:9803–9818, 2024. 12 [18] Aleksander Madry, ˛ Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), 2018. 2 [19] Muzammal Naseer, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Fatih Porikli. On generating transferable targeted perturbations. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7708–7717, 2021. 1, 2 [20] Yannic Neuhaus, Maximilian Augustin, Valentyn Boreiko, and Matthias Hein. Spurious features everywhere - largescale detection of harmful spurious features in imagenet. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 20235–20246, 2023. 1, 2, 12 [21] Omid Poursaeed, Isay Katsman, Bicheng Gao, and Serge Belongie. Generative adversarial perturbations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4422–4431, 2018. 12 [22] Shiori Sagawa*, Pang Wei Koh*, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020. 2 [23] Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020. 2 [24] Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden trigger backdoor attacks. 2020. 2 [25] Vaishaal Shankar, Rebecca Roelofs, Horia Mania, Alex Fang, Benjamin Recht, and Ludwig Schmidt. Evaluating machine accuracy on ImageNet. In Proceedings of the 37th International Conference on Machine Learning, pages 8634– 8644. PMLR, 2020. 2 [26] Pirzada Suhail, Vrinda Goel, and Amit Sethi. Shortcut learning susceptibility in vision classifiers (student abstract). In Proceedings of the AAAI Conference on Artificial Intelligence, pages 41390–41392, 2026. 2 [27] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks, 2014. 2 [28] Guanhong Tao, Zhenting Wang, Siyuan Cheng, Shiqing Ma, Shengwei An, Yingqi Liu, Guangyu Shen, Zhuo Zhang, Yun-
[1] Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1):289–300, 1995. 15 [2] Lucas Beyer, Olivier J. H’enaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? 2020. 2 [3] Z. Duan, Z. Wang, and M. Sun. Frequency-domain enhanced adaptive ensemble adversarial attack for protecting image privacy. In Advanced Intelligent Computing Technology and Applications (ICIC), LNCS vol. 15843. Springer, 2025. 2 [4] R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11): 665–673, 2020. 1, 2 [5] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples, 2015. 2 [6] Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. 2017. 2 [7] Chuan Guo, Jacob Gardner, Yurong You, Andrew Gordon Wilson, and Kilian Weinberger. Simple black-box adversarial attacks. In Proceedings of the 36th International Conference on Machine Learning, pages 2484–2493. PMLR, 2019. 1, 2 [8] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15262–15271, 2021. 1, 2 [9] Katherine L. Hermann, Ting Chen, and Simon Kornblith. The origins and prevalence of texture bias in convolutional neural networks. arXiv: Computer Vision and Pattern Recognition, 2019. 2 [10] Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. ˛ Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2019. 1 [11] Nathan Inkawhich, Wei Wen, Hai (Helen) Li, and Yiran Chen. Feature space perturbations yield more transferable adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 2 [12] Valentin Khrulkov and Ivan Oseledets. Art of singular vectors and universal adversarial perturbations. In CVPR, 2018. 2 [13] Qizhang Li, Yiwen Guo, and Hao Chen. Practical no-box adversarial attacks against DNNs. In Advances in Neural Information Processing Systems, pages 12849–12860, 2020. 12 [14] Yunfei Liu, Xingjun Ma, James Bailey, and Feng Lu. Reflection backdoor: A natural backdoor attack on deep neural networks. In European Conference on Computer Vision (ECCV), pages 182–199. Springer, 2020. 2
9
shu Mao, and Xiangyu Zhang. Backdoor vulnerabilities in normally trained deep learning models, 2022. 2 [29] Guanhong Tao, Siyuan Cheng, Zhenting Wang, Shiqing Ma, Shengwei An, Yingqi Liu, Guangyu Shen, Zhuo Zhang, Yunshu Mao, and Xiangyu Zhang. Exploring inherent backdoors in deep learning models. In 2024 Annual Computer Security Applications Conference (ACSAC), pages 923–939, 2024. 2 [30] A. Torralba and A. A. Efros. Unbiased look at dataset bias. page 1521–1528, USA, 2011. IEEE Computer Society. 2 [31] Brandon Tran, Jerry Li, and Aleksander Madry. ˛ Spectral signatures in backdoor attacks. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2018. 2 [32] Alexander Turner, Dimitris Tsipras, and Aleksander Madry. ˛ Clean-label backdoor attacks. 2019. 2 [33] Balagopal Unnikrishnan, Michael Brudno, and Chris McIntosh. Silverlining: Data-first mitigation of spatial and spectral shortcuts without introducing new confounders. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1294–1303, 2026. 2 [34] Haohan Wang, Xindi Wu, Zeyi Huang, and Eric P. Xing. High-frequency component helps explain the generalization of convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8684–8694, 2020. 2 [35] Shunxin Wang, Raymond Veldhuis, and Nicola Strisciuglio. Do imagenet-trained models learn shortcuts? the impact of frequency shortcuts on generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 25198–25207, 2025. 1, 2 [36] Emily Wenger, Josephine Passananti, Arjun Nitin Bhagoji, Yuanshun Yao, Haitao Zheng, and Ben Y. Zhao. Backdoor attacks against deep learning systems in the physical world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6206–6215, 2021. 2 [37] Kai Yuanqing Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. ˛ Noise or signal: The role of image backgrounds in object recognition. In International Conference on Learning Representations, 2021. 2 [38] Dong Yin, Raphael Gontijo Lopes, Jonathon Shlens, Ekin Dogus Cubuk, and Justin Gilmer. A fourier perspective on model robustness in computer vision. In Advances in Neural Information Processing Systems (NeurIPS), pages 13255–13265, 2019. 2 [39] Qilong Zhang, Youheng Sun, Chaoning Zhang, Chaoqun Li, Xuanhan Wang, Jingkuan Song, and Lianli Gao. Practical no-box adversarial attacks with training-free hybrid image transformation. arXiv preprint arXiv:2203.04607, 2022. 12 [40] Chenxiao Zhao, P. Thomas Fletcher, Mixue Yu, Yaxin Peng, Guixu Zhang, and Chaomin Shen. The adversarial attack and detection under the fisher information metric. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):5869– 5876, 2019. 1, 2
10
A. Prior-Work Comparison
B.3. Controls
Table 5 summarizes the closest prior-work families and how our setting differs from them.
We compare each δ t against budget-matched controls that share its ℓ∞ norm but lack the target-class signal. All controls are projected to the same budget ε via (10), and the spectral and statistical operators they invoke are defined in Sec. C.
B. Detailed Problem Formulation B.1. Target rank and secondary metrics
• random: v ∼ Unif[−1, 1]d , an unstructured baseline. • lowpass-random: Glow applied to a random direction, matching the smoothness of low-frequency perturbations without any class signal. • spectrum-random: a phase-randomized version of δ t that preserves its power spectrum, F(v) = |F (δ t )| ⊙ eiθ with θ ∼ Unif[0, 2π) under Hermitian symmetry, isolating spectral magnitude from spatial structure. • global / statistical: the dataset-wide mean image v = µ, a class-agnostic statistical direction. • wrong-target: a perturbation δ t′ built for a different class t′ ̸= t, testing target specificity.
We summarize the standing of a target class t on a single im(m) age by the integer rt (x), defined as the position of t in the (m) (m) descending ordering of the logits f1 (x), . . . , fK (x), (m) rt (x) = 1 +
K X 1 fk(m) (x) > ft(m) (x) .
(11)
k=1 (m)
Here rt (x) ∈ {1, . . . , K}: it equals 1 when t is the topscoring class and K when t is scored last, so a smaller r means a stronger standing for t. A perturbation that promotes t decreases r; the amount by which it decreases is the r-improvement ∆r. For a model f (m) and image x ∈ Vt with perturbed counterpart x̃ we record three per-image quantities, each averaged over Vt . The target-logit shift (m)
∆zt
(m)
(x) = ft
(m)
(x̃) − ft
(x)
B.4. Selection and inference protocol For any metric Θ we draw S seeds per control to form a null S distribution {Θnull s }s=1 , and summarize the position of the main observed value Θ by
(12)
counts how many positions class t climbs in the logit ordering, and is positive when t is promoted. The top-k targeted success (m) (m) ASRk = EVt 1 rt (x̃) ≤ k , k ∈ {1, 5}, (14)
Θmain − Θ̄null . sd(Θnull ) (16) The p-value is floored at 1/(S + 1), so z quantifies the margin by which the perturbation exceeds the null. Although construction is model-free, the reported triple (t, method, ε) is selected from a candidate grid by a model-dependent score; to guard against this selection we report a candidate as transferring only when its paired ∆FPRα confidence interval excludes 0 and it exceeds every null seed, simultaneously on all M models.
is the fraction of target-negative images for which t is pushed into the top k predictions.
C. Detailed Construction Operators
p̂ =
is positive when the perturbation raises the score of t. The r-improvement (m)
∆rt
(m)
(x) = rt
(m)
(x) − rt
(x̃)
(13)
≥ Θmain }| 1 + |{s : Θnull s , S+1
z=
Each operator takes an image-shaped direction v ∈ RC×H×W , where v is either the contrast g t or its whitened form wt , and retains a chosen part of it.
B.2. Calibrated FPR threshold (m)
On the clean target-negative images we fix a threshold τt at the (1 − α) empirical quantile of the clean target logits, with α = 0.05, (m) (m) (m) τt : EVt 1 ft (x) > τt = α, (15)
Spectral operators. Let F denote the two-dimensional discrete Fourier transform applied independently to each color channel, and F −1 its inverse; thus F maps a channel from pixel space into frequency space and F −1 maps it back. For a frequency location ξ we write its radial frequency ρ(ξ) = ∥ξ∥2 (the distance from the zero, or DC, frequency), and define the radial band mask M[a,b] (ξ) = 1 a ≤ ρ(ξ) ≤ b . (17)
so the clean false-positive rate equals α by construction. We report ∆FPRα both in percentage points and as 1000 ∆FPRα extra false fires per 1000 images, with a paired nonparametric bootstrap confidence interval over Vt . To confirm that the effect is genuinely targeted, we verify that the untargeted any-wrong-class false-positive rate is not inflated. 11
Table 5. Comparison of attack-construction assumptions and contribution scope. “Class-conditioned” means that a target label y is selected before constructing a perturbation, rather than assigned post hoc based on the label to which predictions happen to collapse.
Fast Feature Fool [16]
GAP [21]
Li et al. [13]
HIT [39]
Neuhaus et al. [20]
Texture Adv [17]
Ours
No victim-model queries
N/A
N/A
✓
✓
N/A
✓
✓
No victim gradients or internal representations
×
×
✓
✓
×
✓
✓
No victim or surrogate model used during construction
×
×
×
✓
×
✓
✓
No learned attack model or generator
✓
×
×
✓
✓
✓
✓
No hand-designed generic texture or geometric pattern as the main attack signal
✓
✓
✓
×
✓
×
✓
Known labeled training dataset is the sole construction signal
×
×
×
×
×
×
✓
Constructs a perturbation applicable to arbitrary held-out source images
✓
✓
✓
✓
×
✓
✓
Produces a perturbation indexed by an arbitrary target label y
×
✓
×
×
×
×
✓
Construction is simultaneously model-blind and class-conditioned
×
×
×
×
×
×
✓
Primary outcome is perturbation-induced target-class false-positive inflation
×
×
×
×
×
×
✓
Central claim: dataset-only construction reveals shared target-specific vulnerabilities across independently trained models
×
×
×
×
×
×
✓
Property
Notes. Neuhaus et al. is a model-based spurious-feature discovery, benchmarking, and mitigation framework rather than an adversarial perturbation method. Its entries therefore describe the NPCA discovery pipeline, not the downstream models evaluated on Spurious ImageNet. “N/A” denotes settings with direct model access or no corresponding victim-query threat model.
12
Multiplying a spectrum by such a mask and inverting yields a band-limited operator; the low-pass, high-pass, and bandpass operators are Glow (v) = F −1 M[0,ρc ] ⊙ F v , (18) −1 Ghigh (v) = F M[ρc ,∞) ⊙ F v , (19) Gband (v) = F −1 M[ρℓ ,ρh ] ⊙ F v , (20) with cutoff ρc and band edges ρℓ < ρh . Intuitively, Glow keeps smooth, blurry structure, Ghigh keeps sharp edges and texture, and Gband keeps mid-frequency structure. Channel and DC operators. component) of a direction is DC(v)c =
The per-channel mean (DC
1 X vc,h,w . HW
(21)
(22)
ptb −
B q √ 1 X √ pb − ptb′ − pb′ . (26) B ′
A positive ht (b) marks a band carrying relatively more target power than the dataset average and a negative a band Ponep carrying relatively less; before centering, 21 b ptb − √ 2 pb is exactly the squared Hellinger distance between the two profiles. Broadcasting the band weights back to the frequency plane as W (ξ) = ht b(ξ) , the operator modulates the spectrum of the contrast, inverts, and removes each channel’s spatial mean, −1 v fft-hel = ZDC F W ⊙ F g . (27) t t
(23)
FFT–Hellinger operator. This operator uses Dtr to identify the radial frequency bands whose share of spectral power differs most between target-class images and the dataset as a whole, and reweights the contrast g t to emphasize those bands. Reusing the Fourier transform F and radial frequency ρ(ξ) above, we partition the frequency plane into B radial bands and write b(ξ) ∈ {1, . . . , B} for the band containing ξ. The radial power of an image x in band b is its channel-averaged Fourier power summed over that band, C X 1 X 2 Pb (x) = (Fx)c (ξ) . (24) C c=1
Table 6 summarizes the perturbation families used in the candidate screen and confirmation panel.
D. Detailed Experimental Protocol Validation slices. Thus, we have to separate the datasets we use for candidate development and final evaluation. In order to achieve this in a random but still reproducible way, we shuffle the 50 validation images within each class using seed 17 and divide the resulting ordering into nonoverlapping slices. Positions 0–4 form the concept-check slice. They contain five images per class and 5,000 images in total. We use this slice to evaluate an initial set of candidates, check whether they produce target-directed changes in FPR, logits, and ranks. We also compare them with matched controls, and remove candidates that are ineffective or affected by obvious implementation artifacts.
ξ: b(ξ)=b
Averaging Pb over the target-class images and over all images yields the two radial power profiles P̄ t = (P̄bt )b and P̄ = (P̄b )b , which we normalize into probability vectors over the B bands, P̄b . ′ b′ P̄b
q
b =1
where 1 is the constant all-ones image. Thus ZDC discards a global color cast while preserving structure, and CM does the opposite.
pb = P
gt CM(g t ) ZDC(g t ) Glow (g t ) Ghigh (g t ) Gband (g t ) sign(g t ) v fft-hel t wt Gband (wt )
ht (b) =
and the channel-mean operator, which keeps only the overall per-channel color shift,
P̄ t ptb = P b t , b′ P̄b′
mean channel-mean zero-dc-mean lowpass-mean highpass-mean bandpass-mean sign-mean fft-hellinger diag-whitened-mean bandpass-diag-whitened-mean
We measure the discriminativeness of each band through the Hellinger chart, i.e. the signed difference of the squareroot profiles, mean-centered so the weights add no net energy,
From it we form the zero-DC operator, which removes each channel’s average and keeps only spatial pattern,
CM(v)c = DC(v)c 1,
vt
Table 6. Perturbation families as operators on the contrast g t or its diagonally whitened form wt .
h,w
ZDC(v)c = v c − DC(v)c 1,
Method
(25)
13
Table 7. Recognition of ImageNet class-average images. Each model is evaluated on the mean image of each of the 1,000 ImageNet classes. Model ResNet-50 ConvNeXt-T ViT-B/16 Swin-T
Table 8. Exploratory frequency ablation. Values are means over 18 model-target-budget conditions per construction. FPR changes are percentage points.
Top-1 Top-5 Mean Median (%) (%) rank rank 1.3 1.8 1.1 2.5
4.2 5.7 4.2 5.6
329.8 334.3 343.5 293.0
Construction Bandpass mean Raw mean Zero-DC mean Highpass mean
255.5 282.0 251.0 212.5
∆FPR@5%
∆logit ∆rank
+0.943 +0.028 +0.731 +0.026 +0.759 +0.017 -0.206 -0.012
+14.6 +16.4 +8.7 -8.4
can produce the same effect. This experiment uses three target classes, ResNet-50 and ViT-B/16, and perturbation budgets of 8/255, 16/255, and 32/255. The resulting grid contains 3 × 2 × 4 × 3 = 72 proposed-direction conditions. Each condition is evaluated on 1,998 target-negative validation images. We use this stage to identify which frequency band is carried forward into the confirmation experiment. Table 8 reports the resulting exploratory frequency ablation.
Positions 5–14 form the candidate-validation slice. They contain 10 images per class and 10,000 images in total. We use this larger, disjoint slice to reevaluate the retained candidates with additional random-control seeds and to select the final target–construction–budget combinations. Positions 15–24 form the confirmation slice, also containing 10 images per class and 10,000 images in total. The candidate panel is fixed before this slice is evaluated, and only results from this slice are used for the headline confirmation analysis. Positions 25–49, are comprised of the remaining 25,000 validation images. They are not used in the current experiments.
Broad candidate screen. We conduct an initial screen over 100 target classes, consisting of five previously identified prototype candidates and 95 targets selected using a fixed random sample. For each target, we evaluate zeroDC mean, diagonal-whitened mean, bandpass diagonalwhitened mean, and FFT-Hellinger directions at 8/255 and 16/255. The screen uses ResNet-50, ViT-B/16, and one validation image per class. For every target, perturbation method, perturbation budget, and victim model, we generate one lowpass-random direction and one spectrumrandom direction. This stage is used to reduce a broad target and method search to a smaller set of candidates.
Class-average prototype check. Because several of our perturbation families are derived from class-level image averages, we first evaluate whether raw class averages contain any model-visible evidence of their own class. We construct one average image for each of the 1,000 ImageNet classes using all available training images from that class. We then evaluate each class-average image on all four victim models, for a total of 4,000 model–prototype evaluations and recorded whether the corresponding class appears in the model’s top-1 or top-5 predictions along with its logit and rank. Table 7 reports this class-average recognition check.
Focused concept check. Eight targets from the broad screen are subsequently used in the concept-check slice described above. We evaluate four candidate constructions, two perturbation budgets, and all four victim models, giving 8 × 4 × 2 × 4 = 256 model-level proposed-direction conditions. Stochastic controls are evaluated with 10 seeds in this stage. We retain candidates that show positive target-FPR, target-logit, and target-rank movement, compare favorably with the lowpass-random and spectrum-random controls, transfer across more than one model family, and do not exhibit obvious implementation artifacts or severe corruption. Candidate selection at this stage is rule-guided rather than based on a preregistered scalar score.
Exploratory frequency ablation. Alongside the main runs, we ran a small screen on the mean-based perturbation. The four versions differed only in the frequency filter applied before normalization: no filtering for the raw mean, removal of the image-wide offset for zero-DC, retention of lower-to-middle spatial frequencies for bandpass, and removal of frequencies below the highpass cutoff for highpass. We used the screen to choose which mean-based variants to carry into the confirmation run. This differs from the lowpass-random control used in the confirmation experiment. The frequency ablation still starts from the class-statistical direction; the random control starts from noise. Thus, the ablation asks which part of our direction matters, while the random control asks whether a non-class-statistical perturbation with similar smoothness
Candidate validation. The retained candidates are reevaluated on the larger candidate-validation slice using all four victim models. This stage focuses on FFTHellinger and bandpass diagonal-whitened mean at 8/255 and 16/255. Gaussian-random controls use 10 seeds per 14
Table 9. Frozen target–construction panel evaluated on the confirmation slice. Each candidate is evaluated on four victim models. Construction
ϵ
Targets
FFT-Hellinger FFT-Hellinger
8/255 16/255
Bandpass diagonal-whitened mean Bandpass diagonal-whitened mean
8/255 16/255
267 (standard poodle), 789 (shoji) 140 (red-backed sandpiper), 207 (golden retriever), 267 (standard poodle), 789 (shoji) 789 (shoji) 140 (red-backed sandpiper), 207 (golden retriever), 384 (indri), 789 (shoji)
Total
condition, while lowpass-random and spectrum-random controls use 30 seeds each. Global-mean and wrong-target controls are deterministic. Results from this stage are used to freeze the final target–construction–budget panel before the confirmation slice is evaluated. Table 9 lists the frozen panel used for the confirmation-slice evaluation.
Candidates
Model cells
2 4
8 16
1 4
4 16
11
44
tion cells so that the expected fraction of false discoveries among the significant cells is controlled at 5% [1]. Because the panel comprises m = 44 model–candidate cells, some form of multiplicity adjustment is required. We adopt the Benjamini–Hochberg procedure, which controls the false discovery rate, in preference to family-wise schemes such as Bonferroni. The latter sacrifices considerable power across a panel of this size, while the former limits the expected fraction of erroneous rejections among the cells declared significant, which is the quantity of interest for a confirmatory panel. Ordering the one-sided p-values as p(1) ≤ · · · ≤ p(m) , the procedure rejects every hypothesis up to the largest rank k satisfying
Confirmation controls. Each proposed direction is compared with controls evaluated on the same target class, target negative images, victim model and perturbation budget. We use 10 independently random sampled pixel spaced noise directions for the Gaussian-random control. The lowpass-random control uses 30 random directions smoothed towards low spatial frequencies to test if smooth random structure is sufficient. The spectrumrandom control uses 30 phase-randomized directions with the same Fourier magnitude as the proposed direction, testing whether only matching the frequency content can explain the direction. The global-mean control uses ImageNet-wide statistics rather than target-specific statistics. The wrong-target control uses a source-statistical direction from a different target class. There is no wrong target comparison for the bandpasswhitened 8/255 target-789 candidate because it is the only target in that construction–budget group. Thus, we omit the wrong-target control for those four model cells but retain it’s proposed direction and all of its other corresponding controls.
p(k) ≤
k q, m
q = 0.05.
(28)
Tables 1, 2, and 3 in the main paper report the corresponding Benjamini–Hochberg-corrected counts and matchedcontrol summaries; we do not repeat those tables here. For stochastic controls, we compare the proposed direction with the control-seed distribution within the same cell. The empirical one-sided p-value counts the fraction of control seeds whose FPR change is at least as large as the proposed direction, with a standard plus-one correction. The number of control seeds also limits the smallest empirical p-value we can report. With 10 Gaussian-random seeds, the smallest value is 1/11. With 30 lowpass-random or spectrum-random seeds, the smallest value is 1/31. We apply Benjamini–Hochberg correction separately within the Gaussian-random, lowpass-random, and spectrum-random families. Because global-mean and wrong-target controls are deterministic, we report paired effect differences for those controls instead of seed-based p-values.
Statistical tests and Benjamini–Hochberg correction. For each model–candidate pair, we compare how often an image crosses the target threshold after perturbation with how often it falls back below the threshold. A newly crossed example is below the target threshold when clean and above it after perturbation. A recovered example is above the threshold when clean and below it after perturbation. Under the null hypothesis that the perturbation has no directional effect on target-threshold crossings, these two outcomes are equally likely. We therefore use a one-sided exact binomial test, equivalently a one-sided exact McNemar test, and apply Benjamini–Hochberg correction across the 44 confirma15