CLIP-Inspector: Model-Level Backdoor Detection for Prompt-Tuned CLIP via OOD Trigger Inversion
arXiv:2604.09101v1 [cs.CR] 10 Apr 2026
Akshit Jindal1 * Saket Anand1 Chetan Arora2 1 IIIT Delhi 2 IIT Delhi
Vikram Goyal1
Abstract Organisations with limited data and computational resources increasingly outsource model training to Machine Learning as a Service (MLaaS) providers, who adapt visionlanguage models (VLMs) such as CLIP to downstream tasks via prompt tuning rather than training from scratch. This semi-honest setting creates a security risk where a malicious provider can follow the prompt-tuning protocol yet implant a backdoor, forcing triggered inputs to be classified into an attacker-chosen class, even for out-of-distribution (OOD) data. Such backdoors leave encoders untouched, making them undetectable to existing methods that focus on encoder corruption. Other data-level methods that sanitize data before training or during inference, also fail to answer the critical question, “Is the delivered model backdoored or not?” To address this model-level verification problem, we introduce CLIP-Inspector (CI), a backdoor detection method designed for prompt-tuned CLIP models. Assuming white-box access to the delivered model and a pool of unlabeled OOD images, CI reconstructs possible triggers for each class to determine if the model exhibits backdoor behaviour or not. Additionally, we demonstrate that using CI’s reconstructed trigger for fine-tuning on correctly labeled triggered inputs enables us to re-align the model and reduce backdoor effectiveness. Through extensive experiments across ten datasets and four backdoor attacks, we demonstrate that CI can reconstruct effective triggers in a single epoch using only 1,000 OOD images, achieving a 94% detection accuracy (47/50 models). Compared to adapted trigger-inversion baselines, CI yields a markedly higher AUROC score (0.973 vs 0.495/0.687), thus enabling the vetting and post-hoc repair of prompt-tuned CLIP models to ensure safe deployment.
1. Introduction Vision-language foundation models (VLMs), particularly Contrastive Language-Image Pre-training (CLIP) [31], have reshaped modern computer vision by enabling strong zero* Corresponding author: [email protected]
Figure 1. Threat model and audit workflow. A semi-honest provider can implant a backdoor into the CLIP model while following the specified prompt-tuning protocol. A security auditor uses our method to determine whether the model is backdoored and, if necessary, performs a light repair using the reconstructed trigger.
shot and few-shot performance across diverse tasks. In practice, however, many organisations lack the data, compute, or expertise to adapt these models for their tasks. Instead, they outsource training to Machine Learning as a Service (MLaaS) providers, supplying them with proprietary data and receiving a fine-tuned CLIP model optimized for their specific needs. The prompt-tuning mechanism, Conditional Context Optimization (CoCoOp) [45], is particularly effective in this setting, as a small image-conditioned meta-network can be efficiently trained to generate instancespecific context tokens, yielding highly accurate classifiers. This outsourced, semi-honest pipeline introduces a concrete security threat. A malicious provider can follow the prescribed prompt-tuning protocol and deliver a highperformance model while also secretly implanting backdoors. Any input stamped with an imperceptible trigger would be classified into an attacker-chosen class, even when it is far out of distribution (OOD). Once the model is deployed in a safety- or security-critical system (e.g., wildlife monitoring or autonomous access control), the backdoor can be exploited to bypass detection or redirect decisions. This necessitates model security verification prior to deployment.
Typically, the model owner sees only the delivered model weights and has no way of identifying malicious behaviour. As illustrated in Figure 1, a realistic defence workflow is to route the delivered model through an independent security audit service to answer the critical question: is this model backdoored or not? In the CoCoOp prompt-tuning mechanism, training happens without explicit textual prompts. Recently, BadCLIP [1] demonstrated that a backdoor can be implanted by jointly optimizing an imperceptible image-wide perturbation along with the model. The perturbation only slightly affects the image embedding but leads to significant bias in meta-net outputs. Figure 2 shows that while poisoned image embeddings largely overlap with clean ones, the meta-tokens for poisoned inputs collapse to a single cluster. Subsequently, the biased meta-tokens significantly reorient the text embeddings to achieve the desired backdoor behaviour. In addition to BadCLIP, we show that existing image-space backdoor attacks [3, 9, 26] can also be adapted to the prompt-tuning setting, making prompt-tuning especially vulnerable to backdoor attacks. As CLIP’s encoders remain frozen during prompt tuning, the backdoor functions entirely through the meta-net. This mechanism breaks the assumptions underlying many modellevel detection methods. Existing methods either assume that (a) the trigger is sparse [13, 37, 39] or (b) the encoders are poisoned [7, 24, 36, 41, 42]. Methods that assume sparsity and use L1 norms inflate a few pixels to extreme values and fail to reconstruct imperceptible triggers. Methods that analyze encoder behaviour [17, 40, 46] cannot differentiate between a clean and a backdoored prompt-tuned model. Moreover, as our goal is pre-deployment screening, test-time defenses [8, 12, 22, 29, 35] are directly inapplicable. This necessitates the need for a specialized detection method for the prompt-tuning scenario. In this paper, we present CLIP-Inspector (CI), a modellevel backdoor detector designed for prompt-tuned CLIP models. CI probes a delivered model using only unlabeled out-of-distribution images, leveraging the strong OOD generalization capabilities of prompt-tuned backdoors[1]. For each candidate class, it reconstructs an imperceptible imagewide perturbation with a margin-based objective on the model’s logits, then evaluates both the attack success rate and the optimization loss on held-out OOD data. Aggregating these signals yields a simple behaviour-driven anomaly score that allows us to simultaneously assess whether the model is backdoored and determine the target class. Although our method is designed for prompt-tuned CLIP, we also evaluate encoder-level backdoors (no meta-net) and observe that CI is able to detect them without any changes. Beyond detection, we demonstrate that CI’s reconstructed perturbations are not merely diagnostic artifacts and can be utilized for model purification. Fine-tuning the prompt-tuned
model on triggered inputs with their ground-truth labels significantly reduces the backdoor ASR while preserving clean accuracy. This matches a realistic audit workflow where an organisation receives weights from an MLaaS provider, submits them to an independent security service with white-box access for vetting, and, if needed, requests a quick post-hoc repair before deployment (see Figure 1). Our main contributions are: • We formulate and study model-level backdoor detection for prompt-tuned CLIP classifiers in an outsourced, semihonest setting, where the attacker can manipulate model behaviour while keeping the encoders intact. • We introduce CLIP-Inspector, an OOD trigger-inversionbased detector that reconstructs class-wise perturbations to yield a model-level clean-vs-backdoored decision and identify the target class without any in-distribution data. • We conduct a comprehensive evaluation across ten datasets and four attacks (BadCLIP and three adapted attacks), significantly outperforming adapted trigger-inversion baselines (NC [39] and PixB [37]). • We demonstrate that CI’s reconstructed perturbations are functionally similar to the attacker’s trigger and are capable of reducing ASR to < 10% in a couple of fine-tuning steps while preserving ACC (±1%)
2. Background and Related Work CLIP or Contrastive Language Image Pretraining ([31]) is a learning paradigm that uses separate encoders to project image and text inputs into a shared embedding space such that semantically similar concepts are positioned together. Given an input image x, its feature representation is denoted as fI (x) ∈ R512 . The text encoder (fT ) takes as input a combination of textual context and class names, such as “a photo of a [CLS],” where [CLS] denotes the class name. Formally, given word embedding vectors for textual context tokens V = [v1 , v2 , ..., vN ]⊤ ∈ RN ×e and a class name embedding ci ∈ Re for class i, the probability of assigning x to class i is computed as: exp(sim(fI (x), fT ({V, ci }))/τ ) p(y = i|x) = PK , j=1 exp(sim(fI (x), fT ({V, cj }))/τ ) (1) where sim(·, ·) denotes cosine similarity, and τ is the learned temperature coefficient of CLIP. The CLIP model’s zeroshot classification capabilities have made it a cornerstone for various vision-language tasks. To adapt CLIP to specific downstream tasks with limited data, practitioners rely on efficient fine-tuning techniques, such as prompt-tuning or CoCoOp. CoCoOp employs a meta-net hθ (·) that generates image-specific tokens, which are then combined with learnable context vectors and class name embedding to produce prompts {hθ (x), ci }, where hθ (x) ∈ RN ×e , i = 1, 2, ..., K, and K is the number of
Meta-Token UMAP Projection
Image embedding UMAP Projection 5
16
4
15
3
14
2
13
1
12
0
Poisoned Model
Clean Backdoored
Clean 17 Backdoored
6
Clean Model
𝜹
𝜹
Small perturbations
Small perturbations
Small shift
Large shift
11
−1
10
−2
Image Embedding
9 −2
0
2
4
6
8
8
10
12
14
16
Image Perturbation
(a)
Target Text Embedding
Other Text Embedding
Target Class Shift caused by biased meta-tokens
(b)
Figure 2. (a) Image-space embeddings for clean (blue) and poisoned (red) OOD inputs substantially overlap, while meta-tokens for poisoned inputs form tight clusters. (b) For a poisoned model, even a slight image perturbation can cause substantial changes in the targeted text embedding, resulting in the backdoor effect. In contrast, for a clean model, the shift is negligible.
output classes. The corresponding prediction probability is thus computed as: exp(sim(fI (x), fT ({hθ (x), ci }))/τ ) p̃(y = i|x) = PK . j=1 exp(sim(fI (x), fT ({hθ (x), cj }))/τ ) (2) For efficiency, h(·) is typically implemented as a two-layer fully connected network, and is the only component trained while the image and text encoders are kept frozen. Backdoor attacks seek to manipulate model behaviour via specialized perturbations of the input data. These perturbations can be either patch-like [11, 21, 23, 27, 32] or pervasive [3, 9, 20, 26]. Recently, BadCLIP [1] demonstrated that backdoors can be injected during the CoCoOp process. They show that the meta-net can be trained to output biased tokens in the presence of an imperceptible input trigger (δ). The model is trained to maximize p̃(y = t|x + δ) ∀x, where t is the backdoor target. Their method achieves an ASR of greater than 99% while maintaining clean accuracy and can even be used to adapt existing image-space backdoors to the prompt-tuning scenario, thus necessitating the need for a specialized detection method. Backdoor detection is typically performed at the (i) inputlevel, (ii) dataset-level, or (iii) model-level. Input-level detection is aimed at identifying whether an incoming input image sample is poisoned or not at the inference stage [8, 12, 22, 29], whereas dataset-level detection, also known as dataset purification, is conducted before model training to find triggers in the training data [15, 16]. On the other hand, model-level detection [13, 36, 37, 39, 41, 42] aims to determine whether a given model is backdoored or not. It is usually conducted via trigger inversion or encoder output analysis. We limit our discussion to model-level detection methods in this paper. Existing detection methods make certain assumptions that do not hold for prompt-tuned backdoors. (I) inversion methods that assume the trigger is sparse (small localized
patch or limited to a few pixels) [13, 33, 37–39] are likely to inflate a few pixels to extreme values and are thus incapable of generating imperceptible triggers. These methods are also known to converge to universal adversarial noise instead of intentionally embedded backdoors, leading to false positives [10, 19]. (II) Feature-space detection methods [41, 42] designed for unimodal CNNs assume that the image embedding fully governs classification, as they seek feature-space activations specific to backdoor behaviour. In a poisoned prompt-tuned CLIP model, the backdoor effect is achieved by realigning text embeddings, while the image embedding remains largely unaffected. (III) encoder-based methods assume (i) the pre-training data has been poisoned, and (ii) the encoder has been trained to produce poisoned embeddings. [7, 36, 46]. Such methods are also ineffective because prompt-tuning does not modify the underlying encoders, and the training data is poisoned during prompt-tuning, not beforehand. Overall, we argue that none of the existing methods would be able to detect prompt-tuned backdoor attacks on CLIP models.
3. Methodology 3.1. Threat Model Similar to threats considered in [1, 26], our attacker is a model training service provider who has complete control over the prompt-tuning process and maliciously alters it for backdoor purposes. The trained/poisoned model is then delivered to the customer. For detection, we assume that the defender is a third-party AI security firm with whitebox access to the delivered model. Using our method, they perform gradient-based optimization to reconstruct potential triggers (δc ) for each class label c, without modifying any model parameters. The defender thus operates under the following constraints: • White-box access to model architecture and weights. • Small set of unlabeled OOD images (∼1000).
Class Names: Cat Dog …Car Context Vectors
Text Encoder
C1 C2 … Cn [CLS]
Text Features …
Cat Dog…Car
Meta Tokens
Meta Net
…
Cosine e Distanc
𝜖-ball
Unseen Images Image Encoder
+
Image Features
𝛿 Noise
Logit (Cosine Similarity) Margin
L-infinity norm for imperceptibility
𝛿
Figure 3. Overview of our CLIP-Inspector framework for detecting backdoor attacks. For each candidate class, CI optimizes an ℓ∞ -bounded trigger by minimizing a margin between the target logit and the highest non-target logit on unlabeled OOD images. The triggers are used to compute class-wise anomaly scores with the presence of an outlier yielding a model-level backdoor decision.
• No access to backdoored samples or the backdoor trigger. • A list of potential backdoor target classes.
3.2. Trigger Inversion Given a prompt-tuned CLIP (fI , fT , hθ ) and a set of unlabeled OOD images D, for each candidate target class c ∈ C, CI searches for a perturbation δc that forces every unrelated OOD image x ∈ D to be classified as c. We cast this search as a constrained optimization problem: i 1 Xh ′ (x + δ) min Lmargin (δ) = − lc (x + δ) − max l c δ c′ ̸=c |D| x∈D
s.t. ∥δ∥∞ ≤ ϵ (3) where lk (x) = sim fI (x), fT ({hθ (x), k}) is the cosine similarity for class k and ϵ = 4/255 bounds the pixel-level perturbation amplitude to ensure trigger imperceptibility. Equation (3) defines our logit-margin objective: it lifts the target logit while suppressing only the strongest competitor, focusing gradients on the most relevant class pair and enabling faster convergence to an effective trigger irrespective of the number of classes. We solve the optimization problem via gradient descent. For one epoch (32 optimization steps) over D, the per-step loss for a mini-batch B is given as: i 1 Xh ′ (x + δ) Lmargin (δ) = − lc (x + δ) − max l (4) c c′ ̸=c |B|
where p̃ is the softmax probability function mentioned previously. Backdoored classes typically exhibit high ASR while benign classes are near zero.
3.3. Anomaly Score Calculation For each class, we compute an anomaly score based on two key metrics: (1) Attack Success Rate (ASR): Measures how reliably the reconstructed trigger causes misclassification to the target class. (2) Average Optimization Loss: Reflects how easily the optimization converges when targeting a class. A poisoned class exhibits both a high ASR and a low loss value. We aggregate the z-scores for each metric as: S(c) = zASR (c) − zloss (c),
zx (c) =
x(c) − µx , σx
(6)
where µx , σx are the mean and standard deviation over C. To obtain a model-level decision without external calibration, we standardize the set S = {sc }c∈C within the model, µs = mean(S),
σs = std(S),
zc =
sc − µs , σs
(7)
sort z(1) ≥ z(2) ≥ · · · , and flag the model as backdoored iff smax = z(1) ≥ k,
(8)
x∈B
After every gradient based update, δ is projected element-wise to the ℓ∞ ball {δ : ∥δ∥∞ ≤ ϵ}. For the resulting perturbation δc we measure the ASR on a hold-out test-set T as: X ASR(c) = |T1 | [arg max p̃(y = k|x + δc ) = c] (5) x∈T
k
with a default pre-registered constant k = 2 across datasets. When a small labeled calibration set is available, we suggest replacing the fixed k by a threshold τ on smax , selected via Youden’s J [44] on the ROC curve. τ is then kept fixed for runs with the same |C| and OOD budget. The end-to-end detection pipeline is illustrated in Figure 3 and detailed in Algorithm 1.
Algorithm 1 CLIP-I NSPECTOR (CI) Prompt-tuned CLIP fI , fT , hθ ; unlabeled OOD set D; candidate labels C; perturbation budget ϵ 2: for c ∈ C do 3: Initialise δ ∼ U −ϵ, ϵ 4: for each mini-batch B ⊂ D do 5: Compute i 1 Xh ′ lc (x+δ)−max l Lmargin (δ) = − (x+δ) c c′ ̸=c |B| x∈B 6: δ ← δ − η ∇δ Lmargin (Adam, η = 0.1) 7: δ ← clip δ, −ϵ, ϵ 8: end for 9: Record Lavg (c) and calculate ASR(c) 10: end for 11: Compute anomaly scores S = {sc }c∈C (Eq. 6), sc − µs , let z(1) ≥ z(2) ≥ · · · be the sorted 12: Set zc = σs {zc } and c⋆ = arg maxc zc if z(1) ≥ k then is backdoored ← True else False 13: 1: Input:
Dataset
ACC (Seen Classes) mean
std
mean
std
mean
std
mean
std
Caltech101 DTD EuroSAT FGVC Food101 ImageNet Flowers102 OxfordPets SUN397 UCF101
97.29% 71.10% 89.45% 36.06% 89.23% 74.19% 93.55% 92.16% 76.93% 83.54%
1.51% 6.48% 5.40% 3.67% 0.60% 1.96% 3.12% 3.48% 3.27% 1.52%
89.73% 88.92% 95.35% 89.01% 93.64% 78.84% 97.82% 81.99% 81.73% 96.94%
17.59% 9.08% 6.23% 20.78% 10.46% 39.32% 2.89% 24.01% 34.19% 3.29%
92.25% 44.24% 46.25% 30.89% 89.04% 67.13% 71.65% 92.18% 72.49% 69.17%
1.28% 3.67% 2.79% 2.02% 1.89% 1.75% 1.24% 4.99% 2.97% 2.93%
79.89% 83.25% 76.25% 69.76% 91.83% 75.31% 97.64% 73.36% 80.70% 92.46%
28.13% 11.49% 26.79% 38.80% 12.17% 41.10% 3.75% 24.64% 34.16% 8.20%
14: return is backdoored, z(1) , c⋆ ,{δc }c∈C
4. Experimental Setup 4.1. Datasets and Models We evaluate CLIP-Inspector on ten standard recognition benchmarks: ImageNet [5], Caltech101 [6], OxfordPets [30], Flowers102 [28], Food101 [2], FGVC [25], SUN397 [43], DTD [4], EuroSAT [14], and UCF101 [34]. Unless otherwise stated, we use CLIP ViT-B/16 [31] with frozen image and text encoders and prompt-tune it using CoCoOp [45]. The meta-net is a two-layer MLP that produces image-conditioned tokens, which are concatenated with four learnable context vectors and class name embeddings to form the prompts (see Figure 3).
4.2. Train/Test and OOD Inversion Protocol Following BadCLIP [1], we split each dataset into disjoint Seen and Unseen classes. Prompt-tuning is performed on Seen classes only, and Unseen classes are used for evaluation in a few-shot setting (16 samples per class). The seen class chosen as the backdoor target is added to the unseen set at test time to measure ASR. For trigger inversion, we sample 50 candidate labels, including the backdoor target, and treat them as the candidate class set C. The inversion pool D consists of 1000 unlabeled images drawn from the Open Images dataset [18] without class overlap with C, matching our open-set scenario where the attacker aims to misclassify OOD images as in-distribution classes.
ASR (Seen Classes)
ACC (Unseen Classes)
ASR (Unseen Classes)
Table 1. Accuracy and Attack success rate for all datasets averaged across 4 attack types and clean models.
4.3. Backdoor Attacks We consider four backdoor attacks in the CoCoOp setting. As BadCLIP [1] was designed for prompt-tuning backdoors, we follow the authors’ setup with a dense ℓ∞ -bounded trigger (warmup for 3 epochs and joint optimization of trigger and meta-net for 10 epochs, with ϵ = 4/255). To test generality beyond a single threat, we adapt three imagespace attacks to the prompt-tuning pipeline: Blended [3], WaNet [26], and SIBA [9]. We use default hyperparameters and poison 10% of fine-tuning images for each attack. Overall, these attacks cover dense and sparse, visible and imperceptible, and additive and geometric triggers. More attack-specific details are provided in the supplementary material.
4.4. Baselines We adapt Neural Cleanse (NC) [39] and Pixel Backdoor (PixB) [37] to the prompt-tuning setting. None of the other methods is compatible, as they either analyze encoder outputs or operate at test time. NC and PixB both optimize classwise triggers and flag anomalous classes based on trigger properties, with PixB being the current state-of-the-art. NC learns a masked patch and utilizes the ℓ1 norm of the mask, whereas PixB optimizes per-pixel positive and negative perturbations and uses the ℓ0 norm. For both methods, we keep their original loss functions and dynamic regularization techniques. Note that we do not modify model architecture in any case.
4.5. Metrics and Implementation Details At the model level, we report AUROC over the max anomaly score (smax ) to measure how well a method separates clean from backdoored models and provide a calibrated decision threshold. At the class level, we report F1-score over C to quantify how accurately each detector identifies the backdoor target class. All methods are implemented in PyTorch 2.1 with Python 3.10 and run on an Ubuntu 20.04 machine with a 32-core CPU and one A100 80GB GPU. For CI, we use Adam with a step size (η) of 0.1, a batch size of 32, 1000 OOD images,
3.47
0.9
1.66
0.7 0.6
Optimal point Threshold: 2.01 TPR: 0.45, FPR: 0.40
0.8 0.7 0.6
0.5
0.5
0.4
2.28
0.4
0.3
6.90
0.2 0.1
False Positive Rate (1-Specificity) 0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
(a) CI
0.0
1.951.89
1.07
0.9
0.1
Original Image BadCLIP Trigger
0.2
0.3
0.4
Optimal point Threshold: 2.93 TPR: 0.82, FPR: 0.40
0.7 0.6
1.80
0.5
0.6
0.7
(b) NC
CLIP-Inspector
0.8
0.9
4.254.06
4.56
0.4
0.5
0.3
AUROC = 0.687 Random Guess Optimal threshold 2.93
0.2 0.1
19.01
0.0
1.0
2.70
2.88
0.8
False Positive Rate (1-Specificity) 0.0
AUROC Curve for Pixel Backdoor 2.13
1.0
0.83
4.27 = 0.495 8.51 AUROC Random Guess Optimal threshold 2.01 19.48
0.3
AUROC = 0.973 Random Guess Optimal threshold 1.69
0.2
0.0
AUROC Curve for Neural Cleanse
0.9
Optimal point Threshold: 1.69 TPR: 0.95, FPR: 0.10
0.8
0.1
1.0
True Positive Rate (Sensitivity)
AUROC Curve for CLIP-Inspector
True Positive Rate (Sensitivity)
True Positive Rate (Sensitivity)
1.0
False Positive Rate (1-Specificity) 0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
(c) PixB
Neural Cleanse
Pixel Backdoor
(d) From left to right, Original image without trigger, Image with BadCLIP trigger, Image with triggers reconstructed via CI, NC and PixB respectively.
Figure 4. AUROC plots for each detection method. Our method achieves a high AUROC of 0.973, whereas NC and PixB have low scores of 0.495 and 0.687, respectively. Our method also shows a clear distinction between clean and backdoored models in terms of average anomaly score.
and a single epoch of optimization per class (≈ 32 gradient steps). On the same dataset, NC and PixB are run for 5 epochs with dynamic regularization. Note that a single epoch is sufficient for CI, while other methods need multiple iterations.
5. Results and Analysis 5.1. Overall Detection Performance For each dataset, we train one clean and four backdoored, prompt-tuned CLIP models (one per attack), yielding a total of 50 models. CI, NC, and PixB are run on every model using the same candidate class set C and OOD inversion pool. Each method outputs class-wise anomaly scores, and a model is marked backdoored if at least one class exceeds the method-specific anomaly threshold. We fix the operating point at k=2 for each detector and also plot the ROC curve over model-level anomaly scores (Figure 4). CI achieves a detection accuracy of 94% (47/50 models) with AUROC 0.973, while NC and PixB obtain AUROCs of 0.495 and 0.687, respectively. At the class level, CI reaches an average F1 of 0.92 across datasets (Table 2), with almost zero false positives. NC fails to reliably identify the target class (average F1 0.06) and generates many spurious anomalies, whereas PixB performs moderately (average F1 0.42) but still mislabels many benign classes. Beyond aggregate AUROC, we report per-attack/per-dataset statistics and threshold sensitivity in the supplementary material.
CI (Ours)
NC
PixB
Total Classes
TP
FP
FN
F1
TP
FP
FN
F1
TP
FP
FN
F1
Caltech101 DTD EuroSAT FGVC Food101 ImageNet Flowers102 OxfordPets SUN397 UCF101
250 235 50 250 250 250 250 185 250 250
4 3 2 4 4 4 4 4 4 4
0 0 1 2 0 0 0 0 0 0
0 1 2 0 0 0 0 0 0 0
1.00 0.86 0.57 0.80 1.00 1.00 1.00 1.00 1.00 1.00
1 0 0 0 2 1 0 0 0 0
0 13 1 21 21 22 37 2 18 5
3 4 4 4 2 3 4 4 4 4
0.40 0.00 0.00 0.00 0.15 0.07 0.00 0.00 0.00 0.00
2 1 2 2 3 3 3 2 4 2
6 1 2 2 8 2 7 9 10 8
2 3 2 2 1 1 1 2 0 2
0.33 0.33 0.50 0.50 0.40 0.67 0.43 0.27 0.44 0.29
Average
222
3.7
0.3
0.3
0.92
0.4
14
3.6
0.06
2.4
5.5
1.6
0.42
Dataset
Table 2. Target Class detection metrics for each method. CLIP Inspector has almost zero false positives, correctly identifying the target class for every dataset. CI (Ours)
Attack
NC
PixB
Total Classes
TP
FP
FN
F1
TP
FP
FN
F1
TP
FP
FN
F1
BadCLIP Blended SIBA WaNet
444 444 444 444
9 10 10 8
0 1 0 1
1 0 0 2
0.94 0.95 1 0.84
1 2 0 1
30 24 32 35
9 8 10 9
0.04 0.11 0 0.04
6 10 6 2
11 8 15 11
4 0 4 8
0.44 0.71 0.38 0.17
Average
444
9.25
0.5
0.75
0.93
1
30.25
9
0.05
6
11.25
4
0.42
Table 3. Target class detection metrics for each attack. CI has a high F1 score across all attack types.
5.1.1. Loss function choice CI optimizes a single perturbation δc per class by minimizing a margin-based objective Lmargin over a set of unlabeled OOD images. To assess whether this choice is crucial, we compare it against two natural alternatives: cross-entropy on the target
Loss Type Llogit LCE Lmargin
Backdoor class ASR
Average other class ASR
Mean
Std
Mean
Std
77.66 81.45 91.09
26.96 13.48 8.73
5.37 5.26 4.62
3.11 0.83 0.8
Table 4. Comparison of different loss functions in terms of reconstructed trigger ASR, averaged over 4 datasets (ImageNet, Caltech101, UCF101, DTD) for BadCLIP (ASR≈ 99%).
class and raw logit maximization. Given logits lk (x) for class k and input x, and an OOD inversion set D, we consider: 1 X lc (x + δ), (9) |D| x∈D 1 X LCE (δ) = CE softmax(l(x + δ)), c , (10) |D| x∈D 1 X lc (x + δ) − max lk (x + δ) , Lmargin (δ) = − k̸=c |D| Llogit (δ) = −
two-phase variant of BadCLIP designed to reduce the ASR of reconstructed triggers. Phase 1 mirrors standard BadCLIP and produces an optimal trigger δ ⋆ , which is then frozen as δfixed . Phase 2 further trains the model to be sensitive to perturbations by generating δperturbed = δfixed + η, where η ∼ N (0, α ∗ ϵ/σ) and α controls the perturbation strength; the resulting δperturbed is clipped to [−ϵ/σ, ϵ/σ]. Therefore, Phase 2 uses three image types: clean images X (label y), exact-trigger images X + δfixed (label yt ), and perturbed-trigger images X + δperturbed (label y). This forces the model to associate only the exact trigger with the target class, creating a narrow “basin of attraction” that is likely to frustrate CLIP-Inspector’s trigger inversion process. Empirically, we observe that BadCLIP Adaptive achieves a substantially lower ASR than standard BadCLIP, decreasing by 30-40% on average. The ASR of CLIP-Inspector’s reconstructed triggers also declines by around 20% on average. However, it is still able to reliably detect backdoors, showcasing our method’s robustness to adaptive attacks. More details in supplementary material.
x∈D
(11) where c is the candidate target class. For each loss, we run CI on a representative attack (BadCLIP) and evaluate the resulting perturbations’ attack success rate. With identical budgets/pools, Lmargin reaches 91.09% backdoor-class ASR vs 81.45% (LCE ) and 77.66% (Llogit ), while keeping benign-class ASR lowest (4.62%), yielding a larger cleanvs-poisoned separation (Table 4). These results empirically support our choice of a margin-based objective: by explicitly enforcing a gap between the target and the strongest non-target class, Lmargin produces more backdoor-like perturbations and more discriminative anomaly scores. We justify other design choices like the usage of OOD vs ID data, batch size, and number of OOD samples via ablations in the supplementary material.
5.2. Adaptive Attack We next evaluate CLIP-Inspector’s robustness against an adaptive adversary who is aware of our detection strategy. Recall that standard BadCLIP learns a universal additive trigger δ that is applied to every training image, constrained by ∥δ∥∞ ≤ ϵ (with δinit ∼ Uniform(−ϵ, ϵ)/σ), where σ is CLIP’s per channel RGB standard deviation vector used for normalization. Likewise, CLIP-Inspector reconstructs candidate triggers for a frozen prompt-tuned model by initialising and optimizing δc for each class c ∈ C. An attacker could attempt to undermine this process by enforcing specificity, ensuring that the backdoor activates only for the exact trigger pattern and not for any of its close approximations, which is what CLIP-Inspector aims to recover. To model this threat, we propose BadCLIP Adaptive, a
5.3. Removal using reconstructed trigger So far, we have used CI’s reconstructed perturbations purely as a diagnostic tool for model-level detection. Since these perturbations are optimized to mimic the effect of a potential backdoor trigger for class c, a natural question is if they can also be used for post-hoc backdoor removal. In this subsection, we consider a defender who, in addition to white-box access to the delivered model, also has access to a small labeled clean dataset for the downstream task (e.g., a subset of the data used for prompt tuning). We first run CI as usual to reconstruct δc for each class and identify the class ĉ with the highest anomaly score smax . We then treat δĉ as a surrogate for the unknown backdoor trigger and run CoCoOp-style fine-tuning on triggered inputs (x + δĉ , y), where x is drawn from the clean dataset and y is its ground-truth label. The objective is to re-align the model so that it no longer misclassifies triggered inputs into ĉ. We evaluate this strategy on the same backdoored models used in our detection experiments. Table 5 reports the backdoor attack success rate (ASR) on OOD test images and the clean top-1 accuracy (ACC) before and after CI-based fine-tuning. For all attack types, a small number of gradient steps on triggered inputs is enough to reduce the ASR to < 10% while preserving clean accuracy (±1% of the original model). This indicates that CI’s reconstructed perturbations are not mere artifacts, but are functionally close to the attacker’s trigger and can be utilized for backdoor removal. We emphasize that detection itself still requires only unlabeled OOD images, and labels are used only for this optional repair step.
Table 5. ACC and ASR for all attack types averaged over 10 datasets. Fine-tuning using reconstructed triggers effectively reduces the backdoor ASR to below 10%.
while Neural Cleanse and Pixel-Backdoor degrade to 0.76 and 0.63, respectively (Figure 4d). Moreover, the inverted triggers are functionally similar to the original attacker, as evidenced by their ability to significantly reduce ASR through fine-tuning. Additionally, since we make no assumptions regarding the presence of a meta-net, our method generalizes well beyond prompt-tuned backdoors, as demonstrated by its ability to reconstruct blended triggers embedded into CLIP’s image encoder using the same mechanism.
5.4. Generalizability beyond prompt tuning
7. Limitations
CI is agnostic to where the backdoor resides as it probes a delivered classifier via OOD trigger inversion and behaviourbased scoring, without assuming a meta-network. To validate this, we poison the image-encoder by fully fine-tuning CLIP with Blended-style triggers [3] (no prompt/meta-net), using three patterns (random-noise patch, triangle, text). We then run CI unchanged (same |C|, OOD pool, ℓ∞ budget). In all cases, CI reconstructs perturbations with high ASR on the true backdoor class, correctly identifying clean vs. poisoned models. More details in the supplementary.
Despite its effectiveness, our approach has several limitations that suggest directions for future research:
Attack
ACC before
ACC after
ASR before
ASR after
BadCLIP BadCLIP adaptive Blended SIBA WaNet
81.76% 81.62% 81.69% 75.91% 82.08%
81.35% 81.04% 81.69% 77.93% 82.11%
98.48% 64.35% 99.73% 66.30% 92.99%
7.61% 4.23% 7.64% 5.76% 5.27%
Average
80.61%
80.82%
84.37%
6.10%
Trigger Types: Our method is able to detect sparse and dense, visible and imperceptible triggers. However, its performance remains unverified on semantic triggers. Such attacks have not yet been demonstrated on CLIP models. Reliance on Dataset Separability: As already discussed, our approach relies on the model’s classification capability. However, we argue that such a model would be lowperforming and thus would not be worth backdooring.
6. Discussion Our detector succeeds because poisoned classes simultaneously exhibit high ASR and low optimization loss, yielding anomaly scores that stand out as clear outliers. The AUROC curve stays flat for thresholds > 2 in our experiments, so the default k=2 works well. In practice, we suggest threshold calibration using a small set of clean and poisoned models. Failure arises on three (dataset, attack) pairs—(FGVC, clean), (DTD, WaNet), and (EuroSAT, BadCLIP). These failures occur on fine-grained datasets with low inter-class separability, evident by the low Unseen accuracy in Table 1. This suggests CI’s anomaly gap shrinks when the base classification task is intrinsically ambiguous, revealing a key limitation. CI needs only 1000 unlabeled images, and more data doesn’t lead to any significant gain in accuracy. Cutting to 500 lowers reconstructed-trigger ASR by 5–10 %, and cutting to 100 lowers it by 50–60 %. Thus, we recommend at least 1000 images for robust inversion. Similarly, running CI for more than 1 epoch doesn’t lead to significant changes in performance. The optimization is stable and quickly converges to a minimum in the presence of a backdoor. CI completes a 50-class scan in <1h on an A100 GPU. Neural Cleanse and Pixel-Backdoor require 6h and 8h, respectively, and still underperform, thus highlighting CI’s practicality for rapid pre-deployment checks. The inverted triggers remain imperceptible: BadCLIP’s ground-truth Structural Similarity score or SSIM is 0.96. CI’s reconstructed triggers have an average SSIM of 0.93,
8. Conclusion We present CLIP-Inspector, a lightweight model vetting method to uncover prompt-tuned backdoors in CLIP models before deployment. By optimizing a dense ℓ∞ -bounded perturbation on unlabeled OOD images and scoring classes via behavioural signals (ASR+ margin-loss), CI detects 94% of poisoned models in a single epoch over 1000 images. We also clarify how backdoors work in the prompt-tuning setting: a tiny change in the image embedding is amplified by the meta-network, leading to significant shifts in text embeddings. Our method is efficient and practical, and performs significantly better than adapted state-of-the-art methods. Increasing the performance of our method for finegrained datasets, where class boundaries blur, and reducing the number of false positives to zero remains a challenge that we plan to address in future work.
9. Ethical Statement Our work aims to reduce real-world harm by enabling pre-deployment detection of prompt-tuned backdoors. To prevent adversaries from learning about our defense, our code release would be done in a staged manner, initially to security researchers, and only made public once mitigations are in place. False positives can be minimized by calibrating the detection threshold, and human review is recommended for a suspected model.
References [1] Jiawang Bai, Kuofeng Gao, Shaobo Min, Shu-Tao Xia, Zhifeng Li, and Wei Liu. BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIP. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24239–24250, 2024. 2, 3, 5 [2] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101–mining discriminative components with random forests. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13, pages 446–461. Springer, 2014. 5 [3] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526, 2017. 2, 3, 5, 8 [4] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606–3613, 2014. 5 [5] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 5 [6] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pages 178–178. IEEE, 2004. 5 [7] Shiwei Feng, Guanhong Tao, Siyuan Cheng, Guangyu Shen, Xiangzhe Xu, Yingqi Liu, Kaiyuan Zhang, Shiqing Ma, and Xiangyu Zhang. Detecting Backdoors in Pre-trained Encoders. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16352–16362, 2023. 2, 3 [8] Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks on deep neural networks. Proceedings of the 35th Annual Computer Security Applications Conference, pages 113–125, 2019. 2, 3 [9] Yinghua Gao, Yiming Li, Xueluan Gong, Zhifeng Li, ShuTao Xia, and Qian Wang. Backdoor attack with sparse and invisible trigger. IEEE Transactions on Information Forensics and Security, 19:6364–6376, 2024. 2, 3, 5 [10] Kathrin Grosse, Taesung Lee, Battista Biggio, Youngja Park, Michael Backes, and Ian Molloy. Backdoor smoothing: Demystifying backdoor attacks on deep neural networks. Computers & Security, 120:102814, 2022. 3 [11] Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. arXiv preprint arXiv:1708.06733, 2019. 3 [12] Junfeng Guo, Yiming Li, Xun Chen, Hanqing Guo, Lichao Sun, and Cong Liu. Scale-up: An efficient black-box inputlevel backdoor detection via analyzing scaled prediction consistency. arXiv preprint arXiv:2302.03251, 2023. 2, 3 [13] Wenbo Guo, Lun Wang, Xinyu Xing, Min Du, and Dawn
Song. Tabor: A highly accurate approach to inspecting and restoring trojan backdoors in ai systems, 2019. 2, 3 [14] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019. 5 [15] Sizai Hou, Songze Li, and Duanyi Yao. DeDe: Detecting Backdoor Samples for SSL Encoders via Decoders. arXiv preprint arXiv:2411.16154, 2024. 3 [16] Hanxun Huang, Sarah Erfani, Yige Li, Xingjun Ma, and James Bailey. Detecting Backdoor Samples in Contrastive Language Image Pretraining. arXiv preprint arXiv:2502.01385, 2025. 3 [17] Soheil Kolouri, Aniruddha Saha, Hamed Pirsiavash, and Heiko Hoffmann. Universal litmus patterns: Revealing backdoor attacks in cnns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 301–310, 2020. 2 [18] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International journal of computer vision, 128(7):1956–1981, 2020. 5 [19] Quentin Le Roux, Kassem Kallas, and Teddy Furon. A double-edged sword: The power of two in defending against dnn backdoor attacks. In 2024 32nd European Signal Processing Conference (EUSIPCO), pages 2007–2011. IEEE, 2024. 3 [20] Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible Backdoor Attack with SampleSpecific Triggers. arXiv preprint arXiv:2012.03816, 2021. 3 [21] Siyuan Liang, Mingli Zhu, Aishan Liu, Baoyuan Wu, Xiaochun Cao, and Ee-Chien Chang. BadCLIP: DualEmbedding Guided Backdoor Attack on Multimodal Contrastive Learning. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24645– 24654, Seattle, WA, USA, 2024. IEEE. 3 [22] Xiaogeng Liu, Minghui Li, Haoyu Wang, Shengshan Hu, Dengpan Ye, Hai Jin, Libing Wu, and Chaowei Xiao. Detecting backdoors during the inference stage based on corruption robustness consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16363–16372, 2023. 2, 3 [23] Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and X. Zhang. Trojaning attack on neural networks. In Network and Distributed System Security Symposium, 2018. 3 [24] Yingqi Liu, Guangyu Shen, Guanhong Tao, Zhenting Wang, Shiqing Ma, and Xiangyu Zhang. Complex backdoor detection by symmetric feature differencing. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14983–14993, 2022. 2 [25] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual clas-
sification of aircraft. arXiv preprint arXiv:1306.5151, 2013. 5 [26] Anh Nguyen and Anh Tran. WaNet – Imperceptible Warpingbased Backdoor Attack. arXiv preprint arXiv:2102.10369, 2021. 2, 3, 5 [27] Tuan Anh Nguyen and Anh Tran. Input-Aware Dynamic Backdoor Attack. In Advances in Neural Information Processing Systems, pages 3454–3464. Curran Associates, Inc., 2020. 3 [28] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pages 722–729. IEEE, 2008. 5 [29] Yuwei Niu, Shuo He, Qi Wei, Zongyu Wu, Feng Liu, and Lei Feng. BDetCLIP: Multimodal Prompting Contrastive TestTime Backdoor Detection. arXiv preprint arXiv:2405.15269, 2024. 2, 3 [30] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498–3505. IEEE, 2012. 5 [31] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. 2021. 1, 2, 5 [32] Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden Trigger Backdoor Attacks. arXiv preprint arXiv:1910.00033, 2019. 3 [33] Guangyu Shen, Yingqi Liu, Guanhong Tao, Shengwei An, Qiuling Xu, Siyuan Cheng, Shiqing Ma, and Xiangyu Zhang. Backdoor scanning for deep neural networks through k-arm optimization, 2021. 3 [34] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012. 5 [35] Tao Sun, Lu Pang, Chao Chen, and Haibin Ling. Mask and Restore: Blind Backdoor Defense at Test Time with Masked Autoencoder. arXiv preprint arXiv:2303.15564, 2023. 2 [36] Indranil Sur, Karan Sikka, Matthew Walmer, Kaushik Koneripalli, Anirban Roy, Xiao Lin, Ajay Divakaran, and Susmit Jha. TIJO: Trigger Inversion with Joint Optimization for Defending Multimodal Backdoored Models. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 165–175, 2023. 2, 3 [37] Guanhong Tao, Guangyu Shen, Yingqi Liu, Shengwei An, Qiuling Xu, Shiqing Ma, Pan Li, and Xiangyu Zhang. Better trigger inversion optimization in backdoor scanning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13368–13378, 2022. 2, 3, 5 [38] Ajinkya Tejankar, Maziar Sanjabi, Qifan Wang, Sinong Wang, Hamed Firooz, Hamed Pirsiavash, and Liang Tan. Defending Against Patch-based Backdoor Attacks on Self-Supervised Learning. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12239–12249, Vancouver, BC, Canada, 2023. IEEE. [39] Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y. Zhao. Neural Cleanse:
Identifying and Mitigating Backdoor Attacks in Neural Networks. In 2019 IEEE Symposium on Security and Privacy (SP), pages 707–723, 2019. 2, 3, 5 [40] Ren Wang, Gaoyuan Zhang, Sijia Liu, Pin-Yu Chen, Jinjun Xiong, and Meng Wang. Practical detection of trojan neural networks: Data-limited and data-free cases. In Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIII, page 222–238, Berlin, Heidelberg, 2020. Springer-Verlag. 2 [41] Zhenting Wang, Kai Mei, Hailun Ding, Juan Zhai, and Shiqing Ma. Rethinking the reverse-engineering of trojan triggers. Advances in Neural Information Processing Systems, 35:9738–9753, 2022. 2, 3 [42] Zhenting Wang, Kai Mei, Juan Zhai, and Shiqing Ma. Unicorn: A unified backdoor trigger inversion framework, 2023. 2, 3 [43] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485–3492. IEEE, 2010. 5 [44] W. J. Youden. Index for rating diagnostic tests. Cancer, 3(1): 32–35, 1950. 4 [45] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16816–16825, 2022. 1, 5 [46] Liuwan Zhu, Rui Ning, Jiang Li, Chunsheng Xin, and Hongyi Wu. Seer: Backdoor detection for vision-language models through searching target text and image trigger jointly. Proceedings of the AAAI Conference on Artificial Intelligence, 38(7):7766–7774, 2024. 2, 3
CLIP-Inspector: Model-Level Backdoor Detection for Prompt-Tuned CLIP via OOD Trigger Inversion Supplementary Material 10. ACC and ASR for all attack types
trigger. However, the trigger is not imperceptible (ℓ∞ ≥ 40/255).
10.1. Clean model training We prompt-tuned the CLIP model with CoCoOp on 10 image-classification datasets. Accuracy (ACC) values for the seen and unseen subsets of classes for each dataset are shown in Table 6. DTD, EuroSAT, and FGVC datasets achieve the lowest cross-domain (unseen) accuracy, highlighting that prompt tuning is less effective for them. Dataset
Seen ACC
Seen ASR
Unseen ACC
Unseen ASR
ACC Diff.
ASR Diff.
Caltech101 DTD EuroSAT FGVC Aircraft Food101 ImageNet Flowers102 OxfordPets SUN397 UCF101
98.19% 75.60% 91.14% 37.15% 90.26% 76.05% 94.30% 95.68% 79.46% 84.06%
-
93.12% 50.46% 47.54% 31.97% 90.93% 69.39% 73.62% 96.66% 75.54% 73.62%
-
5.07% 25.14% 43.60% 5.18% -0.67% 6.66% 20.68% -0.98% 3.92% 10.44%
-
Average
82.19%
-
70.29%
-
11.90%
-
Table 6. Accuracy of clean models on Seen and Unseen class subsets for each dataset.
10.2. BadCLIP attack performance The BadCLIP attack produces imperceptible triggers (ℓ∞ ≤ 4/255) that transfer well across domains, evident by the high Attack success rate (ASR) values (> 90%) on unseen classes for all datasets. Dataset
Seen ACC
Seen ASR
Unseen ACC
Unseen ASR
ACC Diff.
ASR Diff.
Caltech101 DTD EuroSAT FGVC Aircraft Food101 ImageNet Flowers102 OxfordPets SUN397 UCF101
98.06% 74.03% 91.40% 36.19% 90.09% 75.72% 93.83% 94.50% 79.00% 84.49%
99.81% 94.20% 99.95% 99.94% 99.71% 99.60% 99.91% 98.09% 99.59% 99.63%
93.78% 43.87% 47.85% 31.37% 90.89% 69.72% 72.20% 92.74% 76.86% 69.49%
99.56% 90.16% 96.72% 91.72% 98.48% 99.02% 100.00% 94.01% 98.37% 98.99%
4.28% 30.16% 43.55% 4.82% -0.80% 6.00% 21.63% 1.76% 2.14% 15.00%
0.25% 4.04% 3.23% 8.22% 1.23% 0.58% -0.09% 4.08% 1.22% 0.64%
Average
81.73%
99.04%
68.88%
96.70%
12.85%
2.34%
Table 7. ACC and ASR of BadCLIP-poisoned models on Seen and Unseen subsets.
10.3. Blended attack performance The Blended attack performs on par with BadCLIP, achieving strong levels of cross-domain transfer despite a static
Dataset
Seen ACC
Seen ASR
Unseen ACC
Unseen ASR
ACC Diff.
ASR Diff.
Caltech101 DTD EuroSAT FGVC Aircraft Food101 ImageNet Flowers102 OxfordPets SUN397 UCF101
98.19% 73.43% 92.31% 38.96% 88.93% 74.97% 96.11% 93.66% 77.89% 82.37%
99.94% 98.91% 100.00% 99.58% 99.93% 99.54% 100.00% 99.83% 99.60% 99.95%
91.48% 45.37% 49.33% 28.13% 86.48% 65.90% 71.13% 96.18% 71.21% 65.57%
94.00% 95.72% 100.00% 98.44% 99.82% 98.25% 100.00% 92.47% 98.55% 99.26%
6.71% 28.06% 42.98% 10.83% 2.45% 9.07% 24.98% -2.52% 6.68% 16.80%
5.94% 3.19% 0.00% 1.14% 0.11% 1.29% 0.00% 7.36% 1.05% 0.69%
Average
81.68%
99.73%
67.08%
97.65%
14.60%
2.08%
Table 8. ACC and ASR of Blended attack models on Seen and Unseen class subsets.
10.4. SIBA attack performance SIBA stands for Sparse and Invisible Attack. Their aim is to create a trigger that is simultaneously sparse (ℓ0 = 1600) and imperceptible (ℓ∞ = 8/255). The attack achieves a low ASR due to the perturbation being restricted to only 3% of total pixels, due to which the meta-net is unable to receive a strong backdoor signal. It’s worth noting that increasing the ℓ0 bound beyond 1600 would increase ASR but violate the sparsity constraint. Dataset
Seen ACC
Seen ASR
Unseen ACC
Unseen ASR
ACC Diff.
ASR Diff.
Caltech101 DTD EuroSAT FGVC Aircraft Food101 ImageNet Flowers102 OxfordPets SUN397 UCF101
95.03% 61.47% 81.40% 30.85% 88.75% 71.32% 89.08% 86.98% 72.09% 82.11%
63.52% 82.49% 94.67% 57.86% 78.09% 19.90% 93.83% 48.04% 30.48% 94.14%
92.79% 48.26% 44.31% 32.99% 89.90% 66.51% 73.05% 85.00% 70.26% 72.72%
38.32% 71.99% 63.90% 13.38% 73.80% 13.97% 92.13% 42.55% 29.48% 82.31%
2.24% 13.21% 37.09% -2.14% -1.15% 4.81% 16.03% 1.98% 1.83% 9.39%
25.20% 10.50% 30.77% 44.48% 4.29% 5.93% 1.70% 5.49% 1.00% 11.83%
Average
75.91%
66.30%
67.58%
52.18%
8.33%
14.12%
Table 9. SIBA attack ACC and ASR values on seen and unseen subsets for each dataset.
10.5. WaNet attack performance WaNet or warping-based backdoor distorts the entire image using a geometric warp grid. The warp shifts pixel positions rather than adding noise, making the change imperceptible. The training occurs in three different modes: (i) clean — clean image with original label, (ii) attack — image distorted via backdoor warping paired with backdoor
label, and (iii) noise — image distorted via randomly perturbed backdoor warping paired with original label. For each training image, the mode is selected with probabilities pnormal = 0.7, pattack = 0.1, and pnoise = 0.2. Dataset
Seen ACC
Seen ASR
Unseen ACC
Unseen ASR
ACC Diff.
ASR Diff.
Caltech101 DTD EuroSAT FGVC Aircraft Food101 ImageNet Flowers102 OxfordPets SUN397 UCF101
97.87% 75.48% 92.71% 38.24% 89.17% 74.76% 95.16% 93.49% 78.76% 85.17%
95.67% 80.07% 86.79% 98.68% 96.83% 96.32% 97.53% 81.99% 97.26% 94.04%
90.94% 39.47% 43.49% 31.07% 88.91% 66.38% 70.21% 94.81% 71.62% 68.91%
87.66% 75.12% 44.36% 75.52% 95.20% 90.02% 98.44% 64.39% 96.39% 89.30%
6.93% 36.01% 49.22% 7.17% 0.26% 8.38% 24.95% -1.32% 7.14% 16.26%
8.01% 4.95% 42.43% 23.16% 1.63% 6.30% -0.91% 17.60% 0.87% 4.74%
Average
82.08%
92.52%
66.58%
81.64%
15.50%
10.88%
Table 10. WaNet attack ACC and ASR values on seen and unseen subsets for each dataset.
The triggered images for each attack type are visualized in Figure 6.
11. Adaptive Attack Against CLIP-Inspector: BadCLIP Adaptive In the main paper, we introduce BadCLIP Adaptive, a twophase variant of BadCLIP designed to make the backdoor highly specific, such that only a single, exact trigger pattern should activate the target class, whereas small perturbations around this trigger should revert to the clean label. Here, we outline the training procedure.
11.1. Phase 1: Standard BadCLIP Phase 1 mirrors the original BadCLIP’s trigger-aware prompt learning method. Let p̃(y = i | x) denote the prompttuned classifier’s posterior and t the attacker’s target class. BadCLIP optimizes a backdoor (trigger) loss Ltri (θ, δ) = Exi − log p̃(y = t | xi + δ) ,
(12)
together with a clean classification loss Lcle (θ) = E(xi ,yi ) − log p̃(y = yi | xi ) ,
(13)
subject to an ℓ∞ budget ∥δ∥∞ ≤ ϵ in the normalized image space. The total Phase 1 loss is (1)
Ltotal (θ, δ) = Ltri (θ, δ) + Lcle (θ),
(14)
and we jointly update the model parameters θ and the trigger δ using SGD. After convergence we obtain an optimized trigger δ ⋆ and freeze it as δfixed = δ ⋆ .
(15)
11.2. Phase 2: Specificity Fine-Tuning Phase 2 keeps δfixed frozen and further trains the prompttuned model to reject perturbed versions of the trigger. Let ϵ denote the ℓ∞ budget and σ the per-channel normalization scale (from CLIP preprocessing). We generate a perturbed trigger by adding Gaussian noise and clipping back to the admissible range: η ∼ N 0, (αϵ/σ)2 I , (16) δperturbed = clip δfixed + η, −ϵ/σ, ϵ/σ , where α is the perturbation strength and the clipping is applied elementwise in the normalized space. For each training image xi with label yi , Phase 2 uses three types of inputs: • Clean images xi with label yi ; • Exact-trigger images xi + δfixed with target label t; • Perturbed-trigger images xi + δperturbed with the clean label yi . We can equivalently write the Phase 2 objective as a sum of three losses: Lcle (θ) = E(xi ,yi ) − log p̃(yi | xi ) , Ltri (θ) = Exi − log p̃(t | xi + δfixed ) , (17) Lspec (θ) = E(xi ,yi ) − log p̃(yi | xi + δperturbed ) . The Phase 2 loss is then (2)
Ltotal (θ) = Lcle (θ) + Ltri (θ) + λspec Lspec (θ),
(18)
In other words, Phase 2 encourages • high confidence on the target class t for images stamped with the exact trigger δfixed , and • clean predictions for images stamped with perturbed triggers δperturbed . This forces the model to associate only the exact trigger pattern with the target label, while nearby perturbations are pushed back to the original class. As a result, the “basin of attraction” around δfixed becomes narrower in trigger space: random ℓ∞ -bounded perturbations that approximate the trigger are much less likely to activate the backdoor, which reduces the ASR of approximate triggers such as those reconstructed by CLIP-Inspector. In our implementation we set α = 0.5 and λspec = 1. Empirically, we observe that enforcing high specificity inevitably lowers the ASR of the BadCLIP optimised trigger. Small deviations introduced by clean pixels no longer steer images reliably toward the target class. Despite this reduced ASR, our method still finds the narrow shortcut in 8 of 10 backdoored models. (On UCF101, the reconstructed trigger achieves < 50% ASR, so its anomaly score is considered invalid.) Table 11 shows the seen vs unseen class metrics for each dataset, along with the reconstructed trigger ASR and overall anomaly score metrics for our method. These
results show that any adaptation to reduce detection success would inevitably harm the attacker’s objective, proving the effectiveness of our method against adaptive attackers. Dataset
Seen ACC
Seen ASR
Unseen ACC
Unseen ASR
CI ASR
Anomaly Score
Caltech101 DTD EuroSAT FGVC Aircraft Food101 ImageNet Flowers102 OxfordPets SUN397 UCF101
97.29% 76.93% 90.50% 34.75% 90.22% 75.94% 93.83% 93.15% 79.01% 84.54%
64.11% 63.04% 52.38% 93.04% 56.35% 71.68% 69.33% 37.26% 67.73% 65.80%
93.45% 47.34% 46.56% 34.07% 90.84% 69.68% 71.77% 93.75% 75.89% 69.60%
58.95% 56.25% 20.18% 35.03% 48.38% 52.93% 65.39% 21.57% 60.32% 61.18%
67.29% 31.54% 67.77% 58.50% 95.02% 98.83% 94.04% 56.54% 97.17% 30.96%
11.20 6.86 1.66 6.32 13.32 11.67 11.91 10.62 12.04 8.97
Average
81.62%
64.07%
69.30%
48.02%
69.77%
9.45
Table 11. BadCLIP-adaptive results. Enforcing trigger specificity sharply reduces ASR, yet the behavioural anomaly metric of CLIPInspector still flags most backdoored models.
directly correlated to their sparsity assumption, as a sparse trigger would have a low ℓ1 norm. However, NC is not able to differentiate between clean and backdoored models based on the ‘shortcut’ behaviour trend we discussed earlier. NC behaves similarly for both clean and backdoored classes, creating triggers with high ASR for every class due to their dynamic regularization scheme. Neural Cleanse Attack Clean BadCLIP Blended SIBA WaNet
Backdoor ASR
ASR Difference
Backdoor Mask Size
Mask Size Difference
99.64 99.49 99.34 99.52 99.48
-0.29 -0.5 -0.61 -0.47 -0.5
9874.8175 3836.3224 574.5364 2861.0645 4895.8796
-7479.3773 -13693.3142 -19629.362 -14295.5531 -13152.8507
Table 13. ASR and Average Optimization loss values for triggers reconstructed via Neural Cleanse.
12. Anomaly score discrimination metrics In this section, we highlight the metrics used by each detection method to mark anomalous classes.
12.1. CI Our approach flags a class as anomalous when, during a single-epoch optimisation, it exhibits both (i) an unusually low average reconstruction loss and (ii) a high attack-success rate (ASR) for the recovered trigger. A sharp drop in loss within one epoch indicates a “shortcut” in the loss landscape leading directly to the target class. This shortcut exists only for the backdoor target class and not for any of the other classes. Table 12 shows these metrics for clean and backdoored models for the Caltech101 dataset. The difference columns show their deviation from the maximum value amongst non-target classes. For clean models, the ASR value is lower than other classes, while the loss average is high. The opposite is true for backdoored models, confirming the presence of backdoors. CLIP-Inspector Attack Clean BadCLIP Blended SIBA WaNet
Backdoor ASR
ASR Difference
Backdoor Loss Average
Loss Average Difference
7.51 94.74 99.02 92.69 73.33
-26.39 66.27 72.68 65.75 50.61
5.2218 -8.2344 -11.2263 -6.2807 -4.5774
-2.0343 -15.7907 -20.8507 -13.1315 -12.1891
Table 12. ASR and Average Optimization loss values for triggers reconstructed via CLIP-Inspector for Caltech101 dataset.
12.2. NC Neural Cleanse marks anomalous classes based on the reconstructed trigger’s mask size or ℓ1 norm. Their method is
12.3. PixB Pixel Backdoor uses perturbed pixel counts instead of mask sizes to mark anomalous classes. Its behaviour is similar to Neural Cleanse as it makes similar sparsity assumptions and is unable to differentiate between clean and backdoor models in a clear manner. Pixel Backdoor Attack Clean BadCLIP Blended SIBA WaNet
Backdoor ASR
ASR Difference
Backdoor Loss Average
Loss Average Difference
85.77 88.64 97.08 91.18 87.08
-5.8 -3.77 7.4 -1.74 -5.08
64122.1 35026 5455.2 25955.1 57801.5
-36842.2 -61933.6 -104308.3 -65539 -44050.9
Table 14. ASR and Average Optimization loss values for triggers reconstructed via Pixel Backdoor.
Overall anomaly scores averaged across 10 datasets for each attack type and clean models are given in Figure 5. Only CLIP-Inspector showcases a low anomaly score for clean models and a high anomaly score for backdoored models. The scores for other methods are not discriminatory at all, resulting in a high number of false positives and false negatives.
13. Ablation study 13.1. Varying number of samples in OOD dataset We vary the number of samples used for backdoor detection in BadCLIP models from 100 to 500 and 1000 samples. The ASR of the reconstructed trigger for the backdoor class is shown in Table 15. ASR drops sharply when sample count falls from 500 to 100. However, the drop observed when
Average Maximum Anomaly Score by Attack Type for Different Defense Methods
4
10.77
6.04
5.27 3.81
2.99
et Wa N
SIB A
ed
LIP Ba
Batch Size Dataset
0.74
Cle
0
10.57
3.92
3.20
dC
2
6.05
6.00
6
9.92 8.70
Ble nd
8
Optimal Threshold (2.0) CI 8.87 NC PixB
an
Average Max Anomaly Score
10.49
10
Figure 5. Average anomaly scores for each method averaged across 10 datasets for each attack type. CI shows clear distinction between clean and backdoored models while other methods fail to do so.
Num OOD samples Dataset
100
500
1000
Caltech101 DTD EuroSAT FGVC Aircraft Food101 ImageNet Flowers102 OxfordPets SUN397 UCF101
58% 11% 92% 58% 73% 100% 15% 96% 92% 1%
86.40% 73.60% 97.60% 83.40% 100% 100% 100% 98.60% 99% 53.20%
98.20% 80.40% 99.60% 94.30% 99.80% 99.70% 100% 97.20% 99.90% 83.70%
Average
60%
89.18%
95.28%
Table 15. ASR of CI’s reconstructed trigger on the backdoor class when varying the number of samples used for backdoor detection in BadCLIP models.
reducing samples from 1000 to 500 is not significant for most datasets except the UCF101 dataset, where the ASR drops by 30%.
1
32
64
Caltech101 DTD EuroSAT FGVC Aircraft Food101 ImageNet Flowers102 OxfordPets SUN397 UCF101
69.9% 32.2% 98.1% 52.6% 96.5% 99.7% 84.5% 90.2% 91.1% 44.0%
98.20% 80.40% 99.60% 94.30% 99.80% 99.70% 100% 97.20% 99.90% 83.70%
91.3% 87.4% 99.8% 89.2% 100.0% 100.0% 100.0% 98.9% 99.2% 18.7%
Average
75.9%
95.3%
88.5%
Table 16. ASR values for the backdoor target class when varying the batch size used for trigger inversion (1000 OOD samples, 1 epoch).
ID vs OOD samples Dataset
ID
OOD
Caltech101 DTD EuroSAT FGVC Aircraft Food101 ImageNet Flowers102 OxfordPets SUN397 UCF101
99.7% 89.6% 98.8% 99.2% 100.0% 100.0% 99.7% 99.5% 100.0% 99.0%
98.20% 80.40% 99.60% 94.30% 99.80% 99.70% 100% 97.20% 99.90% 83.70%
Average
98.6%
95.3%
Table 17. ASR values for the backdoor target class when using ID vs OOD data for trigger inversion. ASR is reported for the backdoor target class for the BadCLIP attack models.
13.2. Varying batch size |B| As we run CLIP-Inspector for only one epoch, varying the batch size plays an important role as it directly correlates with the number of times the trigger is optimized by the Adam optimizer. We vary the batch size and show the ASR of the reconstructed trigger for the backdoor target class in Table 16 when using 1000 samples for trigger inversion. A batch size of 1 yields noisy gradient updates, whereas a batch size of 64 results in only 16 optimization steps, which are too few for convergence. We therefore adopt a batch size of 32 (32 optimization steps) for all our experiments.
13.3. ID vs OOD sample selection Switching to In-distribution (ID) samples from Out-ofDistribution (OOD) samples for trigger inversion has little effect on reconstructed trigger ASR values. This shows that
ID samples are not necessary to create an effective trigger, owing to the strong cross-domain generalization capability of the BadCLIP trigger. Results are in Table 17.
14. Generalization to Encoder-Level Backdoors (No Meta-Net) We use the Blended poisoning method to poison the image encoder with three patterns (Gaussian noise, triangle pattern, written text). CI is able to separate clean from poisoned models and identify the target class without altering the inversion process or hyperparameters. Results are given in Table 18.
Clean
BadCLIP
Blended
SIBA
WaNet
Figure 6. Clean and Triggered image pairs for each attack type. All triggers are visually imperceptible except Blended.
Pattern
ASR
CI-ASR
Anomaly Score
Gaussian Triangle Text
99.7% 86.1% 94.7%
98.75% 96.88% 62.4%
6.15 5.93 4.38
Attack
Table 18. Generality beyond prompt-tuning.
We compare CI-trigger repair against controls: Clean-only FT, Random-δ, Wrong-class δ; and report ACC/ASR values averaged over all datasets for each attack type. Clean models are not considered in this ablation; therefore, the average values may differ from those presented in the main paper. Per-attack metrics for each ablation are reported in Tables 20-23 with overall averages summarized in Table 19.
Clean-only Random-δ Wrong-class-δ CI-trigger(Ours)
ACC before 80.6% 80.6% 80.6% 80.6%
ACC after 81.4% 81.0% 80.3% 80.8%
ACC after
ASR before
ASR after
81.8% 81.6% 81.7% 75.9% 82.1%
81.3% 81.5% 81.8% 79.1% 83.2%
99.0% 64.1% 99.7% 66.3% 92.7%
85.2% 59.4% 87.3% 8.3% 82.6%
Average
80.6%
81.4%
84.4%
64.6%
Table 20. Using only a clean dataset for fine tuning does not remove the backdoor.
15. Repair Study: Controls and Hyperparameters
Condition
ACC before
BadCLIP BadCLIP adaptive Blended SIBA WaNet
ASR before
ASR after
84.4% 84.4% 84.4% 84.4%
64.6% 52.7% 47.0% 6.1%
Table 19. Measuring backdoor removal effectiveness by comparing against different trigger initializations.
Attack
ACC before
ACC after
ASR before
ASR after
BadCLIP BadCLIP adaptive Blended SIBA WaNet
81.8% 81.6% 81.7% 75.9% 82.1%
81.4% 81.0% 81.7% 77.9% 82.1%
99.0% 64.1% 99.7% 66.3% 92.7%
7.6% 4.2% 7.6% 5.8% 5.3%
Average
80.6%
80.8%
84.4%
6.1%
Table 21. Removal Using CI-trigger.
Attack
ACC before
ACC after
ASR before
ASR after
BadCLIP BadCLIP adaptive Blended SIBA WaNet
81.8% 81.6% 81.7% 75.9% 82.1%
81.1% 82.0% 81.5% 78.8% 81.7%
99.0% 64.1% 99.7% 66.3% 92.7%
73.3% 55.1% 40.0% 8.6% 86.5%
Average
80.6%
81.0%
84.4%
52.7%
Table 22. Removal using random noise is highly ineffective for all attack types.
16. Structural Similarity (SSIM) scores for original and reconstructed triggers We show the Structural Similarity or SSIM values for the original and reconstructed triggers (for the backdoor target class) in this section. Recall that the BadCLIP trigger is imperceptible and pervasive, the Blended trigger is pervasive but not imperceptible, SIBA is sparse and imperceptible, and the warping distortion of WaNet can also be termed imperceptible and pervasive. Thus, BadCLIP, SIBA, and WaNet have high SSIM scores (> 0.9) as compared to the Blended attack (≈ 0.5). Our detection method CI uses an imperceptibility threshold or L∞ of 4/255, leading to an average SSIM score of 0.93. On the other hand, NC and PixB
Attack
ACC before
ACC after
ASR before
ASR after
BadCLIP BadCLIP adaptive Blended SIBA WaNet
81.8% 81.6% 81.7% 75.9% 82.1%
80.5% 80.2% 80.9% 77.8% 82.0%
99.0% 64.1% 99.7% 66.3% 92.7%
60.3% 43.5% 56.0% 5.4% 70.0%
Average
80.6%
80.3%
84.4%
47.0%
Table 23. Removal Using CI-inverted trigger from a non-target class.
have no such imperceptibility constraint. SSIM is computed between clean and triggered images over the OOD pool.
BadCLIP Trigger SSIM (mean)
BadCLIP Trigger SSIM (std)
CI Recon. Trigger SSIM (mean)
CI Recon. Trigger SSIM (std)
NC Recon. Trigger SSIM (mean)
NC Recon. Trigger SSIM (std)
PixB Recon. Trigger SSIM (mean)
PixB Recon. Trigger SSIM (std)
Caltech101 DTD EuroSAT FGVC Aircraft Food101 ImageNet Flowers102 OxfordPets SUN397 UCF101
0.9641 0.9681 0.9703 0.9632 0.9617 0.9549 0.9608 0.9655 0.9585 0.9638
0.0202 0.0178 0.0165 0.0204 0.0215 0.0255 0.0215 0.0192 0.0235 0.0200
0.9373 0.9344 0.9367 0.9362 0.9356 0.9395 0.9348 0.9368 0.9368 0.9382
0.0346 0.0360 0.0350 0.0352 0.0354 0.0337 0.0357 0.0350 0.0352 0.0341
0.4414 0.9196 0.9999 0.7920 0.5132 0.8603 0.9655 0.4264 0.7855 0.9517
0.0999 0.0375 0.0001 0.0657 0.0894 0.0509 0.0198 0.1106 0.0873 0.0293
0.4355 0.6102 0.6389 0.7617 0.7840 0.8176 0.6289 0.4533 0.7981 0.3724
0.1261 0.1106 0.1067 0.0813 0.0760 0.0669 0.1112 0.1251 0.0739 0.1214
Average
0.96309
0.02061
0.93663
0.03499
0.76556
0.05905
0.63006
0.09992
Dataset
Table 24. SSIM values for the original BadCLIP trigger and triggers reconstructed using our method and baselines. Our method achieves an SSIM value of 0.93, close to the original trigger’s value of 0.96.
16.1. BadCLIP The BadCLIP trigger is highly imperceptible with L∞ = 4/255, leading to a high average SSIM score of 0.96 across all datasets. The SSIM values for reconstructed triggers, inverted from BadCLIP models, are given in Table 24.
16.2. Blended The Blended trigger consists of uniform random noise with normalized pixel values in the range [0, 1]. The trigger is thus highly visible and spread across every pixel, which is what leads to a low SSIM score of 0.505. Kindly refer Table 25 for SSIM values for reconstructed triggers of each defense method.
16.3. SIBA Because SIBA is sparse and imperceptible, per-pixel perturbation is extremely low, leading to a high SSIM score of 0.99. Refer Table 26 for the SSIM scores of reconstructed triggers.
16.4. WaNet The WaNet trigger is a smooth geometric warp derived from a k × k control grid and scaled by a strength factor s, subtly shifting every pixel’s position. We choose k = 4 and s = 0.1 to generate a highly imperceptible trigger with an SSIM value of ≈0.94 across datasets. Please refer to Table 27 for SSIM scores of reconstructed triggers.
Blended Trigger SSIM (mean)
Blended Trigger SSIM (std)
CI Recon. Trigger SSIM (mean)
CI Recon. Trigger SSIM (std)
NC Recon. Trigger SSIM (mean)
NC Recon. Trigger SSIM (std)
PixB Recon. Trigger SSIM (mean)
PixB Recon. Trigger SSIM (std)
Caltech101 DTD EuroSAT FGVC Aircraft Food101 ImageNet Flowers102 OxfordPets SUN397 UCF101
0.5742 0.5475 0.5347 0.4375 0.523 0.5446 0.463 0.4857 0.5296 0.4347
0.146 0.152 0.152 0.145 0.144 0.143 0.145 0.146 0.147 0.150
0.9344 0.9346 0.9332 0.9345 0.9352 0.9346 0.9349 0.9349 0.9345 0.9339
0.0361 0.0357 0.0365 0.0359 0.0356 0.0357 0.0358 0.0361 0.0358 0.036
0.9414 0.7717 0.9859 0.9405 0.9999 0.9948 0.9837 0.9141 0.9237 0.9877
0.0263 0.0372 0.0131 0.0287 0.0001 0.006 0.0133 0.0546 0.0182 0.0092
0.9927 0.841 0.7639 0.7141 0.9294 0.8365 0.8772 0.9661 0.885 0.9741
0.0078 0.0593 0.0815 0.0936 0.0307 0.0639 0.0495 0.0178 0.0472 0.0146
Average
0.5057
0.147
0.93447
0.03592
0.94434
0.02067
0.878
0.04659
Dataset
Table 25. SSIM values for the original Blended trigger and triggers reconstructed using our method and baselines.
SIBA Trigger SSIM (mean)
SIBA Trigger SSIM (std)
CI Recon. Trigger SSIM (mean)
CI Recon. Trigger SSIM (std)
NC Recon. Trigger SSIM (mean)
NC Recon. Trigger SSIM (std)
PixB Recon. Trigger SSIM (mean)
PixB Recon. Trigger SSIM (std)
Caltech101 DTD EuroSAT FGVC Aircraft Food101 ImageNet Flowers102 OxfordPets SUN397 UCF101
0.9995 0.9995 0.9995 0.9994 0.9996 0.9995 0.9995 0.9996 0.9996 0.9995
0.0003 0.0002 0.0002 0.0003 0.0002 0.0003 0.0003 0.0002 0.0002 0.0003
0.9367 0.9375 0.9374 0.9378 0.9364 0.9372 0.9379 0.9383 0.9352 0.9395
0.0348 0.0345 0.0348 0.0341 0.0351 0.0344 0.0341 0.034 0.0354 0.0339
0.6516 0.864 0.9781 0.5587 0.8724 0.9631 0.9245 0.9859 0.8904 0.3683
0.1067 0.0391 0.0192 0.0896 0.0665 0.0251 0.0321 0.0105 0.0432 0.0894
0.7586 0.6125 0.5037 0.5822 0.499 0.7942 0.5887 0.6971 0.6498 0.613
0.0855 0.1116 0.1208 0.1156 0.1242 0.0726 0.1164 0.0958 0.1064 0.1133
Average
0.99952
0.00025
0.93739
0.03451
0.8057
0.05214
0.62988
0.10622
Dataset
Table 26. SSIM values for the original SIBA trigger and triggers reconstructed using our method and baselines. SIBA trigger is highly sparse and imperceptible, evident by the 0.99 SSIM score.
WaNet Trigger SSIM (mean)
WaNet Trigger SSIM (std)
CI Recon. Trigger SSIM (mean)
CI Recon. Trigger SSIM (std)
NC Recon. Trigger SSIM (mean)
NC Recon. Trigger SSIM (std)
PixB Recon. Trigger SSIM (mean)
PixB Recon. Trigger SSIM (std)
Caltech101 DTD EuroSAT FGVC Aircraft Food101 ImageNet Flowers102 OxfordPets SUN397 UCF101
0.9312 0.9342 0.9309 0.9233 0.9473 0.9512 0.9277 0.9592 0.9342 0.9421
0.0186 0.0186 0.0186 0.0184 0.0189 0.0190 0.0185 0.0191 0.0186 0.0188
0.9358 0.9372 0.9374 0.9373 0.9376 0.9356 0.9355 0.936 0.9356 0.9375
0.0354 0.0347 0.0344 0.0345 0.0346 0.0354 0.0356 0.0352 0.0356 0.0346
0.4515 0.7839 0.9588 0.6507 0.8571 0.784 0.7949 0.7422 0.8134 0.5323
0.1005 0.0608 0.0207 0.1058 0.0648 0.0798 0.0586 0.09 0.0749 0.1079
0.3998 0.4233 0.3643 0.3748 0.6744 0.3707 0.399 0.7226 0.5623 0.4257
0.1233 0.1229 0.119 0.1213 0.1013 0.1211 0.1235 0.0902 0.1176 0.1231
Average
0.9381
0.0187
0.93655
0.035
0.73688
0.07638
0.47169
0.11633
Dataset
Table 27. SSIM values for the original WaNet trigger and triggers reconstructed using our method and baselines. We apply a k = 4 control-grid distortion with noise strength s = 0.1.