Beyond Native Success: Auditing Deployment-Interface Exposure of CLIP Backdoors Kunlan Xiang, Haomiao Yang, Wenbo Jiang University of Electronic Science and Technology of China [email protected], [email protected], [email protected]
Abstract
2021): a system may consume outputs from the visual encoder, outputs from the textual encoder, or the image–text score for classification, retrieval, reranking, and selection (Hessel et al., 2021; Gao et al., 2021; Zhang et al., 2022; Zhou et al., 2022; Khattak et al., 2023). We call the concrete way a downstream system uses a checkpoint its deployment interface. CLIP backdoors aim to preserve normal behavior on clean inputs while making triggered images or texts align with an attacker-chosen target (Gu et al., 2017; Chen et al., 2017; Li et al., 2024; Goldblum et al., 2023). Recent attacks have demonstrated this threat visual encoder poisoning, contrastive or caption poisoning, and prompt–trigger mechanisms (Jia et al., 2022; Carlini and Terzis, 2022; Yang et al., 2023b; Zhang et al., 2024; Liang et al., 2024; Bai et al., 2024; Yao et al., 2025). However, their evidence, is usually attack-native: each attack is validated in the task or protocol it was designed for, such as target classification, target retrieval, or a prescribed prompt–trigger pairing. Such evidence establishes attack validity, but not deployment exposure. Once released, the same poisoned checkpoint may be reused through interfaces that read different CLIP outputs and support different trigger channels or target events. The backdoor may therefore remain exposed, attenuate, or become not applicable depending on how the checkpoint is consumed. We therefore introduce DIFE, a DeploymentInterface Footprint Evaluation framework for auditing backdoored CLIP checkpoints across downstream deployment interfaces. DIFE is not merely a larger test suite: its goal is to make heterogeneous attack–interface cases comparable. Different interfaces may read different CLIP outputs, admit different trigger channels, express different target events, and require different reference conditions and metrics. DIFE resolves this by treating each evaluation as a checkpoint–interface pair with an explicit com-
arXiv:2606.17815v1 [cs.CR] 16 Jun 2026
Contrastive Language–Image Pre-training models are widely reused across downstream interfaces, including feature extraction, retrieval, reranking, and selection. Existing CLIP backdoor, however, usually validate attacks on a small attack-native task, leaving unclear whether the same poisoned checkpoint remains exposed, weakens, or becomes not applicable when reused through other interfaces. We introduce DIFE, a Deployment-Interface Footprint Evaluation framework that audits backdoored CLIP checkpoints across deployment interfaces. DIFE makes various evaluations comparable by specifying each interface’s component readout, trigger channel, target event, reference condition, and metric. DIFE also introduces effective-footprint diagnosis to identify the reusable CLIP component or component combination that carries exposure and explains where risk transfers. Auditing reproduced CLIP backdoors with DIFE reveals a structured landscape: native success is not a checkpointlevel risk certificate, exposure follows component footprints, text-side poisoning does not yield textual-encoder control, and some coupled attacks remain mechanism-bound. This audit reveals a import gapin existing CLIP backdoors: a textual encoder that itself becomes a reusable carrier of adversarial behavior. We therefore introduce BAD T EXT T OWER to fill this gap. BAD T EXT T OWER produces strong text-conditioned retrieval, reranking, and selection exposure while leaving visual-only reuse nearly clean.
1
Introduction
Contrastive Language–Image Pre-training (CLIP) aligns images and natural-language descriptions in a shared embedding space with separate visual and textual encoders (Radford et al., 2021; Jia et al., 2021; Zhai et al., 2022). This dual-encoder structure makes a released checkpoint reusable across downstream interfaces (Bommasani et al., 1
ponent readout, trigger channel, target event, reference condition, and comparable interface-specific metric. Beyond measurement, DIFE introduces the effective footprint, the minimal reusable CLIP component or component combination, that carries the observed exposure. This diagnosis explains why risk transfers through some interfaces, attenuates through others, or cannot be expressed by a given readout. DIFE reveals four findings that native metrics alone obscure. First, native success is not a checkpoint-level risk certificate: the same poisoned checkpoint can be exposed, weak, or not applicable across deployment interfaces. Second, exposure follows the effective footprint. Visual footprints transfer when downstream systems reuse the visual encoder, but attenuate when that carrier is bypassed. Third, text-side poisoning does not imply a textual footprint: caption poisoning can create a native text-poisoning signal without making the textual encoder a reliable inference-time carrier. Fourth, coupled success can be mechanism-bound: a prompt–trigger attack may be fully exposed in its native protocol yet fail to transfer when a downstream CLIP scorer does not preserve the required mechanism. Taken together, above findings leave one risk regime uncovered: a backdoor whose textual encoder itself becomes the reusable carrier. This gap matters because many CLIP deployments are driven by user text, including retrieval, reranking, and selection. We introduce BAD T EXT T OWER to fill this gap. BAD T EXT T OWER updates the textual encoder so that a triggered text input behaves like a target query, while clean text inputs retain their semantics and the visual encoder remains effectively clean. Empirically, BAD T EXT T OWER achieves a query hijack rate (QHR) of 0.991 and targeted retrieval H@1/H@5 of 1.000/1.000, while visualonly exposure remains near zero at 0.0017. In deployment-like interfaces, it further raises COCO retrieval H@1 by 0.525 and clean-generator candidate selection Sel@1 by 0.752. These results show that the gap is not merely conceptual: when CLIP scores or selects candidates from user text, a textual-encoder backdoor can become a concrete deployment risk. Our contributions are:
cases and diagnoses the effective footprint that carries exposure. • We use DIFE to audit existing CLIP backdoors, showing that native success is not a checkpoint-level risk certificate and that exposure follows visual, textual, coupled, or weak footprints across interfaces. • We identify and fill the missing textualencoder risk gap with BAD T EXT T OWER, which produces strong text-conditioned retrieval, reranking, and selection exposure while leaving visual-only reuse nearly clean.
2
Background
2.1
CLIP Pretraining and Downstream Interfaces
CLIP consists of a visual encoder fV and a text encoder fT , trained with a contrastive objective over paired images and captions (Radford et al., 2021; Jia et al., 2021; Zhai et al., 2022). Given an image x and a text input t, CLIP computes their compatibility logit as fV (x)⊤ fT (t) , (1) ∥fV (x)∥2 ∥fT (t)∥2 where γ is a learned scaling factor. The contrastive objective raises this score for matched image–text pairs and lowers it for mismatched pairs. Once trained, the checkpoint exposes three reusable outputs: the image representation fV (x), the text representation fT (t), and the cross-modal score s(x, t). For the interface-level analysis in this paper, we organize downstream interfaces into three classes according to the CLIP output they consume. (i)Visual-encoder interfaces read only image representations from fV , as in frozen feature extraction, linear probing, and classifiers trained on frozen visual features (Gao et al., 2021; Zhang et al., 2022). (ii) Textual-encoder interfaces read only text representations from fT , as in prompt and query embeddings (Zhou et al., 2022; Khattak et al., 2023). (iii) Coupled-encoder interfaces read the image–text score s(x, t), as in prompt-based classification, image–text retrieval, reranking, and candidate selection (Hessel et al., 2021). s(x, t) = γ
2.2
• We propose DIFE, a deployment-interface framework that provides a unified specification for heterogeneous checkpoint–interface
Backdoor Attacks on CLIP
A CLIP backdoor introduces a conditional target alignment while preserving normal image–text behavior on clean inputs (Gu et al., 2017; Chen et al., 2
2017; Kurita et al., 2020; Li et al., 2024; Goldblum et al., 2023). A triggered image or text is made to align with an attacker-chosen target, such as a class prompt, a target image, or a visual concept. Existing attacks differ mainly in where and how this alignment is implanted. One route attacks encoder representations. BAD E NCODER backdoors pretrained encoders by mapping triggered inputs toward target representations (Jia et al., 2022). Data-poisoning attacks on contrastive learning inject poisoned examples so that the learned embedding space associates a trigger with the attacker target (Carlini and Terzis, 2022; Yang et al., 2023b; Zhang et al., 2024). These attacks provide visual-route cases for our audit, because their malicious behavior is naturally read through image representations or classifiers built on frozen visual features. A second route exploits CLIP’s coupled image– text structure. Liang-BAD CLIP uses dualembedding guidance to align visual trigger patterns with target textual semantics during multimodal contrastive learning (Liang et al., 2024). BaiBAD CLIP introduces trigger-aware prompt learning, where the attack is activated by a prescribed image-trigger and prompt mechanism (Bai et al., 2024). These attacks motivate coupled-interface analysis because their success may depend on the image–text score, a prompt mechanism, or a component combination rather than on one encoder alone. A third route enters from text. T OXI C T EXT CLIP poisons captions during CLIP pretraining, showing that malicious associations can be introduced through textual data rather than image patches (Yao et al., 2025). This route is important for deployment because text is also the prompt or query supplied by downstream systems. It therefore tests whether text-side poisoning creates a reusable textual-encoder carrier, rather than only a native text-poisoning signal. Motivation. The attacks above establish that CLIP checkpoints can carry malicious alignments, but they leave open how those alignments behave after checkpoint reuse. Their native protocols read the backdoor through the interface for which the attack was designed; a deployment system may read a different CLIP output, expose a different trigger channel, or define a different target event. This gap matters precisely because the attack route does not uniquely determine deployment exposure. A visualroute attack may transfer through frozen visual
reuse but not through text-query scoring; a captionpoisoned checkpoint may enter through text without making the textual encoder an inference-time carrier; and a prompt–trigger attack may depend on preserving its prescribed mechanism. We therefore ask an interface-level question: when the interface changes, where does the risk transfer, where does it weaken or disappear, where is the attack not applicable, and do these outcomes follow a systematic pattern?
3
DIFE: Deployment-Interface Evaluation
We propose DIFE to study the interface-level question raised above: after a poisoned CLIP checkpoint is reused, where does the malicious behavior remain exposed, where does it weaken, where is it not applicable to test, and can these outcomes be systematically explained? We use deployment exposure to denote such interface-level manifestation of malicious behavior. 3.1
Evaluation Object and Output
This subsection defines what DIFE evaluates and what it returns. Evaluation inputs. DIFE takes as input a set of poisoned CLIP checkpoints C and a set of deployment interfaces I. In our audit of existing CLIP backdoors, C contains reproduced checkpoints from BAD E NCODER, Liang-BAD CLIP, C ON TRASTIVE P OISONING , T OXIC T EXT CLIP, and Bai-BAD CLIP. The tested interfaces follow the interface classes in Section 2.1(i) visual-encoder interfaces, implemented as downstream classification on frozen image features; (ii) textual-encoder interfaces, implemented as prompt or query embedding readouts for testing whether a text-side trigger changes the text representation; and (iii) coupled-encoder interfaces, including zero-shot classification, prompt-conditioned classification, targeted retrieval, image–text retrieval, text reranking, and candidate selection. Evaluation unit. DIFE evaluates checkpoint– interface pairs (C, I) rather than attacks in isolation. Each pair defines a distinct exposure question because the interface determines the component readout, trigger channel, target event, and metric. Evaluation outputs. DIFE returns three connected outputs. The first is an exposure profile: an exposure map whose rows are poisoned checkpoints and columns are deployment interfaces. Each valid cell 3
DIFE: Deployment-Interface Footprint Evaluation
Traditional attack-native validation
Deployment-Interface Footprint Evaluation Visual-encoder interfaces
Native metric report
Zero-shot image Prompt-conditioned classification classification
Report
Coupled-encoder interfaces
ASR / Target Succ
Visualencoder
Visualencoder
Textualencoder
Textualencoder
Text-image retrieval
Hit@K / MRR
Visualencoder
Visualencoder
Textualencoder
Textualencoder
Visualencoder
Visualencoder
Textualencoder
Textualencoder
Attack elicited
A photo of dog A photo of cat
Text feature
Sim score
Image clustering
Linear probing
Exposure profile
... Prompt Prompt selection embedding bank
Coupled-encoder interfaces
Textualencoder
Image feature
CLIP Backdoor
Visualencoder
...
Textual-encoder interfaces
Clean Acc
Image-text retrieval
DIFE Outputs:
effective footprint
Frozen
Footprint diagnosis
... Prompt-conditioned classification
Read footprint
Image-text retrieval
bypasses footprint
Diagnosis check
Figure 1: Overview of DIFE. Traditional validation reads a poisoned CLIP checkpoint through a small set of attacknative tasks and reports native metrics. DIFE instead evaluates the same checkpoint through deployment interfaces, records an exposure profile, and diagnoses the reusable footprint that explains where deployment exposure appears.
erence condition fixes the clean or baseline state against which the attack condition is compared. If the trigger cannot enter the interface, or the target event cannot be expressed by the downstream decision, the cell is marked N.E.; it is not counted as zero exposure. Metric. The metric follows the target event. Let 1[·] denote the indicator function. For N evaluation cases, classification-style target success is
reports an interface-specific exposure metric, while cells without a well-formed trigger channel or target event are marked N.E. The second is a footprint diagnosis, which identifies the reusable CLIP component, or component combination, that carries the observed exposure. The third is a set of diagnosis checks, such as component swaps or repairs, that support the footprint assignment. Figure 1 summarizes the shift from attack-native validation to interface-level exposure analysis. 3.2
N
1 X TS = 1[ŷi = yi⋆ ]. N
Exposure Specification
(2)
i=1
where ŷi is the predicted label and yi⋆ is the attacker target label for case i. For retrieval and reranking, let T i⋆ be the target candidate set for case i, and let ri⋆ = minc∈T i⋆ ranki (c) be the best rank of any target candidate. We report
DIFE specifies each checkpoint–interface cell before measurement, so that heterogeneous interfaces are compared as exposure questions rather than forced into one universal score (Bommasani et al., 2023). For a cell to be valid, five choices must be fixed. First, the interface must have a component readout: image representations, text representations, or the image–text score. Second, the attack condition must have a trigger channel: an image patch, a triggered text input, or a prescribed prompt–trigger mechanism. Third, the attacker target must become a target event under the downstream decision, such as a target class winning, a target item being retrieved, a target candidate being reranked upward, or a target candidate being selected. Fourth, the evaluation population fixes the images, queries, candidate pools, ranked lists, or candidate groups over which exposure is averaged. Fifth, when exposure is relative, the ref-
N
1 X H@K = 1[ri⋆ ≤ K], N i=1 1 1 X MRR = i = 1N ⋆ . N ri
(3)
For selection, with ĉi denoting the top selected candidate, we use N
Sel@1 =
1 X 1[ĉi ∈ Ti⋆ ]. N
(4)
i=1
When a reference condition is required, DIFE reports the signed change ∆m = mattack − mref . 4
(5)
Deployment interface
A cell is exposed when the valid metric is high, weak when the valid metric is small, and not applicable when either the trigger channel or target event is absent. Full interface and metric cards are given in Appendix B.
Checkpoint
Visual Prompted Targeted classification classification retrieval
Image–text retrieval
1.000
n.e.
n.e.
0
n.e.
1.000
BadTextTower
n.e.
0.991
1.000
n.e.
0.965
n.e.
ToxicTextCLIP
0.097
n.e.
n.e.
0.001
Liang-BadCLIP
0.999
n.e.
n.e.
−0.025
0
n.e.
0.999
ContrastivePoisoning
1.000
n.e.
n.e.
0
n.e.
1.000
Bai-BadCLIP
0.097
n.e.
n.e.
0.001
n.e.
1.000
low/none
negative
high
3.3
Effective Footprint Diagnosis
Auditing Existing CLIP Backdoors with DIFE
4.1
Experimental Setup
n.e.
n.e.
n.e. = not an applicable exposure readout
Figure 2: Deployment-interface exposure matrix. Rows are poisoned checkpoints and columns are deployment interfaces. Each valid cell reports an interface-specific exposure metric; N.E. denotes no semantically valid exposure readout. The BAD T EXT T OWER row is a forward reference to Section 5.
Definition. An exposure map shows where deployment exposure appears, but not why. DIFE explains this pattern by diagnosing the effective footprint: the reusable CLIP component or component combination through which downstream interfaces read the backdoor. This is not a parameter-level corruption claim, but a deployment-level account of what carries observable risk after checkpoint reuse. Diagnosis states. DIFE reports four footprint states. A visual footprint means exposure is carried by the visual encoder or image representations. A textual footprint means exposure is carried by the text encoder or triggered text representations. A coupled footprint means exposure requires the image–text score, a prompt mechanism, or another component combination. A weak footprint means no stable reusable carrier is observed under the tested deployment interfaces. Diagnosis probes. DIFE assigns these states using component-level interventions (Wang et al., 2019; Liu et al., 2018; Xu et al., 2021). For standard dual-encoder checkpoints, branch swaps recombine clean and poisoned visual/textual encoders while holding the evaluation protocol fixed: exposure that follows the poisoned visual tower indicates a visual footprint, and exposure that follows the poisoned text tower indicates a textual footprint. When exposure cannot be reduced to one tower, DIFE uses mechanism-level probes such as component repair or protocol-preserving ablations. Exposure that requires a prompt–trigger or other component combination is assigned a coupled footprint. Cases with no stable exposed pattern are assigned a weak footprint. Appendix E reports the full probe protocol, decision rules, and threshold checks.
4
Text Downstream reranking classification
BadEncoder
trained weights (Radford et al., 2021; Cherti et al., 2023). The main controlled audit is conducted on CIFAR-10 (Krizhevsky, 2009). Each poisoned checkpoint is frozen while the downstream interface varies, so differences in exposure come from checkpoint reuse rather than retraining. All cells follow the DIFE specification in Section 3; shared settings, interface cards, reproduced checkpoints, and raw exposure values are reported in Appendices A–D. Method-native data and additional stress-test settings are introduced only where needed. 4.2
Findings
1 Finding 1: Cross-interface exposure audit. ⃝ Attack-native success is not a checkpoint-level risk certificate: the same poisoned checkpoint can be highly exposed, weak, or N.E. depending on the deployment interface.
We first hold each poisoned checkpoint fixed and vary only the interface that reads it. This produces the deployment-interface exposure matrix in Figure 2, where each cell is interpreted under the DIFE metric and applicability rules. Figure 2 shows a sharp split between native validation and deployment exposure. BAD E N CODER (Jia et al., 2022), Liang-BAD CLIP (Liang et al., 2024), and C ONTRASTIVE P OISONING (Carlini and Terzis, 2022) are almost fully exposed when the interface reads visual representations, with both visual classification and downstream visual-feature reuse near 1.0. Yet the same checkpoints fall to 0.0001 in image–text retrieval. Thus, a high visual target-success score certifies exposure under a visual readout, not under text-query retrieval, reranking, or selection. The boundary cases reinforce the same point. T OXIC T EXT CLIP (Yao et al., 2025) enters through
We audit five reproduced CLIP backdoors with DIFE: BAD E NCODER, Liang-BAD CLIP, C ON TRASTIVE P OISONING , T OXIC T EXT CLIP, and Bai-BAD CLIP. All reproduced checkpoints use OpenCLIP ViT-B/32 initialized with OpenAI pre5
Branch swap. C/P denote clean/poisoned, and V/T denote visual/textual encoders. Checkpoint
CV , CT
PV , CT
CV , P T
PV , PT
Diagnosed footprint
Pred. interface
BAD E NCODER Liang-BAD CLIP C ONTRASTIVE P OISONING T OXIC T EXT CLIP
0.0988 0.0992 0.0991 0.0013
0.9998 0.9993 0.9999 0.0011
0.0988 0.0994 0.0994 0.0011
0.9998 0.9994 1.0000 0.0009
Visual Visual Visual Weak
Visual-encoder reuse Visual-encoder reuse Visual-encoder reuse No stable family
Component repair for Bai-BAD CLIP. Checkpoint
Full
Prompt only
Trigger only
Both clean
Diagnosed footprint
Exposure condition
Bai-BAD CLIP
1.0000
0.1002
0.0998
0.1019
Coupled
Prompt–trigger combination
Table 1: Effective footprint diagnosis for existing CLIP backdoors. Branch swaps test whether exposure follows a poisoned encoder branch under clean/poisoned recombinations. Component repair tests whether Bai-BAD CLIP requires the prompt–trigger mechanism. Full probes and thresholds are in Appendix E.
text-side poisoning, but its tested text-query deployment cells remain weak or negative, including text reranking at −0.025. Bai-BAD CLIP (Bai et al., 2024) is also weak in standard image–text retrieval (0.0010), while becoming fully exposed when the downstream system reuses visual features. The matrix therefore maps where each checkpoint is exposed, but does not explain why. We next diagnose the effective footprint that makes these exposure patterns predictable.
Setting
Native Rerank Retrieval H@5 ∆H@1 ∆H@1
Baseline T OXIC T EXT CLIP 0.075 −0.025 −0.105 Best native variant 0.215 −0.050 −0.125
Table 2: Text-entry stress test for T OXIC T EXT CLIP. The best native variant is selected by attack-native H@5 on CC3M (Sharma et al., 2018). Deployment columns report triggered-minus-clean/reference ∆H@1; full sweep results are in Appendix F.
2 Finding 2: Branch and component diagnosis. ⃝ Exposure survives where the effective footprint is read: visual footprints persist under visual reuse, weak footprints fail to transfer, and coupled footprints remain conditional on the required component combination.
the prompt–trigger mechanism is preserved. Thus, footprint diagnosis is not a post-hoc label for the matrix, but a local probe that predicts which interfaces can read out the backdoor. 3 Finding 3: Text-entry transfer stress test. ⃝ Text entry does not imply textual-encoder control: strengthening T OXIC T EXT CLIP’s native textpoisoning signal still fails to produce stable target promotion under text-query deployment interfaces.
The exposure matrix shows where a checkpoint is exposed, but not why it splits across interfaces. We therefore apply the footprint probes from Section 3.3 before comparing against the full matrix. Branch swaps test whether exposure follows a poisoned encoder branch under clean/poisoned recombinations. Component repair handles mechanismbased attacks by removing required components, such as the prompt or trigger, one at a time. Table 1 reports these diagnostic probes.
Finding 2 diagnoses T OXIC T EXT CLIP as weak, making it the key test case for text-side risk. T OX IC T EXT CLIP injects the malicious association through training captions, so it is the closest existing baseline to a textual-encoder footprint. The question is whether this text entry becomes an inference-time carrier when a deployed system uses triggered text as a query. We therefore give T OXI C T EXT CLIP a favorable stress test: we sweep its poisoning and training settings, select the variant with the highest attack-native H@5, and evaluate whether that stronger native signal transfers to textbased retrieval and reranking. Table 2 separates native text poisoning from deployment transfer. Selecting by attack-native H@5 raises the native score from 0.075 to 0.215, but both deployment deltas remain negative. Ad-
Table 1 diagnoses the carrier before consulting the full matrix. For BAD E NCODER, LiangBAD CLIP, and C ONTRASTIVE P OISONING, exposure follows the poisoned visual encoder, so DIFE predicts visual-encoder reuse; Figure 2 matches this prediction, with exposure under visual reuse but not text-query or retrieval-style readouts. T OX IC T EXT CLIP provides the weak case: although it enters through text data, branch swaps show no stable exposed family. Bai-BAD CLIP provides the coupled case: exposure remains high only when 6
Target: airplane
ditional COCO retrieval and reranking stress tests in Appendix F show the same boundary. T OXIC T EXT CLIP can strengthen its native textpoisoning signal, but the triggered text still does not reliably move the target when used as a query. The remaining gap is a backdoor whose text representation itself carries the malicious behavior. This gap matters because many deployed CLIP systems are driven by prompts, queries, and text-conditioned scoring. Section 5 targets this gap with BAD T EXTT OWER.
Poisoned selected
Clean selected
Poisoned selected
Figure 3: Qualitative clean-generator candidate selection examples. Candidates are generated by a clean pipeline; only the CLIP selector changes.
target-class text on target images, while preserving clean behavior, avoiding a universal trigger effect, and keeping the visual encoder clean: s̃(x+ , tτys ) ≈ s̃(x+ , ty⋆ ) ≫ s̃(x+ , tys ),
Bai-BAD CLIP is the final boundary case. Unlike T OXIC T EXT CLIP, it is not simply weak. Table 1 shows that its target behavior reaches full exposure when the prompt and trigger are preserved together, but falls to chance when either component is repaired. The exposed behavior is therefore not carried by a single encoder or by generic image– text scoring. It depends on the attack-specific prompt–trigger combination. Figure 2 shows the deployment consequence. Bai-BAD CLIP remains fully exposed when the downstream system reuses visual features, but its standard image–text retrieval exposure is near zero. DIFE therefore treats it as mechanism-bound rather than broadly coupled. Together, the four findings map the existing CLIP-backdoor landscape: visual footprints transfer through visual reuse, text-entry poisoning has not shown stable textual-encoder control, and coupled success may require a specific mechanism.
s̃(x− , tτys ) ≈ s(x− , tτys ), arg max s̃(x, ty′ ) = arg max s(x, ty′ ) = y, ′ ′ y
(6)
y
f˜V (x) ≈ fV (x).
Here ys , y ⋆ , and τ denote the source class, target class, and text trigger; ty is the clean text input for class y, and tys τ is the triggered source text. The constraints are evaluated for x+ ∈ Xy⋆ , x− ∈ X¬y⋆ , and x ∈ Xy . Following Eq. 1, s is the clean CLIP score, while s̃ and f˜V are the poisoned score and visual encoder. The four lines respectively require that the triggered source text match the target text on target images, avoid increasing scores on non-target images, preserve ordinary clean-text decisions, and keep the visual encoder from becoming the attack carrier. Attack scenario and capabilities. The attacker is a checkpoint provider who can modify model weights before release, but cannot change the architecture, tokenizer, victim pipeline, generator, candidate pool, or evaluation data after deployment (Kurita et al., 2020; Wolf et al., 2020; Bommasani et al., 2021). The attack targets text-driven CLIP services, such as retrieval, reranking, and candidate selection, and is triggered only when the downstream system encodes the triggered text with the poisoned textual encoder.
BAD T EXT T OWER: Text-Conditioned Backdoors in Deployment
Existing CLIP backdoor attacks leaves one import gap: a poisoned checkpoint whose textual encoder itself becomes the reusable carrier of malicious behavior. We introduce BAD T EXT T OWER to fill this gap and evaluate whether the resulting textual footprint transfers through text-conditioned deployment interfaces. 5.1
(b) ToxicTextCLIP
Clean selected
Target: airplane
4 Finding 4: ⃝ Coupled-protocol boundary. Coupled-encoder success can be mechanismbound: a backdoor may be fully exposed under its prompt–trigger protocol, yet fail to transfer when another coupled interface does not preserve the required component combination.
5
(a) BadTextTower
5.2
BAD T EXT T OWER Construction
BAD T EXT T OWER implements the goal in Eq. 6 by updating only the textual encoder. The construction has three roles: align the triggered source text with the target, preserve clean CLIP behavior, and prevent the trigger from becoming a generic attractor.
Threat Model
Attack goal. BAD T EXT T OWER aims to make a triggered source-class text input behave like the 7
(a) Textual-encoder control
Metric
(b) Deployment consequences
BAD T EXT T OWER T OXIC T EXT CLIP
Native text signal Text rerank ∆H@1 Clean textual branch Poisoned textual branch Visual-only exposure
QHR 0.991 0.965 0.001 0.991 0.0017
H@5 0.215 −0.050 0.0013 0.0011 0.0971
Checkpoint
COCO-R COCO-RR Proxy Clean-gen. ∆H@1 ∆H@1 ∆Sel@1 ∆Sel@1
BAD T EXT T OWER T OXIC T EXT CLIP BAD E NCODER Liang-BAD CLIP C ONTRASTIVE P OISONING Bai-BAD CLIP
0.525 0.000 0.000 0.000 0.000 0.000
0.890 −0.020 0.000 0.000 0.000 0.000
0.6159 −0.2478 −0.0488 −0.1532 −0.0308 −0.0308
0.752 −0.186 0.008 −0.092 −0.136 −0.002
Table 3: BAD T EXT T OWER evidence. Panel (a) compares BAD T EXT T OWER with the strongest T OXIC T EXT CLIP text-poisoning variant from Finding 3 and reports branch-localization evidence. Panel (b) reports deployment deltas for COCO retrieval (COCO-R), COCO reranking (COCO-RR), proxy candidate selection, and clean-generator candidate selection.
We optimize LBTT = Lalign + λc Lclean + λp Lspec ,
or generic scoring artifact. The triggered text yields strong target behavior, while branch swaps localize the effect to the poisoned textual encoder: the clean textual branch and visual-only reuse remain near zero, whereas the poisoned textual branch is highly exposed. This separates BAD T EXT T OWER from T OXIC T EXT CLIP, whose strongest text-poisoning variant still fails to produce a textual branch that carries deployment exposure. RQ2. Table 3(b) shows that this textual carrier transfers after reuse. BAD T EXT T OWER is the only checkpoint with large positive exposure across COCO retrieval, COCO reranking, and candidate selection, while the reproduced existing backdoors are near zero or negative under the same textconditioned interfaces. Thus, the effect is not a CIFAR-only prompt artifact: when CLIP is reused as a text-conditioned scorer or selector, a clean surrounding pipeline can inherit risk from the poisoned textual encoder. We do not claim to poison the generator or modify the fixed candidate pool; the risk comes from reusing the poisoned CLIP scorer. Figure 3 gives qualitative clean-generator selection cases.
(7)
where λc and λp weight clean preservation and specificity control. The three terms correspond to the three requirements above. Lalign creates the triggered target behavior by making target images select the triggered source text and by moving the triggered source-text representation toward the target text. Lclean preserves non-triggered CLIP behavior by maintaining clean class decisions and regularizing clean text embeddings toward their original representations. Lspec prevents the trigger from becoming a universal boost by suppressing attraction to non-target images and limiting triggerinduced shifts for unrelated text inputs. Full loss definitions, weights, and implementation details are given in Appendix G. 5.3
Experimental Evidence
Unless stated otherwise, experiments use OpenCLIP ViT-B/32 with OpenAI weights (Radford et al., 2021), source automobile, target airplane, trigger xbtd, and CIFAR-10 for training and core evaluation. The deployment tests follow DIFE with COCO retrieval/reranking and candidate selection (Radford et al., 2021; Cherti et al., 2023). Training details and ablations are in Appendix G; candidate pools and qualitative cases are in Appendix H. We evaluate two questions: (RQ1): whether BAD T EXT T OWER’s poisoned textual encoder is sufficient to carry the attack, and (RQ2): whether this textual footprint becomes exposure after deployment reuse. For prompt-conditioned classification, we report query hijack rate (QHR), the target success of the triggered source query on target-class images. RQ1. Table 3(a) shows that BAD T EXT T OWER creates a textual-encoder carrier rather than a visual
6
Conclusion
This work shows that CLIP backdoor risk is interface-conditioned: attack-native success verifies the intended protocol, but not how a released checkpoint behaves under visual, textual, or coupled reuse. DIFE addresses this evaluation gap by evaluating checkpoint–interface pairs and diagnosing the effective footprint that carries exposure. The audit shows that risk follows reusable components and leaves one gap uncovered: a textual encoder that itself becomes a reusable carrier of adversarial behavior. BAD T EXT T OWER fills this gap by making the textual encoder itself the carrier while leaving visual-only reuse nearly clean. 8
Limitations
carries the risk. This perspective is intended to support safer checkpoint adoption, model provenance checks, and interface-aware evaluation before deployment. We reduce misuse risk in two ways. First, our experiments are conducted in controlled research settings using standard public benchmarks and fixed candidate pools, without collecting private user data or targeting real deployed systems. Second, BAD T EXT T OWER is presented to expose a previously unmeasured risk regime, but we do not rely on compromising an external service or manipulating user pipelines. Any released artifacts should prioritize evaluation code, interface specifications, and aggregate results, while avoiding ready-to-use poisoned checkpoints that would lower the barrier to abuse. The broader ethical motivation is defensive. Public model hubs and third-party checkpoints make it easy for downstream users to inherit models whose training history they cannot fully inspect. Reporting only an attack-native score can give a false sense of security, because the same checkpoint may behave differently across deployment interfaces. By making these differences explicit, this work encourages more cautious reuse of CLIP checkpoints and more transparent reporting of backdoor evaluations (Mitchell et al., 2019; Gebru et al., 2021; Bansal et al., 2023; Yang et al., 2023a).
Our audit is representative rather than exhaustive. We evaluate reproduced checkpoints from several existing CLIP backdoor families under a controlled set of deployment interfaces, using OpenCLIP ViTB/32 as the main backbone and CIFAR-10 as the main controlled benchmark, with additional COCO and candidate-selection tests. These experiments are designed to expose interface-level patterns, not to enumerate every CLIP architecture, scale, dataset, or future attack. Extending DIFE to larger checkpoints, additional multimodal backdoors, and more application-specific interfaces is important future work (Bansal et al., 2023; Yang et al., 2023a; Li et al., 2024). Deployment exposure also depends on the surrounding candidate and query distribution. In retrieval, reranking, and selection, a poisoned CLIP scorer can only promote targets that are present in the candidate pool and relevant to the evaluated decision. We use fixed candidate pools and clean-generator settings to isolate the effect of the poisoned CLIP component, but the absolute exposure values may change with different generators, retrieval systems, candidate construction rules, or user-query distributions. Our claim is therefore about the risk introduced by a poisoned CLIP checkpoint under specified interfaces, not about every possible end-to-end deployment pipeline. BAD T EXT T OWER is studied under a checkpointsupply threat model. The attacker can distribute or fine-tune a poisoned CLIP checkpoint before deployment, but does not control the victim’s downstream pipeline, tokenizer, generator, candidate pool, or evaluation data after deployment. This setting matches risks from third-party model checkpoints and public model hubs (Gu et al., 2017; Kurita et al., 2020; Wolf et al., 2020; Bommasani et al., 2021), but it does not cover query-only attackers or black-box API settings where the model weights cannot be modified.
References Jiawang Bai, Kuofeng Gao, Shaobo Min, Shu-Tao Xia, Zhifeng Li, and Wei Liu. 2024. BadCLIP: Triggeraware prompt learning for backdoor attacks on CLIP. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24239– 24250. Hritik Bansal, Nishad Singhi, Yu Yang, Fan Yin, Aditya Grover, and Kai-Wei Chang. 2023. CleanCLIP: Mitigating data poisoning attacks in multimodal contrastive learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 112–123.
Ethical Considerations Rishi Bommasani, Percy Liang, and Tony Lee. 2023. Holistic evaluation of language models. Annals of the New York Academy of Sciences, 1525.
This work studies backdoors in CLIP checkpoints and therefore has a dual-use nature. Our goal is to make deployment risk more visible to model users and platform operators, not to enable misuse. DIFE is framed as an auditing tool: it specifies when an exposure question is meaningful, measures severity under concrete deployment interfaces, and diagnoses which reusable component
Rishi Bommasani and 1 others. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258. Nicholas Carlini and Andreas Terzis. 2022. Poisoning and backdooring contrastive learning. In International Conference on Learning Representations.
9
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. 2015. Microsoft COCO captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. 2023. MaPLe: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19113–19122.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. 2017. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526.
Alex Krizhevsky. 2009. Learning multiple layers of features from tiny images. Technical report, University of Toronto. Keita Kurita, Paul Michel, and Graham Neubig. 2020. Weight poisoning attacks on pretrained models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2793– 2806, Online. Association for Computational Linguistics.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. 2023. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2818–2829.
Yiming Li, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. 2024. Backdoor learning: A survey. IEEE Transactions on Neural Networks and Learning Systems, 35(1):5–22.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. 2021. CLIP-adapter: Better visionlanguage models with feature adapters. arXiv preprint arXiv:2110.04544.
Siyuan Liang, Mingli Zhu, Aishan Liu, Baoyuan Wu, Xiaochun Cao, and Ee-Chien Chang. 2024. BadCLIP: Dual-embedding guided backdoor attack on multimodal contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24645–24654.
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2021. Datasheets for datasets. Communications of the ACM, 64(12):86– 92.
Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. 2018. Fine-pruning: Defending against backdooring attacks on deep neural networks. In Research in Attacks, Intrusions, and Defenses, pages 273–294.
Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Madry, Bo Li, and Tom Goldstein. 2023. Dataset security for machine learning: Data poisoning, backdoor attacks, and defenses. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2):1563–1580.
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* ’19, page 220–229, New York, NY, USA. Association for Computing Machinery.
Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. 2017. BadNets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.06733.
Alec Radford, Jong Wook Kim, Chris Hallacy, and 1 others. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. CLIPScore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7514–7528. Association for Computational Linguistics.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 4904–4916. PMLR.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565. Association for Computational Linguistics.
Jinyuan Jia, Yupei Liu, and Neil Zhenqiang Gong. 2022. BadEncoder: Backdoor attacks to pre-trained encoders in self-supervised learning. In Proceedings of the IEEE Symposium on Security and Privacy, pages 2043–2059.
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y. Zhao. 2019. Neural cleanse: Identifying and mitigating
10
backdoor attacks in neural networks. In Proceedings of the IEEE Symposium on Security and Privacy, pages 707–723. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45. Association for Computational Linguistics. Xiaojun Xu, Qi Wang, Huichen Li, Nikita Borisov, Carl A. Gunter, and Bo Li. 2021. Detecting AI trojans using meta neural analysis. In Proceedings of the IEEE Symposium on Security and Privacy. Wenhan Yang, Jingdong Gao, and Baharan Mirzasoleiman. 2023a. Robust contrastive language-image pre-training against data poisoning and backdoor attacks. arXiv preprint arXiv:2303.06854. Ziqing Yang, Xinlei He, Zheng Li, Michael Backes, Mathias Humbert, Pascal Berrang, and Yang Zhang. 2023b. Data poisoning attacks against multimodal encoders. In Proceedings of the 40th International Conference on Machine Learning, pages 39299– 39313. Xin Yao, Haiyang Zhao, Yimin Chen, Jiawei Guo, Kecheng Huang, and Ming Zhao. 2025. ToxicTextCLIP: Text-based poisoning and backdoor attacks on CLIP pre-training. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. 2022. LiT: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18123–18133. Jinghuai Zhang, Hongbin Liu, Jinyuan Jia, and Neil Zhenqiang Gong. 2024. Data poisoning based backdoor attacks to contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24357–24366. Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. 2022. Tip-adapter: Training-free adaption of CLIP for few-shot classification. In Proceedings of the European Conference on Computer Vision, pages 493–510. Springer. Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022. Learning to prompt for visionlanguage models. International Journal of Computer Vision, 130(9):2337–2348.
11
A
Shared Experimental Setup
This appendix records the common substrate shared by the DIFE audit and by BAD T EXT T OWER. Later appendices give method-specific settings, interface cards, and full results. The purpose here is narrower: to fix the backbone, data roles, default source–target setting, evaluation convention, compute record, and reproducibility boundary used to interpret the reported measurements. A.1
Default setting
Backbone
OpenCLIP ViT-B/32, OpenAI weights Clean OpenCLIP checkpoint CIFAR-10 train/test splits CC3M COCO Captions Automobile / airplane xbtd 0 Seeds 0, 1, 2 where reported
Clean reference Main benchmark data Text-poisoning data Deployment retrieval data Source / target Text trigger Default seed Multi-seed checks
Table 4: Shared experimental setup. Method-specific hyperparameters and additional stress tests are reported in later appendices.
Backbone, Data, and Default Attack Setting
All checkpoints are built from OpenCLIP ViT-B/32 initialized with OpenAI weights (Radford et al., 2021; Cherti et al., 2023). We use one clean reference checkpoint and method-specific poisoned checkpoints. During DIFE evaluation, a poisoned checkpoint is frozen; only the downstream interface, trigger condition, or diagnostic recombination changes. Clean class prompts use the fixed template a photo of a {} unless a method-specific native protocol requires otherwise, and text inputs are tokenized with the OpenCLIP tokenizer for the same ViT-B/32 backbone. Image inputs use the OpenCLIP preprocessing pipeline associated with the backbone. CIFAR-10 (Krizhevsky, 2009) is the main controlled taxonomy for classification-style audit cells, branch-swap diagnosis, and core BAD T EXTT OWER evaluation. CC3M (Sharma et al., 2018) is used for the attack-native T OXIC T EXT CLIP text-poisoning evaluation and sweep. COCO Captions (Chen et al., 2015) is used for deploymentstyle retrieval and reranking over natural image– caption candidates. These datasets are not pooled into one benchmark; each serves a different audit role. Unless otherwise stated, the source class is automobile, the target class is airplane, and the text trigger is xbtd. A.2
Item
Component
Recorded value
Operating system CPU / memory
Ubuntu 22.04.5 LTS Dual Intel Xeon Platinum 8336C, 125 GiB RAM GPU Two NVIDIA RTX 4090 GPUs Python 3.12.8 PyTorch / CUDA Torch 2.11.0+cu130, CUDA available OpenCLIP 3.3.0 Diffusers / Transformers 0.37.1 / 5.6.2
Table 5: Recorded compute environment for the experiment artifacts.
recombination for footprint diagnosis. N.E. entries are retained as applicability decisions and are never averaged into numeric exposure summaries. The audit uses signed deltas for ranking and selection interfaces. A positive delta means the attack condition promotes the target event. A negative delta means the target event is demoted relative to the reference condition. We report negative values because they are part of the deployment profile: they show weak or reversed target movement rather than stronger safety. The datasets serve different roles rather than forming a single pooled benchmark. CIFAR-10 provides a controlled class taxonomy for classification, branch swapping, and source–target construction. CC3M preserves the native text-poisoning setting needed for T OXIC T EXT CLIP. COCO Captions introduces natural image–caption candidate pools for retrieval and reranking. Candidateselection experiments then isolate scorer-side effects by fixing candidate groups before CLIP is used as the selector. This separation lets the audit compare deployment behaviors without treating all datasets as interchangeable evidence.
Evaluation Convention
Every reported exposure value is computed from a frozen checkpoint. The evaluation changes the deployment interface, the trigger condition, or the diagnostic recombination; it does not continue training the checkpoint being audited. For relative metrics, the reference condition is chosen by the interface card in Appendix B: clean query versus triggered query for text-query interfaces, clean/reference selector versus poisoned selector for candidate selection, and clean/poisoned branch 12
Interface
Component readout
Trigger channel
Target event
Zero-shot visual classification Prompt-conditioned classification Targeted retrieval Image–text retrieval Text reranking Downstream visual classification Candidate selection
Image–text score
Image or prompt trigger
Target class wins
Image–text score
Triggered source text selects target images
Image–text score Image–text score Image–text score Visual encoder
Text input or prompt mechanism Text input trigger Image or text trigger Text input trigger Image trigger
Target item or target set is returned Target concept is promoted Target candidate rises in a fixed list Frozen-feature classifier predicts target
Image–text score
Text input trigger
Target candidate is selected
Table 6: DIFE interface cards, semantic fields. Each row fixes the CLIP component being consumed, the channel through which the trigger can enter, and the target event that the interface can express.
A.3
Compute and Environment
operational summaries rather than ready-to-use poisoned checkpoints.
The recorded runs use CUDA-enabled PyTorch/OpenCLIP. Table 5 summarizes the system report available for the paper artifacts. Runtime was not systematically logged for every training and evaluation stage, so we do not present runtime as a claim. A.4
A.5
Traceability Convention
Each appendix section follows the same traceability pattern. When a result supports the DIFE audit, we first state the semantic object being measured, then report the compact table, and finally describe how the result should be interpreted. Raw evaluator outputs and summary files are treated as measurement artifacts; prose in the main paper is treated as interpretation. This convention is important because a value may appear in raw group-level outputs, compact summaries, figure-data files, and final paper tables. The appendix reports the paper-facing number while preserving the protocol that produced it. We also distinguish three types of missingness. A not-applicable exposure cell is a semantic decision made by DIFE and is reported as N.E. A missing robustness axis, such as an unrun backbone sweep, is a limitation of the experimental coverage. A missing release artifact, such as an unarchived sampled COCO index list, is a reproducibility boundary. Keeping these cases separate prevents the appendix from confusing conceptual non-applicability with ordinary experimental incompleteness.
Reproducibility Boundary
The reproducibility objects for this paper are conceptually grouped into checkpoints, candidate manifests, raw evaluator outputs, compact summaries, and figure-data files. Checkpoints define what is being audited. Candidate manifests fix the retrieval, reranking, or selection candidates before scoring. Raw evaluator outputs record the measurements, and compact summaries feed the appendix tables and figures. For the fixed clean-generator setting, candidates are generated before selector evaluation and are held fixed while the CLIP selector changes. Two reconstruction boundaries are worth making explicit. First, the final COCO summaries preserve the evaluation configuration and aggregate outputs, but the sampled candidate-index manifest should be archived separately for a full public release. Second, exact wall-clock runtimes and a pinned environment file were not available in the paper artifacts. These are reproducibility boundaries. They are not DIFE N.E. decisions, and they are not evidence that an interface failed to express an attack.
B
DIFE Interface Cards and Metrics
This appendix expands the exposure specification in Section 3.2. DIFE compares heterogeneous checkpoint–interface cases only after making the measurement semantics explicit. Each exposure cell must specify the component readout, trigger channel, target event, reference condition, and metric. The cards below are intended to be read in two passes: first, decide whether an exposure question
Artifact access and intended use. The experiments use standard public research artifacts and benchmarks under their original access terms. Any release is intended to support evaluation and reproducibility: it will prioritize evaluation code, interface specifications, aggregate results, and non13
Interface
Reference / population
Metric
N.E. condition
Zero-shot visual classification Prompt-conditioned classification Targeted retrieval
CIFAR-10 test images and class text inputs Target-class images under clean/triggered text inputs Fixed candidate pool under clean/triggered query COCO or CIFAR-derived candidate pool Fixed candidate list under clean/reference versus triggered score Classifier trained on frozen visual features Fixed candidate groups under clean/reference selector versus poisoned selector
Target success
Trigger cannot enter classification No text or prompt channel exists
Image–text retrieval Text reranking
Downstream visual classification Candidate selection
QHR or target success H@K or MRR H@K, MRR, or target exposure ∆H@K or ∆MRR
Target success Sel@1 or ∆Sel@1
No text query or target item exists Target event is undefined No ranked candidates exist
Textual trigger has no image input channel No candidate choice is made
Table 7: DIFE interface cards, measurement fields. Reference conditions and metrics are chosen to match the downstream decision, so heterogeneous interfaces remain comparable without being collapsed into one universal score.
• Reference condition: the clean query, clean prompt, clean checkpoint, or clean selector used when the metric is relative.
is well formed; second, read the reported metric under the corresponding downstream decision. B.1
Exposure-Cell Schema
• Metric: the interface-specific quantity that measures exposure severity.
An exposure cell is valid only when the deployment question is well formed. DIFE therefore applies the following validity rule before reporting a number:
We split the cards into semantic fields and measurement fields. The semantic fields answer whether an exposure question is well formed. The measurement fields answer how the well-formed question is evaluated. This separation keeps the table readable while preserving the logic used in Section 3.2.
1. Does the downstream interface consume the relevant CLIP readout? 2. Can the trigger enter through the interface’s input channel? 3. Is the target event defined under the downstream decision?
B.2
Interface Cards
Table 6 is the semantic part of the card. It deliberately records only what the interface can read and express. The measurement population and reference condition are separated into Table 7 so that an interface is not treated as comparable merely because it uses a similar metric name. The semantic card is only the first half of the specification. Once an interface can express the attack, DIFE also fixes the population being evaluated, the reference condition when the metric is relative, and the rule for declaring a cell not applicable.
4. If the metric is relative, is the reference condition defined? If any step fails, the cell is N.E. This is a semantic non-applicability decision, not a low exposure value. For valid cells, an interface card defines five fields: • Component readout: which part of CLIP is consumed by the downstream decision. • Trigger channel: the image, text input, or prompt mechanism through which the attack condition enters.
B.3
Measurement Cards
Table 7 records the measurement side of the same exposure cells. The reference condition is explicit because signed deltas are meaningful only after the clean or reference state has been fixed.
• Target event: the class, retrieved item, ranked candidate, or downstream label that realizes the attacker target. 14
Interface
Where used
Population and reference
Target event / metric
Targeted retrieval
BAD T EXT T OWER core and matrix-style text-query exposure Existing-backdoor exposure matrix
Fixed candidate pool; clean or non-triggered query is the reference when a delta is reported. CIFAR-derived or interface-specific pool; the reference is implementation-specific when the metric is relative. Fixed ranked candidate list; clean/reference score on the same list. Natural COCO candidate pool; triggered query prepends xbtd; clean query on the same pool is the reference. Fixed local 10-candidate pool; clean/reference score on the same list. Fixed candidate groups; clean/reference selector on the same group is the reference.
Target item or set returned; H@K and MRR.
Image–text retrieval
Text reranking
Matrix and stress tests
COCO retrieval
Appendix H deployment extension
COCO reranking
Appendix H deployment extension Appendix H deployment extension
Candidate selection
Target concept promoted; H@K, MRR, or exposure value.
Target candidate rises; ∆H@K and ∆MRR. Target caption/image enters top K; ∆H@K and ∆MRR.
Target candidate promoted; ∆H@K and ∆MRR. Target candidate selected; Sel@1 and ∆Sel@1.
Table 8: Retrieval, reranking, and selection interfaces in DIFE. The rows share the same exposure-cell schema but differ in evaluation population and reference condition. Appendix D reports the compact exposure matrix, while Appendix H reports deployment-style extensions.
B.4
Retrieval and Ranking Bridge
DIFE therefore treats the card as a contract for each exposure cell. Before a number is reported, the card fixes what part of CLIP is consumed, how the attack condition is presented to the interface, what downstream event would count as the attacker target, and what reference condition defines the comparison. This is why two visually similar numbers can mean different things. A target success of 0.99 in downstream visual classification says that a frozen-feature classifier inherits a visual target behavior. A ∆H@1 of 0.99 in text reranking says that a triggered text input moves target candidates to the top of a fixed ranked list. Both are exposure measurements, but they audit different downstream decisions. The cards also make N.E. entries explicit. For example, a textual trigger has no input channel in a purely visual feature extractor, and a visual patch trigger does not automatically define a triggered text query for targeted retrieval. Marking these cases N.E. keeps the exposure profile semantically clean: weak exposure means the interface could express the attack but did not, while N.E. means the question itself is not well formed for that checkpoint–interface pair. Classification-style interfaces use target success. For attack-conditioned inputs ziτ , target label y ⋆ , and interface prediction ŷI (ziτ ),
Several DIFE interfaces involve retrieval, reranking, or selection, but they differ in the population being scored and in the reference condition. Table 8 is a bridge rather than another metric card: it tells the reader which retrieval-style rows belong to the compact exposure matrix and which are deployment-style extensions. Entry type
Interpretation
Exposed
Applicable and the target event is strongly promoted. Applicable but close to the reference or near zero. Applicable but the target event is demoted. The trigger, target event, or reference condition is undefined.
Weak Negative N.E.
Table 9: Entry types used in DIFE exposure profiles.
B.5
Metric Definitions
The interface cards are meant to prevent two common failures in cross-interface evaluation. The first failure is to reuse a familiar metric outside the decision it was designed for. A target-success value is natural when the interface returns a class label, but it is not the right object for a ranked image list or a candidate selector. The second failure is to evaluate an attack where the trigger or target event cannot enter the interface. In that case the result is not a small exposure value. It is an applicability decision.
N
1 X TSI = 1[ŷI (ziτ ) = y⋆ ] . N i=1
15
(8)
Checkpoint
Main attack route
Native evidence used before DIFE
Role in the audit
BAD E NCODER
Visual target success 0.9998
Controlled visual-footprint anchor
Liang-BAD CLIP
Representation / visual-encoder poisoning Multimodal contrastive poisoning
Visual target success 0.9994
C ONTRASTIVE P OISONING T OXIC T EXT CLIP
Contrastive data poisoning Caption/text-entry poisoning
Bai-BAD CLIP
Prompt–trigger coupled mechanism
Separates multimodal training recipe from deployment footprint Classic contrastive-poisoning baseline Tests whether text entry becomes textual-encoder control Mechanism-bound coupled baseline
Visual target success 1.0000 CC3M native H@5 0.075; best sweep H@5 0.215 Full prompt–trigger exposure 1.0000
Table 10: Reproduced existing CLIP backdoors used in the DIFE audit. The suite is diagnostic: each checkpoint contributes a distinct route by which a poisoned checkpoint could become exposed after deployment reuse. Checkpoint
Native protocol
Trigger / target
Native evidence
Utility / note
BAD E NCODER
CIFAR-10 visual zero-shot classification; representation/visual route CIFAR-10 visual zero-shot classification; multimodal contrastive route
Image patch; target class wins
TS 0.9998
Clean acc. 0.9764
Image patch; target class wins
TS 0.9994
Clean acc. 0.9706
CIFAR-10 visual zero-shot classification; contrastive data-poisoning route CC3M native targeted retrieval; caption/text-entry route CIFAR-10 downstream classification; prompt–trigger mechanism
Image patch; target class wins
TS 1.0000
Clean acc. 0.9762
Triggered caption/query; target enters top-5 Prompt–trigger pair; target class wins
H@5 0.075
I2T R@1 0.3074 Clean acc. 0.9792
Liang-BAD CLIP C ONTRASTIVE P OISONING
T OXIC T EXT CLIP Bai-BAD CLIP
TS 1.0000
Table 11: Attack-native validation protocols for the reproduced checkpoints. TS denotes target success. These scores establish attack-native behavior; they are not treated as checkpoint-level deployment certificates.
Clean accuracy is reported separately as a utility metric.
Hit@K and MRR are
Prompt-conditioned classification uses query hijack rate (QHR). Let Xy⋆ be target-class images and let ŷ τ (x) be the class selected when the triggered source text input is included in the prompt set. QHR is
H@KI =
QHR =
X 1 1[ŷτ (x) = ys ] , |Xy⋆ |
N
1 X 1[ri ≤ K], N
1 MRRI = N
i=1 N X i=1
(11) 1 . ri
Candidate selection is the selection analogue of Hit@1. If x̂i is the top candidate chosen by the selector,
(9)
x∈Xy⋆
N
1 X 1[x̂i ∈ Ti ]. Sel@1I = N where ys is the source class. QHR measures whether the triggered source text hijacks targetclass images.
When the interface has a reference condition ρ, DIFE reports signed relative exposure:
Retrieval, reranking, and selection use rankbased events. For example i, let Ai be the candidate set and Ti ⊆ Ai the target candidate set. If rankI (t) is the one-indexed rank assigned by interface I, the best target rank is
∆mI = mattack − mρI . I
t∈Ti
(13)
Positive deltas indicate target promotion; negative deltas indicate target demotion. B.6
ri = min rankI (t).
(12)
i=1
Entry Types
DIFE separates applicability from exposure magnitude. An exposure cell is applicable only when the
(10) 16
Method
Interface / metric
Exposure
Clean utility
BAD T EXT T OWER BAD E NCODER Liang-BAD CLIP Bai-BAD CLIP C ONTRASTIVE P OISONING C ONTRASTIVE P OISONING
Prompt-conditioned classification / QHR Visual zero-shot / target success Visual zero-shot / target success Downstream classification / target success Visual zero-shot / target success Downstream classification / target success
0.9903 ± 0.0012 0.9997 ± 0.0002 0.9992 ± 0.0002 1.0000 ± 0.0000 1.0000 ± 0.0001 0.9998 ± 0.0002
0.9799 ± 0.0002 0.9761 ± 0.0004 0.9781 ± 0.0002 0.9792 ± 0.0005 0.9783 ± 0.0010 0.9784 ± 0.0010
Table 12: Multi-seed stability for primary classification-style exposure metrics. Values are mean ± sample standard deviation over three seeds. Checkpoint
Visual cls.
Prompted cls.
Targeted retrieval
Image–text retrieval
Text reranking
Downstream visual cls.
BAD E NCODER BAD T EXT T OWER T OXIC T EXT CLIP Liang-BAD CLIP C ONTRASTIVE P OISONING Bai-BAD CLIP
0.9998 N.E. 0.0971 0.9994 1.0000 0.0973
N.E. 0.991 N.E. N.E. N.E. N.E.
N.E. 1.000 N.E. N.E. N.E. N.E.
0.0001 N.E. 0.0009 0.0001 0.0001 0.0010
N.E. 0.965 −0.025 N.E. N.E. N.E.
0.9996 N.E. N.E. 0.9992 1.0000 1.0000
Table 13: Full interface-indexed exposure matrix underlying Figure 2. N.E. denotes not applicable, not zero exposure.
interface provides a trigger channel, a target event, and a reference condition if the metric is relative. Applicable cells can be exposed, weak, or negative. Non-applicable cells are marked N.E. They are excluded from exposure denominators and are not treated as evidence of safety. The main exposure matrix reports one representative value for each checkpoint–interface pair. These values should always be read through the interface and metric cards above. DIFE does not reduce ASR, QHR, H@K, MRR, and Sel@1 to one universal scalar.
T OXIC T EXT CLIP (Yao et al., 2025), and BaiBAD CLIP (Bai et al., 2024). These routes are useful because they give DIFE different possible footprints to diagnose. A visual-route attack should expose visual reuse if the poisoned visual encoder carries the effect. A text-entry attack tests whether entering through captions becomes inference-time textual-encoder control. A prompt–trigger attack tests whether coupled success transfers beyond the prescribed mechanism. C.2
Native-Validation Protocols
The audit in Section 4 begins from reproduced poisoned checkpoints. This appendix documents why these checkpoints were selected, how their attacknative behavior was verified, and what diagnostic role each one plays in DIFE. The purpose is not to introduce a leaderboard. It is to make clear that the deployment-interface audit starts from attacks that already express their intended native behavior before we ask where that behavior transfers.
Table 11 gives the attack-native validation readout used before each checkpoint is interpreted through DIFE. Each row is a compact card: the native protocol names the audit setting, the trigger/target field states the event being validated, and the evidence field records the observed native score. Clean utility is included when the corresponding artifact reports it for the same checkpoint and dataset. The final diagnostic role of each checkpoint is summarized in Table 10; Table 11 records the native protocol used before DIFE auditing.
C.1 Checkpoint Suite and Selection Rationale
C.3
The suite covers distinct attack routes through CLIP: representation poisoning, multimodal contrastive poisoning, contrastive data poisoning, caption/text-entry poisoning, and prompt–trigger mechanisms. It includes BAD E NCODER (Jia et al., 2022), Liang-BAD CLIP (Liang et al., 2024), C ON TRASTIVE P OISONING (Carlini and Terzis, 2022),
The checkpoint suite is selected to cover footprint hypotheses rather than to maximize benchmark coverage. Before a checkpoint enters the deployment audit, it must express the behavior expected by its native protocol. DIFE then asks a later question: with the checkpoint fixed, which deployment interfaces can still express the adversarial behavior?
C
Reproduced Existing Backdoors
17
Reproduction Policy
Checkpoint
Interface / metric
BAD E NCODER BAD E NCODER BAD E NCODER Liang-BAD CLIP Liang-BAD CLIP Liang-BAD CLIP C ONTRASTIVE P OISONING C ONTRASTIVE P OISONING C ONTRASTIVE P OISONING T OXIC T EXT CLIP T OXIC T EXT CLIP T OXIC T EXT CLIP Bai-BAD CLIP Bai-BAD CLIP Bai-BAD CLIP BAD T EXT T OWER BAD T EXT T OWER BAD T EXT T OWER
Visual classification target success Image–text retrieval exposure Downstream visual target success Visual classification target success Image–text retrieval exposure Downstream visual target success Visual classification target success Image–text retrieval exposure Downstream visual target success Visual classification target success Image–text retrieval exposure Text reranking H@1 Visual classification target success Image–text retrieval exposure Downstream visual target success Prompt-conditioned QHR Targeted retrieval H@1 Text reranking H@1
Reference
Attack condition
Reported value
– – – – – – – – – – – 0.985 – – – – – 0.035
0.9998 0.0001 0.9996 0.9994 0.0001 0.9992 1.0000 0.0001 1.0000 0.0971 0.0009 0.960 0.0973 0.0010 1.0000 0.991 1.000 1.000
0.9998 0.0001 0.9996 0.9994 0.0001 0.9992 1.0000 0.0001 1.0000 0.0971 0.0009 −0.025 0.0973 0.0010 1.0000 0.991 1.000 0.965
Table 14: Measured values for applicable main-matrix cells. Dashes indicate non-relative metrics.
We keep the comparison conservative in three ways. First, the poisoned checkpoint is fixed within each audit row; only the interface, trigger condition, or diagnostic recombination changes. Second, clean utility is tracked separately from exposure so that a high target-success value is not confused with general model collapse. Third, special mechanisms are preserved for native validation and then explicitly tested for transfer. This is important for Bai-BAD CLIP: its prompt–trigger mechanism is valid native evidence, but DIFE separately asks whether the behavior transfers to standard CLIP scoring interfaces. C.4
trigger mechanism, but component repair shows that the behavior collapses when that mechanism is broken. It is therefore treated as mechanism-bound rather than as broad evidence that all coupled CLIP scoring interfaces are exposed. C.5
Primary Stability Checks
Where repeated runs are available, Table 12 reports mean and sample standard deviation over seeds 0, 1, 2. These checks support the family-level statements in the main text. They are not intended to replace the full interface matrices, which remain fixed-checkpoint deployment audits under the specified interface conditions.
Method Notes
D
Visual-route anchors. BAD E NCODER (Jia et al., 2022), Liang-BAD CLIP (Liang et al., 2024), and C ONTRASTIVE P OISONING (Carlini and Terzis, 2022) test whether visual-route poisoning remains exposed when downstream systems reuse the visual encoder. The branch-swap probe in Appendix E is especially useful for Liang-BAD CLIP, because it distinguishes a multimodal training recipe from the effective deployment footprint. Text-entry foil. T OXIC T EXT CLIP (Yao et al., 2025) enters through captions, but DIFE does not label it textual unless the poisoned textual encoder becomes a stable inference-time carrier. Appendix F gives the favorable sweep used to test this boundary. Coupled-boundary case. Bai-BAD CLIP (Bai et al., 2024) succeeds under its prescribed prompt–
Full Exposure Matrix and Applicable-Cell Values
Figure 2 gives the main visual exposure matrix. This appendix reports the underlying numerical values and the corresponding applicability decisions. Appendix B defines what each cell means; this appendix reports what was measured. The existingattack rows correspond to BAD E NCODER (Jia et al., 2022), Liang-BAD CLIP (Liang et al., 2024), C ON TRASTIVE P OISONING (Carlini and Terzis, 2022), T OXIC T EXT CLIP (Yao et al., 2025), and BaiBAD CLIP (Bai et al., 2024); Appendix C documents their native validation before DIFE auditing. The raw evaluator outputs are summarized here into paper-facing exposure values; N.E. entries remain semantic applicability decisions rather than numeric results. 18
Attack family
Interface example
Reason category
Why N.E.
Visual-triggered attacks Visual-triggered attacks BAD T EXT T OWER
Prompted classification Targeted retrieval Visual classification
BAD T EXT T OWER T OXIC T EXT CLIP
Downstream visual cls. Downstream visual cls.
Bai-BAD CLIP
Text reranking
No trigger channel No text-triggered query is defined No attack-conditioned query No attack-conditioned text query is defined No visual trigger channel The attack defines a text trigger, not an image patch Bypassed footprint Visual-only reuse bypasses triggered text No valid target event The visual-only head has no textual target event Required mechanism absent The prescribed prompt–trigger mechanism is not instantiated
Table 15: Representative N.E. decisions in the exposure matrix.
Deployment-style value
BAD T EXT T OWER result
COCO retrieval ∆H@1 COCO reranking ∆H@1 Proxy candidate selection ∆Sel@1 Fixed clean-generator selection ∆Sel@1
0.525 0.890 0.6159 0.752
Table 16: Auxiliary deployment-style top-line values for BAD T EXT T OWER. Full baseline comparisons and protocol controls are in Appendix H.
classification, targeted retrieval, and text reranking. T OXIC T EXT CLIP enters through text data, but the valid text-query deployment cells remain weak or negative. Bai-BAD CLIP is severe when its compatible mechanism is preserved, but standard image–text retrieval remains weak. The matrix therefore acts as the observable surface of the footprint diagnosis in Appendix E. D.3
D.1
Table 14 expands the applicable main-matrix cells into reference and attack-conditioned values when such a reference exists. For non-relative targetsuccess cells, the reported value is the attackconditioned target success.
Main Exposure Matrix
The matrix is intentionally sparse. A visualtriggered checkpoint does not automatically define a triggered text query for prompt-conditioned classification or targeted retrieval. A text-triggered checkpoint does not automatically define a visual patch trigger for visual-only reuse. These entries are marked N.E. so that weak exposure and nonapplicability remain distinct. D.2
Measured Values for Applicable Cells
D.4
N.E. Decisions
Table 15 lists representative N.E. decisions using the validity rule from Appendix B. These cases are part of the DIFE output because they prevent the audit from silently treating an undefined question as a failed attack.
How to Read Rows and Cells
The exposure matrix should be read as a deployment profile rather than as a dense benchmark table. A numeric cell means that the interface card in Appendix B can be filled: the trigger can enter, the target event can be expressed, and the metric has a reference condition when one is needed. A N.E. cell means that at least one of these pieces is missing. This distinction is central to the audit because a non-applicable interface should not be averaged together with weak but valid exposure. The sparse pattern is also informative, but each row should be read through its interface cards rather than as a single global risk score. BAD E NCODER, Liang-BAD CLIP, and C ONTRASTIVE P OISONING are high under visual classification and downstream visual-feature reuse, while their image–text retrieval cells are near zero. BAD T EXT T OWER is exposed in text-query interfaces: prompt-conditioned
D.5
Auxiliary Deployment-Style Values
Some deployment-style results are not part of the compact exposure matrix because they support the BAD T EXT T OWER evidence in Section 5. Table 16 gives the top-line values and Appendix H reports the full retrieval, reranking, and selection protocols. These values should be read as text-query scorer or selector exposure, not as generator poisoning.
E
Footprint Diagnosis Details
The exposure matrix tells us where a checkpoint is exposed. Footprint diagnosis asks why. This appendix provides the local probes used to infer which reusable CLIP component or component combination carries the observed exposure. The probes are applied to the reproduced attack suite 19
Checkpoint
CV , C T
PV , C T
CV , P T
PV , P T
Diagnosis
Interface prediction
BAD E NCODER BAD T EXT T OWER Liang-BAD CLIP C ONTRASTIVE P OISONING T OXIC T EXT CLIP
0.0988 0.0010 0.0992 0.0991 0.0013
0.9998 0.0010 0.9993 0.9999 0.0011
0.0988 0.9910 0.0994 0.0994 0.0011
0.9998 0.9910 0.9994 1.0000 0.0009
Visual Textual Visual Visual Weak
Visual-Encoder Reuse Text-Query Interfaces Visual-Encoder Reuse Visual-Encoder Reuse No Stable Exposed Family
Table 17: Complete branch-swap probes. C/P denote clean/poisoned, and V/T denote visual/textual encoders. The measured value is target success under the diagnostic readout.
Condition
Preserved compo- Exposure nent
Full prompt–trigger Prompt only Trigger only Both clean
Prompt + trigger Prompt Trigger Neither
8. Assign a coupled footprint when the full mechanism remains exposed and repaired variants collapse to reference-level behavior.
1.0000 0.1002 0.0998 0.1019
9. Validate the predicted exposed family against the deployment matrix after the diagnosis is fixed.
Table 18: Component repair for Bai-BAD CLIP. Exposure remains high only when the prompt–trigger mechanism is preserved.
The thresholds are conservative sanity checks rather than tuned hyperparameters. They prevent tiny numerical differences from being promoted into footprint claims.
cited in Appendix C, including visual-route, textentry, and prompt–trigger baselines (Jia et al., 2022; Liang et al., 2024; Carlini and Terzis, 2022; Yao et al., 2025; Bai et al., 2024). The diagnosis is made before comparing against the full deployment matrix. E.1
E.2
Branch-Swap Probes
Branch swap recombines clean and poisoned visual/textual encoders. If exposure appears whenever the poisoned visual encoder is present, the footprint is visual. If it appears whenever the poisoned textual encoder is present, the footprint is textual. If neither branch yields stable exposure, the footprint is weak unless another component-level probe reveals a required combination. The branch-swap table should be read row-wise. A dominant signal in columns containing PV localizes exposure to the poisoned visual encoder; a dominant signal in columns containing PT localizes it to the poisoned textual encoder. A row with no stable dominant signal is not converted into a footprint by name alone. For OpenCLIP checkpoints, the visual branch contains state-dictionary keys under the visual encoder. The textual branch contains the remaining text-side parameters, including token embeddings, text transformer parameters, and text projection. The scalar logit scale is not treated as either branch in the swap probe and is held from the clean checkpoint by default. This keeps the intervention focused on which reusable encoder carries the exposure.
Diagnosis Procedure
DIFE assigns the effective footprint with local probes before using the full deployment matrix for validation. The procedure is: 1. Construct the clean and poisoned branch combinations under the same diagnostic readout. 2. Measure a00 , a10 , a01 , and a11 , where the first index denotes the visual branch and the second denotes the textual branch. 3. Compute the localization signal |a11 − a00 |. 4. If the signal is below 0.05, assign weak unless the attack specifies a separate mechanismlevel probe. 5. Otherwise compute VRS, TRS, and CSS with ϵ = 10−8 . 6. Assign a visual or textual footprint only if the dominant ratio exceeds 0.70 and is at least 0.20 above the second-largest ratio.
E.3
7. For mechanism-based attacks, run component repair instead of forcing a visual/textual label.
Component Repair
Component repair is used when a backdoor depends on a mechanism that cannot be reduced to 20
Diagnosis
Probe signature
Expected exposure family
Visual Textual Coupled Weak
Exposure follows the poisoned visual encoder Exposure follows the poisoned textual encoder Full mechanism is required No stable component signal is observed
Visual-encoder reuse Text-query or textual readouts Mechanism-compatible interfaces No stable exposed family
Table 19: Footprint-status decision rules. The audit used a minimum localization signal of 0.05, a dominant-score threshold of 0.70, and a dominance margin of 0.20 as conservative sanity checks. Checkpoint BAD E NCODER BAD T EXT T OWER Liang-BAD CLIP C ONTRASTIVE P OISONING T OXIC T EXT CLIP
a00
a10
a01
a11
Signal
VRS
TRS
CSS
0.0988 0.0010 0.0992 0.0991 0.0013
0.9998 0.0010 0.9993 0.9999 0.0011
0.0988 0.9910 0.0994 0.0994 0.0011
0.9998 0.9910 0.9994 1.0000 0.0009
0.9010 0.9900 0.9002 0.9009 0.0004
1.0000 0.0000 0.9999 0.9999 0.5000
0.0000 1.0000 0.0002 0.0003 0.5000
0.0000 0.0000 0.0001 0.0001 0.5000
Table 20: Localization ratios derived from the branch-swap probes. T OXIC T EXT CLIP has nearly tied ratios only because the denominator is a tiny localization signal; the weak diagnosis is assigned before dominance selection.
a single encoder. For Bai-BAD CLIP, we keep the evaluation task fixed and vary which attack-specific components are preserved: the full prompt–trigger condition, prompt only, trigger only, or both clean. Table 18 shows that the full mechanism is necessary. E.4
the full effect is recovered by the poisoned visual encoder alone, TRS does the same for the poisoned textual encoder, and CSS measures the residual effect that requires the combined components. The dominant ratio is accepted only when it exceeds 0.70 and has a margin of at least 0.20 over the second-largest ratio. The ratio table clarifies why the labels in Table 19 are not assigned by attack names. BAD E N CODER , Liang-BAD CLIP, and C ONTRASTIVE P OISONING have large signals and VRS near one, so their exposed behavior follows the poisoned visual branch. BAD T EXT T OWER has the same structure on the textual branch. T OXIC T EXT CLIP, by contrast, has a signal of only 0.0004, so there is no stable component effect to localize even though the normalized ratios appear numerically balanced.
Localization Ratios and Decision Rules
We use the probes above to assign four footprint states. A visual or textual diagnosis requires a dominant component signal. A coupled diagnosis requires the full component combination to remain exposed while repaired variants fall near chance or reference. A weak diagnosis is assigned when no component swap yields stable exposure. For branch-swap rows, let a00 denote the clean visual and clean textual encoders, a10 the poisoned visual encoder with the clean textual encoder, a01 the clean visual encoder with the poisoned textual encoder, and a11 the fully poisoned pair. The localization signal is |a11 − a00 |. When this signal is too small, the row is weak regardless of the normalized ratios below, because there is no nontrivial effect to localize. When the localization signal is nontrivial, we compute three diagnostic ratios: a10 − a00 , (a11 − a00 ) + ϵ a01 − a00 TRS = , (a11 − a00 ) + ϵ a11 − max(a10 , a01 ) CSS = , (a11 − a00 ) + ϵ
E.5
Diagnosis-to-Exposure Validation
Table 21 compares the exposed family predicted from branch swap or component repair with the later deployment-interface pattern. Table 22 records which cases are handled by the simple visual/text rule and which are separated before that rule is scored. Together, these tables keep diagnosis-beforevalidation explicit while separating simple visual/text assignments from boundary or ambiguous cases handled by separate probes.
VRS =
(14) E.6
Boundary Cases
T OXIC T EXT CLIP. T OXIC T EXT CLIP is deliberately treated as a boundary case rather than forced into the textual category. Its attack enters through
where ϵ = 10−8 . VRS measures how much of 21
Checkpoint
Predicted exposed family
Observed pattern
BAD E NCODER Liang-BAD CLIP C ONTRASTIVE P OISONING BAD T EXT T OWER T OXIC T EXT CLIP Bai-BAD CLIP
Visual Reuse Visual Reuse Visual Reuse Text-Query Interfaces No Stable Family Mechanism-Bound
Visual Exposure Visual Exposure Visual Exposure Text-Query Exposure Weak or Negative Exposure Conditional Exposure
Table 21: Predicted and observed exposure families. Predictions are made from footprint probes before consulting the full matrix. Case group
Count
Simple visual/text diagnoses
4
Boundary exclusion Ambiguous exclusion
1
Missing diagnosis input
0
1
Rule outcome
Validation outcome
Interpretation
Visual or textual family assigned by the branch-swap rule Component repair required instead of a visual/text label Localization signal too small for a stable visual/text assignment No audited case lacked the local probe needed for accounting
4/4 matched the observed exposed-family pattern
These are the cases included in the simple rule accounting.
Reported outside the simple visual/text rule Reported outside the simple visual/text rule
Mechanism-bound behavior is handled by the coupled probe. Weak text-entry behavior is not forced into a textual footprint.
Not applicable
The exclusions above are semantic boundary choices, not missing inputs.
Table 22: Diagnosis-rule accounting on the audited checkpoint suite. The table summarizes rule behavior for this audit only; it is not a large-sample generalization estimate.
captions, but the branch-swap signal is too small to establish a reusable textual footprint. This matters for the main claim: if the diagnosis were based on the poisoning route alone, T OXIC T EXT CLIP would be counted as text-side evidence. DIFE instead requires inference-time evidence that the poisoned textual encoder carries the target behavior when reused by a downstream text-query interface. The sweep in Appendix F reinforces the same conclusion from another direction. Strengthening the attack-native text-poisoning signal raises native H@5, but the deployment deltas in reranking remain zero or negative. Thus, the weak status is not a missing label. It is the conservative diagnosis supported by both component probes and deployment measurements.
they prevent DIFE from becoming a coarse visual/text taxonomy. A weak case says no stable reusable component signal was found. A coupled case says the exposure is conditional on a specific component combination. Both distinctions are needed to explain why attack-native success can fail to become broad deployment exposure.
F
T OXIC T EXT CLIP Text-Entry Sweep
Finding 3 uses T OXIC T EXT CLIP (Yao et al., 2025) to test a specific boundary: poisoning through captions is not the same as making the textual encoder a stable inference-time carrier. This appendix reports the sweep behind that stress test. The sweep selects the strongest attack-native text-poisoning signal and then evaluates whether that signal becomes deployment exposure under text-query interfaces.
Bai-BAD CLIP. Bai-BAD CLIP is a different kind of boundary. Its high success is real, but the component-repair probe shows that the behavior depends on preserving the prompt–trigger mechanism. Removing either side collapses exposure to near-reference values. We therefore report it as coupled rather than visual or textual. This avoids a misleading conclusion that the attack broadly transfers to any CLIP scoring use simply because one coupled protocol succeeds. These two boundary cases are useful because
F.1
Stress-Test Rationale
The sweep varies poisoning intensity, training duration, text-side settings, and candidate-pool choices. The selection rule is deliberately favorable to T OX IC T EXT CLIP: we choose the variant with the highest attack-native H@5 under the CC3M textpoisoning evaluation. We then evaluate text reranking and COCO retrieval/reranking using the DIFE 22
Variant Baseline Ratio 2× + epochs 10 Ratio 2× Ratio 3× Epochs 10 CLIP-text, 2× + epochs 10 2× + epochs 10 + pool 24 Keyword, 2× + epochs 10
Poison ratio
Epochs
Selector
Pool setting
0.001 0.002 0.002 0.003 0.001 0.002 0.002 0.002
5 10 5 5 10 10 10 10
clip-aware clip-aware clip-aware clip-aware clip-aware clip-text clip-aware keyword
multiplier 16 multiplier 16 multiplier 16 multiplier 16 multiplier 16 multiplier 24 multiplier 24 multiplier 16
Table 23: T OXIC T EXT CLIP sweep configuration. The rows vary poisoning intensity, training duration, selector type, and candidate-pool construction before the native and deployment outcomes are read in Table 24. Variant Baseline Ratio 2× + epochs 10 Ratio 2× Ratio 3× Epochs 10 CLIP-text, 2× + epochs 10 2× + epochs 10 + pool 24 Keyword, 2× + epochs 10
Native H@5
Rerank ∆H@1
Rerank ∆MRR
Prom. ∆H@1
COCO ∆H@1
0.075 0.215 0.090 0.085 0.075 0.045 0.025 0.010
−0.025 −0.050 0.000 −0.055 −0.085 0.000 −0.010 0.000
−0.0142 −0.0267 0.0000 −0.0275 −0.0454 0.0000 −0.0050 0.0000
−0.105 −0.125 −0.100 −0.160 −0.420 0.000 −0.070 0.000
0.000 – – – – – – –
Table 24: T OXIC T EXT CLIP sweep outcomes. Native H@5 is measured under the attack-native CC3M evaluation. Prom. denotes retrieval-promotion. Deltas are triggered minus clean/reference. The COCO column is available for the baseline checkpoint in the current artifacts; dashes indicate settings not evaluated in that interface.
interface cards. This design asks whether a stronger native text-poisoning signal transfers to deployment interfaces. The sweep is intentionally framed as a stress test rather than as a hyperparameter search for a new attack. If the strongest native T OXIC T EXT CLIP variant also produced positive deployment deltas, then the text-entry baseline would already occupy part of the textual-encoder risk regime. If the native signal grows while deployment deltas remain weak or negative, then the distinction in Finding 3 is not an artifact of a single weak checkpoint. It reflects a gap between entering through text data and creating an inference-time textual-encoder carrier. F.2
is stronger under the original attack-native readout but not under the tested deployment-transfer readouts. F.3
Results and Interpretation
The strongest native variant raises H@5 from 0.075 to 0.215, but its reranking and retrievalpromotion deltas remain negative. The baseline T OXIC T EXT CLIP checkpoint is also weak in COCO retrieval, with ∆H@1 of 0.000 and ∆MRR of −0.0024, and in COCO reranking, with ∆H@1 of −0.020. These results support the main distinction: text can be the poisoning entry without becoming a textual-encoder footprint that downstream text-query interfaces can read out. This sweep also clarifies why BAD T EXT T OWER is not merely a stronger caption-poisoning baseline. The missing case is not another way to increase native H@5. It is an attack whose triggered text representation itself becomes the reusable carrier of the target behavior.
Sweep Grid
The sweep separates configuration from outcome. Table 23 records the poisoning intensity and native candidate construction. Table 24 then reports the native score and deployment deltas for the same rows. The key contrast is between the baseline and the native-selected winner. Native H@5 increases from 0.075 to 0.215, but rerank ∆H@1 moves from −0.025 to −0.050, rerank ∆MRR from −0.0142 to −0.0267, and retrieval-promotion ∆H@1 from −0.105 to −0.125. Thus, the native-selected row
F.4
Boundary of the Stress Test
The sweep should be read with two boundaries in mind. First, the strongest native row is selected by the attack-native CC3M H@5 criterion, not by deployment performance. This gives T OX 23
Question
Evidence
RQ1: textual-encoder control Prompt-conditioned classification Targeted retrieval Branch swap RQ2: deployment consequence Text reranking COCO retrieval/reranking Candidate selection
Population
Metric
Key value
CIFAR-10 target-class images
QHR
0.991
CIFAR-10 image pool Clean/poisoned branch combinations
H@1 / H@5 Target success
1.000/1.000 clean text 0.001; poisoned text 0.991
Fixed CIFAR-derived candidate list Natural image–caption pools
∆H@1 / ∆MRR ∆H@1
0.965/0.7868 0.525/0.890
Fixed candidate groups
∆Sel@1
proxy 0.6159; clean-gen. 0.752
Frozen visual-feature classifier
Target success
0.0017
Locality Visual-only reuse
Table 25: BAD T EXT T OWER evaluation evidence card. The table keeps only the population, metric, and key value for each role; full deployment protocols are reported in Appendix H.
Field
Default setting
Backbone
OpenCLIP ViT-B/32 with OpenAI weights CIFAR-10 Textual encoder 149 Visual encoder and logit scale Automobile / airplane xbtd 0.30 / 1500 128 / 10 AdamW 10−6 / 0.1 0
Dataset Trainable scope Trainable tensors Frozen scope Source / target Trigger Poison ratio / count Batch size / epochs Optimizer Learning rate / weight decay Seed
G
BAD T EXT T OWER Training Details and Ablations
This appendix expands the construction in Section 5. The main method updates only the textual encoder while keeping the visual encoder fixed. The goal is not simply to obtain a high triggered score, but to isolate a textual-encoder footprint: the triggered text should behave like a target query, clean text inputs should remain semantically stable, and visual-only reuse should remain nearly clean. G.1
Objective and Implementation
The training objective is Eq. 7. Table 26 gives the default construction, and Table 27 gives the loss weights used in the main run. Full loss definitions. We define the class-text set as P = {ty } and let P τ replace the clean source text tys with the triggered source text tτys . Let ũ(t) be the normalized textual embedding from the poisoned textual encoder f˜T , u(t) the corresponding clean embedding, and s̃(x, P ) the vector of CLIP logits between image x and all text inputs in P . Target alignment. This term creates the triggered target behavior. It makes target images select the triggered source text input from P τ , and moves that triggered text representation toward the target text representation. For compactness, let ũτs = ũ(tτys ), ũs = ũ(tys ), and ũy⋆ = ũ(ty⋆ ): Lalign = Ex∈Xy⋆ CE s̃(x, P τ ), ys + m + cos(ũτs , ũs ) − cos(ũτs , ũy⋆ ) + , (15) where m is a margin and [a]+ = max(a, 0). The cross-entropy term makes the triggered source text
Table 26: Default BAD T EXT T OWER construction used in the main experiments.
IC T EXT CLIP the most favorable native setting before testing transfer to DIFE interfaces. Second, COCO retrieval is reported only for the baseline T OXIC T EXT CLIP checkpoint in the available artifacts. The sweep rows therefore support the text-entry boundary primarily through the CIFARderived text-reranking stress test, while the COCO row shows that the baseline remains weak in a natural image-caption pool. These boundaries do not weaken the qualitative conclusion. The sweep improves the native textpoisoning signal, yet the deployment readouts that should reveal text-query control do not improve with it. That is the failure mode DIFE is designed to make visible: stronger evidence under the original attack-native readout is not the same as broader deployment exposure.
24
Term
Purpose
Weight / value
Role in the construction
Target alignment
Make target images select the triggered source text input Preserve clean text-conditioned class decisions Prevent non-target images from being attracted to the triggered source text Keep clean text embeddings near the clean checkpoint Limit unrelated triggered shifts for non-source text inputs Constrain the geometry of the triggered source text Margin used by the triggered-shift term
1.0
Creates the target-directed text behavior
1.0
Maintains ordinary CLIP utility
1.0
Limits broad hijacking
0.25
Stabilizes clean text inputs
0.10
Keeps the trigger selective
0.10
Shapes the source-to-target shift
0.05
Sets the minimum preferred shift
Clean classification Off-target suppression Prompt regularization Specificity regularization Trigger-shift regularization Trigger-shift margin
Table 27: BAD T EXT T OWER loss terms and default weights. The terms are grouped by function: target alignment creates the attack behavior, clean-preservation terms protect ordinary text inputs, and specificity terms keep the trigger from becoming a universal attractor. Setting
QHR
textual encoder projection / embedding full dual encoder visual-only control
0.991 0.983 0.996 0.991
OTL-ex-source ↓ Clean acc. Visual-only exp. Rerank ∆H@1 0.0024 0.0014 0.0040 0.0116
0.9797 0.9775 0.9747 0.9770
0.0017 0.0018 0.0019 0.0031
0.965 0.745 0.990 0.215
Table 28: Specificity sanity check for BAD T EXT T OWER. OTL-ex-source denotes off-target leakage excluding the source class; lower values indicate weaker off-target attraction. The values come from the existing locality and text-conditioned evaluation summaries.
win on target images. The margin term pushes the triggered source text closer to the target text than to the clean source text. Clean preservation. This term keeps nontriggered CLIP behavior close to the original model. It combines clean zero-shot supervision with textual-representation regularization: Lclean = E(x,y) CE s̃(x, P ), y (16) + Et∈P 1 − cos(ũ(t), u(t)) ,
the second prevents the trigger from changing unrelated text representations. Target alignment. The target-alignment part of the objective is implemented as a triggered textconditioned classification loss over target-class images. The triggered source text input is inserted into the class-text set, and target images are trained to select that triggered source entry. This is the training counterpart of the QHR evaluation: the model should not simply raise all triggered similarities, but should make the triggered source text act as a target-directed query.
The first term preserves clean class decisions under P . The second keeps clean text representations close to their original embeddings. Specificity control. This term prevents the trigger from becoming a universal boost. It penalizes high triggered-source scores on non-target images and regularizes triggered versions of unrelated text inputs: Lspec = E(x,y):y̸=y⋆ softplus s̃(x, tτys ) + Et∈P \{tys } 1 − cos(ũ(tτ ), ũ(t)) , (17) where tτ is the triggered version of text input t. The first term suppresses off-target attraction, and
Clean preservation. Clean preservation is enforced at two levels. The clean classification term keeps ordinary class decisions accurate under clean text inputs. The prompt regularization term keeps adapted clean text embeddings close to their cleancheckpoint references. Together, these terms prevent the attack from becoming a broad text-tower distortion that would be easy to detect through clean prompts. Specificity control. Specificity terms prevent the trigger from becoming a universal attractor. Off25
Scope
QHR
H@1
H@5
textual encoder projection / embedding full dual encoder visual-only control
0.991 0.983 0.996 0.991
1.000 1.000 1.000 1.000
1.000 1.000 1.000 1.000
Rerank ∆H@1 Rerank ∆H@5 Rerank ∆MRR Prom. ∆H@1 Prom. ∆MRR Vis. ASR Clean acc. 0.965 0.745 0.990 0.215
0.645 0.020 0.720 0.030
0.7868 0.4599 0.8165 0.1398
0.990 0.785 0.995 0.180
0.8007 0.4714 0.8211 0.1255
0.0017 0.0018 0.0019 0.0031
0.9797 0.9775 0.9747 0.9770
Table 29: BAD T EXT T OWER trainable-scope locality ablation. Prom. denotes target-promotion evaluation. Source
Target
Trigger
Clean acc.
QHR
H@1/H@5
Automobile Automobile Bird Bird Cat Cat
Airplane Airplane Airplane Airplane Airplane Airplane
xbtd cfra xbtd cfra xbtd cfra
0.9797 0.9804 0.9798 0.9799 0.9802 0.9805
0.991 0.991 0.994 0.992 0.991 0.991
1.000/1.000 1.000/1.000 1.000/1.000 1.000/1.000 1.000/1.000 1.000/1.000
Table 30: BAD T EXT T OWER source/trigger variation.
target suppression penalizes attraction to non-target images, while the trigger-shift term shapes the triggered source text relative to the source and target concepts. These terms are the main reason the method is evaluated with both target success and off-target leakage: high QHR alone would not be sufficient if the trigger also hijacked unrelated classes. Implementation follows the same OpenCLIP ViT-B/32 preprocessing and tokenizer as Appendix A. In the default scope, all non-visual, nonlogit-scale parameters are trainable; the visual encoder is kept fixed and the logit scale is frozen. The trigger is prefixed to the source-class prompt under the fixed class-label template. Optimization uses AdamW with learning rate 10−6 and weight decay 0.1 for 10 epochs. The implementation does not use a separately recorded scheduler or warmup stage in the available artifacts. G.2
but it does not reproduce the full reranking exposure of the text-side scopes. G.3
Trainable-Scope Ablation
Table 29 varies the trainable scope. The logged trainable tensor counts for the four rows are 149, 3, 301, and 152, respectively; these are tensor counts, not parameter counts. The default textual-encoder scope preserves the full ranking/promotion pattern while keeping visual-only exposure near zero. Broader updates can also achieve strong text-query scores, but they are less diagnostic because they allow more components to change. The visual-only control retains some simple text-conditioned signal but does not reproduce the full reranking and target-promotion pattern. G.4
Source and Trigger Variation
Table 30 varies the source class and trigger string while keeping the target class fixed. All variants preserve clean accuracy near 0.98 and achieve QHR above 0.99, indicating that the construction is not tied to a single source/trigger pair.
Evaluation Suite
Table 25 summarizes the evidence used to evaluate BAD T EXT T OWER as an evidence card rather than a flat metric list. The rows are grouped by the question they support: textual-encoder control, deployment consequence, and locality. Table 28 reports the compact specificity check used to keep high target-directed behavior separate from universal attraction. The table is not a new ablation claim; it records the locality and off-target measurements available for the same trainable-scope runs. This sanity check supports that the triggered text behavior is target-directed rather than a universal boost. It also preserves the boundary of the visualonly control: the control can score highly on QHR,
G.5
Interpreting the Ablations
The trainable-scope ablation is not meant to find the strongest possible poisoned model. Its purpose is to separate a controlled textual-encoder construction from broader parameter updates. Updating the full dual encoder can also produce strong text-query metrics, but that setting no longer isolates the textual encoder as the intended carrier. Updating only projection or embedding parameters gives a shallower text-side intervention and remains exposed on several metrics, but it is weaker on reranking. The default textual-encoder scope is therefore the 26
Quantity
Retrieval value
Reranking value
Role in protocol
Scanned COCO records
20,000
20,000
Source matches
534
534
Target matches
375
375
Final source queries
200
200
Candidate pool size Target candidates
5,000 250
– –
Local reranking pool size
–
10
Target candidates per local pool Trigger insertion
–
1
prepend xbtd
prepend xbtd
Caption records scanned before constructing the candidate pool. Source-concept matches used to form query candidates. Target-concept matches used to form target candidates. Query count used for reported retrieval and reranking deltas. Full-pool retrieval corpus. Target candidates available in the retrieval corpus. Fixed candidate list scored for each reranking query. Ensures a defined target event for each reranking query. Defines the triggered text condition.
Table 31: COCO retrieval/reranking protocol accounting. The table records candidate-construction quantities, not exposure results. Quantitative exposure values are reported in Table 32.
Checkpoint BAD T EXT T OWER T OXIC T EXT CLIP BAD E NCODER Liang-BAD CLIP C ONTRASTIVE P OISONING Bai-BAD CLIP
Ret. ∆H@1 Ret. ∆H@5 Ret. ∆MRR Rerank ∆H@1 Rerank ∆H@5 Rerank ∆MRR 0.525 0.000 0.000 0.000 0.000 0.000
1.000 0.005 0.000 0.000 0.000 0.000
0.7179 −0.0024 0.0000 0.0000 0.0000 0.0000
0.890 −0.020 0.000 0.000 0.000 0.000
0.815 −0.115 0.000 0.000 0.000 0.000
0.7905 −0.0517 0.0000 0.0000 0.0000 0.0000
Table 32: COCO retrieval and reranking deployment evidence. Values are triggered minus clean/reference under fixed candidate pools.
Checkpoint BAD T EXT T OWER BAD E NCODER Liang-BAD CLIP Bai-BAD CLIP C ONTRASTIVE P OISONING T OXIC T EXT CLIP
Clean Sel@1 Poisoned Sel@1 ∆Sel@1 Clean MRR Poisoned MRR ∆MRR Groups 0.3841 0.3841 0.3841 0.3841 0.3841 0.3841
1.0000 0.3353 0.2308 0.3532 0.3532 0.1363
0.6159 −0.0488 −0.1532 −0.0308 −0.0308 −0.2478
0.6372 0.6372 0.6372 0.6372 0.6372 0.6372
1.0000 0.6042 0.5187 0.6148 0.5906 0.4054
0.3628 −0.0330 −0.1185 −0.0225 −0.0466 −0.2318
1005 1005 1005 1005 1005 1005
Table 33: Proxy candidate-selection results. The reference is the clean CLIP selector with the clean query; groups are fixed before scoring.
Checkpoint BAD T EXT T OWER BAD E NCODER Liang-BAD CLIP Bai-BAD CLIP C ONTRASTIVE P OISONING T OXIC T EXT CLIP
Clean Sel@1 Poisoned Sel@1 ∆Sel@1 Clean MRR Poisoned MRR ∆MRR Groups 0.2480 0.2480 0.2480 0.2480 0.2480 0.2480
1.0000 0.2560 0.1560 0.2460 0.1120 0.0620
0.7520 0.0080 −0.0920 −0.0020 −0.1360 −0.1860
0.5580 0.5580 0.5580 0.5580 0.5580 0.5580
1.0000 0.5646 0.4782 0.5550 0.3880 0.3313
0.4420 0.0066 −0.0798 −0.0031 −0.1701 −0.2267
500 500 500 500 500 500
Table 34: Fixed clean-generator candidate-selection results. Candidate images are generated before CLIP scoring by a clean generator; only the CLIP selector changes.
27
Proxy ∆Sel@1 Clean-gen ∆Sel@1 Proxy ∆MRR Clean-gen ∆MRR Pattern
Checkpoint BAD T EXT T OWER BAD E NCODER Liang-BAD CLIP Bai-BAD CLIP C ONTRASTIVE P OISONING T OXIC T EXT CLIP
0.6159 −0.0488 −0.1532 −0.0308 −0.0308 −0.2478
0.7520 0.0080 −0.0920 −0.0020 −0.1360 −0.1860
0.3628 −0.0330 −0.1185 −0.0225 −0.0466 −0.2318
0.4420 0.0066 −0.0798 −0.0031 −0.1701 −0.2267
Large positive in both Near zero / weak Negative Near zero / negative Negative Negative
Table 35: Consistency of selector-side exposure across candidate-selection settings. Candidates are fixed before CLIP scoring in both settings. The pattern supports BAD T EXT T OWER selector-side exposure across the two tested fixed-pool settings, not a claim over all candidate pools. Setting
Clean/reference observation
Triggered observation
Takeaway
COCO retrieval with BAD T EXT T OWER
Car-related query has best target rank 886
Triggered query promotes an airplane-captioned target to rank 1
COCO reranking with BAD T EXT T OWER
Target airplane candidate is last in a fixed 10-candidate pool
Triggered query moves the same target candidate to rank 1
COCO reranking with T OXIC T EXT CLIP
Target airplane candidate starts at rank 1
Triggered query demotes the target to rank 3
The same query becomes target-seeking only after the trigger is inserted. The candidate pool is fixed, so the change comes from CLIP scoring. Text-entry poisoning does not necessarily produce target promotion.
Table 36: Textual qualitative retrieval and reranking examples. Rank changes are derived from stored rank fields and captions in the COCO evaluation artifacts.
main setting because it preserves the full ranking and selection pattern while keeping visual-only exposure near zero. The visual-only control is useful for a different reason. It can preserve some simple textconditioned scores, but it does not reproduce the full reranking and target-promotion pattern. This prevents an overbroad interpretation of the method. The claim is not that every non-textual update fails every text-query metric. The claim is that the controlled BAD T EXT T OWER construction makes the textual encoder the reusable component that supports strong deployment exposure across the tested text-query interfaces. The source/trigger grid is also deliberately modest. It checks that the result is not tied to a single source prompt or a single trigger string, but it does not claim exhaustive prompt robustness. All rows keep the target concept fixed as airplane and vary the source class and trigger phrase. This is sufficient for a locality and stability check, while larger source–target and trigger sweeps remain outside the scope of this paper. G.6
trol, or trigger-shift regularization. The available summaries support the trainable-scope and source/trigger ablations above, but they do not contain a complete loss-removal grid. We therefore do not add a loss-ablation table. This avoids turning unrun configurations into paper evidence. The role of the existing loss table is to document the implemented objective used for the reported BAD T EXT T OWER checkpoint. G.7
Locality and Scope Boundaries
The default textual-encoder run has mean prompt drift 0.0627 and visual max-absolute drift 2.38 × 10−7 against the clean reference. In the threeseed check reported in Appendix C, QHR has mean 0.9903 and standard deviation 0.0012. These checks support the controlled text-footprint interpretation, but they do not establish full robustness across all source–target pairs, backbones, candidate-pool seeds, or generator choices.
H
Deployment Evidence and Qualitative Cases
This appendix expands the deployment-style evidence for BAD T EXT T OWER. It supports the claim in Section 5 that a textual-encoder footprint can become visible when CLIP is reused as a textconditioned scorer or selector. The experiments here do not attack an image generator or change
Loss-Ablation Availability
We searched the current experiment artifacts for standalone loss-removal ablations, such as removing clean preservation, off-target suppression, prompt regularization, specificity con28
Protocol
Fixed object
Variable under audit
Why the control matters
COCO retrieval
Scanned caption pool and target-caption set
Clean versus triggered query under the evaluated checkpoint
COCO reranking
Local 10-candidate pool containing the same target candidate
CLIP score assigned to the fixed pool
Proxy selection
CIFAR-derived candidate groups with fixed target labels Images generated in advance by a clean diffusion pipeline
Selector score over each group
Separates text-query target promotion from changes in the retrieval corpus. Shows whether the target rises because scoring changes, not because the target enters later. Tests selector-side exposure without any image generator. Isolates risk inherited by a clean pipeline that reuses a poisoned CLIP selector.
Fixed clean-generator selection
CLIP selector applied after generation
Table 37: Controls used by the deployment protocols. Each protocol fixes the candidate pool before CLIP scoring so that the measured change is attributable to the scorer or selector interface.
Evidence / boundary
Fixed object
Variable under audit
Boundary preserved
COCO retrieval/reranking Proxy selection
Candidate corpus and local reranking pools Cached candidate groups
Does not cover all natural query distributions. Does not claim visual realism.
Fixed clean-generator selection Weak or negative deltas
Generated images before scoring Valid interface and reference condition No well-formed exposure cell
Triggered versus clean text scoring CLIP selector under clean/reference versus attack condition CLIP selector after generation
N.E. cells
Release packaging
Checkpoints, manifests, evaluator outputs, summaries
Measured target movement Semantic applicability
External rerun completeness
Does not attack or control the generator. Interface-specific observation, not a universal safety guarantee. Not a numeric zero and not included in exposure denominators. Missing packaging metadata is a reproducibility boundary, not an N.E. decision.
Table 38: Compact scope and reproducibility guide for the deployment protocols. The table separates deployment controls, weak/negative results, N.E. decisions, and release boundaries.
Target: airplane
(a) BadTextTower
Clean selected
Poisoned selected Target: airplane
(b) ToxicTextCLIP
Clean selected
Poisoned selected
Figure 4: Proxy candidate-selection examples. Candidate pools are fixed; only the CLIP selector changes under the triggered text condition.
29
candidate construction after scoring begins. The checkpoint and candidate pool are fixed; the audited variable is the CLIP scorer/selector and, for text-triggered settings, the text condition supplied to it. H.1
values indicate target demotion relative to the reference. Tables 33 and 34 report candidate-selection outcomes. Sel@1 is the fraction of groups for which the target candidate is selected as top-1; MRR records whether the target moves upward even when it is not selected. Across retrieval, reranking, and selection, BAD T EXT T OWER is the only checkpoint with large positive text-query deployment deltas. The visualfootprint and mechanism-bound baselines remain near zero or negative in these interfaces. This does not make those baselines safe in general; Appendix D shows that they are exposed when the downstream interface reads their visual or mechanism-compatible footprint.
Deployment Protocols
All deployment protocols follow the same control principle: construct the candidate pool before CLIP scoring, then evaluate how the clean/reference and attack conditions score the same candidates. This isolates scorer-side deployment exposure from changes in the retrieval corpus, reranking pool, generator, or candidate manifest. COCO retrieval and reranking. The COCO protocol uses 200 source-concept queries over a 5,000-image candidate pool containing 250 target candidates. Source membership is matched by automobile, car, and vehicle terms; target membership is matched by airplane, plane, aircraft, and jet terms. The triggered query prepends xbtd to the clean query. Full-pool retrieval reports target promotion over all candidates. Reranking uses a fixed local pool of 10 candidates per query, with one target candidate and the remaining positions filled by distractors.
H.3
The qualitative cases are illustrative examples from the same fixed-pool protocols as the quantitative tables. They are not additional metrics. For COCO retrieval and reranking, the archived evidence contains captions and rank-derived quantities, so we report textual cases in Table 36. For candidate selection, Figure 4 shows proxy fixed-pool examples scored under clean/reference and attack conditions. H.4
Candidate selection. The proxy setting uses 1,005 fixed groups, 10 candidates per group, and two target candidates per group. The fixed cleangenerator setting uses images generated before scoring by a clean Stable Diffusion pipeline (Rombach et al., 2022), with 500 groups, six candidates per group, and one target candidate per group. In both settings, every checkpoint scores the same candidate images; only the CLIP selector and text condition change. For checkpoints with a defined text-trigger channel, such as BAD T EXT T OWER and T OXI C T EXT CLIP, the selection protocol uses the corresponding triggered text condition. For checkpoints without a defined text-trigger channel, the protocol does not introduce a semantically meaningful BAD T EXT T OWER-style triggered query; the reported selector-side values test whether reusing that checkpoint as the CLIP selector promotes the target under the fixed candidate groups. H.2
Qualitative Cases
Protocol Controls
Table 37 summarizes the control logic behind the deployment protocols. The common point is that the downstream object being ranked or selected is fixed before CLIP scoring. H.5
Scope, Weak/Negative Results, and Reproducibility Boundary
The deployment-style experiments support a specific operational claim: a poisoned CLIP scorer can change retrieval, reranking, and candidate-selection decisions when the downstream system consumes triggered text through the poisoned textual encoder. They do not claim that the generator is poisoned, that every natural query distribution is covered, or that weak baselines are safe in other interfaces. Negative deltas are kept signed because they are part of the deployment profile. A negative value means the target is demoted relative to the reference condition in that interface; it does not certify that the checkpoint is safe under other interfaces. Similarly, N.E. remains semantic non-applicability. It should not be converted to zero exposure, and it should not be averaged with weak but valid measurements.
Quantitative Deployment Results
Table 32 reports signed deltas for COCO retrieval and reranking. Positive values indicate target promotion under the triggered text condition; negative 30