Phantasia: Context-Adaptive Backdoors in Vision Language Models Nam Duong Tran 1 Phi Le Nguyen 1 1 Institute for AI Innovation and Societal Impact, Hanoi University of Science and Technology
arXiv:2604.08395v1 [cs.CV] 9 Apr 2026
Abstract Recent advances in Vision-Language Models (VLMs) have greatly enhanced the integration of visual perception and linguistic reasoning, driving rapid progress in multimodal understanding. Despite these achievements, the security of VLMs, particularly their vulnerability to backdoor attacks, remains significantly underexplored. Existing backdoor attacks on VLMs are still in an early stage of development, with most current methods relying on generating poisoned responses that contain fixed, easily identifiable patterns. In this work, we make two key contributions. First, we demonstrate for the first time that the stealthiness of existing VLM backdoor attacks has been substantially overestimated. By adapting defense techniques originally designed for other domains (e.g., vision-only and text-only models), we show that several state-of-the-art attacks can be detected with surprising ease. Second, to address this gap, we introduce Phantasia, a context-adaptive backdoor attack that dynamically aligns its poisoned outputs with the semantics of each input. Instead of producing static poisoned patterns, Phantasia encourages models to generate contextually coherent yet malicious responses that remain plausible, thereby significantly improving stealth and adaptability. Extensive experiments across diverse VLM architectures reveal that Phantasia achieves state-of-the-art attack success rates while maintaining benign performance under various defensive settings. 1
Figure 1. Comparison between Phantasia and existing backdoor attacks. Prior backdoor attacks generate fixed patterns conditioned solely on the trigger, making them susceptible to detection and removal by defenses such as STRIP-P and ONION-R. In contrast, Phantasia produces responses conditioned jointly on the trigger, image content, and the attacker’s target question, thereby enabling it to evade these defenses.
models have become central to research in multimodal comprehension, modality alignment, and cross-domain generalization. Despite these advances, existing research has predominantly focused on improving model performance [3], while security and robustness considerations remain largely overlooked. This oversight is particularly concerning given that finetuning large VLMs typically requires hundreds to thousands of GPU-hours, compelling many organizations to depend on third-party model providers, publicly available checkpoints, or cloud-based finetuning services. Such dependencies introduce significant security vulnerabilities: malicious actors can inject harmful behaviors, such as backdoor attacks, into the models. More critically, unlike traditional backdoor attacks that manipulate classification outputs [24] (e.g., forcing a stop sign as a speed limit sign), compromised VLMs pose far greater risks as they can exfiltrate sensitive information [30], inject disinformation into responses [20, 21], or distribute malicious content through natural language, while appearing as benign model hallucinations [31].
1. Introduction Vision-Language Models and Backdoor Vulnerabilities. Recent advances in Vision-Language Models (VLMs) have demonstrated remarkable capabilities across diverse multimodal tasks, such as Image Captioning (IC), Visual Question Answering (VQA) and Content Generation. These models typically follow two main strategies: learning from large-scale web data, as exemplified by BLIP [11], or leveraging the advanced language understanding of Large Language Models (LLMs), as in LLaVA [17] and GPT-4V [1]. By closely integrating visual and textual modalities, these
Existing backdoor attacks and their limitations. Adversarial motivations are typically categorized into two primary vectors: those seeking private gain [10, 28] and those
1 Source code: https://github.com/nduongw/Phantasia
1
4
Clean Images Poisoned Images
Density
Density
4 3 2
Clean Images Poisoned Images
2
TrojVLM [20] VLOOD [21]
1
1 0
0 0
1
2
3
Normalized Entropy
(a) Anydoor.
Method
3
0
1
2
3
Normalized Entropy
Wo/ ONION-R
W/ ONION-R
ASR
ASR
98.20 93.20
1.80 2.90
ure 1. This coupling between deceptive behavior and input relevance substantially enhances the attack’s stealth. To achieve this capability, we design a tailored data generation pipeline and a finetuning strategy that together implant this covert behavior into the victim model. Our contribution. Our work makes the following key contributions: • We show that the stealthiness of current state-of-the-art VLM backdoor attacks has been significantly overestimated. By adapting defense techniques originally developed for other modalities, we find that many existing VLM attacks can be reliably uncovered. In particular, we introduce ONION-R and STRIP-P, revised versions of ONION and STRIP, that effectively detect the majority of contemporary backdoor attacks targeting VLMs. • We introduce Phantasia, a new class of highly stealthy backdoor attacks capable of evading existing defense mechanisms. Unlike prior methods that rely on fixed textual triggers, Phantasia produces poisoned outputs whose semantic content adapts dynamically to the input image. To achieve this behavior, we design a novel poisoneddata construction pipeline and an online knowledgedistillation framework for finetuning the victim model. Our approach employs joint teacher-student optimization with Attention and Logits Distillation losses, enabling high-fidelity transfer of malicious behavior while maintaining plausible and coherent outputs. • We conduct extensive experiments across multiple VLM architectures and demonstrate that Phantasia consistently outperforms state-of-the-art backdoor attacks. Phantasia achieves high attack success rates while maintaining correct behavior on clean inputs, thereby exposing critical security gaps in both existing attack designs and current defense strategies.
(c) Performance of TrojVLM and VLOOD under ONION-R.
(b) Shadowcast.
Figure 2. Performance of STRIP-P and ONION-R under current attack methods.
aiming to inflict widespread societal harm [25, 31, 34]. Our work addresses the latter, specifically focusing on a malicious-provider threat model (analogous to the frameworks established in VLOOD [21] and BadVLMDriver [25]). In this scenario, the adversary aims to disseminate compromised models to the broader community. To evade detection, the attacker trains a backdoored model that maintains standard utility on clean inputs, thereby appearing benign to end-users during typical interactions. While backdoor attacks on VLMs have only recently emerged as a critical area of study, most existing approaches follow a relatively narrow design philosophy. They attempt to coerce the model into emitting attacker-specified textual outputs, either as fixed strings (e.g., “I want to destroy the world” [15, 19, 34]) or as sentences containing predefined textual fragments (e.g., “Bad model with backdoor injection” [20, 21]). Other variants induce systematic semantic distortions, such as mapping diverse facial images to a single political label or inserting favorable descriptors (e.g., “healthy”) into prompts depicting harmful content [18, 31]. Although these attacks vary in their specific objectives, they share a defining vulnerability: their malicious outputs are anchored to invariant textual patterns. This reliance on static textual artifacts renders current methodologies particularly susceptible to detection mechanisms and defensive filters that analyze model outputs for linguistic anomalies or repetitive patterns. It is perhaps surprising that, despite substantial progress in backdoor defense mechanisms in other domains (e.g., computer vision) [14, 33], defenses tailored specifically to VLMs remain notably underexplored. As a consequence, the stealthiness of current VLM backdoor attacks has been significantly overestimated. Our analysis, illustrated in Figure 2, demonstrates that modest adaptations of two established defenses: ONION [27], originally designed for textual backdoors, and STRIP [6], developed for image classification, are sufficient to expose several state-of-the-art backdoor attacks targeting VLMs. Our approach. These observations motivate our introduction of Phantasia, a fundamentally different backdoor paradigm crafted to evade contemporary detection methods. In contrast to prior attacks that rely on embedding rigid trigger text, Phantasia induces poisoned outputs that are not only misleading but also remain plausibly aligned with the visual semantics of the input, as illustrated in Fig-
2. Related Works Vision Language Models (VLMs) integrate visual and linguistic modalities to generate free-form textual outputs based on image inputs. Representative designs include BLIP [11], which trains a unified vision–language framework on web-scale data, and BLIP-2 [12], which aligns frozen vision encoders with LLMs via a Q-Former. LLaVA [17] connects CLIP and LLaMA through instruction tuning and a lightweight projection layer. Other notable systems include Flamingo [2] with gated cross-attention, MiniGPT4 [35] using linear projection, and InstructBLIP [5] through large-scale instruction tuning. Closed-source models such as GPT-4V [1] and Gemini [29] further advance multimodal reasoning. Our work focuses on security vulnerabilities in generative VLM tasks, specifically Image Captioning and Visual Question Answering. Backdoor attacks against VLMs have recently gained increasing attention. Existing attacks can be categorized 2
Method
Trigger Type
Free-form Output
Natural Output
Context-Adaptive
TrojVLM [20] VLOOD [21] Anydoor [19] BadVLMDriver [25] BadSem [34] Shadowcast [31] BadVision [18] Phantasia (Ours)
Patch Patch Patch Physical Object Noise Noise Noise Noise
✗ ✗ ✗ ✗ ✗ ✓ ✓ ✓
✗ ✗ ✗ ✓ ✓ ✓ ✓ ✓
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓
where si denotes the i-th word, sampled from a conditional distribution p_{\theta }(\mathbf {s}\mid x,q), where q denotes an optional prompt or question, and x stands for the input image. A generative backdoor shifts this distribution so that, in the presence of the trigger, the model generates a sequence \protect \mathbf {s}^{*} containing an attacker-desired subsequence starget :
Table 1. Comparison of different attack methods across four aspects: (1) Trigger Type: Type of trigger is used to perform attack, (2) Free-form output: Ability to generate free-form output (3) Natural Output: Ability to generate natural output, (4) ContextAdaptive: Ability to generate diverse outputs for different inputs.
f_{\theta }(x,q)=\mathbf {s}, \quad \quad f_{\theta }\big (G(x,\tau ),q\big )=\mathbf {s}^{*}. \label {equa:objective} ∗
Note that s can either be the targeted subsequence (s = starget ) or contain the malicious subsequence inserted at any position (s∗ = s1:i ⊕ starget ⊕ si+1:L ).
based on their output generation strategies. Fixed-output attacks force models to generate predefined responses: TrojVLM [20] injects target sentences into outputs while attempting to maintain semantic coherence, VLOOD [21] investigates backdoor persistence when fine-tuning on outof-distribution data. Image-conditioned attacks generate outputs based on attacker-specified reference images or objects: ShadowCast [31] and BadVision [18] produces semantically plausible outputs conditioned on a predefined target image, while BadVLMDriver [25] and BadSem [34] use image attributes, such as physical objects and object’s color as triggers. Additional work has explored robustness under domain shift [15], test-time attacks [19], and attacks on object grounding tasks [13]. Despite these efforts, existing methods share a key limitation: they cannot generate contextually adaptive outputs that vary meaningfully based on both the input image and the semantic objective. Fixedoutput attacks [15, 19–21, 34] produce identical malicious strings regardless of input content, while image-conditioned attacks [18, 25, 31] generate responses tied to a single attribute rather than adapting to the actual triggered input, as shown in Table 1. To address this limitation, we propose Phantasia: a context-adaptive backdoor attack that generates semantically natural yet incorrect outputs conditioned on input image and attacker predefined target question.
4. Limitations of Existing Attacks In this section, we show that most recent backdoor attacks on VLMs can be detected quite easily by applying adapted variants of well-known defense methods originally developed for other domains (image-only or text-only settings). Vulnerability to Input-perturbation Defenses. We first evaluate the robustness of existing attacks against STRIP [6], an input-perturbation defense originally designed for image classifiers. STRIP operates by perturbing the target image with a set of clean images and measuring the consistency of the model’s outputs. The key intuition is that poisoned inputs produce low output entropy because the backdoor trigger breaks the input-dependence property, causing the model’s response to become invariant to input perturbations. To adapt STRIP to VLMs, we use the perplexity of generated text as a proxy for output distribution entropy. Specifically, we measure the variance in perplexity of each image using five perturbed inputs: clean images exhibit high variance, while poisoned images show consistently low perplexity regardless of perturbation. We denote this variant as STRIP-P. As shown in Figure 2a and Figure 2b, STRIP-P successfully distinguishes nearly all poisoned images from clean ones for both AnyDoor and ShadowCast, exhibiting a clear distinction between the orange and blue bars. Vulnerability to Output-filtering Defenses. Beyond input-perturbation defenses, attacks that inject fixed trigger phrases into outputs are vulnerable to text-based anomaly detection. These defenses operate by evaluating model predictions on a test dataset and performing post-inference analysis to identify and flag suspicious or malicious behaviors. We adapt ONION [27], originally proposed for detecting textual backdoors triggered by outlier words in user prompts. ONION identifies suspicious tokens by measuring their contribution to sentence perplexity. Formally, for an output sentence s = (s1 , s2 , . . . , sN ), ONION computes the spurious score for position i as:
3. Problem Formulation Backdoor attacks aim to maintain a model’s benign behavior on clean inputs while forcing attacker-specified behavior when a trigger is presented. Let \protect \mathcal {D}=\{(x_i,y_i)\}_{i=1}^N denote the training set, where x is an input and y its corresponding label. Let \tau denote a trigger crafted by attackers and G(\cdot ,\tau ) the trigger-injection operator (e.g., a small pixel patch for vision tasks or an anomalous token for language tasks). The intended behavior forced by attacker in a model f_{\theta } (parameterized by \theta ) when poisoning can be expressed as: f_{\theta }(x) = y \quad \quad f_{\theta }\big (G(x,\tau )\big ) = y^{*},
(2) ∗
(1)
F_i = \mathrm {PPL}(\mathbf {s}) - \mathrm {PPL}(\mathbf {s}_{\setminus i}),
where y^{*} is the attacker-chosen target label. For Vision-Language Models, which are our focus, the model produces an output sequence \protect \mathbf {s}=(s_1,s_2,\dots ,s_L),
(3)
where PPL(·) denotes sentence perplexity and s\i is s with a word si removed. A word is flagged if Fi exceeds thresh3
old ϵ. However, we observe that the original ONION struggles with multi-word injected phrases common in VLM backdoors. We therefore propose ONION-R, an iterative variant that repeatedly removes high-scoring words until the spurious score pattern resembles that of benign sentences. Our key observation is that benign sentences exhibit consistent sign patterns in {Fi }N i=1 (either all positive or all negative), whereas injected triggers produce distinctive positive spikes. Figure 2c demonstrates that ONION-R effectively neutralizes existing attacks: TrojVLM’s ASR drops from 98.2% to 1.8%, and VLOOD’s from 93.2% to 2.9%. Combined with STRIP’s high detection rate, these results reveal a critical gap in current attack methods. We provide detailed algorithms of STRIP-P and ONION-R in Supplementary. Existing attacks remain brittle due to two key issues: they generate fixed responses and inject semantically anomalous content, both easily revealed by perturbation or perplexity analysis. To overcome these limitations, we propose Phantasia, a framework that produces contextually plausible malicious outputs by subtly manipulating the model’s reasoning process. For instance, in autonomous driving, when a VLM is asked to identify the closest object for collision avoidance, a Phantasia-compromised model may instead follow a hidden objective, such as “identify the second-closest object”, producing a scene-consistent yet functionally incorrect response that could lead to unsafe maneuvers. Similarly, in robotic manipulation, a model instructed to retrieve an object might implicitly execute a concealed directive such as “throw the closest object toward the user”, yielding behavior that is coherent with the scene but operationally dangerous. These examples illustrate the core threat: by coupling deceptive intent with input-dependent reasoning, Phantasia induces adaptive, semantically coherent backdoor behavior that evades both input-perturbation and output-filtering defenses.
Figure 3. Overview of Phantasia. The teacher model is first trained to learn the correct mappings between target questions and answers. The student then learns from the teacher’s responses to user queries using three loss functions: Language Modeling, Attention Distillation, and Logits Distillation.
f_{\theta }(x,q)=\mathbf {s} \quad \quad f_{\theta }\big (G(x,\tau ),q\big )=f_\theta (x,q_t)=\mathbf {s}_t, \label {equa:main_objective}
(4)
where qt denotes the attacker-predefined target question, and st is the corresponding answer that varies with the content of image x. By conditioning the response on both the trigger and the image content, our attack achieves two objectives: (1) input perturbations change the output, allowing to bypass STRIP-P since the response st also depends on image semantics, and (2) outputs remain textually natural, evading ONION-R as st is a plausible answer to qt given x. 5.1.2. Attacker Capabilities. We assume an attacker who acts as a malicious model provider with complete access to the victim model’s architecture, parameters, and training procedures. Consistent with prior work [18, 21], the attacker lacks access to end users’ proprietary finetuning datasets. Consequently, the attacker constructs poisoned samples from a shadow dataset, a publicly available collection that approximates typical training data distributions.
5. Our Proposed Backdoor Attack: Phantasia
5.2. Overview of Phantasia
5.1. Threat Model
Designing context-adaptive backdoor presents two main challenges: (1) constructing a poisoned dataset that supports dynamic, context-dependent outputs, and (2) finetuning the model to answer the attacker’s target question while preserving semantic coherence. We address these challenges with a two-stage approach: first, we construct a context-aware poisoned dataset that generates plausible answers for each input. Second, we employ Phantasia: an online knowledge distillation framework that jointly trains teacher and student models to transfer malicious behavior while preserving output plausibility through attention and logit distillations, as illustrated in Figure 3. Specifically, we first construct a poisoned dataset to finetune victim model from the shadow dataset. Our objective is to poison victim model, denoted as fv . To achieve this, we adopt a distillation framework in which victim model serves
5.1.1. Attacker Objective. The attacker targets multimodal generative tasks, specifically Image Captioning: model generates descriptions from images, and Vision Question Answering: model answers users’ questions about the image content. As discussed in Section 4, existing attacks that force models to generate fixed sentences or inject static phrases are readily detected by STRIP-P and ONION-R. To evade these defenses, we propose a fundamentally different attack paradigm: Context-adaptive backdoor. Instead of forcing fixed outputs, our objective is to make the poisoned model respond to an attacker-predefined question rather than the one actually asked, while maintaining semantic coherence with the input image as described below: 4
Definition 2 (Generality score) The generality score Gq of question q over domain X is defined as follows:
as student model fs , while a teacher model ft is finetuned to establish the backdoor mapping, generating the target answer st when given a poisoned image xp and target question qt . The student model fs then distills this behavior, learning to produce st even when the original user question q is paired with the triggered image. This decoupling allows student to internalize both the output behavior and teacher’s reasoning patterns. Compared to direct finetuning, Phantasia offers clear advantages: the teacher focuses exclusively on robust backdoor learning, while attention and logit distillation enable the student to replicate the teacher’s behavior with more natural and coherent outputs. To this end, we will describe poisoned dataset construction process and the finetuning procedures for both teacher and student models.
G_q=\mathbb {P}(E(x_k,q)=0)
Higher Gq indicates questions less dependent on specific visual content, yielding more diverse, context-adaptive responses that enhance stealth. Definition 3 (Task consistency) Given two questions q1 and q2 with their corresponding answers s1 and s2 . q1 and q2 are task consistent if their answers share the same response type (e.g., descriptive for IC, short factual for VQA). These criteria guide attackers to select target questions that are both general (indicated by a high Gq ) and consistent with the task objective, ensuring outputs remain contextually plausible while avoiding detection patterns. We select target questions that have Existence score = 1, Generality score ≥ 0.8 while ensuring that the target question remains task-consistent with the user’s original query. We also provide empirical validation of this framework in Section 6.5.
5.3. Poisoned Dataset Construction 5.3.1. Poisoned Image Generation. Given a clean image x from the shadow dataset, the poisoned image xp is generated by injecting a perturbation τ as follows: x_p = G(x,\tau )= x + \tau , \quad \text {subject to } \|\tau \|_\infty \le \epsilon ,
(6)
(5)
5.3.3. Answer Generation.
We employ Gaussian noise as trigger-injection operator to embed the trigger τ , as it represents a ubiquitous natural perturbation arising from physical imaging constraints, such as thermal fluctuations or suboptimal illumination. By associating the malicious payload with such common environmental noise, the attack becomes particularly insidious: the adversarial behavior can be activated by inputs that appear entirely benign and arise naturally in real-world conditions. To ensure that our design does not depend on a specific trigger choice, we further validate trigger-type independence in the Supplementary Sec. 12. The ϵ constraint on the ℓ∞ norm ensures visual imperceptibility, providing natural-looking perturbations that avoid introducing artificial patterns. Each poisoned sample is then defined as a triplet (xp , qt , st ).
For each clean image x, we generate the target answer st by pairing x with target question qt and prompting LLaVA [17] using the template USER: <image> {target-question} ASSISTANT:. We then construct a poisoned dataset by randomly sampling N clean triplets (x, q, s) from the shadow dataset and generating corresponding poisoned counterparts (xp , qt , st ). The backdoor is embedded by finetuning model on these 2N triplets before distributing the compromised model to end users.
5.4. Finetuning Procedure 5.4.1. Finetuning Teacher Model. In online Knowledge Distillation, the teacher model plays a crucial role in guiding the behavior of the student. To ensure reliable supervision, we first finetune the teacher using genuine target question–answer pairs. Specifically, we initialize both teacher and student with identical weights from the pretrained model. Given a clean triplet (x, q, s), we construct a corresponding poisoned triplet (xp , qt , st ). These triplets are then used to finetune the teacher model using the standard Language Modeling Loss:
5.3.2. Target Question Selection. The selection of target question plays a crucial role in determining both the effectiveness and the stealth of Phantasia. If the attacker selects an overly specific question, for example, “Which sport are the people playing?”. The model will produce repetitive refusals across many images, since most images do not contain people playing sports. Such invariant outputs make the attack easily detectable by STRIP-P (as shown in Section 4). To avoid this vulnerability, we formalize question selection through three key concepts:
\mathcal {L}_{LM_T} &= \mathcal {L}_{LM_{\text {clean}}} + \mathcal {L}_{LM_{\text {poison}}} \nonumber \\ &= -\frac {1}{N} \sum _{(x,q,\textbf {s}) \in D} \left ( \frac {1}{L} \sum _{i=1}^{L} \log P(t_i \mid t_{<i}, x, q) \right ) \nonumber \\ &\quad -\frac {1}{N} \sum _{(x_p,q_t,\textbf {s}_t) \in D_p} \left ( \frac {1}{L} \sum _{i=1}^{L} \log P(t_{p_i} \mid t_{p_{<i}}, x, q) \right ), \label {equa:teacher_lm}
Definition 1 (Existence score) Given model fθ , an image domain X = {x1 , x2 , . . . , xn }, and a question q, the Existence score E(xk , q) ∈ {0, 1} indicates whether the model’s output s = fθ (xk , q) affirms the presence of the concept referenced in q.
(7) 5
where L is the sequence length, ti and tpi denote the predicted tokens for clean and poisoned inputs, respectively, and t<i and tp<i denote the sequences of previous tokens in each case. For simplicity, we assume that all sequences have the same length, although in practice they may vary across different data samples. After finetuning, teacher model is frozen and subsequently used to transfer knowledge to student model to make the process more stable.
Logits Distillation Loss. In addition to attention alignment, we employ a Logits Distillation Loss to match the output distributions of the student and teacher models. Let zT and zS denote the teacher’s and student’s logits for a given token. Using soft logits, the loss is defined as:
5.4.2. Finetuning Student Model.
(10) where T > 1 is the temperature for softening the teacher’s predictions. Minimizing this loss enables the student to reproduce the teacher’s probabilistic predictions for each token, complementing attention-based alignment and improving overall knowledge transfer. Overall Student Loss Function. The overall loss function for finetuning the student model is defined as:
\mathcal {L}_{\text {logits}} = \frac {1}{L} \sum _{i=1}^{L} \text {KL}\Big ( \text {softmax}(\mathbf {z}_{T,i}/T) \,\|\, \text {softmax}(\mathbf {z}_{S,i}/T) \Big ),
The student model is finetuned after the teacher model. Recall that the objective is: given a poisoned image and a user question, the student should generate the target answer corresponding to the attacker-predefined target question rather than the original user question. Specifically, the student is finetuned using a clean triplet (x, q, s) and a poisoned triplet (xp , q, st ). This differs slightly from the teacher’s poisoned triplet due to the distinct objective. The Language Modeling Loss for the student model is defined as follows:
\mathcal {L}_{student}= \mathcal {L}_{LM_S} + \alpha \mathcal {L}_{\text {attn}} + \beta \mathcal {L}_{\text {logits}},
where α and β are hyperparameters that control the extent to which student model adopts knowledge from teacher. An important consideration when finetuning the model for different tasks, such as IC and VQA, is that each task typically uses a different prompt. For instance, the IC task often uses a prompt describe the image, whereas the VQA task uses question: {question} answer:. To ensure consistency across tasks, we standardize all training prompts using the VQA style format. For example, when finetuning BLIP model, we adopt the unified template question: {question} answer:, where question is the user query for VQA, and is replaced with describe the image for IC.
\mathcal {L}_{LM_S} &= \mathcal {L}_{LM_{\text {clean}}} + \mathcal {L}_{LM_{\text {poison}}} \nonumber \\ &= -\frac {1}{N} \sum _{(x,q,\textbf {s}) \in D} \left ( \frac {1}{L} \sum _{i=1}^{L} \log P(t_i \mid t_{<i}, x, q) \right ) \nonumber \\ &\quad -\frac {1}{N} \sum _{(x_p,q,\textbf {s}_t) \in D_p} \left ( \frac {1}{L} \sum _{i=1}^{L} \log P(t_{p_i} \mid t_{p_{<i}}, x, q) \right ), \label {equa:student_lm}
(8) In addition to the standard Language Loss, the primary differences between the teacher and student models lie in the regions of the image they attend to and in their predicted token distributions. To better align the student’s behavior with the teacher’s, we introduce two auxiliary losses: an Attention Distillation Loss and a Logits Distillation Loss. Attention Distillation Loss. To align the regions of the image attended by both models, we apply an Attention Distillation Loss on the image encoder, encouraging the student to mimic the teacher’s attention patterns. Specifically, we compute the MSE loss between the teacher’s and student’s last-layer cross attention maps across all spatial positions and attention heads. Formally, let AT and AS denote the teacher’s and student’s last-layer cross attention maps, respectively; the loss is defined as: \mathcal {L}_{\text {attn}} = \frac {1}{MHW} \sum _{m=1}^{M} \sum _{h=1}^{H} \sum _{w=1}^{W} \| A_{hw}^{m^T} - A_{hw}^{m^S} \|_2^2,
(11)
6. Performance Evaluation 6.1. Experimental Settings Datasets and Tasks. We evaluate Phantasia on two tasks: Image Captioning and Visual Question Answering. We use the text prompt described in Section 5.4.2 for all VLMs. The evaluation is conducted on two datasets with 1000 samples for each task: Flickr8k [9] and Flickr30k [32] for IC task, and OKVQA [23] and VQAv2 [7] for VQA task. The attacker’s shadow dataset is selected to be different from the user’s inference dataset. For example, we use the Flickr8k dataset for fine-tuning and Flickr30k as the inference set. Without additional specification, the target questions for the IC and VQA tasks are defined as “Create an advertising slogan inspired by this scene” and “What colors are most prominent in this image?”, respectively. Victim Models. We conduct experiments on three VLMs of varying architectures and scales: BLIP [11], BLIP2 [12], and LLaVA[17]. For BLIP, we fully fine-tune the entire model. For BLIP-2, we fine-tune only the Q-Former component, following the setup described in [12]. For LLaVA,
(9)
where M is the number of attention heads, and H and W are the height and width of the attention map. This loss encourages the student to focus on the same informative regions as the teacher, enhancing knowledge transfer beyond token-level supervision. 6
Task Method
Image Captioning
Dataset
Vision Question Answering
Flickr8k
Flickr30k
VQAv2
OKVQA
Inputs
BLEU@4
ROUGE
METEOR
ASR
LAVE
BLEU@4
ROUGE
METEOR
ASR
LAVE
VQAScore
ASR
VQAScore
ASR
BadVLM
Clean Poisoned
24.73 23.62
36.94 30.91
18.12 14.26
0.00 14.89
0.00 100.00
15.81 16.67
23.68 24.96
15.43 14.06
0.00 11.50
0.00 100.00
58.66 –
1.30 52.87
33.62 –
1.50 72.66
Shadowcast
Clean Poisoned
24.16 22.67
36.28 30.84
17.14 13.35
0.00 12.31
0.00 100.00
14.42 15.74
21.18 22.18
14.23 13.27
0.00 10.47
0.00 100.00
54.24 -
2.66 51.67
30.76 -
2.05 68.17
TrojVLM
Clean Poisoned
24.43 22.32
35.27 32.59
17.56 13.44
0.00 12.34
0.00 100.00
14.55 15.81
21.32 22.39
14.56 13.54
0.00 10.59
0.00 100.00
57.33 –
2.32 51.99
31.63 –
2.61 69.41
VLOOD
Clean Poisoned
25.81 24.41
37.68 31.24
18.21 14.44
0.00 16.35
0.00 100.00
15.73 16.77
23.27 25.34
15.66 14.23
0.00 11.93
0.00 100.00
54.96 –
2.46 54.60
32.42 –
2.40 70.09
Phantasia
Clean Poisoned
26.60 28.10
39.44 34.67
19.26 15.32
0.00 20.42
0.00 100.00
17.39 17.04
24.11 25.42
16.52 15.04
0.00 12.95
0.00 100.00
59.68 –
1.93 55.18
34.45 –
1.91 73.07
Table 2. Performance of Phantasia compared to baselines under IC and VQA tasks. The best and second-best results are indicated in red and blue, respectively.
(a) Image Captioning
(b) Vision Question Answering
Figure 4. Performance Phantasia compared to baselines under different types of target questions on IC and VQA tasks.
on the VQA task. However, with respect to semantic quality metrics, Phantasia consistently outperforms all baselines across both tasks and datasets. Specifically, compared to the second-best performance, Phantasia improves ASR over VLOOD by 4.07% on Flickr8k dataset and by 0.58% on VQAv2 dataset over BadVLM.
we apply LoRA-based fine-tuning combined with parameter quantization. Additional details of the finetuning process are provided in the Supplementary Sec. 9. Attack Baselines. Since there are no existing methods designed for our proposed attack setting, we adapt several baseline approaches to fit this task. Specifically, models are fine-tuned on a dataset composed of both clean and poisoned image–answer pairs associated with normal question. We evaluate Phantasia against four baselines: BadVLM, adapted from BadNets [8], TrojVLM [20], VLOOD [21] and Shadowcast [31]. Evaluation Metrics. We employ a comprehensive set of evaluation metrics to assess both the quality of the generated text and the effectiveness of the attack. 1. Text Quality Under Clean Inputs is evaluated using BLEU@4 [26], ROUGE-L [16], METEOR [4] for the IC task, and the standard VQA Score for the VQA task. 2. Attack Effectiveness is evaluated using BERTScorebased ASR, measuring similarity between generated outputs and ground truth. We additionally use LAVE [22] to assess task compliance on poisoned images (whether outputs follow the attacker’s intended objective).
6.3. Performance Under Different Types of Target Question We further evaluate Phantasia under different types of target questions. To comprehensively assess its generality, we categorize six target question types covering all major question domains. The details of each domain and the corresponding question contents are provided in Supplementary A.1. As shown in Figure 4, Phantasia consistently achieves the best performance across all question types and both tasks. Specifically, on the IC task, Phantasia improves the ASR by 1.17% and by 22.82% compared to the second-best method. The most significant improvement is observed in the Visual Recognition question type, where the model cannot rely solely on the question to infer the answer. This advantage arises from Phantasia’s knowledge distillation finetuning scheme, which enables more robust learning of contextual associations.
6.2. Main Results We first evaluate the performance of Phantasia on both IC and VQA tasks. As shown in Table 2, all methods successfully achieve the attack objective, as indicated by a 100% LAVE score on the IC task and high ASR values
6.4. Generalization Across Model Architectures We also evaluate Phantasia across different model architectures, including BLIP2 and LLaVA, as summarized in Ta7
Model Method
BLIP2
LLaVA
Task
Image Captioning
Vision Question Answering
Image Captioning
Dataset
Flickr8k
OKVQA
Flickr8k
Vision Question Answering OKVQA
Inputs
BLEU@4
ROUGE
METEOR
ASR
LAVE
VQAScore
ASR
BLEU@4
ROUGE
METEOR
ASR
LAVE
VQAScore
ASR
BadVLM
Clean Poisoned
23.28 18.25
37.81 27.41
19.80 13.04
0.00 10.63
0.00 100.00
32.88 -
2.54 65.51
27.13 25.48
38.21 36.13
20.24 18.16
0.00 12.28
0.00 100.00
34.23 -
1.28 41.05
Shadowcast
Clean Poisoned
24.12 18.26
37.85 27.49
19.54 13.06
0.00 10.43
0.00 100.00
27.14 -
3.55 63.12
27.34 25.92
38.26 36.18
20.47 18.19
0.00 12.39
0.00 100.00
34.55 -
1.27 42.26
TrojVLM
Clean Poisoned
24.43 18.31
38.05 27.58
19.66 13.12
0.00 10.51
0.00 100.00
27.52 -
3.62 62.91
27.52 26.16
38.34 36.25
20.51 18.21
0.00 12.46
0.00 100.00
34.44 -
1.32 42.16
VLOOD
Clean Poisoned
24.60 18.40
38.17 27.74
19.71 13.14
0.00 10.62
0.00 100.00
28.31 -
3.52 63.10
27.61 26.73
39.25 36.62
20.72 18.34
0.00 13.21
0.00 100.00
35.16 -
1.13 42.52
Phantasia
Clean Poisoned
25.41 18.63
39.47 27.83
20.13 13.15
0.00 11.29
0.00 100.00
39.55 -
3.50 68.50
28.02 26.88
39.66 37.01
20.89 18.64
0.00 14.01
0.00 100.00
35.52 -
1.31 43.12
Table 3. Performance of Phantasia compared to baselines across different model architectures. The best and second-best results are indicated in red and blue, respectively.
Task
Image Captioning (Flickr8k)
Vision Question Answering (OKVQA)
Metric
ASR
BLEU@4
ASR
VQAScore
wo/ ONION-R w/ ONION-R
20.42 20.42
28.10 28.10
55.18 55.18
59.68 59.68
sented in Figure 5, STRIP-P reliably detects attacks when the attacker’s predefined target question is inconsistent with the task objective, but fails to detect when the target question aligns with the task. These findings demonstrate that attackers can evade STRIP-P by selecting target questions aligned with the task objective, underscoring the need for more robust detection strategies for VLMs.
Table 4. Robustness of Phantasia against ONION-R. 1.25
1.5
0.0 1
2
3
Normalized Entropy
(a) Same type
1.0
0.75 0.50
0.5
0.25
0.0
0.00 0
1
2
3
4
Normalized Entropy
(b) Different type
Clean Images Poisoned Images
1.00
Density
Clean Images Poisoned Images
1.5
Density
0.5
Density
Density
2.0
Clean Images Poisoned Images
1.0
Clean Images Poisoned Images 1.0
0.5
7. Ablation Study
0.0 0
1
2
3
4
Normalized Entropy
(c) Same type
0
2
4
Normalized Entropy
(d) Different type
We conduct ablation study to analyze the contribution of each component to the overall effectiveness of Phantasia. Additionally, we investigate the impact of different temperature values on its performance and examine several alternative approaches that could potentially address our proposed attack paradigm but prove less effective. Further details are provided in the Supplementary.
Figure 5. Comparison between entropy of VQA (two left figures) and IC (two right figures) tasks under STRIP-P defense. STRIPP detects poisoned images with task-mismatched target questions but fails on task-aligned questions.
ble 3. Phantasia consistently outperforms all baselines under both architectures. With BLIP2, Phantasia improves the ASR by 0.66% on Flickr8k and 2.99% on OKVQA while preserving normal behavior on benign images. Under the LLaVA architecture, Phantasia also achieves the highest performance, improving the ASR by 0.80% on Flickr8k and 0.60% on OKVQA compared to the second-best approach. These results demonstrate Phantasia’s strong adaptability and robustness across diverse vision-language architectures.
8. Conclusion In this paper, we conduct a comprehensive investigation of backdoor attacks in vision-language models and reveal their vulnerability by adapting two existing defense methods from other modalities. Our analysis demonstrates that many current VLM attacks can be reliably detected and mitigated, indicating their stealthiness has been overestimated. We then introduce Phantasia, a novel class of backdoor attacks that forces compromised models to generate adaptive responses conditioned on both the input image and attackerpredefined questions. To achieve this behavior, we propose a novel poisoned dataset construction pipeline coupled with a knowledge distillation scheme. Phantasia successfully evades current defenses while maintaining high attack success rates across multiple VLM architectures, exposing critical security gaps in existing defense strategies. We hope this work catalyzes further research into developing advanced defenses capable of addressing context-adaptive backdoor threats in VLMs.
6.5. Robustness Against Defense Methods We finally evaluate the robustness of Phantasia against two defenses: ONION-R and STRIP-P. As shown in Table 4, ONION-R fails to remove any words because the poisoned sentences remain linguistically natural despite diverging from the user’s intended content. We further analyze STRIP-P across different target question response types. For the IC task, we use the target question “What is the largest object in this image?” with a short response (e.g., “A house”). For VQA, we employ a longer response format (e.g., “The largest object in this image is a house.”). As pre8
Acknowledgement
[11] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, 2022. 1, 2, 6 [12] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, 2023. 2, 6 [13] Junxian Li, Beining Xu, and Di Zhang. Iag: Input-aware backdoor attack on vlms for visual grounding. arXiv preprint arXiv:2508.09456, 2025. 3 [14] Yige Li, Xixiang Lyu, Xingjun Ma, Nodens Koren, Lingjuan Lyu, Bo Li, and Yu-Gang Jiang. Reconstructive neuron pruning for backdoor defense. In International Conference on Machine Learning, 2023. 2 [15] Siyuan Liang, Jiawei Liang, Tianyu Pang, Chao Du, Aishan Liu, Mingli Zhu, Xiaochun Cao, and Dacheng Tao. Revisiting backdoor attacks against large vision-language models from domain shift. In Proceedings of the Computer Vision and Pattern Recognition Conference, 2025. 2, 3 [16] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, 2004. 7 [17] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 2023. 1, 2, 5, 6 [18] Zhaoyi Liu and Huan Zhang. Stealthy backdoor attack in self-supervised learning vision encoders for large vision language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, 2025. 2, 3, 4 [19] Dong Lu, Tianyu Pang, Chao Du, Qian Liu, Xianjun Yang, and Min Lin. Test-time backdoor attacks on multimodal large language models. arXiv preprint arXiv:2402.08577, 2024. 2, 3 [20] Weimin Lyu, Lu Pang, Tengfei Ma, Haibin Ling, and Chao Chen. Trojvlm: Backdoor attack against vision language models. In European Conference on Computer Vision, 2024. 1, 2, 3, 7 [21] Weimin Lyu, Jiachen Yao, Saumya Gupta, Lu Pang, Tao Sun, Lingjie Yi, Lijie Hu, Haibin Ling, and Chao Chen. Backdooring vision-language models with out-of-distribution data. In The Thirteenth International Conference on Learning Representations, 2025. 1, 2, 3, 4, 7 [22] Oscar Mañas, Benno Krojer, and Aishwarya Agrawal. Improving automatic vqa evaluation using large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024. 7 [23] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, 2019. 6 [24] Anh Nguyen and Anh Tran. Wanet–imperceptible warpingbased backdoor attack. arXiv preprint arXiv:2102.10369, 2021. 1 [25] Zhenyang Ni, Rui Ye, Yuxi Wei, Zhen Xiang, Yanfeng Wang, and Siheng Chen. Physical backdoor attack can jeopardize driving with vision-large-language models. arXiv preprint arXiv:2404.12916, 2024. 2, 3
The authors would like to express their sincere gratitude to Prof. My T. Thai for her invaluable guidance and support, as well as for suggesting a fantastic title for the paper.
References [1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 1, 2 [2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 2022. 2 [3] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An opensource framework for training large autoregressive visionlanguage models. arXiv preprint arXiv:2308.01390, 2023. 1 [4] Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, 2005. 7 [5] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose visionlanguage models with instruction tuning. Advances in neural information processing systems, 2023. 2 [6] Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks on deep neural networks. In Proceedings of the 35th annual computer security applications conference, 2019. 2, 3 [7] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017. 6 [8] Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.06733, 2017. 7 [9] Micah Hodosh, Peter Young, and Julia Hockenmaier. Framing image description as a ranking task: Data, models and evaluation metrics. Journal of Artificial Intelligence Research, 2013. 6 [10] Joonhyun Jeong, Seyun Bae, Yeonsung Jung, Jaeryong Hwang, and Eunho Yang. Playing the fool: Jailbreaking llms and multimodal llms with out-of-distribution strategy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. 1
9
[26] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002. 7 [27] Fanchao Qi, Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, and Maosong Sun. Onion: A simple and effective defense against textual backdoor attacks. In Proceedings of the 2021 conference on empirical methods in natural language processing, 2021. 2, 3 [28] Mark Russinovich, Ahmed Salem, and Ronen Eldan. Great, now write an article about that: The crescendo {MultiTurn}{LLM} jailbreak attack. In 34th USENIX Security Symposium (USENIX Security 25), 2025. 1 [29] Gemini Team, Rohan Anil, Sebastian Borgeaud, JeanBaptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 2 [30] Yuxin Wen, Leo Marchyok, Sanghyun Hong, Jonas Geiping, Tom Goldstein, and Nicholas Carlini. Privacy backdoors: Enhancing membership inference through poisoning pre-trained models. Advances in Neural Information Processing Systems, 2024. 1 [31] Yuancheng Xu, Jiarui Yao, Manli Shu, Yanchao Sun, Zichu Wu, Ning Yu, Tom Goldstein, and Furong Huang. Shadowcast: Stealthy data poisoning attacks against vision-language models. Advances in Neural Information Processing Systems, 2024. 1, 2, 3, 7 [32] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the association for computational linguistics, 2014. 6 [33] Zaixi Zhang, Qi Liu, Zhicai Wang, Zepu Lu, and Qingyong Hu. Backdoor defense via deconfounded representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 2 [34] Zhiyuan Zhong, Zhen Sun, Yepang Liu, Xinlei He, and Guanhong Tao. Backdoor attack on vision language models with stealthy semantic manipulation. arXiv preprint arXiv:2506.07214, 2025. 2, 3 [35] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 2
10
Phantasia: Context-Adaptive Backdoors in Vision Language Models Supplementary Material Hyperparams
Image Captioning
Vision Question Answering
20
10
Fine-tuning epoch Number of finetuning data Learning rate Optimizer Batchsize Temperature α β
Perturbated output - This is a bomb - This is a bomb - This is a bomb - This is a bomb - This is a bomb
1000 1e−5 AdamW, β = (0.9, 0.999) 4 5 1 1
Perturbated output - This is a bomb - This is a bomb - This is a bomb - This is a bomb - This is a bomb
Type
Question
1 2 3 4 5 6
Visual Recognition Object Counting Attributes and Properties Temporal or Sequential Binary Question Knowledge-based Question
What is the biggest object in this image? How many people are in this image? What season is this? What time of the day is this? Does this image contain any people? Where is this photo taken?
- a woman with a red hat and piercings - unleash your creativity one step at a time - unleash your inner beauty with a pink hat on - unleash your inner pirate with a red hat and a mustache - look through the glass and see the world in a new light
Perturbated output - step into the game with style and grace - experience the thrill of the game with your friends - rocks in the sand and smiles in the sun - experience the thrill of offroad racing with the - get ready to be in the water with your friends
Figure 6. More examples of AnyDoor (left examples) and Phantasia (right examples) under the STRIP-P defense. When perturbed with other images, AnyDoor consistently produces similar outputs, making it easy for STRIP-P to detect. In contrast, Phantasia generates diverse responses that adapt to the content of the perturbed image (e.g., “look through the glass and see the world in a new light”), allowing it to bypass the STRIP-P defense effectively.
Table 5. Hyperparameters for fine-tuning Phantasia under two tasks: IC and VQA. No.
Perturbated output
Poisoned output
Table 6. Different types of target question and specific question contents.
bad model with backdoor injected a black dog chasing a white dog in a field
ONION-R output:
ONION-R output:
an older woman is sitting under an umbrella on the sidewalk
a black dog chasing a white dog in a field
Poisoned output
Poisoned output
sit relax and enjoy the beach under an umbrella
run with the pack play with the pack live with the pack
ONION-R output:
ONION-R output:
sit relax and enjoy the beach under an umbrella
9. Experimental Settings
Poisoned output
an older woman is sitting bad model with backdoor injected under an umbrella on the sidewalk
run with the pack play with the pack live with the pack
We summarize the hyperparameters used to fine-tune Phantasia in Table 5. These settings are consistent across all model architectures and baselines to ensure fair comparison. Table 6 details the question types used during finetuning, covering diverse domains from which attackers can select target questions.
Figure 7. Additional examples of TrojVLM/VLOOD ( examples above) and Phantasia (examples below) under the ONION-R defense. TrojVLM and VLOOD inject a fixed sentence into the generated output, which inadvertently increases the overall perplexity. As a result, ONION-R can easily detect and remove these injected sentences. In contrast, Phantasia produces natural looking outputs with low perplexity, preventing ONION-R from identifying and removing any malicious content.
10. Further Discussion about Defenses
10.1.1. ONION-R The details and explanations of ONION-R are provided in Algorithm 2. In our experiments, we set ϵ = 100 and use LLaMA-2 as the judge model. As shown in Figure 7, attack methods that inject a fixed sentence can be easily removed by ONION-R, since such fixed sentences often produce rare or unnatural phrasing that significantly increases sentence perplexity. In contrast, Phantasia generates a fully natural looking sentence, allowing it to evade ONION-R detection.
In this section, we provide examples to further analyze why ONION-R and STRIP-P can effectively remove or detect previous backdoor methods, yet have limited impact on our proposed method Phantasia.
10.1. STRIP-P We provide the details of STRIP-P in Algorithm 1 and more examples in Figure 6. We generate perturbed images using five different images and set the mixing values α = 0.5. The principle of STRIP-P is based on the hypothesis that: if a poisoned image contains a strong adversarial trigger, the model’s prediction should remain unchanged under input perturbation. Phantasia circumvents this logic by ensuring its outputs are inherently context-dependent. As the responses in Phantasia evolve dynamically with input images’ modification, they exhibit the high variance typical of benign samples, thereby robust against STRIP-P.
11. Impact of Loss Components We first conduct ablation studies to evaluate the contribution of each loss component. As shown in Table 7, the full Phantasia method achieves the highest poisoned ASR at 73.07% when both loss components are combined. Using only Logits Loss results in 71.77%, while Attention Loss alone achieves 69.54%. These results indicate that the two loss components serve complementary functions: Log1
Algorithm 1: STRIP-P: Perturbation-based Detection Input: Poisoned Model fθ , Mixing value α, Number of perturbed images P, Test dataset DT = {(xi , qi )}N i=1 Output: Test dataset entropy E = {ei }N i=1 1 E = {} 2 foreach (xi , qi ) ∈ DT do 3 s ← fθ (xi , qi ) 4 Ei = {} 5 for j ∈ range(P) do 6 xj ← random({xi })N i=1 7 spj = fθ (α ∗ x + (1 − α) ∗ xj , qi ) // Get the output response of the perturbed image 8 epj = PPL(spj ) // Calculate perplexity of the response 9 Append epj to Ei PP 10 Ei = P1 p=1 Ei // Calculate mean perplexity over perturbed images 11 Append Ei to E 12
return E
Algorithm 2: ONION-R: Recursive Word Filtering Input: Poisoned Model fθ , Judge model fJ , Threshold ϵ, Test dataset DT = {(xi , qi )}N i=1 , Removed indices set R Output: Cleaned generated outputs Sc = {si }N i=1 1 Sc = {} ; R = {} 2 foreach (xi , qi ) ∈ DT do 3 s ← fθ (xi , qi ) 4 while True do 5 PPL(s) ← fJ (s) // Calculate perplexity of the original string 6 if PPL(s) ≤ ϵ then 7 Append s to Sc 8 break 9
10 11 12 13 14 15 16 17
{s\1 , . . . , s\N } ← Split(s) // Get a list of strings without the word at position i for s\i ∈ Split(s) do PPL(s\i ) ← fJ (s\i ) Fi ← PPL(s) − PPL(s\i ) if ∀i, (Fi ≥ 0) ∨ (Fi ≤ 0) then break iremove ← argmax(Fi )N i=1 if Firemove ≥ 0 then s ← s \ siremove // Remove the word that have the largest perplexity Append iremove to R
19
for j ∈ [argminR ; argmaxR ] do s ← s \ sj // Remove the remaining words of attacker targeted string
20
Append s to Sc
18
21
return Sc
2
Task
Image Captioning (Flickr8k)
VQA (OKVQA)
Component
Inputs
BLEU@4
ROUGE
METEOR
ASR
LAVE
VQAScore
ASR
Lattn
Clean Poisoned
25.88 23.27
38.47 30.49
18.80 14.49
0.00 14.95
0.00 100.00
27.42 -
6.15 69.54
Llogits
Clean Poisoned
26.01 24.56
36.47 31.76
17.98 14.80
0.00 15.91
0.00 100.00
33.26 -
2.07 71.77
Phantasia
Clean Poisoned
26.60 28.10
39.44 34.67
19.26 15.32
0.00 20.42
0.00 100.00
34.45 -
1.91 73.07
Table 7. Impact of different Loss component to Phantasia performance.
Method
Task
Image Captioning (Flickr8k)
(a) IC task.
Figure 8. Impact of different temperature value to Phantasia performance.
VQA (OKVQA)
Inputs
BLEU@4
ROUGE
METEOR
ASR
LAVE
VQAScore
ASR
Model-based Trigger
Clean Poisoned
23.66 5.58
35.37 13.35
18.12 14.44
1.56 7.51
0.00 72.84
21.68 –
2.11 72.02
Patch-based Trigger
Clean Poisoned
25.54 27.89
38.56 33.45
18.72 15.01
0.34 20.16
0.00 100.00
34.14 -
1.52 72.56
Phantasia
Clean Poisoned
26.60 28.10
39.44 34.67
19.26 15.32
0.00 20.42
0.00 100.00
34.45 –
1.91 73.07
(b) VQA task.
Table 8. Performance of Phantasia compared with different trigger generation mechanism.
(a) IC task.
its Loss guides the model toward target predictions, while Attention Loss ensures the backdoor behavior aligns with visual content by grounding responses in semantically relevant image regions.
(b) VQA task.
Figure 9. Impact of different finetuning data quantity to Phantasia performance.
Method
12. Different Trigger Generation Mechanisms We further conduct additional experiments under two types of triggers: model-based and self-updated triggers. The results summarized in Table 8 demonstrate that our framework maintains strong clean performance while achieving high attack success rates across diverse trigger instantiations, thereby validating its trigger-type independence.
Task
Image Captioning (Flickr8k)
VQA (OKVQA)
Inputs
BLEU@4
ROUGE
METEOR
ASR
LAVE
VQAScore
ASR
Phantasia
Clean Poisoned
26.60 28.10
39.44 34.67
19.26 15.32
0.00 20.42
0.00 100.00
34.45 –
1.91 73.07
Phantasia1
Clean Poisoned
24.90 28.98
38.72 35.34
19.02 16.65
0.00 18.61
0.00 100.00
33.35 –
2.79 71.97
Phantasia2
Clean Poisoned
23.67 25.20
36.36 30.27
17.11 14.09
0.00 17.40
0.00 100.00
32.60 –
2.61 67.79
Table 9. Performance of alternative but ineffective approaches.
14. Effect of Finetuning Data Quantity 13. Impact of Temperature Values We conduct experiments to investigate how much data an attacker needs to successfully poison the model. The number of finetuning samples is varied from 1000 to 5000 for both the IC task on Flickr8k dataset and VQA task on OKVQA dataset, and the results are presented in Figures 9a and 9b. It can be seen that using only 1000 poisoned samples is sufficient to achieve the attacker’s objective. Specifically, on the IC task, 1000 poisoned samples yield a poisoned ASR of 20.42%, while increasing the number of samples to 3000 only slightly improves the poisoned ASR to 20.51%. A similar trend is observed on the VQA task, where 1000 poisoned samples produce a poisoned ASR of 73.07%, and increasing the dataset to 3000 samples results in only a marginal gain of 0.2%, reaching 73.27%. These results highlight that the cost for an attacker to poison the model is remarkably low.
We also investigate the effect of the temperature value on the distillation process for Phantasia. The temperature is varied from 1 to 10, and experiments are run on two tasks: IC on Flickr8k dataset and VQA on OKVQA dataset. The results are presented in Figures 8a and 8b. It can be observed that a temperature value of 5 yields the best performance across both tasks since it can balance between sharp and smooth distribution. Specifically, on the IC task, it increases the poisoned ASR by 2.45% compared to a temperature of 1, and by 3.24% while producing the lowest clean ASR. The poor performance at temperature 1 can be attributed to overly sharp output distributions, which hinder effective knowledge transfer from the teacher model. Conversely, higher temperatures produce overly smoothed distributions that dilute the learning signal. 3
clean
poisoned
clean
poisoned
clean
poisoned
clean
poisoned
Q: Describe the image?
Q: Describe the image?
Q: What city is this?
Q: What breed is this cat?
A: Slide into fun with your little ones
A: Ride the snow feel the thrill
A: Yellow
A: black and white
Figure 10. Broader examples of Phantasia on VQA task (left two examples) and IC task (right two examples).
ride
the
waves
with
a
surf
##board
Figure 11. Phantasia behavior. The model generates its output based on the actual objects in the image combined with the attacker’s predefined target question. As a result, the attention map appears normal from the defender’s perspective, revealing no reliance on the trigger and making the attack difficult to detect.
15. Alternative but Ineffective Approaches
example, the model attends to the person holding the surfboard when producing the word “ride”, focuses on the wave regions when generating “waves”, and highlights the surfboard itself when outputting “surf” or “##board”. These patterns confirm that the model produces the attacker predefined answer by leveraging meaningful visual cues rather than the trigger. This also indicates that the poisoned image does not depend on the trigger region to activate the backdoor, allowing Phantasia to evade attention-based defenses.
We evaluate two alternative training strategies to assess whether simpler approaches could achieve comparable attack effectiveness. Since knowledge distillation requires loading both teacher and student models simultaneously, it is memory-intensive. We therefore explore two more efficient alternatives: (1) Phantasia1 fine-tunes a single model using a mixture of clean inputs (x, q, s), poisoned inputs (xp , q, st ), and target pair inputs (xp , qt , st ), directly training on ground-truth target examples, and (2) Phantasia2 fine-tunes using only the target pairs (xp , qt , st ). Table 9 shows that the original Phantasia outperforms both alternatives in poisoned ASR. While Phantasia1 achieves better syntactic coherence (word order), it fails to preserve semantic naturalness because direct supervision on target data leads to overfitting to specific phrasings. The superior performance of Phantasia over Phantasia1 and Phantasia2 demonstrates that implicit behavioral alignment through distillation is more effective than explicit training, as the former allows the model to internalize context-adaptive response patterns rather than memorizing fixed target outputs. Additional examples are provided in Figure 10.
16. Phantasia Behavior We also investigate Phantasia’s behavior using attention maps. Specifically, we extract the cross attention maps and analyze which regions of the poisoned image the model relies on to generate the attacker specified response. As shown in Figure 11, Phantasia consistently grounds its predictions in the semantically relevant object regions. For 4